DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models

Thanks for the interesting work! Can you offer some insight on how ART compares to FDFO (Finite Difference Flow Optimization), and which works better in which case?

Thank you for your comment. Regarding FDFO, I see the following limitations:

1. It still relies on SDE sampling. Its methodological design cannot avoid ODE-to-SDE conversion because it needs two distinct trajectories originating from the same initial noise to construct a finite-difference pair. SDE sampling cannot perfectly preserve the marginal distributions, resulting in a loss of distributional information that directly affects training efficiency. Moreover, the estimated guidance acts on the SDE transition mean along the trajectory, which differs from the ODE transition mean, whereas inference uses ODE sampling. This introduces some training–inference mismatch. ART, by contrast, is designed to support different sampling methods during both training and inference, including ODE sampling.
2. Its reward guidance is based on finite differences between two nearby points, which requires the points to be sufficiently close in the sample space. We instead compute Cov(x₀, r), without imposing a proximity requirement. Coincidentally, I just realized that Cov(x₀, r) is a more general form of its finite-difference calculation: with only two points, our formulation reduces to a finite difference. In this sense, its reward-guidance construction is a special case of our approach. As the sample size increases, the direction estimate undoubtedly becomes more accurate and its variance decreases. Covariance may therefore be the more fundamental formulation.
3. Why do we not require the two points used to construct guidance to be close? This comes from our insight into restoration. We estimate a more accurate reward-improving direction and apply it directly to all states, then use the restoration arising from the second MSE update within each rollout batch to correct errors. We believe simplicity and efficiency are essential to scalability. Coincidentally, we have already conducted a closely related ablation in Figure 14 on page 20 of the appendix: aggregating information from all samples, applying the guidance to all states, and incorporating restoration performs substantially better than applying guidance locally within a pair of trajectories.
4. FDFO appears to require four updates per rollout, whereas we need only two. In terms of computational cost, this should halve the GPU hours spent on the training phase. Additionally, FDFO doesn't utilize the insight of restoration for stablization, its SPO trick may introduce additional complexity which may hampers scalability.
5. FDFO has not yet been validated on video generation. In our experience, methods can behave very differently on images and videos. If time permits, we will conduct additional video experiments with FDFO and update the paper.

Hope these insights helpful for your understanding!