Recommended
Recommended
More Related Content
Similar to Denoising MCMC for Accelerating Diffusion-Based Generative Models
Similar to Denoising MCMC for Accelerating Diffusion-Based Generative Models (20)
Recently uploaded
Recently uploaded (20)
Denoising MCMC for Accelerating Diffusion-Based Generative Models
- 1. Denoising MCMC for Accelerating Diffusion-Based Generative Models Beomsu Kim and Jong Chul Ye Kim Jaechul Graduate School of AI Korea Advanced Institute of Science and Technology (KAIST) Bioimaging, Signal Processing, and Learning Lab (BiSPL) The Fortieth International Conference on Machine Learning, 2023, Honolulu, Hawai’i
- 2. Diffusion Models 2 Image Source : Score-Based Generative Modeling through Stochastic Differential Equations, ICLR, 2021.
- 7. Denoising MCMC 7 𝑝 𝑥, 𝑡 ≔ 𝑝𝑡 𝑥 ⋅ 𝑝(𝑡) 𝑥𝑛, 𝑡𝑛 𝑛=1 𝑁 ∼ 𝑝(𝑥, 𝑡) such that 𝑥𝑛 ∼ 𝑝𝑡𝑛 (𝑥) Run reverse-S/ODE from t = 𝑡𝑛 to 0 for 𝑥𝑛 MCMC Diffusion Model
- 8. Denoising MCMC 8 𝑝 𝑥, 𝑡 ≔ 𝑝𝑡 𝑥 ⋅ 𝑝(𝑡) 𝑥𝑛, 𝑡𝑛 𝑛=1 𝑁 ∼ 𝑝(𝑥, 𝑡) such that 𝑥𝑛 ∼ 𝑝𝑡𝑛 (𝑥) Run reverse-S/ODE from t = 𝑡𝑛 to 0 for 𝑥𝑛 MCMC Diffusion Model Gibbs Sampling
- 9. Denoising MCMC 9 Current MCMC iterate (𝑥𝑛, 𝑡𝑛) 𝑡𝑛+1 ∼ 𝑞𝜙(𝑡|𝑥𝑛+1) Langevin MCMC Time Classifier 𝑞𝜙(𝑡|𝑥) 𝑥𝑛+1 = 𝑥𝑛 + 𝜂 2 ⋅ 𝑠𝜃 𝑥𝑛, 𝑡𝑛 + 𝜂 ⋅ 𝜖 ∼ 𝑝(𝑥|𝑡𝑛)
- 10. Quantitative Results – CIFAR10 10
- 11. Quantitative Results – CelebA-256 11
- 12. Quantitative Results – SOTA 12
- 13. Qualitative Results – CelebA-256 13
- 14. Qualitative Results – FFHQ-1024 14
- 15. Conclusion • Proposed DMCMC by combining MCMC with diffusion • MCMC is used to find good initialization points near the image manifold, leading to faster generation • DMCMC successfully accelerated six reverse S/ODE integrators on CIFAR10, CelebA-HQ-256, FFHQ-1024 • Achieved SOTA results in limited NFE setting 15
Editor's Notes
- Hello, my name is Beomsu Kim, and I will give an overview of our research on Denoising MCMC for Accelerating Diffusion-Based Generative Models.
- Diffusion model is a method for generating data from noise by solving a SDE or an ODE. Specifically, given a forward SDE which sends data to prior noise, there exists a reverse SDE/ODE. The reverse SDE or ODE can be solved numerically after learning the time-dependent score function by solving denoising score matching.
- However, Gaussian noise is generally far away from data. So, diffusion models usually need to integrate the reverse SDE or ODE over a long time interval to generate samples.
- So, when there is not enough computation budget, diffusion models tend to generate poor-quality samples, as shown in the current figure.
- Our method, Denoising MCMC, DMCMC in short, uses Markov Chain Monte Carlo to generate initialization points that are near the image manifold. Then, reverse SDE or ODE is applied to the initialization points to generate clean data.
- Since those initialization points lie near the image manifold, high-quality samples can be produced with little computation budget.
- To achieve this, we first define a joint distribution of noisy data x and diffusion time t, denoted p(x,t). Here, p(t) is a custom prior on diffusion time. Then, we use MCMC to generate samples from p(x,t). Finally, we use a diffusion model to denoise each x_n and obtain clean data.
- Here, to sample from p(x,t) we use Gibbs sampling.
- Specifically, suppose we currently have the MCMC iterate x_n, t_n. Then, x is updated via Langevin MCMC using the pre-trained score function. Next, t is updated with a diffusion time classifier, which predicts the diffusion time for a given sample.
- On CIFAR10, we verified whether DLG, a variant of DMCMC is able to accelerate diffusion sampling. The blue line denotes the performance of reverse-S/ODE sampler without DLG, and the orange line denotes the performance with DLG. We observed DLG was able to greatly accelerate all six samplers considered in this paper.
- We observed a similar trend on CelebA-HQ-256 as well. Dot denotes performance without DLG, cross denotes performance with DLG, and dotted line denotes performance improvement due to DLG.
- Surprisingly, DLG was able to achieve state-of-the-art performance in the limited NFE setting on CIFAR10.
- Qualitatively, we observed remarkable improvements in sample quality regardless of the reverse-S/ODE solver used.
- Qualitatively, we observed improvements on the FFHQ-1024 dataset as well. This demonstrates the scalability of DMCMC.