Neurips2021 presentation

1. Noise2Score: Tweedie’s Approach to Self-Supervised Imag
e Denoising without Clean Images
Kwanyoung Kim, Jong Chul Ye
Bio Imaging, Signal Processing & Learning Lab
@KAIST

2. Supervised Learning Approaches
• CNN as a direct inverse operation
• Most simplest and fastest method
• Supervised learning with lots of data
: feed-forward network

3. Self-Supervised
Feed-forward CNN?

4. Noise2Noise
Input Target
Output
L2 loss
Self-Supervised Learning: Noise2X
Lehtinen et al, ICML 2018

5. Input Target
L2 loss
Output
Noise2Self
Self-Supervised Learning: Noise2X
Batson et al, ICML 2019

6. Self-Supervised Learning: SURE
Soltanayev et al, NeurIPS, 2018
Self-supervised denoising using Stein Unsupervised Risk Estimator (SURE)
Divergence-based
penalty
Autoencoder loss

7. Supervised learning, Noise2X, SURE
Supervised learning
Noise2X : Noise2Noise, Noise2Self, Noise2Void, etc.
:samples of the noisy-clean image pair for training data
: target z is related to in unique ways depending on algorithms.
 most of the algorithmic choices are heuristic
SURE
: divergence penalty is added to compensate for use of y

8. Any unified mathematical
framework?

9. Yes! Noise2Score
Score function
Noise2X, SURE Noise2Score
Kim and Ye, NeurIPS, 2021

10. Tweedie’s formula for general exponential family
• Using the Bayes’ rule, the posterior density:
• Probability distribution of exponential family:
B Efron, Journal of the American Statistical Association, 2011

11. Tweedie’s formula for general exponential family
• The closed form solution for the posterior mean:
• Bayes optimal solution  posterior mean

12. Case1. Gaussian noise
• Tweedie’s formula calculates the posterior mean of x given y.
• Gaussian noise removal
Score function

13. Proof:

14. Case2. Poisson noise

15. Case3. Gamma noise

16. Tweedie’s formula of exponential family distribution for
image denoising
As long as we can compute the score function,
optimal denoising can be achieved by using Tweedie’s formula.

17. How to estimate the
score function?

18. Score function
Training data
≃
Score function
Score model
Figure courtesy from https://yang-song.github.io/blog/2021/score/

19. Score matching
Figure courtesy from https://yang-song.github.io/blog/2021/score/
Hyvärinen et al, JMLR, 2005

20. Score matching
Figure courtesy from https://yang-song.github.io/blog/2021/score/

21. Denoising Score matching
Figure courtesy from https://yang-song.github.io/blog/2021/score/

22. Denoising Autoencoder (DAE)
• DAE is to learn to recover data from the perturbed data
• DAE can be used to estimate the score function of data
Alain et al, JMLR, 2014
Equal to Tweedie’s formula for Gaussian noises

23. Amortized – Residual DAE (AR-DAE)
• The residual from of the DAE:
• Addresses instability and reduce approximation error
• Directly estimate the score function:
Lim et al, ICML, 2020

24. Relation to SURE
By using AR-DAE transform
Score matching cost by Hyvärinen et al, JMLR, 2005

25. Noise2Score
Score function
Kim and Ye, NeurIPS, 2021

30. Conclusion
• Noise2Score: a novel unified framework for self-supervised image
denoising.
• Our Noise2Score can be applied to any image denoising problem from
exponential family noises.
• Identical neural network training can be used regardless of noise model.

31. Thank you!
Jong Chul Ye
E-mail:
jong.ye@kaist.ac.kr
Kwanyoung Kim
E-mail:
cubeyoung@kaist.ac.kr