Not Enough Measurements, Too Many Measurements

Not Enough Measurements,
Too Many Measurements
Michael McCann
Department of Computational
Mathematics, Science and Engineering
Michigan State University
UM CSP Seminar, Oct. 22, 2020

2
My collaborators
●
Michael Unser and
BIG (EPFL)
– Kyong Jin
– Laurène Donati
– Harshit Gupta
●
Sai Ravishankar
and SLIM (MSU)
– Avrajit Ghosh

3
Goal: reconstruct meaningful
images from measurements
what we have:
measurements
what we want: image
The Physics

7
Not Enough Measurements
●
measurements are expensive
– time, radiation exposure, …
●
want to take M << N measurements
●
it’s hopeless!

8
Circa 2008-2016, the next slide
would have been...

10
Too Many Measurements!
●
e.g., The Cancer Imaging Archive
– 2223 subjects x hundreds of images
– includes Patient CT Projection Data Library
(AKA Mayo Clinic Data)

11
Goal: reconstruct meaningful
images from measurements PLUS a
training set
what we have:
training
what we have:
measurements
what we want: image
The Physics

12
Supervised image reconstruction

13
Supervised image reconstruction
●
What do we pick for D?
●
Where does the training data come from?
●
How do we solve the fitting problem?
●
What is the structure of F?

15
Themes in designing F
●
augment a direct method
– e.g., mix together different FBPs (Pelt et al. 2013)
denoise an FBP (Jin et al. 2017)

16
●
●
take inspiration from variational methods
– unrolling, plug-and-play, ...
– e.g., learn the regularization (Aggarwal et al. 2019;
Gupta et al. 2018)
learn the gradient (Adler et al. 2017),
learn filters and nonlinearities (Hammernik et al.
2017)

17
●
●
– e.g., learn the regularization (Aggarwal et al. 2019; Gupta et
al. 2018)
learn filters and nonlinearities (Hammernik et al. 2017)
●
throw away variational methods
– e.g., learn the entire measurement to image mapping (Zhu et
al. 2018)

18
●
●
– e.g., learn the regularization (Aggarwal et al. 2019; Gupta et al. 2018)
●
– e.g., learn the entire measurement to image mapping (Zhu et al. 2018)
●
work in the data domain
– e.g., learn to inpaint missing measurements (Ghani et al. 2019)

19
●
●
– e.g., learn the regularization (Aggarwal et al. 2019; Gupta et al. 2018)
●
– e.g., learn the entire measurement to image mapping (Zhu et al. 2018)
●
work in the data domain
– e.g., learn to inpaint missing measurements (Ghani et al. 2019)
●
many more...

20
Results: biomedical images
(Jin, McCann, Froustey, Unser 2017)

21

22

24
Perspectives
●
It’s not about H, it’s about 𝓗

25
Perspectives
●
●
If we care about MSE, let’s do challenges
– fastMRI, Low Dose CT Grand Challenge, others?

26
Perspectives
●
●
If we care about MSE, let’s do challenges
– fastMRI, Low Dose CT Grand Challenge, others?
●
Let’s not forget data fidelity and robustness
ground truth FBPConvNet

27
Summary so far
X-ray CT
low-dose X-ray CT
(“impossible”)

28
Summary so far
X-ray CT
supervised
low-dose X-ray CT

29
Next topic
X-ray CT
supervised
low-dose X-ray CT

30
Next topic
X-ray CT
supervised
low-dose X-ray CT

31
Next topic
X-ray CT
supervised
low-dose X-ray CT

32
Single-particle cryo-EM
●
~10^6 projections of a single particle
●
random orientations, (electron) optical effects,
SNR ≈ 0 dB
●
subnanometer resolution

33
●
●
SNR ≈ 0 dB
●

34
●
●
SNR ≈ 0 dB
●

35
●
●
SNR ≈ 0 dB
●
right: Wu et al. 2020

36
what we have:
measurements
what we want:
image
The Physics
unknown
nuisance

37
Cryo-EM reconstruction
●
projection matching (Penczek et al. 1994)
– # of unknowns grows with data :(
●
marginalized maximum likelihood (Sigworth 1998)
– marginalization is computationally heavy
– what most software currently does
●
method of moments (Kam 1980; Sharon et al. 2020)
– avoids above problems
– makes efficient use of data

38
Generative adversarial networks
●
given samples of a distribution, generate new
samples from the same distribution
image: Goodfellow et al. 2014

39
Generative adversarial networks
●
given samples of a distribution, generate new
samples from the same distribution
●
parameterization of the generator is flexible
image: Goodfellow et al. 2014

40
Typical GAN
(Gupta, McCann, Donati, Unser. 2020)

41
Typical GAN

42
Typical GAN

44
thisprojectiondoesnotexist.com
Not obviously useful!
(Gupta et al. 2020)

45
CryoGAN
(Gupta et al. 2020)

48
for more on CryoGAN, SPACE Webinar: https://
www.youtube.com/watch?v=J6UZBeU3Bm0
(Gupta et al. 2020)

50
Back to low-dose X-ray CT
(Jin et al. 2017)

51
(Jin et al. 2017)

52
(Jin et al. 2017)

53
A “that’s funny…”moment
figures and table: Adler et al. 2017

54
Can we learn a regularizer from
data?
●
recall supervised reconstruction
●
our proposed “architecture”

56
Why learn a regularizer?
●
can’t be worse than sparsity-based
reconstruction

57
●
reconstruction
●
results in a a convex problem
– robust, includes data fidelity, decades of theory

58
●
reconstruction
●
●
joins proven architecture with data adaptivity

59
●
can’t be worse than sparsity-based reconstruction
●
●
joins proven architecture with data adaptivity
●
gives a hope of interpreting the learned part

60
Related work
●
Main approach: relax the l1 term
– Peyré et al. 2011; Mairal et al. 2012; Sprechmann et al. 2013; Chen et al. 2014

61
Related work
●

62
Related work
●

63
Related work
●

65
Our approach (outline)
1. solve the lower problem at the current W

66
2. find a (local) closed-form solution, x*(W)

67
2. find a (local) closed-form solution, x*(W)
3. substitute x*(W) into the upper level problem,
compute a gradient w.r.t. W, and descend

68
1. Solve the lower level problem
●
standard, convex problem
– ADMM, Chambolle-Pock
●
need fast, accurate solutions
– hyperparameter selection is tough

69
2. Find a closed-form solution

70
●
uniqueness? no, but

71
●
uniqueness? no, but
●
intuition: in a region (of W-space) where the
sign pattern of Wx*(W) does not change,
||Wx||1 is linear

72
●
uniqueness? no, but
●
intuition: in a region (of W-space) where the
sign pattern of Wx*(W) does not change,
||Wx||1 is linear
– see McCann and Ravishankar 2020 for A = I

73
●
from McCann and Ravishankar 2020

74
●
●
even better, Ali and Tibshirani 2019

75
●
●
even better, Ali and Tibshirani 2019
– unique minimum norm solution when b=0

77
3. Find the gradient
●
with 1d signals, pytorch can autograd these
expressions
– Agrawal et al. 2019 makes it even easier

78
3. Find the gradient
●
with 1d signals, pytorch can autograd these
expressions
– Agrawal et al. 2019 makes it even easier
●
with images, things get tough
– W is potentially million X million
– no more SVDs or explicit inverses
– our solution: by hand + pytorch

79
Early experiments
●
image denoising
●
W is a set of 8, 3x3 convolutions
●
compare to
– BM3D, TV, DCT
– unsupervised learned regularizer
●
training
– SGD with increasing batch size
– takes ~hours

83
Absolute summed filter responses
and filters
●
Learned filters
– Are not orthonormal (neither ortho- nor -normal)
– Penalize edges less than DCT

85
Taking a step back
●
CNN performance on image reconstruction is
due to
– the CNN (architecture)
– training

86
Taking a step back
●
due to
– training
●
remove the training: deep image prior
– Reinhard Heckel 1W-MINDS seminar:
https://www.youtube.com/watch?v=AvJgmbeupGY

87
Taking a step back
●
due to
– training
●
remove the training: deep image prior
– Reinhard Heckel 1W-MINDS seminar:
https://www.youtube.com/watch?v=AvJgmbeupGY
●
remove the CNN: learned regularizer

88
Thanks for your attention!
michael.thompson.mccann@gmail.com
slides:

89
References
●
M. Wu, G. C. Lander, and M. A. Herzik, “Sub-2 Angstrom resolution structure determination using single-
particle cryo-EM at 200 keV,” Journal of Structural Biology: X, vol. 4, p. 100020, 2020, doi:
10.1016/j.yjsbx.2020.100020.
●
Pawel A. Penczek, Robert A. Grassucci, and Joachim Frank. “The ribosome at improved resolution:
New techniques for merging and orientation refinement in 3D cryo-electron microscopy of biological
particles”. In:Ultramicroscopy 53.3 (1994), pp. 251–270.
●
Fred J Sigworth. “A maximum-likelihood approach to single-particle image refinement”. In:Journal of
structural biology 122.3 (1998), pp. 328–339.
●
Z. Kam, “The reconstruction of structure from electron micrographs of randomly oriented particles,” Journal
of Theoretical Biology, vol. 82, no. 1, Art. no. 1, Jan. 1980, doi: 10.1016/0022-5193(80)90088-0.
●
N. Sharon, J. Kileel, Y. Khoo, B. Landa, and A. Singer, “Method of moments for 3D single particle ab initio
modeling with non-uniform distribution of viewing angles,” Inverse Problems, vol. 36, no. 4, Art. no. 4, Feb.
2020, doi: 10.1088/1361-6420/ab6139.
●
I. Goodfellow et al., “Generative Adversarial Nets,” in Advances in Neural Information Processing Systems
27, Ed.Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger Curran Associates,
Inc., 2014, pp. 2672–2680.
●
J. Adler and O. Öktem, “Solving ill-posed inverse problems using iterative deep neural networks,” Inverse
Problems, vol. 33, no. 12, p. 124007, Nov. 2017.

90
References
●
H. Gupta, M. T. McCann, L. Donati, and M. Unser, “CryoGAN: A New Reconstruction Paradigm for
Single-Particle Cryo-EM via Deep Adversarial Learning,” bioRxiv, Mar. 2020, doi:
10.1101/2020.03.20.001016.
●
G. Peyré and J. M. Fadili, “Learning analysis sparsity priors,” in Sampling Theory and Applications,
Singapore, Singapore, May 2011, p. 4.
●
J. Mairal, F. Bach, and J. Ponce, “Task-driven dictionary learning,”IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol. 34, no. 4, pp. 791–804, Apr. 2012.
●
P. Sprechmann, R. Litman, T. Ben Yakar, A. M. Bronstein, and G. Sapiro, “Supervised sparse analysis
and synthesis operators,” in Advances in Neural Information Processing Systems 26, 2013, pp. 908–
916.
●
M. T. McCann and S. Ravishankar, “Supervised Learning of Sparsity-Promoting Regularizers for
Denoising,” arXiv:2006.05521 [eess.IV], 2020-06-09.
●
A. Ali and R. J. Tibshirani, “The Generalized Lasso Problem and Uniqueness,” Electronic Journal of
Statistics, vol. 13, no. 2, Art. no. 2, 2019, doi: 10.1214/19-ejs1569.
●
A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Z. Kolter, “Differentiable Convex
Optimization Layers,” in Advances in Neural Information Processing Systems 32, Ed.H. Wallach, H.
Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett Curran Associates, Inc., 2019, pp.
9562–9574.

Not Enough Measurements, Too Many Measurements

Recommended

Recommended

More Related Content

What's hot

What's hot (14)

Similar to Not Enough Measurements, Too Many Measurements

Similar to Not Enough Measurements, Too Many Measurements (20)

Recently uploaded

Recently uploaded (20)

Not Enough Measurements, Too Many Measurements