Julie Delon speaks about imaging science and presents the Plug and Play approach to address inverse problems related to image and video restoration.

1. Plug-and-Play methods for
inverse problems in imaging
Julie Delon

3. x ∈ ℝn

4. Inverse problems in imaging
[The Mandalorian, Disney, 2020]

5. Inverse problems
image x additive noise
A = In
blur
A =
a0 a1 … an−1
an−1 a0 … an−2
a1 a2 … a0
missing pixels
A =
(
? 0 … 0
0 ? … 0
0 0 … ?)
Goal : estimate from the observation
x y
ℝn
ℝd
ℝd
ℝd×n
observation noise 𝒩(0,σ2
)
unknown
degradation operator
y = Ax + n

6. Classical approaches
•Least squares
not always one-to-one
and its eigenvalues
may be small.
= (A*A)−1
A*y
̂
x = argminx∥y − Ax∥2
2

7. From inverse problems to optimization
arg min
x
1
2σ2
∥y − Ax∥2
2 + U(x)
Example: Total variation [Rudin-Osher-Fatemi 1992 for A = Id]
U(x) = TV(x) =
∑
i
∥∇xi ∥

8. Neural networks as regularizers
arg min
x
1
2σ2
∥y − Ax∥2
2 + U(x)
64 64 64
. . .
64 64 3
y → fθ(y)
Problem specific, must be trained again for a new inverse problem
̂
θ = arg min
θ∈Θ
1
N
N
∑
i=1
∥fθ (Axi + ni) − xi∥2
Database of images.
(xi)i

9. From inverse problems to optimization
arg min
x
1
2σ2
∥y − Ax∥2
2
F(x,y)
+ U(x)
What if is not
smooth?
U
Gradient Descent xk+1 = xk − γ∇F(xk, y) − γ∇U(xk)
F + U
x
xk xk+1
xk−1

10. Non smooth optim. / Proximal operator
(explicit GD)
xk+1 = xk − γ∇f(xk) for f smooth
arg min
x
f(x) with f convex

11. Non smooth optim. / Proximal operator
arg min
x
f(x) with f convex
(implicit GD)
xk+1 = xk − γ∇f(xk+1)
⇔ xk+1 = arg min
t
1
2
∥t − xk∥2
2 + γf(t)
Property x̄ ∈ arg min f ⇔ x̄ = proxγf(x̄)
Algorithm x0 initialization
xk+1 = proxγf(xk)
for f smooth
proxγf(xk) Defined even for f not smooth!

12. Convex optimization
argmin
x
F(x, y) + U(x)
Numerous sophistications of gradient descent to minimize :
proximal gradient descent, primal-dual methods etc…
→ F + U
Proximal gradient (Forward-Backward splitting)
xk+1 = proxγU(xk − γ∇F(xk, y))
Gradient Descent: xk+1 = xk − γ∇F(xk, y) − γ∇U(xk)
Known convergence properties in the convex case.
[Moreau 1965], [Combettes-Pesquet 2011], [Chambolle-Pock, 2011]…

13. Plug-and-Play
Plug and Play: replace in proximal optimization
schemes by a well chosen denoiser .
proxγU
Dγ
solves a denoising problem for the regularizer !
with .
proxγU U
y = x + n n ∼ 𝒩(0, γ)
arg min
x
1
2σ2
∥y − Ax∥2
2 + U(x)
proxγU(y) = arg min
x
1
2γ
∥y − x∥2
2 + U(x)
y = Ax + n

14. Plug-and-Play optimization methods
Plug-and-Play
Proximal gradient (Forward-Backward splitting)
xk+1 = proxγU(xk − γ∇F(xk, y))
argmin
x
F(x, y) + U(x)

15. Plug-and-Play optimization methods
Plug-and-Play
Proximal gradient (Forward-Backward splitting)
xk+1 = Dγ (xk − γ∇F(xk, y))
argmin
x
F(x, y) + U(x)
• Versatility/flexibility: one network to rule them all!
• Used with optimization schemes and denoisers [Meinhardt et al. 2017, Ryu
et al. 2019, Xu et al. 2020, Sun et al. 2021,…]
• State-of the-art results when the denoiser is powerful enough [DPIR, Zhang
et al. 2021]
≠ ≠
Deblurring Demoisaicing Superresolution
Convergence analysis under appropriate conditions on F and U

16. [Monod et al. 2022]
PnP video - deblurring

17. [Monod et al. 2022]
PnP video - interpolation

18. Plug-and-Play optimization methods
Take home message
• NN for inv. pb.: powerful, power-consuming, new training for each problem.
• Plug-and-Play: plug a powerful denoiser in proximal optimization schemes.
• One NN, trained once, able to solve numerous inverse problems!
• Solution for embeddable systems with limited memory
• Mathematicians: find appropriate conditions on the denoiser and the inverse
problem to ensure convergence
THANKS!
storimaging.github.io