This document summarizes the Alternating Direction Method of Multipliers (ADMM) algorithm. It discusses how ADMM can be used to solve optimization problems of the form minimize f(x) + g(z) subject to Kx - z = 0, where Kx - z decomposes the problem into separable subproblems for x and z. The algorithm alternates between optimizing the augmented Lagrangian Lρ(x, z, u) with respect to x, z, and their dual variables u. Each subproblem can be solved efficiently using proximal operators, often having closed-form solutions in frequency space.

1. Masayuki Tanaka
Aug. 24, 2016
ADMM algorithm in ProxImaL [1]

2. Proximal operator [2]
prox 𝑓,𝜌 𝒗 = arg min
𝒙
𝑓 𝒙 +
𝜌
2
𝒙 − 𝒗 2
2
1

3. Alternating Direction Method of Multipliers
[2]
 ADMM problem
Minimize 𝐼 𝒙, 𝒛 = 𝑓 𝒙 + 𝑔(𝒛) subject to 𝑨𝒙 + 𝑩𝒛 = 𝒄
 Augmented Lagrangian
𝐿 𝜌 𝒙, 𝒛, 𝒚 = 𝑓 𝒙 + 𝑔 𝒛 + 𝒚 𝑇
𝑨𝒙 + 𝑩𝒛 − 𝒄 +
𝜌
2
𝑨𝒙 + 𝑩𝒛 − 𝒄 2
2
= 𝑓 𝒙 + 𝑔 𝒛 +
𝜌
2
𝑨𝒙 + 𝑩𝒛 − 𝒄 + 𝑢 2
2
+ const. with scaled dual variables 𝒖 =
1
𝜌
𝒚
𝐿 𝜌 𝒙, 𝒛, 𝒖 = 𝑓 𝒙 + 𝑔 𝒛 +
𝜌
2
𝑨𝒙 + 𝑩𝒛 − 𝒄 + 𝒖 2
2
 ADMM algorithm
𝒙 𝑘+1 = arg min
𝒙
𝐿 𝜌 (𝒙, 𝒛 𝑘, 𝒖 𝑘) = arg min
𝒙
𝑓 𝒙 +
𝜌
2
𝑨𝒙 + 𝑩𝒛 𝑘 − 𝒄 + 𝒖 𝑘
2
2
𝒛 𝑘+1 = arg min
𝒛
𝐿 𝜌 (𝒙 𝑘+1, 𝒛, 𝒖 𝑘) = arg min
𝒛
𝑔 𝒛 +
𝜌
2
𝑨𝒙 𝑘+1 + 𝑩𝒛 − 𝒄 + 𝒖 𝑘
2
2
𝒖 𝑘+1 = 𝒖 𝑘 + 𝑨𝒙 𝑘+1 + 𝑩𝒛 𝑘+1 − 𝒄
2

4. Problem of Image Processing
 Example of Cost Function in Image Processing
𝐼 𝒙 = 𝒚 − 𝑫𝒙 2
2
+ 𝛼 𝛻𝒙 1
 General Formulation of Cost Function in Image Proces
𝐼 𝒙 =
𝑖
𝑓𝑖(𝒙) +
𝑗
𝑔𝑗(𝑲𝑗 𝒙)
Quadratic form Non-quadratic form
𝐼 𝒙 =
𝑖
𝑓𝑖(𝒙) +
𝑗
𝑔𝑗(𝒛𝑗) 𝑲𝑗 𝒙 − 𝒛𝑗 = 𝟎
𝐼 𝒙 = 𝑓(𝒙) + 𝑔(𝒛) 𝑲𝒙 − 𝒛 = 𝟎 𝑲 =
𝑲1
𝑲2
⋮
𝒛 =
𝒛1
𝒛2
⋮
𝒛𝑗 = 𝑹𝑗 𝒛
3
𝑔 𝒛 =
𝑗
𝑔𝑗(𝑹𝑗 𝒛 )𝑓 𝒙 =
𝑖
𝑓𝑖(𝒙)
ADMM problem formulation

5. Augmented Lagrangian
4
𝐼 𝒙 = 𝑓(𝒙) + 𝑔(𝒛) 𝑲𝒙 − 𝒛 = 𝟎 𝑲 =
𝑲1
𝑲2
⋮
𝒛 =
𝒛1
𝒛2
⋮
𝒛𝑗 = 𝑹𝑗 𝒛𝑔 𝒛 =
𝑗
𝑔𝑗(𝑹𝑗 𝒛 )𝑓 𝒙 =
𝑖
𝑓𝑖(𝒙)
ADMM problem formulation
𝐿 𝜌 𝒙, 𝒛, 𝒖 = 𝑓 𝒙 + 𝑔 𝒛 +
𝜌
2
𝑨𝒙 + 𝑩𝒛 − 𝒄 + 𝒖 2
2
𝒖𝑗 = 𝑹𝑗 𝒖
𝑨𝒙 + 𝑩𝒛 − 𝒄 + 𝒖 2
2
= 𝑲𝒙 − 𝒛 + 𝒖 2
2
=
𝑲1
𝑲2
⋮
𝒙 −
𝒛1
𝒛2
⋮
+
𝒖1
𝒖2
⋮ 2
2
=
𝑗
𝑲𝑗 𝒙 − 𝒛𝑗 + 𝒖𝑗 2
2
𝐿 𝜌 𝒙, 𝒛, 𝒖 =
𝑖
𝑓𝑖(𝒙) +
𝑗
𝑔𝑗(𝒛𝑗 ) +
𝜌
2
𝑗
𝑲𝑗 𝒙 − 𝒛𝑗 + 𝒖𝑗 2
2

6. ADMM Algorithm
5
𝐿 𝜌 𝒙, 𝒛, 𝒖 =
𝑖
𝑓𝑖(𝒙) +
𝑗
𝑔𝑗(𝒛𝑗 ) +
𝜌
2
𝑗
𝑲𝑗 𝒙 − 𝒛𝑗 + 𝒖𝑗 2
2
𝒙 𝑘+1 = arg min
𝒙
𝐿 𝜌 (𝒙, 𝒛 𝑘, 𝒖 𝑘) = arg min
𝒙
𝑖
𝑓𝑖(𝒙) +
𝜌
2
𝑗
𝑲𝑗 𝒙 − 𝒛𝑗
𝑘
+ 𝒖𝑗
𝑘
2
2
𝒛𝑗
𝑘+1
= arg min
𝒛 𝑗
𝐿 𝜌 (𝒙 𝑘+1, 𝒛, 𝒖 𝑘) = arg min
𝒛 𝑗
𝑔𝑗(𝒛𝑗 ) +
𝜌
2
𝑲𝑗 𝒙 𝑘+1 − 𝒛𝑗 + 𝒖𝑗
𝑘
2
2
𝒖𝑗
𝑘+1
= 𝒖𝑗
𝑘
+ 𝑲𝑗 𝒙 𝑘+1 − 𝒛𝑗
𝑘+1
𝒛𝑗, 𝒖𝑗 can be independently calculated.
= prox 𝑔 𝑗,𝜌 (𝑲𝑗 𝒙 𝑘+1
+ 𝒖𝑗
𝑘
)
It usually has the closed form solution in the frequency
domain.

7. References
[1] F. Heide et al, ProxImaL: Efficient Image Optimization using Proximal
Algorithms, SIGGRAPH 2016.
https://graphics.stanford.edu/~niessner/heide2016proximal.html
[2] S. Boyd et al, Distributed Optimization and Statistical Learning via the
Alternating Direction Method of Multipliers, Foundations and Trends in
Machine Learning 2011.
http://web.stanford.edu/~boyd/papers/admm_distr_stats.html
6