ICCV2009: MAP Inference in Discrete Models: Part 6: Recent Advances in Convex Relaxations
1. MAP Inference in Discrete Models
Recent Advances in Convex
Relaxations
M. Pawan Kumar, Stanford University
2. Outline
• Revisiting the LP relaxation
• Rounding Schemes and Move Making
• Beyond the LP relaxation
3. Linear Programming Relaxation
min Ty
ya;i [0,1]
∑i ya;i = 1
∑k yab;ik = ya;i
No reason why we can’t solve this*
*memory requirements, time complexity
4. Linear Programming Relaxation
Primal formulation is useful
Easier to analyze
LP better than a large class of relaxations
- QP (Ravikumar, Lafferty 2006)
- SOCP (Muramatsu, Suzuki 2003)
Kumar, Kolmogorov and Torr, NIPS 2007
5. Linear Programming Relaxation
Primal fractional solution is useful
Multiplicative Bounds
Type of Problem Bound
Potts 2
Truncated Linear 2 + √2
Truncated Quadratic O(√M)
General Metric O(log |L|)
6. Outline
• Revisiting the LP relaxation
• Rounding Schemes and Move Making
• Beyond the LP relaxation
7. Randomized Rounding
0 y’a;0 y’a;i y’a;k y’a;h = 1
y’a;i = ya;0 + ya;1 + … + ya;i
Choose an interval of length L’
8. Randomized Rounding
r
0 y’a;0 y’a;i y’a;k y’a;h = 1
y’a;i = ya;0 + ya;1 + … + ya;i
Generate a random number r (0,1]
9. Randomized Rounding
r
0 y’a;0 y’a;i y’a;k y’a;h = 1
y’a;i = ya;0 + ya;1 + … + ya;i
Assign label next to r (if within the interval)
10. Move Making
• Initialize the labeling
• Choose interval I of L’ labels
• Each variable can
• Retain old label
• Choose a label from I
• Choose best labeling
Va Vb Iterate over intervals
Truncated Convex Models
11. Two Problems
• Choose interval I of L’ labels
• Each variable can
• Retain old label
• Choose a label from I
• Choose best labeling
Large L’ => Non-submodular
Va Vb Non-submodular
12. First Problem
Va Vb Submodular problem
Ishikawa, 2003; Veksler, 2007
15. First Problem
am+1 bm+1
am+2 bm+2
am+2 bm+2
an bn
Va Vb t
16. First Problem
am+1 bm+1
am+2 bm+2
am+2 bm+2
an bn
Va Vb t
17. First Problem
am+1 bm+1
am+2 bm+2
am+2 bm+2
an bn
Va Vb t
18. First Problem
am+1 bm+1
am+2 bm+2
am+2 bm+2
an bn
Va Vb t
19. First Problem
am+1 bm+1
am+2 bm+2
am+2 bm+2
an bn
Va Vb t
Model unary potentials exactly
20. First Problem
am+1 bm+1
am+2 bm+2
am+2 bm+2
an bn
Va Vb t
Similarly for Vb
21. First Problem
am+1 bm+1
am+2 bm+2
am+2 bm+2
an bn
Va Vb t
Model convex pairwise costs
22. First Problem
Wanted to model
ab;ik = wab min{ d(i-k), M }
For all li, lk I
Have modelled
ab;ik = wab d(i-k)
For all li, lk I
Va Vb
Overestimated pairwise potentials
23. Second Problem
• Choose interval I of L’ labels
• Each variable can
• Retain old label
• Choose a label from I
• Choose best labeling
Va Vb Non-submodular problem !!
24. Second Problem
am+1 bm+1
am+2 bm+2
an bn
Va Vb t
Previous labels may not lie in interval
25. Second Problem
s
ua ub
am+1 bm+1
am+2 bm+2
an bn
Va Vb t
ua and ub : unary potentials for previous labels
26. Second Problem
s
ua ub
Pab
M Mb
am+1 ab m+1
am+2 bm+2
an bn
Va Vb t
Pab : pairwise potential for previous labels
27. Second Problem
s
ua ub
Pab
M M
am+1 ab bm+1
am+2 bm+2
an bn
Va Vb t
wab d(i-k)
28. Second Problem
s
ua ub
Pab
M M
am+1 ab bm+1
am+2 bm+2
an bn
Va Vb t
wab ( d(i-m-1) + M )
29. Second Problem
s
ua ub
Pab
M M
am+1 ab bm+1
am+2 bm+2
an bn
Va Vb t
Pab
30. Graph Construction
Find st-MINCUT.
Retain old labeling
if energy increases.
am+1 bm+1
am+2 bm+2
an bn
Va Vb t
ITERATE
31. Move Making
LP Bounds In General?
Kumar and Torr, NIPS 08 Kumar and Koller, UAI 09
Type of Problem Bound
Potts 2
Truncated Linear 2 + √2
Truncated Quadratic O(√M)
General Metric O(log |L|)
32. Outline
• Revisiting the LP relaxation
• Rounding Schemes and Move Making
• Beyond the LP relaxation
33. LP over a Frustrated Cycle
0 1 0 0 1 0 0 1 0
l1
0 0 0 0 0 0
l0
0 1 0 0 1 0 0 1 0
Va Vb Vb Vc Vc Va
Optimal labeling has energy = 1
One takes label l0, two take label l1
One takes label l1, two take label l0
34. LP optimal solution
0.5 0 0.5 0.5 0 0.5 0.5 0 0.5
l1
0.5 0.5 0.5 0.5 0.5 0.5
l0
0.5 0 0.5 0.5 0 0.5 0.5 0 0.5
Va Vb Vb Vc Vc Va
Optimal fractional labeling has energy = 0
Need tighter relaxations
40. Cycle Inequalities
Generalizes to cycles of arbitrary length
Barahona and Mahjoub, 1986
Generalizes to arbitrary label sets
Chopra and Rao, 1991
Sontag and Jaakkola, 2007
Modifies the primal
But weren’t we solving the dual?
41. Modifying the Dual
Do operations on trees and cycles
Which algorithm? Which cycles?
Kumar and Torr, 2008
TRW-S All cycles of length 3 and 4
Komodakis and Paragios, 2008
Dual Decomposition All frustrated cycles
Sontag et al, 2008
MPLP Iteratively add cycles
Maximum increase in the dual
