Upcoming SlideShare
×

# ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2

632 views
548 views

Published on

0 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

Views
Total views
632
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
17
0
Likes
0
Embeds 0
No embeds

No notes for slide
• Move to front
• correct
• correct
• Chi square
• Chi square
• ### ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2

1. 1. MAP Estimation Algorithms in M. Pawan Kumar, University of Oxford Pushmeet Kohli, Microsoft Research Computer Vision - Part II
2. 2. Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg Image (D) i i,j n = number of pixels
3. 3. Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg i i,j Unary Cost (c i ) Dark ( negative ) Bright (positive) n = number of pixels
4. 4. Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg i i,j Discontinuity Cost (c ij ) n = number of pixels
5. 5. Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg i i,j Global Minimum (x * ) x * = arg min E(x) x How to minimize E(x)? n = number of pixels
6. 6. Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Recent Advances and Open Problems Connection between st-mincut and energy minimization?
7. 7. Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
8. 8. The st-Mincut Problem Source Sink v 1 v 2 2 5 9 4 2 1 <ul><li>Graph (V, E, C) </li></ul><ul><ul><li>Vertices V = {v 1 , v 2 ... v n } </li></ul></ul><ul><ul><li>Edges E = {(v 1 , v 2 ) ....} </li></ul></ul><ul><ul><li>Costs C = {c (1, 2) ....} </li></ul></ul>
9. 9. The st-Mincut Problem Source Sink v 1 v 2 2 5 9 4 2 1 What is a st-cut?
10. 10. The st-Mincut Problem Source Sink v 1 v 2 2 5 9 4 2 1 What is a st-cut? An st-cut ( S , T ) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T 5 + 2 + 9 = 16
11. 11. The st-Mincut Problem What is a st-cut? An st-cut ( S , T ) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T What is the st-mincut? st-cut with the minimum cost Source Sink v 1 v 2 2 5 9 4 2 1 2 + 1 + 4 = 7
12. 12. How to compute the st-mincut? Source Sink v 1 v 2 2 5 9 4 2 1 Solve the dual maximum flow problem In every network, the maximum flow equals the cost of the st-mincut Min-cutMax-flow Theorem Compute the maximum flow between Source and Sink Constraints Edges: Flow < Capacity Nodes: Flow in = Flow out
13. 13. Maxflow Algorithms Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Source Sink v 1 v 2 2 5 9 4 2 1 Algorithms assume non-negative capacity Flow = 0
14. 14. Maxflow Algorithms Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Source Sink v 1 v 2 2 5 9 4 2 1 Algorithms assume non-negative capacity Flow = 0
15. 15. Maxflow Algorithms Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Source Sink v 1 v 2 2-2 5-2 9 4 2 1 Algorithms assume non-negative capacity Flow = 0 + 2
16. 16. Maxflow Algorithms Source Sink v 1 v 2 0 3 9 4 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 2
17. 17. Maxflow Algorithms Source Sink v 1 v 2 0 3 9 4 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 2
18. 18. Maxflow Algorithms Source Sink v 1 v 2 0 3 9 4 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 2
19. 19. Maxflow Algorithms Source Sink v 1 v 2 0 3 5 0 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 2 + 4
20. 20. Maxflow Algorithms Source Sink v 1 v 2 0 3 5 0 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 6
21. 21. Maxflow Algorithms Source Sink v 1 v 2 0 3 5 0 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 6
22. 22. Maxflow Algorithms Source Sink v 1 v 2 0 2 4 0 2+1 1-1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 6 + 1
23. 23. Maxflow Algorithms Source Sink v 1 v 2 0 2 4 0 3 0 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 7
24. 24. Maxflow Algorithms Source Sink v 1 v 2 0 2 4 0 3 0 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 7
25. 25. History of Maxflow Algorithms [Slide credit: Andrew Goldberg] Augmenting Path and Push-Relabel n: # nodes m: # edges U: maximum edge weight Algorithms assume non-negative edge weights
26. 26. History of Maxflow Algorithms [Slide credit: Andrew Goldberg] Augmenting Path and Push-Relabel n: # nodes m: # edges U: maximum edge weight Algorithms assume non-negative edge weights
27. 27. Augmenting Path based Algorithms a 1 a 2 1000 1 Sink Source 1000 1000 1000 0 Ford Fulkerson: Choose any augmenting path
28. 28. Augmenting Path based Algorithms a 1 a 2 1000 1 Sink Source 1000 1000 1000 0 Bad Augmenting Paths Ford Fulkerson: Choose any augmenting path
29. 29. a 1 a 2 1000 1 Sink Source 1000 1000 1000 0 Augmenting Path based Algorithms Bad Augmenting Path Ford Fulkerson: Choose any augmenting path
30. 30. a 1 a 2 999 0 Sink Source 1000 1000 999 1 Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting path
31. 31. Augmenting Path based Algorithms a 1 a 2 999 0 Sink Source 1000 1000 999 1 Ford Fulkerson: Choose any augmenting path n: # nodes m: # edges We will have to perform 2000 augmentations! Worst case complexity: O (m x Total_Flow) (Pseudo-polynomial bound: depends on flow)
32. 32. Augmenting Path based Algorithms Dinic: Choose shortest augmenting path n: # nodes m: # edges Worst case Complexity: O (m n 2 ) a 1 a 2 1000 1 Sink Source 1000 1000 1000 0
33. 33. Maxflow in Computer Vision <ul><li>Specialized algorithms for vision problems </li></ul><ul><ul><li>Grid graphs </li></ul></ul><ul><ul><li>Low connectivity (m ~ O(n)) </li></ul></ul><ul><li>Dual search tree augmenting path algorithm </li></ul><ul><li>[Boykov and Kolmogorov PAMI 2004] </li></ul><ul><ul><li>Finds approximate shortest augmenting paths efficiently </li></ul></ul><ul><ul><li>High worst-case time complexity </li></ul></ul><ul><ul><li>Empirically outperforms other algorithms on vision problems </li></ul></ul>
34. 34. Maxflow in Computer Vision <ul><li>Specialized algorithms for vision problems </li></ul><ul><ul><li>Grid graphs </li></ul></ul><ul><ul><li>Low connectivity (m ~ O(n)) </li></ul></ul><ul><li>Dual search tree augmenting path algorithm </li></ul><ul><li>[Boykov and Kolmogorov PAMI 2004] </li></ul><ul><ul><li>Finds approximate shortest augmenting paths efficiently </li></ul></ul><ul><ul><li>High worst-case time complexity </li></ul></ul><ul><ul><li>Empirically outperforms other algorithms on vision problems </li></ul></ul><ul><ul><li>Efficient code available on the web </li></ul></ul>http://www.adastral.ucl.ac.uk/~vladkolm/software.html
35. 35. Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
36. 36. St-mincut and Energy Minimization E: {0,1} n -> R Minimizing a Qudratic Pseudoboolean function E(x) Functions of boolean variables Pseudoboolean? Polynomial time st-mincut algorithms require non-negative edge weights E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) c ij ≥0 i,j i T S st-mincut
37. 37. So how does this work? <ul><li>Construct a graph such that: </li></ul><ul><li>Any st-cut corresponds to an assignment of x </li></ul><ul><li>The cost of the cut is equal to the energy of x : E(x) </li></ul>E(x) Solution T S st-mincut
38. 38. Graph Construction Sink (1) Source (0) a 1 a 2 E(a 1 ,a 2 )
39. 39. Graph Construction Sink (1) Source (0) a 1 a 2 E(a 1 ,a 2 ) = 2a 1 2
40. 40. Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 2 5 Sink (1) Source (0)
41. 41. Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 2 5 9 4 Sink (1) Source (0)
42. 42. Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 2 5 9 4 2 Sink (1) Source (0)
43. 43. Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)
44. 44. Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)
45. 45. Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2 a 1 + 5 ā 1 + 9 a 2 + 4 ā 2 + 2 a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 a 1 = 1 a 2 = 1 E (1,1) = 11 Cost of cut = 11 Sink (1) Source (0)
46. 46. Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2 a 1 + 5 ā 1 + 9 a 2 + 4 ā 2 + 2 a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0) a 1 = 1 a 2 = 0 E (1,0) = 8 st-mincut cost = 8
47. 47. Energy Function Reparameterization Two functions E 1 and E 2 are reparameterizations if E 1 ( x ) = E 2 ( x ) for all x For instance: E 1 (a 1 ) = 1+ 2a 1 + 3ā 1 E 2 (a 1 ) = 3 + ā 1 a 1 ā 1 1+ 2a 1 + 3ā 1 3 + ā 1 0 1 4 4 1 0 3 3
48. 48. Flow and Reparametrization a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)
49. 49. Flow and Reparametrization a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0) 2a 1 + 5ā 1 = 2(a 1 +ā 1 ) + 3ā 1 = 2 + 3ā 1
50. 50. Flow and Reparametrization Sink (1) Source (0) a 1 a 2 E(a 1 ,a 2 ) = 2 + 3ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 0 3 9 4 2 1 2a 1 + 5ā 1 = 2(a 1 +ā 1 ) + 3ā 1 = 2 + 3ā 1
51. 51. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 2 + 3ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 0 3 9 4 2 1 9a 2 + 4ā 2 = 4(a 2 +ā 2 ) + 5ā 2 = 4 + 5ā 2
52. 52. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 2 + 3ā 1 + 5a 2 + 4 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1 9a 2 + 4ā 2 = 4(a 2 +ā 2 ) + 5ā 2 = 4 + 5ā 2
53. 53. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1
54. 54. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1
55. 55. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1 3ā 1 + 5a 2 + 2a 1 ā 2 = 2(ā 1 +a 2 +a 1 ā 2 ) +ā 1 +3a 2 = 2(1+ā 1 a 2 ) +ā 1 +3a 2 F1 = ā 1 +a 2 +a 1 ā 2 F2 = 1+ā 1 a 2 a 1 a 2 F1 F2 0 0 1 1 0 1 2 2 1 0 1 1 1 1 1 1
56. 56. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 0 1 3 0 0 3 3ā 1 + 5a 2 + 2a 1 ā 2 = 2(ā 1 +a 2 +a 1 ā 2 ) +ā 1 +3a 2 = 2(1+ā 1 a 2 ) +ā 1 +3a 2 F1 = ā 1 +a 2 +a 1 ā 2 F2 = 1+ā 1 a 2 a 1 a 2 F1 F2 0 0 1 1 0 1 2 2 1 0 1 1 1 1 1 1
57. 57. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 0 1 3 0 0 3 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 No more augmenting paths possible
58. 58. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 0 1 3 0 0 3 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Total Flow Residual Graph (positive coefficients) bound on the optimal solution Inference of the optimal solution becomes trivial because the bound is tight
59. 59. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 0 1 3 0 0 3 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 a 1 = 1 a 2 = 0 E (1,0) = 8 st-mincut cost = 8 Total Flow bound on the optimal solution Inference of the optimal solution becomes trivial because the bound is tight Residual Graph (positive coefficients)
60. 60. Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg i i,j Global Minimum (x * ) x * = arg min E(x) x How to minimize E(x)?
61. 61. How does the code look like? Sink (1) Source (0) Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1)
62. 62. How does the code look like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) a 1 a 2 fgCost( a 1 ) Sink (1) Source (0) fgCost( a 2 ) bgCost( a 1 ) bgCost( a 2 )
63. 63. How does the code look like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) a 1 a 2 fgCost( a 1 ) Sink (1) Source (0) fgCost( a 2 ) bgCost( a 1 ) bgCost( a 2 ) cost(p,q)
64. 64. How does the code look like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) a 1 a 2 fgCost( a 1 ) Sink (1) Source (0) fgCost( a 2 ) bgCost( a 1 ) bgCost( a 2 ) cost(p,q) a 1 = bg a 2 = fg
65. 65. Image Segmentation in Video Image Flow Global Optimum s t = 0 = 1 E(x) x * n-links st-cut
66. 66. Image Segmentation in Video Image Flow Global Optimum
67. 67. Dynamic Energy Minimization E B computationally expensive operation E A Recycling Solutions Can we do better? Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07) S B S A minimize
68. 68. Dynamic Energy Minimization E B computationally expensive operation E A cheaper operation Kohli & Torr (ICCV05, PAMI07) 3 – 100000 time speedup! Reuse flow Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07) S B S A minimize Simpler energy E B* differences between A and B A and B similar Reparametrization
69. 69. Dynamic Energy Minimization Reparametrized Energy Kohli & Torr (ICCV05, PAMI07) Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07) E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Original Energy E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 7a 1 ā 2 + ā 1 a 2 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 + 5a 1 ā 2 New Energy New Reparametrized Energy
70. 70. Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
71. 71. Minimizing Energy Functions <ul><li>General Energy Functions </li></ul><ul><ul><li>NP-hard to minimize </li></ul></ul><ul><ul><li>Only approximate minimization possible </li></ul></ul><ul><li>Easy energy functions </li></ul><ul><ul><li>Solvable in polynomial time </li></ul></ul><ul><ul><li>Submodular ~ O(n 6 ) </li></ul></ul>Space of Function Minimization Problems Submodular Functions NP-Hard MAXCUT Functions defined on trees
72. 72. Submodular Set Functions Set function f  2 |E|  ℝ 2 |E| = #subsets of E Let E= {a 1 ,a 2 , .... a n } be a set
73. 73. Submodular Set Functions Set function f  2 |E|  ℝ is submodular if E A B f( A ) + f( B )  f( A  B ) + f( A  B ) for all A , B  E 2 |E| = #subsets of E Let E= {a 1 ,a 2 , .... a n } be a set Important Property Sum of two submodular functions is submodular
74. 74. Minimizing Submodular Functions <ul><li>Minimizing general submodular functions </li></ul><ul><ul><li>O(n 5 Q + n 6 ) where Q is function evaluation time </li></ul></ul><ul><ul><li>[Orlin, IPCO 2007] </li></ul></ul><ul><li>Symmetric submodular functions </li></ul><ul><ul><li>E ( x ) = E ( 1 - x ) </li></ul></ul><ul><ul><li>O(n 3 ) [ Queyranne 1998] </li></ul></ul><ul><li>Quadratic pseudoboolean </li></ul><ul><ul><li>Can be transformed to st-mincut </li></ul></ul><ul><ul><li>One node per variable [ O(n 3 ) complexity] </li></ul></ul><ul><ul><li>Very low empirical running time </li></ul></ul>
75. 75. Submodular Pseudoboolean Functions <ul><li>All functions for one boolean variable (f: {0,1} -> R) are submodular </li></ul><ul><li>A function of two boolean variables (f: {0,1} 2 -> R) is submodular if </li></ul><ul><li>f(0,1) + f(1,0)  f(0,0) + f(1,1) </li></ul><ul><li>A general pseudoboolean function f  2 n  ℝ is submodular if all its projections f p are submodular i.e. </li></ul>f p (0,1) + f p (1,0)  f p (0,0) + f p (1,1) Function defined over boolean vectors x = {x 1 ,x 2 , .... x n } Definition :
76. 76. Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0)  θ ij (0,0) + θ ij (1,1) For all ij E(x) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i
77. 77. Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0)  θ ij (0,0) + θ ij (1,1) For all ij E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) c ij ≥0 i,j i Equivalent (transformable) i.e. All submodular QPBFs are st-mincut solvable E(x) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i
78. 78. 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
79. 79. 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
80. 80. 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
81. 81. 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
82. 82. 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
83. 83. Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0)  θ ij (0,0) + θ ij (1,1) For all ij Equivalent (transformable) x in {0,1} n E(x) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i T S st-mincut
84. 84. Minimizing Non-Submodular Functions <ul><li>Minimizing general non-submodular functions is NP-hard. </li></ul><ul><li>Commonly used method is to solve a relaxation of the problem </li></ul>θ ij (0,1) + θ ij (1,0) ≤ θ ij (0,0) + θ ij (1,1) for some ij [Slide credit: Carsten Rother] E(x) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i
85. 85. Minimization using Roof-dual Relaxation pairwise nonsubmodular unary pairwise submodular [Slide credit: Carsten Rother]
86. 86. <ul><li>Double number of variables: </li></ul>Minimization using Roof-dual Relaxation [Slide credit: Carsten Rother]
87. 87. <ul><li>Double number of variables: </li></ul>Minimization using Roof-dual Relaxation Non- submodular Submodular
88. 88. <ul><li>Double number of variables: </li></ul>Minimization using Roof-dual Relaxation <ul><li>is submodular ! Ignore </li></ul><ul><li>(solvable using st-mincut) </li></ul>Property of the problem :
89. 89. <ul><li>Double number of variables: </li></ul>Minimization using Roof-dual Relaxation is the optimal label Property of the solution :
90. 90. Recap <ul><li>Exact minimization of Submodular QBFs using graph cuts. </li></ul><ul><li>Obtaining partially optimal solutions of non-submodular QBFs using graph cuts. </li></ul>
91. 91. But ... <ul><li>Need higher order energy functions to model image structure </li></ul><ul><ul><li>Field of experts [Roth and Black ] </li></ul></ul><ul><li>Many problems in computer vision involve multiple labels </li></ul>c x ϵ Labels L = {l 1 , l 2 , … , l k } Clique c  V E( x ) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) + ∑ θ c ( x c ) i,j i
92. 92. Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions
93. 93. Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions
94. 94. Higher order to Quadratic <ul><li>Simple Example using Auxiliary variables </li></ul>{ 0 if all x i = 0 C 1 otherwise f( x ) = min f( x ) min C 1 a + C 1 ∑ ā x i x ϵ L = {0,1} n x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i = 0 a=0 ( ā=1) f( x ) = 0 ∑ x i ≥ 1 a=1 ( ā=0) f( x ) = C 1
95. 95. Higher order to Quadratic min f( x ) min C 1 a + C 1 ∑ ā x i x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i 1 2 3 C 1 C 1 ∑ x i
96. 96. Higher order to Quadratic min f( x ) min C 1 a + C 1 ∑ ā x i x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i 1 2 3 C 1 C 1 ∑ x i a=1 a=0 Lower envelop of concave functions is concave
97. 97. Higher order to Quadratic min f( x ) min f 1 (x) a + f 2 (x) ā x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i 1 2 3 a=1 Lower envelop of concave functions is concave f 2 (x) f 1 (x)
98. 98. Higher order to Quadratic min f( x ) min f 1 (x) a + f 2 (x) ā x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i 1 2 3 a=1 a=0 Lower envelop of concave functions is concave f 2 (x) f 1 (x)
99. 99. Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions
100. 100. Multi-label to Pseudo-boolean So what is the problem? E b (x 1 ,x 2 , ..., x m ) E m (y 1 ,y 2 , ..., y n ) Multi-label Problem Binary label Problem y i ϵ L = {l 1 , l 2 , … , l k } x i ϵ L = {0,1} such that: <ul><li>Let Y and X be the set of feasible solutions, then </li></ul><ul><li>For each binary solution x ϵ X with finite energy there exists exactly one multi-label solution y ϵ Y </li></ul><ul><li>-> One-One encoding function T:X->Y </li></ul><ul><li>2. arg min E m (y) = T(arg min E b (x)) </li></ul>
101. 101. Multi-label to Pseudo-boolean <ul><li>Popular encoding scheme </li></ul><ul><li>[Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06] </li></ul>
102. 102. Multi-label to Pseudo-boolean <ul><li>Popular encoding scheme </li></ul><ul><li>[Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06] </li></ul>Ishikawa’s result: y ϵ Labels L = {l 1 , l 2 , … , l k } θ ij (y i ,y j ) = g(|y i -y j |) Convex Function g(|y i -y j |) |y i -y j | E( y ) = ∑ θ i (y i ) + ∑ θ ij (y i ,y j ) i,j i
103. 103. Multi-label to Pseudo-boolean <ul><li>Popular encoding scheme </li></ul><ul><li>[Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06] </li></ul>Schlesinger & Flach ’06: y ϵ Labels L = {l 1 , l 2 , … , l k } θ ij ( l i+1 ,l j ) + θ ij (l i ,l j+1 )  θ ij (l i ,l j ) + θ ij (l i+1 ,l j+1 ) Covers all Submodular multi-label functions More general than Ishikawa E( y ) = ∑ θ i (y i ) + ∑ θ ij (y i ,y j ) i,j i
104. 104. Multi-label to Pseudo-boolean <ul><li>Applicability </li></ul><ul><ul><li>Only solves restricted class of energy functions </li></ul></ul><ul><ul><li>Cannot handle Potts model potentials </li></ul></ul><ul><li>Computational Cost </li></ul><ul><ul><li>Very high computational cost </li></ul></ul><ul><ul><li>Problem size = |Variables| x |Labels| </li></ul></ul><ul><ul><li>Gray level image denoising (1 Mpixel image) </li></ul></ul><ul><ul><li>(~2.5 x 10 8 graph nodes) </li></ul></ul>Problems
105. 105. Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
106. 106. St-mincut based Move algorithms <ul><li>Commonly used for solving non-submodular multi-label problems </li></ul><ul><li>Extremely efficient and produce good solutions </li></ul><ul><li>Not Exact: Produce local optima </li></ul>x ϵ Labels L = {l 1 , l 2 , … , l k } E( x ) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i
107. 107. Move Making Algorithms Solution Space Energy
108. 108. Move Making Algorithms Search Neighbourhood Current Solution Optimal Move Solution Space Energy
109. 109. Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move x c ( t ) Key Property Move Space Bigger move space Solution Space Energy <ul><li>Better solutions </li></ul><ul><li>Finding the optimal move hard </li></ul>
110. 110. Moves using Graph Cuts <ul><li>Expansion and Swap move algorithms </li></ul><ul><li>[Boykov Veksler and Zabih, PAMI 2001] </li></ul><ul><li>Makes a series of changes to the solution (moves) </li></ul><ul><li>Each move results in a solution with smaller energy </li></ul>Space of Solutions (x) : L N Move Space (t) : 2 N Search Neighbourhood Current Solution N Number of Variables L Number of Labels
111. 111. Moves using Graph Cuts <ul><li>Expansion and Swap move algorithms </li></ul><ul><li>[Boykov Veksler and Zabih, PAMI 2001] </li></ul><ul><li>Makes a series of changes to the solution (moves) </li></ul><ul><li>Each move results in a solution with smaller energy </li></ul>Current Solution Construct a move function Minimize move function to get optimal move Move to new solution How to minimize move functions?
112. 112. General Binary Moves Minimize over move variables t to get the optimal move x = t x 1 + (1- t ) x 2 New solution Current Solution Second solution E m ( t ) = E( t x 1 + (1- t ) x 2 ) Boykov, Veksler and Zabih, PAMI 2001 Move energy is a submodular QPBF (Exact Minimization Possible)
113. 113. Swap Move <ul><li>Variables labeled α , β can swap their labels </li></ul>[Boykov, Veksler, Zabih]
114. 114. Swap Move Sky House Tree Ground Swap Sky, House [Boykov, Veksler, Zabih] <ul><li>Variables labeled α , β can swap their labels </li></ul>
115. 115. Swap Move <ul><li>Move energy is submodular if: </li></ul><ul><ul><li>Unary Potentials: Arbitrary </li></ul></ul><ul><ul><li>Pairwise potentials: Semimetric </li></ul></ul><ul><li>Variables labeled α , β can swap their labels </li></ul>[Boykov, Veksler, Zabih] θ ij (l a ,l b ) ≥ 0 θ ij (l a ,l b ) = 0 a = b Examples: Potts model, Truncated Convex
116. 116. Expansion Move [Boykov, Veksler, Zabih] [Boykov, Veksler, Zabih] <ul><li>Variables take label  or retain current label </li></ul>
117. 117. Expansion Move Sky House Tree Ground Initialize with Tree Status: Expand Ground Expand House Expand Sky [Boykov, Veksler, Zabih] [Boykov, Veksler, Zabih] <ul><li>Variables take label  or retain current label </li></ul>
118. 118. Expansion Move <ul><li>Move energy is submodular if: </li></ul><ul><ul><li>Unary Potentials: Arbitrary </li></ul></ul><ul><ul><li>Pairwise potentials: Metric </li></ul></ul>[Boykov, Veksler, Zabih] [Boykov, Veksler, Zabih] θ ij (l a ,l b ) + θ ij (l b ,l c ) ≥ θ ij (l a ,l c ) Semi metric + Triangle Inequality <ul><li>Variables take label  or retain current label </li></ul>Examples: Potts model, Truncated linear Cannot solve truncated quadratic
119. 119. General Binary Moves Minimize over move variables t x = t x 1 + (1-t) x 2 New solution First solution Second solution Move functions can be non-submodular!! Move Type First Solution Second Solution Guarantee Expansion Old solution All alpha Metric Fusion Any solution Any solution 
120. 120. Solving Continuous Problems using Fusion Move x = t x 1 + (1-t) x 2 (Lempitsky et al. CVPR08, Woodford et al. CVPR08) x 1 , x 2 can be continuous F x 1 x 2 x Optical Flow Example Final Solution Solution from Method 1 Solution from Method 2
121. 121. Range Moves <ul><li>Move variables can be multi-label </li></ul><ul><li>Optimal move found out by using the Ishikawa </li></ul><ul><li>Useful for minimizing energies with truncated convex pairwise potentials </li></ul>O. Veksler, CVPR 2007 θ ij (y i ,y j ) = min(|y i -y j |,T) |y i -y j | θ ij (y i ,y j ) T x = ( t == 1) x 1 + (t == 2) x 2 +… + (t == k) x k
122. 122. Move Algorithms for Solving Higher Order Energies <ul><li>Higher order functions give rise to higher order move energies </li></ul><ul><li>Move energies for certain classes of higher order energies can be transformed to QPBFs. </li></ul>[Kohli, Kumar and Torr, CVPR07] [Kohli, Ladicky and Torr, CVPR08] E( x ) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) + ∑ θ c ( x c ) i,j i c x ϵ Labels L = {l 1 , l 2 , … , l k } Clique c  V
123. 123. Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
124. 124. Solving Mixed Programming Problems x – binary image segmentation (x i ∊ {0,1}) ω – non-local parameter (lives in some large set Ω ) constant unary potentials pairwise potentials ≥ 0 Rough Shape Prior Stickman Model ω Pose θ i ( ω, x i ) Shape Prior E(x, ω ) = C( ω ) + ∑ θ i ( ω, x i ) + ∑ θ ij ( ω, x i ,x j ) i,j i
125. 125. Open Problems Submodular Functions st-mincut Equivalent <ul><li>Characterization of Problems Solvable using st-mincut </li></ul><ul><ul><li>What functions can be transformed to submodular QBFs? </li></ul></ul>
126. 126. Minimizing General Higher Order Functions <ul><li>We saw how simple higher order potentials can be solved </li></ul><ul><li>How more sophisticated higher order potentials can be solved? </li></ul>
127. 127. Summary Exact Transformation (global optimum) Or Relaxed transformation (partially optimal) Labelling Problem Submodular Quadratic Pseudoboolean Function Move making algorithms Sub-problem T S st-mincut
128. 128. Thanks. Questions?
129. 129. Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P 1 P 2 P 3 5 6 7 8 Pixels Disparity Labels E(x 1 ,x 2 ,x 3 ) = θ 12 (x 1 ,x 2 ) + θ 23 (x 2 ,x 3 ) θ ij (x i ,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0
130. 130. Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P 1 P 2 P 3 5 6 7 8 Pixels Disparity Labels E(x 1 ,x 2 ,x 3 ) = θ 12 (x 1 ,x 2 ) + θ 23 (x 2 ,x 3 ) θ ij (x i ,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0 E(6,7,7) = 1 + 0 = 1
131. 131. Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P 1 P 2 P 3 5 6 7 8 Pixels Disparity Labels Pairwise potential penalize slanted planar surfaces E(x 1 ,x 2 ,x 3 ) = θ 12 (x 1 ,x 2 ) + θ 23 (x 2 ,x 3 ) θ ij (x i ,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0 E(6,7,7) = 1 + 0 = 1 E(6,7,8) = 1 + 1 = 2
132. 132. Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move x E ( x ) x c Transformation function T ( t ) T ( x c , t ) = x n = x c + t
133. 133. Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E ( x ) x c Transformation function T E m Move Energy ( t ) x E m ( t ) = E ( T ( x c , t )) T ( x c , t ) = x n = x c + t
134. 134. Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E ( x ) x c E m ( t ) = E ( T ( x c , t )) Transformation function T E m Move Energy T ( x c , t ) = x n = x c + t minimize t* Optimal Move ( t ) x