MAP Estimation Algorithms in M. Pawan Kumar, University of Oxford Pushmeet Kohli, Microsoft Research Computer Vision -  Pa...
Example: Image Segmentation E(x) =  ∑  c i  x i  +  ∑   c ij  x i (1-x j )   E: {0,1} n   ->   R 0  ->  fg 1  ->  bg Image...
Example: Image Segmentation E(x) =  ∑  c i  x i  +  ∑   c ij  x i (1-x j )   E: {0,1} n   ->   R 0  ->  fg 1  ->  bg i i,j...
Example: Image Segmentation E(x) =  ∑  c i  x i  +  ∑   c ij  x i (1-x j )   E: {0,1} n   ->   R 0  ->  fg 1  ->  bg i i,j...
Example: Image Segmentation E(x) =  ∑  c i  x i  +  ∑   c ij  x i (1-x j )   E: {0,1} n   ->   R 0  ->  fg 1  ->  bg i i,j...
Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms ...
Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms ...
The st-Mincut Problem Source Sink v 1 v 2 2 5 9 4 2 1 <ul><li>Graph (V, E, C) </li></ul><ul><ul><li>Vertices V = {v 1 , v ...
The st-Mincut Problem Source Sink v 1 v 2 2 5 9 4 2 1 What is a st-cut?
The st-Mincut Problem Source Sink v 1 v 2 2 5 9 4 2 1 What is a st-cut? An st-cut ( S , T ) divides the nodes between sour...
The st-Mincut Problem What is a st-cut? An st-cut ( S , T ) divides the nodes between source and sink. What is the cost of...
How to compute the st-mincut? Source Sink v 1 v 2 2 5 9 4 2 1 Solve the dual  maximum flow  problem In every network, the ...
Maxflow Algorithms Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul...
Maxflow Algorithms Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul...
Maxflow Algorithms Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul...
Maxflow Algorithms Source Sink v 1 v 2 0 3 9 4 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink ...
Maxflow Algorithms Source Sink v 1 v 2 0 3 9 4 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink ...
Maxflow Algorithms Source Sink v 1 v 2 0 3 9 4 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink ...
Maxflow Algorithms Source Sink v 1 v 2 0 3 5 0 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink ...
Maxflow Algorithms Source Sink v 1 v 2 0 3 5 0 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink ...
Maxflow Algorithms Source Sink v 1 v 2 0 3 5 0 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink ...
Maxflow Algorithms Source Sink v 1 v 2 0 2 4 0 2+1 1-1 Augmenting Path Based Algorithms <ul><li>Find path from source to s...
Maxflow Algorithms Source Sink v 1 v 2 0 2 4 0 3 0 Augmenting Path Based Algorithms <ul><li>Find path from source to sink ...
Maxflow Algorithms Source Sink v 1 v 2 0 2 4 0 3 0 Augmenting Path Based Algorithms <ul><li>Find path from source to sink ...
History of Maxflow Algorithms [Slide credit: Andrew Goldberg] Augmenting Path  and  Push-Relabel n:  # nodes m:  # edges U...
History of Maxflow Algorithms [Slide credit: Andrew Goldberg] Augmenting Path  and  Push-Relabel n:  # nodes m:  # edges U...
Augmenting Path based Algorithms a 1 a 2 1000 1 Sink Source 1000 1000 1000 0 Ford Fulkerson:  Choose  any  augmenting path
Augmenting Path based Algorithms a 1 a 2 1000 1 Sink Source 1000 1000 1000 0 Bad Augmenting Paths Ford Fulkerson:  Choose ...
a 1 a 2 1000 1 Sink Source 1000 1000 1000 0 Augmenting Path based Algorithms Bad Augmenting Path Ford Fulkerson:  Choose  ...
a 1 a 2 999 0 Sink Source 1000 1000 999 1 Augmenting Path based Algorithms Ford Fulkerson:  Choose  any  augmenting path
Augmenting Path based Algorithms a 1 a 2 999 0 Sink Source 1000 1000 999 1 Ford Fulkerson:  Choose  any  augmenting path  ...
Augmenting Path based Algorithms Dinic:  Choose  shortest  augmenting path  n:  # nodes m:  # edges Worst case Complexity:...
Maxflow in Computer Vision <ul><li>Specialized algorithms for vision problems </li></ul><ul><ul><li>Grid graphs  </li></ul...
Maxflow in Computer Vision <ul><li>Specialized algorithms for vision problems </li></ul><ul><ul><li>Grid graphs  </li></ul...
Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms ...
St-mincut and Energy Minimization E:  {0,1} n   ->   R Minimizing a Qudratic Pseudoboolean function E(x)  Functions of boo...
So how does this work?  <ul><li>Construct a graph such that: </li></ul><ul><li>Any st-cut corresponds to an assignment of ...
Graph Construction Sink (1) Source (0)   a 1 a 2 E(a 1 ,a 2 )
Graph Construction Sink (1) Source (0)   a 1 a 2 E(a 1 ,a 2 ) =  2a 1 2
Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1  +  5ā 1 2 5 Sink (1) Source (0)
Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1  + 5ā 1 +  9a 2  + 4ā 2 2 5 9 4 Sink (1) Source (0)
Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1  + 5ā 1 + 9a 2  + 4ā 2  +  2a 1 ā 2 2 5 9 4 2 Sink (1) Source (0)
Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1  + 5ā 1 + 9a 2  + 4ā 2  + 2a 1 ā 2  +   ā 1 a 2 2 5 9 4 2 1 Sink (1) Source...
Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1  + 5ā 1 + 9a 2  + 4ā 2  + 2a 1 ā 2  +   ā 1 a 2 2 5 9 4 2 1 Sink (1) Source...
Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2 a 1   + 5 ā 1 + 9 a 2   + 4 ā 2   + 2 a 1 ā 2   +   ā 1 a 2 2 5 9 4 2 1 a 1  =...
Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2 a 1   + 5 ā 1 + 9 a 2   + 4 ā 2   + 2 a 1 ā 2   +   ā 1 a 2 2 5 9 4 2 1 Sink (...
Energy Function Reparameterization Two functions E 1  and E 2  are reparameterizations if E 1  ( x ) = E 2  ( x )  for all...
Flow and Reparametrization a 1 a 2 E(a 1 ,a 2 ) = 2a 1  + 5ā 1 + 9a 2  + 4ā 2  + 2a 1 ā 2  +   ā 1 a 2 2 5 9 4 2 1 Sink (1...
Flow and Reparametrization a 1 a 2 E(a 1 ,a 2 ) =  2a 1  + 5ā 1 + 9a 2  + 4ā 2  + 2a 1 ā 2  +   ā 1 a 2 2 5 9 4 2 1 Sink (...
Flow and Reparametrization Sink (1) Source (0)   a 1 a 2 E(a 1 ,a 2 ) =  2   + 3ā 1 + 9a 2  + 4ā 2  + 2a 1 ā 2  +   ā 1 a ...
Flow and Reparametrization Sink (0) Source (1)   a 1 a 2 E(a 1 ,a 2 ) = 2   + 3ā 1 +  9a 2  + 4ā 2  + 2a 1 ā 2  +   ā 1 a ...
Flow and Reparametrization Sink (0) Source (1)   a 1 a 2 E(a 1 ,a 2 ) = 2   + 3ā 1 +  5a 2  + 4   + 2a 1 ā 2  +   ā 1 a 2 ...
Flow and Reparametrization Sink (0) Source (1)   a 1 a 2 E(a 1 ,a 2 ) = 6   + 3ā 1 + 5a 2  + 2a 1 ā 2  +   ā 1 a 2 0 3 5 0...
Flow and Reparametrization Sink (0) Source (1)   a 1 a 2 E(a 1 ,a 2 ) = 6   +  3ā 1 +  5a 2   +  2a 1 ā 2   +   ā 1 a 2 0 ...
Flow and Reparametrization Sink (0) Source (1)   a 1 a 2 E(a 1 ,a 2 ) = 6   +  3ā 1 +  5a 2   +  2a 1 ā 2   +   ā 1 a 2 0 ...
Flow and Reparametrization Sink (0) Source (1)   a 1 a 2 E(a 1 ,a 2 ) = 8   +  ā 1 +  3a 2   + 3ā 1 a 2 0 1 3 0 0 3 3ā 1 +...
Flow and Reparametrization Sink (0) Source (1)   a 1 a 2 0 1 3 0 0 3 E(a 1 ,a 2 ) = 8   + ā 1 + 3a 2  + 3ā 1 a 2 No more a...
Flow and Reparametrization Sink (0) Source (1)   a 1 a 2 0 1 3 0 0 3 E(a 1 ,a 2 ) = 8   + ā 1 + 3a 2  + 3ā 1 a 2 Total Flo...
Flow and Reparametrization Sink (0) Source (1)   a 1 a 2 0 1 3 0 0 3 E(a 1 ,a 2 ) = 8   + ā 1 + 3a 2  + 3ā 1 a 2 a 1  = 1 ...
Example: Image Segmentation E(x) =  ∑  c i  x i  +  ∑   c ij  x i (1-x j )   E: {0,1} n   ->   R 0  ->  fg 1  ->  bg i i,j...
How does the code look like? Sink (1) Source (0)   Graph *g; For all pixels p  /* Add a node to the graph */ nodeID(p) = g...
How does the code look like? Graph *g; For all pixels p  /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set c...
How does the code look like? Graph *g; For all pixels p  /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set c...
How does the code look like? Graph *g; For all pixels p  /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set c...
Image Segmentation in Video Image Flow Global Optimum s t = 0 = 1 E(x)  x * n-links st-cut
Image Segmentation in Video Image Flow Global Optimum
Dynamic Energy Minimization E B computationally expensive operation   E A Recycling Solutions Can we do better? Boykov & J...
Dynamic Energy Minimization E B computationally expensive operation   E A cheaper operation Kohli & Torr (ICCV05, PAMI07) ...
Dynamic Energy Minimization Reparametrized  Energy Kohli & Torr (ICCV05, PAMI07) Boykov & Jolly ICCV’01, Kohli & Torr (ICC...
Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms ...
Minimizing Energy Functions <ul><li>General Energy Functions </li></ul><ul><ul><li>NP-hard to minimize </li></ul></ul><ul>...
Submodular Set Functions Set function f     2 |E|      ℝ 2 |E|  =  #subsets of E Let E= {a 1 ,a 2 , .... a n } be a set
Submodular Set Functions Set function f     2 |E|      ℝ  is  submodular  if E A B f( A ) + f( B )     f( A  B ) + f( ...
Minimizing Submodular Functions <ul><li>Minimizing general submodular functions </li></ul><ul><ul><li>O(n 5  Q + n 6 ) whe...
Submodular Pseudoboolean Functions <ul><li>All functions for one boolean variable (f: {0,1} -> R) are submodular </li></ul...
Quadratic Submodular Pseudoboolean Functions θ ij (0,1) +  θ ij   (1,0)     θ ij   (0,0) +  θ ij   (1,1) For all ij E(x) ...
Quadratic Submodular Pseudoboolean Functions θ ij (0,1) +  θ ij   (1,0)     θ ij   (0,0) +  θ ij   (1,1) For all ij E(x) ...
0  1 0 1 x i x j =  A   + 0  1 0 1 0  1 0 1 0  1 0 1 + + if x 1 =1  add C-A if x 2  = 1  add D-C B+C-A-D    0  is true fr...
0  1 0 1 x i x j =  A   + 0  1 0 1 0  1 0 1 0  1 0 1 + + if x 1 =1  add C-A if x 2  = 1  add D-C B+C-A-D    0  is true fr...
0  1 0 1 x i x j =  A   + 0  1 0 1 0  1 0 1 0  1 0 1 + + if x 1 =1  add C-A if x 2  = 1  add D-C B+C-A-D    0  is true fr...
0  1 0 1 x i x j =  A   + 0  1 0 1 0  1 0 1 0  1 0 1 + + if x 1 =1  add C-A if x 2  = 1  add D-C B+C-A-D    0  is true fr...
0  1 0 1 x i x j =  A   + 0  1 0 1 0  1 0 1 0  1 0 1 + + if x 1 =1  add C-A if x 2  = 1  add D-C B+C-A-D    0  is true fr...
Quadratic Submodular Pseudoboolean Functions θ ij (0,1) +  θ ij   (1,0)     θ ij   (0,0) +  θ ij   (1,1) For all ij Equiv...
Minimizing Non-Submodular Functions <ul><li>Minimizing general non-submodular functions is NP-hard.  </li></ul><ul><li>Com...
Minimization using Roof-dual Relaxation pairwise nonsubmodular unary pairwise submodular [Slide credit: Carsten Rother]
<ul><li>Double number of variables: </li></ul>Minimization using Roof-dual Relaxation [Slide credit: Carsten Rother]
<ul><li>Double number of variables: </li></ul>Minimization using Roof-dual Relaxation Non- submodular Submodular
<ul><li>Double number of variables: </li></ul>Minimization using Roof-dual Relaxation <ul><li>is submodular !  Ignore  </l...
<ul><li>Double number of variables: </li></ul>Minimization using Roof-dual Relaxation is the optimal label Property of the...
Recap <ul><li>Exact minimization of Submodular QBFs using graph cuts. </li></ul><ul><li>Obtaining partially optimal soluti...
But ... <ul><li>Need higher order energy functions to model image structure  </li></ul><ul><ul><li>Field of experts [Roth ...
Transforming problems in QBFs Multi-label  Functions Pseudoboolean Functions Higher order  Pseudoboolean Functions Quadrat...
Transforming problems in QBFs Multi-label  Functions Pseudoboolean Functions Higher order  Pseudoboolean Functions Quadrat...
Higher order to Quadratic <ul><li>Simple Example using Auxiliary variables </li></ul>{ 0  if  all x i  = 0 C 1   otherwise...
Higher order to Quadratic min   f( x ) min   C 1 a +  C 1  ∑   ā  x i x = x,a  ϵ   {0,1} Higher Order Submodular Function ...
Higher order to Quadratic min   f( x ) min   C 1 a +  C 1  ∑   ā  x i x = x,a  ϵ   {0,1} Higher Order Submodular Function ...
Higher order to Quadratic min   f( x ) min   f 1  (x) a +   f 2 (x) ā x = x,a  ϵ   {0,1} Higher Order Submodular Function ...
Higher order to Quadratic min   f( x ) min   f 1  (x) a +   f 2 (x) ā x = x,a  ϵ   {0,1} Higher Order Submodular Function ...
Transforming problems in QBFs Multi-label  Functions Pseudoboolean Functions Higher order  Pseudoboolean Functions Quadrat...
Multi-label to Pseudo-boolean So what is the problem? E b  (x 1 ,x 2 , ..., x m ) E m  (y 1 ,y 2 , ..., y n ) Multi-label ...
Multi-label to Pseudo-boolean <ul><li>Popular encoding scheme  </li></ul><ul><li>[Roy and Cox ’98, Ishikawa ’03, Schlesing...
Multi-label to Pseudo-boolean <ul><li>Popular encoding scheme  </li></ul><ul><li>[Roy and Cox ’98, Ishikawa ’03, Schlesing...
Multi-label to Pseudo-boolean <ul><li>Popular encoding scheme  </li></ul><ul><li>[Roy and Cox ’98, Ishikawa ’03, Schlesing...
Multi-label to Pseudo-boolean <ul><li>Applicability </li></ul><ul><ul><li>Only solves restricted class of energy functions...
Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms ...
St-mincut based Move algorithms <ul><li>Commonly used for solving  non-submodular  multi-label problems </li></ul><ul><li>...
Move Making Algorithms Solution Space Energy
Move Making Algorithms Search Neighbourhood Current Solution Optimal Move Solution Space Energy
Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move x c ( t ) Key Property  Move Space Bigger mo...
Moves using Graph Cuts <ul><li>Expansion and Swap move algorithms </li></ul><ul><li>[Boykov Veksler and Zabih, PAMI 2001] ...
Moves using Graph Cuts <ul><li>Expansion and Swap move algorithms </li></ul><ul><li>[Boykov Veksler and Zabih, PAMI 2001] ...
General Binary Moves Minimize over move variables  t  to get the optimal move  x =  t  x 1  + (1-  t )   x 2 New solution ...
Swap Move <ul><li>Variables labeled  α ,  β  can swap their labels </li></ul>[Boykov, Veksler, Zabih]
Swap Move Sky House Tree Ground Swap Sky, House [Boykov, Veksler, Zabih] <ul><li>Variables labeled  α ,  β  can swap their...
Swap Move <ul><li>Move energy is submodular if: </li></ul><ul><ul><li>Unary Potentials: Arbitrary </li></ul></ul><ul><ul><...
Expansion Move [Boykov, Veksler, Zabih] [Boykov, Veksler, Zabih] <ul><li>Variables take label   or retain current label...
Expansion Move Sky House Tree Ground Initialize with Tree Status: Expand Ground Expand House Expand Sky [Boykov, Veksler, ...
Expansion Move <ul><li>Move energy is submodular if: </li></ul><ul><ul><li>Unary Potentials: Arbitrary </li></ul></ul><ul>...
General Binary Moves Minimize over move variables t  x =  t  x 1  + (1-t)   x 2 New solution First solution Second solutio...
Solving Continuous Problems using Fusion Move x =  t  x 1  + (1-t)   x 2 (Lempitsky et al. CVPR08, Woodford et al. CVPR08)...
Range Moves  <ul><li>Move variables can be multi-label </li></ul><ul><li>Optimal move found out by using the Ishikawa </li...
Move Algorithms for Solving Higher Order Energies  <ul><li>Higher order functions give rise to higher order move energies ...
Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms ...
Solving Mixed Programming Problems x  – binary image segmentation (x i  ∊ {0,1}) ω  – non-local parameter (lives in some l...
Open Problems Submodular Functions st-mincut Equivalent <ul><li>Characterization of Problems Solvable using st-mincut </li...
Minimizing General Higher Order Functions <ul><li>We saw how simple higher order potentials can be solved </li></ul><ul><l...
Summary Exact Transformation (global optimum) Or Relaxed transformation (partially optimal) Labelling Problem Submodular Q...
Thanks. Questions?
Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P 1 P 2 P 3 5 6 7 8 Pixels Disparity Labels E(x 1 ,x 2 ,...
Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P 1 P 2 P 3 5 6 7 8 Pixels Disparity Labels E(x 1 ,x 2 ,...
Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P 1 P 2 P 3 5 6 7 8 Pixels Disparity Labels Pairwise pot...
Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move x E ( x ) x c Transformation function T ( t ...
Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E ( x ) x c Transformation function T E m Mo...
Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E ( x ) x c E m ( t )  =  E ( T ( x c , t ))...
Upcoming SlideShare
Loading in …5
×

ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2

632 views
548 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
632
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
17
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Move to front
  • correct
  • correct
  • Chi square
  • Chi square
  • ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2

    1. 1. MAP Estimation Algorithms in M. Pawan Kumar, University of Oxford Pushmeet Kohli, Microsoft Research Computer Vision - Part II
    2. 2. Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg Image (D) i i,j n = number of pixels
    3. 3. Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg i i,j Unary Cost (c i ) Dark ( negative ) Bright (positive) n = number of pixels
    4. 4. Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg i i,j Discontinuity Cost (c ij ) n = number of pixels
    5. 5. Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg i i,j Global Minimum (x * ) x * = arg min E(x) x How to minimize E(x)? n = number of pixels
    6. 6. Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Recent Advances and Open Problems Connection between st-mincut and energy minimization?
    7. 7. Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
    8. 8. The st-Mincut Problem Source Sink v 1 v 2 2 5 9 4 2 1 <ul><li>Graph (V, E, C) </li></ul><ul><ul><li>Vertices V = {v 1 , v 2 ... v n } </li></ul></ul><ul><ul><li>Edges E = {(v 1 , v 2 ) ....} </li></ul></ul><ul><ul><li>Costs C = {c (1, 2) ....} </li></ul></ul>
    9. 9. The st-Mincut Problem Source Sink v 1 v 2 2 5 9 4 2 1 What is a st-cut?
    10. 10. The st-Mincut Problem Source Sink v 1 v 2 2 5 9 4 2 1 What is a st-cut? An st-cut ( S , T ) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T 5 + 2 + 9 = 16
    11. 11. The st-Mincut Problem What is a st-cut? An st-cut ( S , T ) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T What is the st-mincut? st-cut with the minimum cost Source Sink v 1 v 2 2 5 9 4 2 1 2 + 1 + 4 = 7
    12. 12. How to compute the st-mincut? Source Sink v 1 v 2 2 5 9 4 2 1 Solve the dual maximum flow problem In every network, the maximum flow equals the cost of the st-mincut Min-cutMax-flow Theorem Compute the maximum flow between Source and Sink Constraints Edges: Flow < Capacity Nodes: Flow in = Flow out
    13. 13. Maxflow Algorithms Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Source Sink v 1 v 2 2 5 9 4 2 1 Algorithms assume non-negative capacity Flow = 0
    14. 14. Maxflow Algorithms Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Source Sink v 1 v 2 2 5 9 4 2 1 Algorithms assume non-negative capacity Flow = 0
    15. 15. Maxflow Algorithms Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Source Sink v 1 v 2 2-2 5-2 9 4 2 1 Algorithms assume non-negative capacity Flow = 0 + 2
    16. 16. Maxflow Algorithms Source Sink v 1 v 2 0 3 9 4 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 2
    17. 17. Maxflow Algorithms Source Sink v 1 v 2 0 3 9 4 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 2
    18. 18. Maxflow Algorithms Source Sink v 1 v 2 0 3 9 4 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 2
    19. 19. Maxflow Algorithms Source Sink v 1 v 2 0 3 5 0 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 2 + 4
    20. 20. Maxflow Algorithms Source Sink v 1 v 2 0 3 5 0 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 6
    21. 21. Maxflow Algorithms Source Sink v 1 v 2 0 3 5 0 2 1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 6
    22. 22. Maxflow Algorithms Source Sink v 1 v 2 0 2 4 0 2+1 1-1 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 6 + 1
    23. 23. Maxflow Algorithms Source Sink v 1 v 2 0 2 4 0 3 0 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 7
    24. 24. Maxflow Algorithms Source Sink v 1 v 2 0 2 4 0 3 0 Augmenting Path Based Algorithms <ul><li>Find path from source to sink with positive capacity </li></ul><ul><li>Push maximum possible flow through this path </li></ul><ul><li>Repeat until no path can be found </li></ul>Algorithms assume non-negative capacity Flow = 7
    25. 25. History of Maxflow Algorithms [Slide credit: Andrew Goldberg] Augmenting Path and Push-Relabel n: # nodes m: # edges U: maximum edge weight Algorithms assume non-negative edge weights
    26. 26. History of Maxflow Algorithms [Slide credit: Andrew Goldberg] Augmenting Path and Push-Relabel n: # nodes m: # edges U: maximum edge weight Algorithms assume non-negative edge weights
    27. 27. Augmenting Path based Algorithms a 1 a 2 1000 1 Sink Source 1000 1000 1000 0 Ford Fulkerson: Choose any augmenting path
    28. 28. Augmenting Path based Algorithms a 1 a 2 1000 1 Sink Source 1000 1000 1000 0 Bad Augmenting Paths Ford Fulkerson: Choose any augmenting path
    29. 29. a 1 a 2 1000 1 Sink Source 1000 1000 1000 0 Augmenting Path based Algorithms Bad Augmenting Path Ford Fulkerson: Choose any augmenting path
    30. 30. a 1 a 2 999 0 Sink Source 1000 1000 999 1 Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting path
    31. 31. Augmenting Path based Algorithms a 1 a 2 999 0 Sink Source 1000 1000 999 1 Ford Fulkerson: Choose any augmenting path n: # nodes m: # edges We will have to perform 2000 augmentations! Worst case complexity: O (m x Total_Flow) (Pseudo-polynomial bound: depends on flow)
    32. 32. Augmenting Path based Algorithms Dinic: Choose shortest augmenting path n: # nodes m: # edges Worst case Complexity: O (m n 2 ) a 1 a 2 1000 1 Sink Source 1000 1000 1000 0
    33. 33. Maxflow in Computer Vision <ul><li>Specialized algorithms for vision problems </li></ul><ul><ul><li>Grid graphs </li></ul></ul><ul><ul><li>Low connectivity (m ~ O(n)) </li></ul></ul><ul><li>Dual search tree augmenting path algorithm </li></ul><ul><li>[Boykov and Kolmogorov PAMI 2004] </li></ul><ul><ul><li>Finds approximate shortest augmenting paths efficiently </li></ul></ul><ul><ul><li>High worst-case time complexity </li></ul></ul><ul><ul><li>Empirically outperforms other algorithms on vision problems </li></ul></ul>
    34. 34. Maxflow in Computer Vision <ul><li>Specialized algorithms for vision problems </li></ul><ul><ul><li>Grid graphs </li></ul></ul><ul><ul><li>Low connectivity (m ~ O(n)) </li></ul></ul><ul><li>Dual search tree augmenting path algorithm </li></ul><ul><li>[Boykov and Kolmogorov PAMI 2004] </li></ul><ul><ul><li>Finds approximate shortest augmenting paths efficiently </li></ul></ul><ul><ul><li>High worst-case time complexity </li></ul></ul><ul><ul><li>Empirically outperforms other algorithms on vision problems </li></ul></ul><ul><ul><li>Efficient code available on the web </li></ul></ul>http://www.adastral.ucl.ac.uk/~vladkolm/software.html
    35. 35. Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
    36. 36. St-mincut and Energy Minimization E: {0,1} n -> R Minimizing a Qudratic Pseudoboolean function E(x) Functions of boolean variables Pseudoboolean? Polynomial time st-mincut algorithms require non-negative edge weights E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) c ij ≥0 i,j i T S st-mincut
    37. 37. So how does this work? <ul><li>Construct a graph such that: </li></ul><ul><li>Any st-cut corresponds to an assignment of x </li></ul><ul><li>The cost of the cut is equal to the energy of x : E(x) </li></ul>E(x) Solution T S st-mincut
    38. 38. Graph Construction Sink (1) Source (0) a 1 a 2 E(a 1 ,a 2 )
    39. 39. Graph Construction Sink (1) Source (0) a 1 a 2 E(a 1 ,a 2 ) = 2a 1 2
    40. 40. Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 2 5 Sink (1) Source (0)
    41. 41. Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 2 5 9 4 Sink (1) Source (0)
    42. 42. Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 2 5 9 4 2 Sink (1) Source (0)
    43. 43. Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)
    44. 44. Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)
    45. 45. Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2 a 1 + 5 ā 1 + 9 a 2 + 4 ā 2 + 2 a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 a 1 = 1 a 2 = 1 E (1,1) = 11 Cost of cut = 11 Sink (1) Source (0)
    46. 46. Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2 a 1 + 5 ā 1 + 9 a 2 + 4 ā 2 + 2 a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0) a 1 = 1 a 2 = 0 E (1,0) = 8 st-mincut cost = 8
    47. 47. Energy Function Reparameterization Two functions E 1 and E 2 are reparameterizations if E 1 ( x ) = E 2 ( x ) for all x For instance: E 1 (a 1 ) = 1+ 2a 1 + 3ā 1 E 2 (a 1 ) = 3 + ā 1 a 1 ā 1 1+ 2a 1 + 3ā 1 3 + ā 1 0 1 4 4 1 0 3 3
    48. 48. Flow and Reparametrization a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)
    49. 49. Flow and Reparametrization a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0) 2a 1 + 5ā 1 = 2(a 1 +ā 1 ) + 3ā 1 = 2 + 3ā 1
    50. 50. Flow and Reparametrization Sink (1) Source (0) a 1 a 2 E(a 1 ,a 2 ) = 2 + 3ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 0 3 9 4 2 1 2a 1 + 5ā 1 = 2(a 1 +ā 1 ) + 3ā 1 = 2 + 3ā 1
    51. 51. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 2 + 3ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 0 3 9 4 2 1 9a 2 + 4ā 2 = 4(a 2 +ā 2 ) + 5ā 2 = 4 + 5ā 2
    52. 52. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 2 + 3ā 1 + 5a 2 + 4 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1 9a 2 + 4ā 2 = 4(a 2 +ā 2 ) + 5ā 2 = 4 + 5ā 2
    53. 53. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1
    54. 54. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1
    55. 55. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1 3ā 1 + 5a 2 + 2a 1 ā 2 = 2(ā 1 +a 2 +a 1 ā 2 ) +ā 1 +3a 2 = 2(1+ā 1 a 2 ) +ā 1 +3a 2 F1 = ā 1 +a 2 +a 1 ā 2 F2 = 1+ā 1 a 2 a 1 a 2 F1 F2 0 0 1 1 0 1 2 2 1 0 1 1 1 1 1 1
    56. 56. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 0 1 3 0 0 3 3ā 1 + 5a 2 + 2a 1 ā 2 = 2(ā 1 +a 2 +a 1 ā 2 ) +ā 1 +3a 2 = 2(1+ā 1 a 2 ) +ā 1 +3a 2 F1 = ā 1 +a 2 +a 1 ā 2 F2 = 1+ā 1 a 2 a 1 a 2 F1 F2 0 0 1 1 0 1 2 2 1 0 1 1 1 1 1 1
    57. 57. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 0 1 3 0 0 3 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 No more augmenting paths possible
    58. 58. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 0 1 3 0 0 3 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Total Flow Residual Graph (positive coefficients) bound on the optimal solution Inference of the optimal solution becomes trivial because the bound is tight
    59. 59. Flow and Reparametrization Sink (0) Source (1) a 1 a 2 0 1 3 0 0 3 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 a 1 = 1 a 2 = 0 E (1,0) = 8 st-mincut cost = 8 Total Flow bound on the optimal solution Inference of the optimal solution becomes trivial because the bound is tight Residual Graph (positive coefficients)
    60. 60. Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg i i,j Global Minimum (x * ) x * = arg min E(x) x How to minimize E(x)?
    61. 61. How does the code look like? Sink (1) Source (0) Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1)
    62. 62. How does the code look like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) a 1 a 2 fgCost( a 1 ) Sink (1) Source (0) fgCost( a 2 ) bgCost( a 1 ) bgCost( a 2 )
    63. 63. How does the code look like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) a 1 a 2 fgCost( a 1 ) Sink (1) Source (0) fgCost( a 2 ) bgCost( a 1 ) bgCost( a 2 ) cost(p,q)
    64. 64. How does the code look like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) a 1 a 2 fgCost( a 1 ) Sink (1) Source (0) fgCost( a 2 ) bgCost( a 1 ) bgCost( a 2 ) cost(p,q) a 1 = bg a 2 = fg
    65. 65. Image Segmentation in Video Image Flow Global Optimum s t = 0 = 1 E(x) x * n-links st-cut
    66. 66. Image Segmentation in Video Image Flow Global Optimum
    67. 67. Dynamic Energy Minimization E B computationally expensive operation E A Recycling Solutions Can we do better? Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07) S B S A minimize
    68. 68. Dynamic Energy Minimization E B computationally expensive operation E A cheaper operation Kohli & Torr (ICCV05, PAMI07) 3 – 100000 time speedup! Reuse flow Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07) S B S A minimize Simpler energy E B* differences between A and B A and B similar Reparametrization
    69. 69. Dynamic Energy Minimization Reparametrized Energy Kohli & Torr (ICCV05, PAMI07) Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07) E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Original Energy E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 7a 1 ā 2 + ā 1 a 2 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 + 5a 1 ā 2 New Energy New Reparametrized Energy
    70. 70. Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
    71. 71. Minimizing Energy Functions <ul><li>General Energy Functions </li></ul><ul><ul><li>NP-hard to minimize </li></ul></ul><ul><ul><li>Only approximate minimization possible </li></ul></ul><ul><li>Easy energy functions </li></ul><ul><ul><li>Solvable in polynomial time </li></ul></ul><ul><ul><li>Submodular ~ O(n 6 ) </li></ul></ul>Space of Function Minimization Problems Submodular Functions NP-Hard MAXCUT Functions defined on trees
    72. 72. Submodular Set Functions Set function f  2 |E|  ℝ 2 |E| = #subsets of E Let E= {a 1 ,a 2 , .... a n } be a set
    73. 73. Submodular Set Functions Set function f  2 |E|  ℝ is submodular if E A B f( A ) + f( B )  f( A  B ) + f( A  B ) for all A , B  E 2 |E| = #subsets of E Let E= {a 1 ,a 2 , .... a n } be a set Important Property Sum of two submodular functions is submodular
    74. 74. Minimizing Submodular Functions <ul><li>Minimizing general submodular functions </li></ul><ul><ul><li>O(n 5 Q + n 6 ) where Q is function evaluation time </li></ul></ul><ul><ul><li>[Orlin, IPCO 2007] </li></ul></ul><ul><li>Symmetric submodular functions </li></ul><ul><ul><li>E ( x ) = E ( 1 - x ) </li></ul></ul><ul><ul><li>O(n 3 ) [ Queyranne 1998] </li></ul></ul><ul><li>Quadratic pseudoboolean </li></ul><ul><ul><li>Can be transformed to st-mincut </li></ul></ul><ul><ul><li>One node per variable [ O(n 3 ) complexity] </li></ul></ul><ul><ul><li>Very low empirical running time </li></ul></ul>
    75. 75. Submodular Pseudoboolean Functions <ul><li>All functions for one boolean variable (f: {0,1} -> R) are submodular </li></ul><ul><li>A function of two boolean variables (f: {0,1} 2 -> R) is submodular if </li></ul><ul><li>f(0,1) + f(1,0)  f(0,0) + f(1,1) </li></ul><ul><li>A general pseudoboolean function f  2 n  ℝ is submodular if all its projections f p are submodular i.e. </li></ul>f p (0,1) + f p (1,0)  f p (0,0) + f p (1,1) Function defined over boolean vectors x = {x 1 ,x 2 , .... x n } Definition :
    76. 76. Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0)  θ ij (0,0) + θ ij (1,1) For all ij E(x) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i
    77. 77. Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0)  θ ij (0,0) + θ ij (1,1) For all ij E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) c ij ≥0 i,j i Equivalent (transformable) i.e. All submodular QPBFs are st-mincut solvable E(x) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i
    78. 78. 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
    79. 79. 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
    80. 80. 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
    81. 81. 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
    82. 82. 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
    83. 83. Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0)  θ ij (0,0) + θ ij (1,1) For all ij Equivalent (transformable) x in {0,1} n E(x) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i T S st-mincut
    84. 84. Minimizing Non-Submodular Functions <ul><li>Minimizing general non-submodular functions is NP-hard. </li></ul><ul><li>Commonly used method is to solve a relaxation of the problem </li></ul>θ ij (0,1) + θ ij (1,0) ≤ θ ij (0,0) + θ ij (1,1) for some ij [Slide credit: Carsten Rother] E(x) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i
    85. 85. Minimization using Roof-dual Relaxation pairwise nonsubmodular unary pairwise submodular [Slide credit: Carsten Rother]
    86. 86. <ul><li>Double number of variables: </li></ul>Minimization using Roof-dual Relaxation [Slide credit: Carsten Rother]
    87. 87. <ul><li>Double number of variables: </li></ul>Minimization using Roof-dual Relaxation Non- submodular Submodular
    88. 88. <ul><li>Double number of variables: </li></ul>Minimization using Roof-dual Relaxation <ul><li>is submodular ! Ignore </li></ul><ul><li>(solvable using st-mincut) </li></ul>Property of the problem :
    89. 89. <ul><li>Double number of variables: </li></ul>Minimization using Roof-dual Relaxation is the optimal label Property of the solution :
    90. 90. Recap <ul><li>Exact minimization of Submodular QBFs using graph cuts. </li></ul><ul><li>Obtaining partially optimal solutions of non-submodular QBFs using graph cuts. </li></ul>
    91. 91. But ... <ul><li>Need higher order energy functions to model image structure </li></ul><ul><ul><li>Field of experts [Roth and Black ] </li></ul></ul><ul><li>Many problems in computer vision involve multiple labels </li></ul>c x ϵ Labels L = {l 1 , l 2 , … , l k } Clique c  V E( x ) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) + ∑ θ c ( x c ) i,j i
    92. 92. Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions
    93. 93. Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions
    94. 94. Higher order to Quadratic <ul><li>Simple Example using Auxiliary variables </li></ul>{ 0 if all x i = 0 C 1 otherwise f( x ) = min f( x ) min C 1 a + C 1 ∑ ā x i x ϵ L = {0,1} n x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i = 0 a=0 ( ā=1) f( x ) = 0 ∑ x i ≥ 1 a=1 ( ā=0) f( x ) = C 1
    95. 95. Higher order to Quadratic min f( x ) min C 1 a + C 1 ∑ ā x i x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i 1 2 3 C 1 C 1 ∑ x i
    96. 96. Higher order to Quadratic min f( x ) min C 1 a + C 1 ∑ ā x i x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i 1 2 3 C 1 C 1 ∑ x i a=1 a=0 Lower envelop of concave functions is concave
    97. 97. Higher order to Quadratic min f( x ) min f 1 (x) a + f 2 (x) ā x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i 1 2 3 a=1 Lower envelop of concave functions is concave f 2 (x) f 1 (x)
    98. 98. Higher order to Quadratic min f( x ) min f 1 (x) a + f 2 (x) ā x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i 1 2 3 a=1 a=0 Lower envelop of concave functions is concave f 2 (x) f 1 (x)
    99. 99. Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions
    100. 100. Multi-label to Pseudo-boolean So what is the problem? E b (x 1 ,x 2 , ..., x m ) E m (y 1 ,y 2 , ..., y n ) Multi-label Problem Binary label Problem y i ϵ L = {l 1 , l 2 , … , l k } x i ϵ L = {0,1} such that: <ul><li>Let Y and X be the set of feasible solutions, then </li></ul><ul><li>For each binary solution x ϵ X with finite energy there exists exactly one multi-label solution y ϵ Y </li></ul><ul><li>-> One-One encoding function T:X->Y </li></ul><ul><li>2. arg min E m (y) = T(arg min E b (x)) </li></ul>
    101. 101. Multi-label to Pseudo-boolean <ul><li>Popular encoding scheme </li></ul><ul><li>[Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06] </li></ul>
    102. 102. Multi-label to Pseudo-boolean <ul><li>Popular encoding scheme </li></ul><ul><li>[Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06] </li></ul>Ishikawa’s result: y ϵ Labels L = {l 1 , l 2 , … , l k } θ ij (y i ,y j ) = g(|y i -y j |) Convex Function g(|y i -y j |) |y i -y j | E( y ) = ∑ θ i (y i ) + ∑ θ ij (y i ,y j ) i,j i
    103. 103. Multi-label to Pseudo-boolean <ul><li>Popular encoding scheme </li></ul><ul><li>[Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06] </li></ul>Schlesinger & Flach ’06: y ϵ Labels L = {l 1 , l 2 , … , l k } θ ij ( l i+1 ,l j ) + θ ij (l i ,l j+1 )  θ ij (l i ,l j ) + θ ij (l i+1 ,l j+1 ) Covers all Submodular multi-label functions More general than Ishikawa E( y ) = ∑ θ i (y i ) + ∑ θ ij (y i ,y j ) i,j i
    104. 104. Multi-label to Pseudo-boolean <ul><li>Applicability </li></ul><ul><ul><li>Only solves restricted class of energy functions </li></ul></ul><ul><ul><li>Cannot handle Potts model potentials </li></ul></ul><ul><li>Computational Cost </li></ul><ul><ul><li>Very high computational cost </li></ul></ul><ul><ul><li>Problem size = |Variables| x |Labels| </li></ul></ul><ul><ul><li>Gray level image denoising (1 Mpixel image) </li></ul></ul><ul><ul><li>(~2.5 x 10 8 graph nodes) </li></ul></ul>Problems
    105. 105. Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
    106. 106. St-mincut based Move algorithms <ul><li>Commonly used for solving non-submodular multi-label problems </li></ul><ul><li>Extremely efficient and produce good solutions </li></ul><ul><li>Not Exact: Produce local optima </li></ul>x ϵ Labels L = {l 1 , l 2 , … , l k } E( x ) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i
    107. 107. Move Making Algorithms Solution Space Energy
    108. 108. Move Making Algorithms Search Neighbourhood Current Solution Optimal Move Solution Space Energy
    109. 109. Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move x c ( t ) Key Property Move Space Bigger move space Solution Space Energy <ul><li>Better solutions </li></ul><ul><li>Finding the optimal move hard </li></ul>
    110. 110. Moves using Graph Cuts <ul><li>Expansion and Swap move algorithms </li></ul><ul><li>[Boykov Veksler and Zabih, PAMI 2001] </li></ul><ul><li>Makes a series of changes to the solution (moves) </li></ul><ul><li>Each move results in a solution with smaller energy </li></ul>Space of Solutions (x) : L N Move Space (t) : 2 N Search Neighbourhood Current Solution N Number of Variables L Number of Labels
    111. 111. Moves using Graph Cuts <ul><li>Expansion and Swap move algorithms </li></ul><ul><li>[Boykov Veksler and Zabih, PAMI 2001] </li></ul><ul><li>Makes a series of changes to the solution (moves) </li></ul><ul><li>Each move results in a solution with smaller energy </li></ul>Current Solution Construct a move function Minimize move function to get optimal move Move to new solution How to minimize move functions?
    112. 112. General Binary Moves Minimize over move variables t to get the optimal move x = t x 1 + (1- t ) x 2 New solution Current Solution Second solution E m ( t ) = E( t x 1 + (1- t ) x 2 ) Boykov, Veksler and Zabih, PAMI 2001 Move energy is a submodular QPBF (Exact Minimization Possible)
    113. 113. Swap Move <ul><li>Variables labeled α , β can swap their labels </li></ul>[Boykov, Veksler, Zabih]
    114. 114. Swap Move Sky House Tree Ground Swap Sky, House [Boykov, Veksler, Zabih] <ul><li>Variables labeled α , β can swap their labels </li></ul>
    115. 115. Swap Move <ul><li>Move energy is submodular if: </li></ul><ul><ul><li>Unary Potentials: Arbitrary </li></ul></ul><ul><ul><li>Pairwise potentials: Semimetric </li></ul></ul><ul><li>Variables labeled α , β can swap their labels </li></ul>[Boykov, Veksler, Zabih] θ ij (l a ,l b ) ≥ 0 θ ij (l a ,l b ) = 0 a = b Examples: Potts model, Truncated Convex
    116. 116. Expansion Move [Boykov, Veksler, Zabih] [Boykov, Veksler, Zabih] <ul><li>Variables take label  or retain current label </li></ul>
    117. 117. Expansion Move Sky House Tree Ground Initialize with Tree Status: Expand Ground Expand House Expand Sky [Boykov, Veksler, Zabih] [Boykov, Veksler, Zabih] <ul><li>Variables take label  or retain current label </li></ul>
    118. 118. Expansion Move <ul><li>Move energy is submodular if: </li></ul><ul><ul><li>Unary Potentials: Arbitrary </li></ul></ul><ul><ul><li>Pairwise potentials: Metric </li></ul></ul>[Boykov, Veksler, Zabih] [Boykov, Veksler, Zabih] θ ij (l a ,l b ) + θ ij (l b ,l c ) ≥ θ ij (l a ,l c ) Semi metric + Triangle Inequality <ul><li>Variables take label  or retain current label </li></ul>Examples: Potts model, Truncated linear Cannot solve truncated quadratic
    119. 119. General Binary Moves Minimize over move variables t x = t x 1 + (1-t) x 2 New solution First solution Second solution Move functions can be non-submodular!! Move Type First Solution Second Solution Guarantee Expansion Old solution All alpha Metric Fusion Any solution Any solution 
    120. 120. Solving Continuous Problems using Fusion Move x = t x 1 + (1-t) x 2 (Lempitsky et al. CVPR08, Woodford et al. CVPR08) x 1 , x 2 can be continuous F x 1 x 2 x Optical Flow Example Final Solution Solution from Method 1 Solution from Method 2
    121. 121. Range Moves <ul><li>Move variables can be multi-label </li></ul><ul><li>Optimal move found out by using the Ishikawa </li></ul><ul><li>Useful for minimizing energies with truncated convex pairwise potentials </li></ul>O. Veksler, CVPR 2007 θ ij (y i ,y j ) = min(|y i -y j |,T) |y i -y j | θ ij (y i ,y j ) T x = ( t == 1) x 1 + (t == 2) x 2 +… + (t == k) x k
    122. 122. Move Algorithms for Solving Higher Order Energies <ul><li>Higher order functions give rise to higher order move energies </li></ul><ul><li>Move energies for certain classes of higher order energies can be transformed to QPBFs. </li></ul>[Kohli, Kumar and Torr, CVPR07] [Kohli, Ladicky and Torr, CVPR08] E( x ) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) + ∑ θ c ( x c ) i,j i c x ϵ Labels L = {l 1 , l 2 , … , l k } Clique c  V
    123. 123. Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
    124. 124. Solving Mixed Programming Problems x – binary image segmentation (x i ∊ {0,1}) ω – non-local parameter (lives in some large set Ω ) constant unary potentials pairwise potentials ≥ 0 Rough Shape Prior Stickman Model ω Pose θ i ( ω, x i ) Shape Prior E(x, ω ) = C( ω ) + ∑ θ i ( ω, x i ) + ∑ θ ij ( ω, x i ,x j ) i,j i
    125. 125. Open Problems Submodular Functions st-mincut Equivalent <ul><li>Characterization of Problems Solvable using st-mincut </li></ul><ul><ul><li>What functions can be transformed to submodular QBFs? </li></ul></ul>
    126. 126. Minimizing General Higher Order Functions <ul><li>We saw how simple higher order potentials can be solved </li></ul><ul><li>How more sophisticated higher order potentials can be solved? </li></ul>
    127. 127. Summary Exact Transformation (global optimum) Or Relaxed transformation (partially optimal) Labelling Problem Submodular Quadratic Pseudoboolean Function Move making algorithms Sub-problem T S st-mincut
    128. 128. Thanks. Questions?
    129. 129. Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P 1 P 2 P 3 5 6 7 8 Pixels Disparity Labels E(x 1 ,x 2 ,x 3 ) = θ 12 (x 1 ,x 2 ) + θ 23 (x 2 ,x 3 ) θ ij (x i ,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0
    130. 130. Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P 1 P 2 P 3 5 6 7 8 Pixels Disparity Labels E(x 1 ,x 2 ,x 3 ) = θ 12 (x 1 ,x 2 ) + θ 23 (x 2 ,x 3 ) θ ij (x i ,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0 E(6,7,7) = 1 + 0 = 1
    131. 131. Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P 1 P 2 P 3 5 6 7 8 Pixels Disparity Labels Pairwise potential penalize slanted planar surfaces E(x 1 ,x 2 ,x 3 ) = θ 12 (x 1 ,x 2 ) + θ 23 (x 2 ,x 3 ) θ ij (x i ,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0 E(6,7,7) = 1 + 0 = 1 E(6,7,8) = 1 + 1 = 2
    132. 132. Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move x E ( x ) x c Transformation function T ( t ) T ( x c , t ) = x n = x c + t
    133. 133. Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E ( x ) x c Transformation function T E m Move Energy ( t ) x E m ( t ) = E ( T ( x c , t )) T ( x c , t ) = x n = x c + t
    134. 134. Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E ( x ) x c E m ( t ) = E ( T ( x c , t )) Transformation function T E m Move Energy T ( x c , t ) = x n = x c + t minimize t* Optimal Move ( t ) x

    ×