• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2
 

ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2

on

  • 647 views

 

Statistics

Views

Total Views
647
Views on SlideShare
647
Embed Views
0

Actions

Likes
0
Downloads
13
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Move to front
  • correct
  • correct
  • Chi square
  • Chi square

ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2 ECCV2008: MAP Estimation Algorithms in Computer Vision - Part 2 Presentation Transcript

  • MAP Estimation Algorithms in M. Pawan Kumar, University of Oxford Pushmeet Kohli, Microsoft Research Computer Vision - Part II
  • Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg Image (D) i i,j n = number of pixels
  • Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg i i,j Unary Cost (c i ) Dark ( negative ) Bright (positive) n = number of pixels
  • Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg i i,j Discontinuity Cost (c ij ) n = number of pixels
  • Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg i i,j Global Minimum (x * ) x * = arg min E(x) x How to minimize E(x)? n = number of pixels
  • Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Recent Advances and Open Problems Connection between st-mincut and energy minimization?
  • Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
  • The st-Mincut Problem Source Sink v 1 v 2 2 5 9 4 2 1
    • Graph (V, E, C)
      • Vertices V = {v 1 , v 2 ... v n }
      • Edges E = {(v 1 , v 2 ) ....}
      • Costs C = {c (1, 2) ....}
  • The st-Mincut Problem Source Sink v 1 v 2 2 5 9 4 2 1 What is a st-cut?
  • The st-Mincut Problem Source Sink v 1 v 2 2 5 9 4 2 1 What is a st-cut? An st-cut ( S , T ) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T 5 + 2 + 9 = 16
  • The st-Mincut Problem What is a st-cut? An st-cut ( S , T ) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T What is the st-mincut? st-cut with the minimum cost Source Sink v 1 v 2 2 5 9 4 2 1 2 + 1 + 4 = 7
  • How to compute the st-mincut? Source Sink v 1 v 2 2 5 9 4 2 1 Solve the dual maximum flow problem In every network, the maximum flow equals the cost of the st-mincut Min-cutMax-flow Theorem Compute the maximum flow between Source and Sink Constraints Edges: Flow < Capacity Nodes: Flow in = Flow out
  • Maxflow Algorithms Augmenting Path Based Algorithms
    • Find path from source to sink with positive capacity
    • Push maximum possible flow through this path
    • Repeat until no path can be found
    Source Sink v 1 v 2 2 5 9 4 2 1 Algorithms assume non-negative capacity Flow = 0
  • Maxflow Algorithms Augmenting Path Based Algorithms
    • Find path from source to sink with positive capacity
    • Push maximum possible flow through this path
    • Repeat until no path can be found
    Source Sink v 1 v 2 2 5 9 4 2 1 Algorithms assume non-negative capacity Flow = 0
  • Maxflow Algorithms Augmenting Path Based Algorithms
    • Find path from source to sink with positive capacity
    • Push maximum possible flow through this path
    • Repeat until no path can be found
    Source Sink v 1 v 2 2-2 5-2 9 4 2 1 Algorithms assume non-negative capacity Flow = 0 + 2
  • Maxflow Algorithms Source Sink v 1 v 2 0 3 9 4 2 1 Augmenting Path Based Algorithms
    • Find path from source to sink with positive capacity
    • Push maximum possible flow through this path
    • Repeat until no path can be found
    Algorithms assume non-negative capacity Flow = 2
  • Maxflow Algorithms Source Sink v 1 v 2 0 3 9 4 2 1 Augmenting Path Based Algorithms
    • Find path from source to sink with positive capacity
    • Push maximum possible flow through this path
    • Repeat until no path can be found
    Algorithms assume non-negative capacity Flow = 2
  • Maxflow Algorithms Source Sink v 1 v 2 0 3 9 4 2 1 Augmenting Path Based Algorithms
    • Find path from source to sink with positive capacity
    • Push maximum possible flow through this path
    • Repeat until no path can be found
    Algorithms assume non-negative capacity Flow = 2
  • Maxflow Algorithms Source Sink v 1 v 2 0 3 5 0 2 1 Augmenting Path Based Algorithms
    • Find path from source to sink with positive capacity
    • Push maximum possible flow through this path
    • Repeat until no path can be found
    Algorithms assume non-negative capacity Flow = 2 + 4
  • Maxflow Algorithms Source Sink v 1 v 2 0 3 5 0 2 1 Augmenting Path Based Algorithms
    • Find path from source to sink with positive capacity
    • Push maximum possible flow through this path
    • Repeat until no path can be found
    Algorithms assume non-negative capacity Flow = 6
  • Maxflow Algorithms Source Sink v 1 v 2 0 3 5 0 2 1 Augmenting Path Based Algorithms
    • Find path from source to sink with positive capacity
    • Push maximum possible flow through this path
    • Repeat until no path can be found
    Algorithms assume non-negative capacity Flow = 6
  • Maxflow Algorithms Source Sink v 1 v 2 0 2 4 0 2+1 1-1 Augmenting Path Based Algorithms
    • Find path from source to sink with positive capacity
    • Push maximum possible flow through this path
    • Repeat until no path can be found
    Algorithms assume non-negative capacity Flow = 6 + 1
  • Maxflow Algorithms Source Sink v 1 v 2 0 2 4 0 3 0 Augmenting Path Based Algorithms
    • Find path from source to sink with positive capacity
    • Push maximum possible flow through this path
    • Repeat until no path can be found
    Algorithms assume non-negative capacity Flow = 7
  • Maxflow Algorithms Source Sink v 1 v 2 0 2 4 0 3 0 Augmenting Path Based Algorithms
    • Find path from source to sink with positive capacity
    • Push maximum possible flow through this path
    • Repeat until no path can be found
    Algorithms assume non-negative capacity Flow = 7
  • History of Maxflow Algorithms [Slide credit: Andrew Goldberg] Augmenting Path and Push-Relabel n: # nodes m: # edges U: maximum edge weight Algorithms assume non-negative edge weights
  • History of Maxflow Algorithms [Slide credit: Andrew Goldberg] Augmenting Path and Push-Relabel n: # nodes m: # edges U: maximum edge weight Algorithms assume non-negative edge weights
  • Augmenting Path based Algorithms a 1 a 2 1000 1 Sink Source 1000 1000 1000 0 Ford Fulkerson: Choose any augmenting path
  • Augmenting Path based Algorithms a 1 a 2 1000 1 Sink Source 1000 1000 1000 0 Bad Augmenting Paths Ford Fulkerson: Choose any augmenting path
  • a 1 a 2 1000 1 Sink Source 1000 1000 1000 0 Augmenting Path based Algorithms Bad Augmenting Path Ford Fulkerson: Choose any augmenting path
  • a 1 a 2 999 0 Sink Source 1000 1000 999 1 Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting path
  • Augmenting Path based Algorithms a 1 a 2 999 0 Sink Source 1000 1000 999 1 Ford Fulkerson: Choose any augmenting path n: # nodes m: # edges We will have to perform 2000 augmentations! Worst case complexity: O (m x Total_Flow) (Pseudo-polynomial bound: depends on flow)
  • Augmenting Path based Algorithms Dinic: Choose shortest augmenting path n: # nodes m: # edges Worst case Complexity: O (m n 2 ) a 1 a 2 1000 1 Sink Source 1000 1000 1000 0
  • Maxflow in Computer Vision
    • Specialized algorithms for vision problems
      • Grid graphs
      • Low connectivity (m ~ O(n))
    • Dual search tree augmenting path algorithm
    • [Boykov and Kolmogorov PAMI 2004]
      • Finds approximate shortest augmenting paths efficiently
      • High worst-case time complexity
      • Empirically outperforms other algorithms on vision problems
  • Maxflow in Computer Vision
    • Specialized algorithms for vision problems
      • Grid graphs
      • Low connectivity (m ~ O(n))
    • Dual search tree augmenting path algorithm
    • [Boykov and Kolmogorov PAMI 2004]
      • Finds approximate shortest augmenting paths efficiently
      • High worst-case time complexity
      • Empirically outperforms other algorithms on vision problems
      • Efficient code available on the web
    http://www.adastral.ucl.ac.uk/~vladkolm/software.html
  • Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
  • St-mincut and Energy Minimization E: {0,1} n -> R Minimizing a Qudratic Pseudoboolean function E(x) Functions of boolean variables Pseudoboolean? Polynomial time st-mincut algorithms require non-negative edge weights E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) c ij ≥0 i,j i T S st-mincut
  • So how does this work?
    • Construct a graph such that:
    • Any st-cut corresponds to an assignment of x
    • The cost of the cut is equal to the energy of x : E(x)
    E(x) Solution T S st-mincut
  • Graph Construction Sink (1) Source (0) a 1 a 2 E(a 1 ,a 2 )
  • Graph Construction Sink (1) Source (0) a 1 a 2 E(a 1 ,a 2 ) = 2a 1 2
  • Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 2 5 Sink (1) Source (0)
  • Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 2 5 9 4 Sink (1) Source (0)
  • Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 2 5 9 4 2 Sink (1) Source (0)
  • Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)
  • Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)
  • Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2 a 1 + 5 ā 1 + 9 a 2 + 4 ā 2 + 2 a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 a 1 = 1 a 2 = 1 E (1,1) = 11 Cost of cut = 11 Sink (1) Source (0)
  • Graph Construction a 1 a 2 E(a 1 ,a 2 ) = 2 a 1 + 5 ā 1 + 9 a 2 + 4 ā 2 + 2 a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0) a 1 = 1 a 2 = 0 E (1,0) = 8 st-mincut cost = 8
  • Energy Function Reparameterization Two functions E 1 and E 2 are reparameterizations if E 1 ( x ) = E 2 ( x ) for all x For instance: E 1 (a 1 ) = 1+ 2a 1 + 3ā 1 E 2 (a 1 ) = 3 + ā 1 a 1 ā 1 1+ 2a 1 + 3ā 1 3 + ā 1 0 1 4 4 1 0 3 3
  • Flow and Reparametrization a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0)
  • Flow and Reparametrization a 1 a 2 E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 5 9 4 2 1 Sink (1) Source (0) 2a 1 + 5ā 1 = 2(a 1 +ā 1 ) + 3ā 1 = 2 + 3ā 1
  • Flow and Reparametrization Sink (1) Source (0) a 1 a 2 E(a 1 ,a 2 ) = 2 + 3ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 0 3 9 4 2 1 2a 1 + 5ā 1 = 2(a 1 +ā 1 ) + 3ā 1 = 2 + 3ā 1
  • Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 2 + 3ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 0 3 9 4 2 1 9a 2 + 4ā 2 = 4(a 2 +ā 2 ) + 5ā 2 = 4 + 5ā 2
  • Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 2 + 3ā 1 + 5a 2 + 4 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1 9a 2 + 4ā 2 = 4(a 2 +ā 2 ) + 5ā 2 = 4 + 5ā 2
  • Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1
  • Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1
  • Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 0 3 5 0 2 1 3ā 1 + 5a 2 + 2a 1 ā 2 = 2(ā 1 +a 2 +a 1 ā 2 ) +ā 1 +3a 2 = 2(1+ā 1 a 2 ) +ā 1 +3a 2 F1 = ā 1 +a 2 +a 1 ā 2 F2 = 1+ā 1 a 2 a 1 a 2 F1 F2 0 0 1 1 0 1 2 2 1 0 1 1 1 1 1 1
  • Flow and Reparametrization Sink (0) Source (1) a 1 a 2 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 0 1 3 0 0 3 3ā 1 + 5a 2 + 2a 1 ā 2 = 2(ā 1 +a 2 +a 1 ā 2 ) +ā 1 +3a 2 = 2(1+ā 1 a 2 ) +ā 1 +3a 2 F1 = ā 1 +a 2 +a 1 ā 2 F2 = 1+ā 1 a 2 a 1 a 2 F1 F2 0 0 1 1 0 1 2 2 1 0 1 1 1 1 1 1
  • Flow and Reparametrization Sink (0) Source (1) a 1 a 2 0 1 3 0 0 3 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 No more augmenting paths possible
  • Flow and Reparametrization Sink (0) Source (1) a 1 a 2 0 1 3 0 0 3 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Total Flow Residual Graph (positive coefficients) bound on the optimal solution Inference of the optimal solution becomes trivial because the bound is tight
  • Flow and Reparametrization Sink (0) Source (1) a 1 a 2 0 1 3 0 0 3 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 a 1 = 1 a 2 = 0 E (1,0) = 8 st-mincut cost = 8 Total Flow bound on the optimal solution Inference of the optimal solution becomes trivial because the bound is tight Residual Graph (positive coefficients)
  • Example: Image Segmentation E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) E: {0,1} n -> R 0 -> fg 1 -> bg i i,j Global Minimum (x * ) x * = arg min E(x) x How to minimize E(x)?
  • How does the code look like? Sink (1) Source (0) Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1)
  • How does the code look like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) a 1 a 2 fgCost( a 1 ) Sink (1) Source (0) fgCost( a 2 ) bgCost( a 1 ) bgCost( a 2 )
  • How does the code look like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) a 1 a 2 fgCost( a 1 ) Sink (1) Source (0) fgCost( a 2 ) bgCost( a 1 ) bgCost( a 2 ) cost(p,q)
  • How does the code look like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); end g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) a 1 a 2 fgCost( a 1 ) Sink (1) Source (0) fgCost( a 2 ) bgCost( a 1 ) bgCost( a 2 ) cost(p,q) a 1 = bg a 2 = fg
  • Image Segmentation in Video Image Flow Global Optimum s t = 0 = 1 E(x) x * n-links st-cut
  • Image Segmentation in Video Image Flow Global Optimum
  • Dynamic Energy Minimization E B computationally expensive operation E A Recycling Solutions Can we do better? Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07) S B S A minimize
  • Dynamic Energy Minimization E B computationally expensive operation E A cheaper operation Kohli & Torr (ICCV05, PAMI07) 3 – 100000 time speedup! Reuse flow Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07) S B S A minimize Simpler energy E B* differences between A and B A and B similar Reparametrization
  • Dynamic Energy Minimization Reparametrized Energy Kohli & Torr (ICCV05, PAMI07) Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07) E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Original Energy E(a 1 ,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 7a 1 ā 2 + ā 1 a 2 E(a 1 ,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 + 5a 1 ā 2 New Energy New Reparametrized Energy
  • Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
  • Minimizing Energy Functions
    • General Energy Functions
      • NP-hard to minimize
      • Only approximate minimization possible
    • Easy energy functions
      • Solvable in polynomial time
      • Submodular ~ O(n 6 )
    Space of Function Minimization Problems Submodular Functions NP-Hard MAXCUT Functions defined on trees
  • Submodular Set Functions Set function f  2 |E|  ℝ 2 |E| = #subsets of E Let E= {a 1 ,a 2 , .... a n } be a set
  • Submodular Set Functions Set function f  2 |E|  ℝ is submodular if E A B f( A ) + f( B )  f( A  B ) + f( A  B ) for all A , B  E 2 |E| = #subsets of E Let E= {a 1 ,a 2 , .... a n } be a set Important Property Sum of two submodular functions is submodular
  • Minimizing Submodular Functions
    • Minimizing general submodular functions
      • O(n 5 Q + n 6 ) where Q is function evaluation time
      • [Orlin, IPCO 2007]
    • Symmetric submodular functions
      • E ( x ) = E ( 1 - x )
      • O(n 3 ) [ Queyranne 1998]
    • Quadratic pseudoboolean
      • Can be transformed to st-mincut
      • One node per variable [ O(n 3 ) complexity]
      • Very low empirical running time
  • Submodular Pseudoboolean Functions
    • All functions for one boolean variable (f: {0,1} -> R) are submodular
    • A function of two boolean variables (f: {0,1} 2 -> R) is submodular if
    • f(0,1) + f(1,0)  f(0,0) + f(1,1)
    • A general pseudoboolean function f  2 n  ℝ is submodular if all its projections f p are submodular i.e.
    f p (0,1) + f p (1,0)  f p (0,0) + f p (1,1) Function defined over boolean vectors x = {x 1 ,x 2 , .... x n } Definition :
  • Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0)  θ ij (0,0) + θ ij (1,1) For all ij E(x) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i
  • Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0)  θ ij (0,0) + θ ij (1,1) For all ij E(x) = ∑ c i x i + ∑ c ij x i (1-x j ) c ij ≥0 i,j i Equivalent (transformable) i.e. All submodular QPBFs are st-mincut solvable E(x) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i
  • 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
  • 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
  • 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
  • 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
  • 0 1 0 1 x i x j = A + 0 1 0 1 0 1 0 1 0 1 0 1 + + if x 1 =1 add C-A if x 2 = 1 add D-C B+C-A-D  0 is true from the submodularity of θ ij How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) θ ij (x i ,x j ) = θ ij (0,0) + ( θ ij (1,0)- θ ij (0,0)) x i + ( θ ij (1,0)- θ ij (0,0)) x j + ( θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1-x i ) x j A B C D 0 0 C-A C-A 0 D-C 0 D-C 0 B+C-A-D 0 0
  • Quadratic Submodular Pseudoboolean Functions θ ij (0,1) + θ ij (1,0)  θ ij (0,0) + θ ij (1,1) For all ij Equivalent (transformable) x in {0,1} n E(x) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i T S st-mincut
  • Minimizing Non-Submodular Functions
    • Minimizing general non-submodular functions is NP-hard.
    • Commonly used method is to solve a relaxation of the problem
    θ ij (0,1) + θ ij (1,0) ≤ θ ij (0,0) + θ ij (1,1) for some ij [Slide credit: Carsten Rother] E(x) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i
  • Minimization using Roof-dual Relaxation pairwise nonsubmodular unary pairwise submodular [Slide credit: Carsten Rother]
    • Double number of variables:
    Minimization using Roof-dual Relaxation [Slide credit: Carsten Rother]
    • Double number of variables:
    Minimization using Roof-dual Relaxation Non- submodular Submodular
    • Double number of variables:
    Minimization using Roof-dual Relaxation
    • is submodular ! Ignore
    • (solvable using st-mincut)
    Property of the problem :
    • Double number of variables:
    Minimization using Roof-dual Relaxation is the optimal label Property of the solution :
  • Recap
    • Exact minimization of Submodular QBFs using graph cuts.
    • Obtaining partially optimal solutions of non-submodular QBFs using graph cuts.
  • But ...
    • Need higher order energy functions to model image structure
      • Field of experts [Roth and Black ]
    • Many problems in computer vision involve multiple labels
    c x ϵ Labels L = {l 1 , l 2 , … , l k } Clique c  V E( x ) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) + ∑ θ c ( x c ) i,j i
  • Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions
  • Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions
  • Higher order to Quadratic
    • Simple Example using Auxiliary variables
    { 0 if all x i = 0 C 1 otherwise f( x ) = min f( x ) min C 1 a + C 1 ∑ ā x i x ϵ L = {0,1} n x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i = 0 a=0 ( ā=1) f( x ) = 0 ∑ x i ≥ 1 a=1 ( ā=0) f( x ) = C 1
  • Higher order to Quadratic min f( x ) min C 1 a + C 1 ∑ ā x i x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i 1 2 3 C 1 C 1 ∑ x i
  • Higher order to Quadratic min f( x ) min C 1 a + C 1 ∑ ā x i x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i 1 2 3 C 1 C 1 ∑ x i a=1 a=0 Lower envelop of concave functions is concave
  • Higher order to Quadratic min f( x ) min f 1 (x) a + f 2 (x) ā x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i 1 2 3 a=1 Lower envelop of concave functions is concave f 2 (x) f 1 (x)
  • Higher order to Quadratic min f( x ) min f 1 (x) a + f 2 (x) ā x = x,a ϵ {0,1} Higher Order Submodular Function Quadratic Submodular Function ∑ x i 1 2 3 a=1 a=0 Lower envelop of concave functions is concave f 2 (x) f 1 (x)
  • Transforming problems in QBFs Multi-label Functions Pseudoboolean Functions Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions
  • Multi-label to Pseudo-boolean So what is the problem? E b (x 1 ,x 2 , ..., x m ) E m (y 1 ,y 2 , ..., y n ) Multi-label Problem Binary label Problem y i ϵ L = {l 1 , l 2 , … , l k } x i ϵ L = {0,1} such that:
    • Let Y and X be the set of feasible solutions, then
    • For each binary solution x ϵ X with finite energy there exists exactly one multi-label solution y ϵ Y
    • -> One-One encoding function T:X->Y
    • 2. arg min E m (y) = T(arg min E b (x))
  • Multi-label to Pseudo-boolean
    • Popular encoding scheme
    • [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]
  • Multi-label to Pseudo-boolean
    • Popular encoding scheme
    • [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]
    Ishikawa’s result: y ϵ Labels L = {l 1 , l 2 , … , l k } θ ij (y i ,y j ) = g(|y i -y j |) Convex Function g(|y i -y j |) |y i -y j | E( y ) = ∑ θ i (y i ) + ∑ θ ij (y i ,y j ) i,j i
  • Multi-label to Pseudo-boolean
    • Popular encoding scheme
    • [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]
    Schlesinger & Flach ’06: y ϵ Labels L = {l 1 , l 2 , … , l k } θ ij ( l i+1 ,l j ) + θ ij (l i ,l j+1 )  θ ij (l i ,l j ) + θ ij (l i+1 ,l j+1 ) Covers all Submodular multi-label functions More general than Ishikawa E( y ) = ∑ θ i (y i ) + ∑ θ ij (y i ,y j ) i,j i
  • Multi-label to Pseudo-boolean
    • Applicability
      • Only solves restricted class of energy functions
      • Cannot handle Potts model potentials
    • Computational Cost
      • Very high computational cost
      • Problem size = |Variables| x |Labels|
      • Gray level image denoising (1 Mpixel image)
      • (~2.5 x 10 8 graph nodes)
    Problems
  • Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
  • St-mincut based Move algorithms
    • Commonly used for solving non-submodular multi-label problems
    • Extremely efficient and produce good solutions
    • Not Exact: Produce local optima
    x ϵ Labels L = {l 1 , l 2 , … , l k } E( x ) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) i,j i
  • Move Making Algorithms Solution Space Energy
  • Move Making Algorithms Search Neighbourhood Current Solution Optimal Move Solution Space Energy
  • Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move x c ( t ) Key Property Move Space Bigger move space Solution Space Energy
    • Better solutions
    • Finding the optimal move hard
  • Moves using Graph Cuts
    • Expansion and Swap move algorithms
    • [Boykov Veksler and Zabih, PAMI 2001]
    • Makes a series of changes to the solution (moves)
    • Each move results in a solution with smaller energy
    Space of Solutions (x) : L N Move Space (t) : 2 N Search Neighbourhood Current Solution N Number of Variables L Number of Labels
  • Moves using Graph Cuts
    • Expansion and Swap move algorithms
    • [Boykov Veksler and Zabih, PAMI 2001]
    • Makes a series of changes to the solution (moves)
    • Each move results in a solution with smaller energy
    Current Solution Construct a move function Minimize move function to get optimal move Move to new solution How to minimize move functions?
  • General Binary Moves Minimize over move variables t to get the optimal move x = t x 1 + (1- t ) x 2 New solution Current Solution Second solution E m ( t ) = E( t x 1 + (1- t ) x 2 ) Boykov, Veksler and Zabih, PAMI 2001 Move energy is a submodular QPBF (Exact Minimization Possible)
  • Swap Move
    • Variables labeled α , β can swap their labels
    [Boykov, Veksler, Zabih]
  • Swap Move Sky House Tree Ground Swap Sky, House [Boykov, Veksler, Zabih]
    • Variables labeled α , β can swap their labels
  • Swap Move
    • Move energy is submodular if:
      • Unary Potentials: Arbitrary
      • Pairwise potentials: Semimetric
    • Variables labeled α , β can swap their labels
    [Boykov, Veksler, Zabih] θ ij (l a ,l b ) ≥ 0 θ ij (l a ,l b ) = 0 a = b Examples: Potts model, Truncated Convex
  • Expansion Move [Boykov, Veksler, Zabih] [Boykov, Veksler, Zabih]
    • Variables take label  or retain current label
  • Expansion Move Sky House Tree Ground Initialize with Tree Status: Expand Ground Expand House Expand Sky [Boykov, Veksler, Zabih] [Boykov, Veksler, Zabih]
    • Variables take label  or retain current label
  • Expansion Move
    • Move energy is submodular if:
      • Unary Potentials: Arbitrary
      • Pairwise potentials: Metric
    [Boykov, Veksler, Zabih] [Boykov, Veksler, Zabih] θ ij (l a ,l b ) + θ ij (l b ,l c ) ≥ θ ij (l a ,l c ) Semi metric + Triangle Inequality
    • Variables take label  or retain current label
    Examples: Potts model, Truncated linear Cannot solve truncated quadratic
  • General Binary Moves Minimize over move variables t x = t x 1 + (1-t) x 2 New solution First solution Second solution Move functions can be non-submodular!! Move Type First Solution Second Solution Guarantee Expansion Old solution All alpha Metric Fusion Any solution Any solution 
  • Solving Continuous Problems using Fusion Move x = t x 1 + (1-t) x 2 (Lempitsky et al. CVPR08, Woodford et al. CVPR08) x 1 , x 2 can be continuous F x 1 x 2 x Optical Flow Example Final Solution Solution from Method 1 Solution from Method 2
  • Range Moves
    • Move variables can be multi-label
    • Optimal move found out by using the Ishikawa
    • Useful for minimizing energies with truncated convex pairwise potentials
    O. Veksler, CVPR 2007 θ ij (y i ,y j ) = min(|y i -y j |,T) |y i -y j | θ ij (y i ,y j ) T x = ( t == 1) x 1 + (t == 2) x 2 +… + (t == k) x k
  • Move Algorithms for Solving Higher Order Energies
    • Higher order functions give rise to higher order move energies
    • Move energies for certain classes of higher order energies can be transformed to QPBFs.
    [Kohli, Kumar and Torr, CVPR07] [Kohli, Ladicky and Torr, CVPR08] E( x ) = ∑ θ i (x i ) + ∑ θ ij (x i ,x j ) + ∑ θ c ( x c ) i,j i c x ϵ Labels L = {l 1 , l 2 , … , l k } Clique c  V
  • Outline of the Tutorial The st-mincut problem What problems can we solve using st-mincut? st-mincut based Move algorithms Connection between st-mincut and energy minimization? Recent Advances and Open Problems
  • Solving Mixed Programming Problems x – binary image segmentation (x i ∊ {0,1}) ω – non-local parameter (lives in some large set Ω ) constant unary potentials pairwise potentials ≥ 0 Rough Shape Prior Stickman Model ω Pose θ i ( ω, x i ) Shape Prior E(x, ω ) = C( ω ) + ∑ θ i ( ω, x i ) + ∑ θ ij ( ω, x i ,x j ) i,j i
  • Open Problems Submodular Functions st-mincut Equivalent
    • Characterization of Problems Solvable using st-mincut
      • What functions can be transformed to submodular QBFs?
  • Minimizing General Higher Order Functions
    • We saw how simple higher order potentials can be solved
    • How more sophisticated higher order potentials can be solved?
  • Summary Exact Transformation (global optimum) Or Relaxed transformation (partially optimal) Labelling Problem Submodular Quadratic Pseudoboolean Function Move making algorithms Sub-problem T S st-mincut
  • Thanks. Questions?
  • Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P 1 P 2 P 3 5 6 7 8 Pixels Disparity Labels E(x 1 ,x 2 ,x 3 ) = θ 12 (x 1 ,x 2 ) + θ 23 (x 2 ,x 3 ) θ ij (x i ,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0
  • Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P 1 P 2 P 3 5 6 7 8 Pixels Disparity Labels E(x 1 ,x 2 ,x 3 ) = θ 12 (x 1 ,x 2 ) + θ 23 (x 2 ,x 3 ) θ ij (x i ,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0 E(6,7,7) = 1 + 0 = 1
  • Use of Higher order Potentials Stereo - Woodford et al. CVPR 2008 P 1 P 2 P 3 5 6 7 8 Pixels Disparity Labels Pairwise potential penalize slanted planar surfaces E(x 1 ,x 2 ,x 3 ) = θ 12 (x 1 ,x 2 ) + θ 23 (x 2 ,x 3 ) θ ij (x i ,x j ) = 0 if x i =x j C otherwise { E(6,6,6) = 0 + 0 = 0 E(6,7,7) = 1 + 0 = 1 E(6,7,8) = 1 + 1 = 2
  • Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move x E ( x ) x c Transformation function T ( t ) T ( x c , t ) = x n = x c + t
  • Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E ( x ) x c Transformation function T E m Move Energy ( t ) x E m ( t ) = E ( T ( x c , t )) T ( x c , t ) = x n = x c + t
  • Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E ( x ) x c E m ( t ) = E ( T ( x c , t )) Transformation function T E m Move Energy T ( x c , t ) = x n = x c + t minimize t* Optimal Move ( t ) x