• Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
419
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
17
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Pushmeet KohliICCV 2009
  • 2. Course programme9.30-10.00 Introduction (Andrew Blake)10.00-11.00 Discrete Models in Computer Vision (Carsten Rother)15min Coffee break11.15-12.30 Message Passing: DP, TRW, LP relaxation (Pawan Kumar)12.30-13.00 Quadratic pseudo-boolean optimization (Pushmeet Kohli)1 hour Lunch break14:00-15.00 Transformation and move-making methods (Pushmeet Kohli)15:00-15.30 Speed and Efficiency (Pushmeet Kohli)15min Coffee break15:45-16.15 Comparison of Methods (Carsten Rother)16:30-17.30 Recent Advances: Dual-decomposition, higher-order, etc. (Carsten Rother + Pawan Kumar) All online material will be online (after conference): http://research.microsoft.com/en-us/um/cambridge/projects/tutorial/
  • 3. E(x) = ∑ fi (xi) + ∑ gij (xi,xj) + ∑ hc(xc) i ij c Unary Pairwise Higher Order ∑ ci xi + ∑ dij |xi-xj| E: {0,1}n → R i i,j n = number of pixels Image Segmentation
  • 4. n = Number of Variables Segmentation EnergyNP-Hard MAXCUTCSP Pair-wise O(n3) Submodular Functions Tree Structured O(n6) Space of Problems
  • 5. st-mincut and Pseudo-boolean optimizationMore General Minimization Problems Speed and Efficiency
  • 6. st-mincut and Pseudo-boolean optimizationMore General Minimization Problems Speed and Efficiency
  • 7. Pseudo-boolean function f{0,1}n  ℝ is submodular if f(A) + f(B)  f(A˅B) + f(A˄B) for all A,B ϵ {0,1}n (OR) (AND)Example: n = 2, A = [1,0] , B = [0,1] f([1,0]) + f([0,1])  f([1,1]) + f([0,0]) Property : Sum of submodular functions is submodular Binary Image Segmentation Energy is submodular E(x) = ∑ ci xi + ∑ dij |xi-xj| i i,j
  • 8.  Discrete Analogues of Concave Functions [Lovasz, ’83] Widely applied in Operations Research Applications in Machine Learning  MAP Inference in Markov Random Fields  Clustering [Narasimhan , Jojic, & Bilmes, NIPS 2005]  Structure Learning [Narasimhan & Bilmes, NIPS 2006] Maximizing the spread of influence through a social network [Kempe, Kleinberg & Tardos, KDD 2003]
  • 9.  Polynomial time algorithms  Ellipsoid Algorithm: [Grotschel, Lovasz & Schrijver ‘81]  First strongly polynomial algorithm: [Iwata et al. ’00] [A. Schrijver ’00]  Current Best: O(n5 Q + n6) [Q is function evaluation time] [Orlin ‘07] Symmetric functions: E(x) = E(1-x)  Can be minimized in O(n3) Minimizing Pairwise submodular functions  Can be transformed to st-mincut/max-flow [Hammer , 1965]  Very low empirical running time ~ O(n) E(X) = ∑ fi (xi) + ∑ gij (xi,xj) i ij
  • 10. Source 2 9 Graph (V, E, C) 2 Vertices V = {v1, v2 ... vn}v1 v2 Edges E = {(v1, v2) ....} 1 Costs C = {c(1, 2) ....}5 4 Sink
  • 11. What is a st-cut? Source 2 9 2v1 v2 15 4 Sink
  • 12. What is a st-cut? An st-cut (S,T) divides the nodes between source and sink. Source 2 9 What is the cost of a st-cut? 2 Sum of cost of all edges going v1 v2 from S to T 1 5 4 Sink5 + 1 + 9 = 15
  • 13. What is a st-cut? An st-cut (S,T) divides the nodes between source and sink. Source 2 9 What is the cost of a st-cut? 2 Sum of cost of all edges going v1 v2 from S to T 1 5 4 What is the st-mincut? Sink st-cut with the minimum cost2 + 2 + 4 = 8
  • 14. Construct a graph such that:1. Any st-cut corresponds to an assignment of x2. The cost of the cut is equal to the energy of x : E(x) S st-mincutE(x) T Solution [Hammer, 1965] [Kolmogorov and Zabih, 2002]
  • 15. E(x) = ∑ θi (xi) + ∑ θij (xi,xj) i i,jFor all ij θij(0,1) + θij (1,0)  θij (0,0) + θij (1,1) Equivalent (transformable) E(x) = ∑ ci xi + ∑ cij xi(1-xj) cij≥0 i i,j
  • 16. E(a1,a2) Source (0) a1 a2 Sink (1)
  • 17. E(a1,a2) = 2a1 Source (0) 2 a1 a2 Sink (1)
  • 18. E(a1,a2) = 2a1 + 5ā1 Source (0) 2 a1 a2 5 Sink (1)
  • 19. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 Source (0) 2 9 a1 a2 5 4 Sink (1)
  • 20. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 Source (0) 2 9 a1 a2 2 5 4 Sink (1)
  • 21. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 2 9 1 a1 a2 2 5 4 Sink (1)
  • 22. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 2 9 1 a1 a2 2 5 4 Sink (1)
  • 23. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 2 9 Cost of cut = 11 1 a1 a2 a1 = 1 a2 = 1 2 E (1,1) = 11 5 4 Sink (1)
  • 24. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 2 9 st-mincut cost = 8 1 a1 a2 a1 = 1 a2 = 0 2 E (1,0) = 8 5 4 Sink (1)
  • 25. Solve the dual maximum flow problem Compute the maximum flow between Source Source and Sink s.t. 2 9 Edges: Flow < Capacity 1 Nodes: Flow in = Flow outv1 v2 25 4 Min-cutMax-flow Theorem Sink In every network, the maximum flow equals the cost of the st-mincut Assuming non-negative capacity
  • 26. Flow = 0 Augmenting Path Based Algorithms Source 2 9 1v1 v2 25 4 Sink
  • 27. Flow = 0 Augmenting Path Based Algorithms Source 2 9 1. Find path from source to sink with positive capacity 1v1 v2 25 4 Sink
  • 28. Flow = 0 + 2 Augmenting Path Based Algorithms Source2-2 9 1. Find path from source to sink with positive capacity 1 v1 v2 2. Push maximum possible flow 2 through this path 45-2 Sink
  • 29. Flow = 2 Augmenting Path Based Algorithms Source 0 9 1. Find path from source to sink with positive capacity 1v1 v2 2. Push maximum possible flow 2 through this path3 4 Sink
  • 30. Flow = 2 Augmenting Path Based Algorithms Source 0 9 1. Find path from source to sink with positive capacity 1v1 v2 2. Push maximum possible flow 2 through this path3 4 3. Repeat until no path can be Sink found
  • 31. Flow = 2 Augmenting Path Based Algorithms Source 0 9 1. Find path from source to sink with positive capacity 1v1 v2 2. Push maximum possible flow 2 through this path3 4 3. Repeat until no path can be Sink found
  • 32. Flow = 2 + 4 Augmenting Path Based Algorithms Source 0 5 1. Find path from source to sink with positive capacity 1v1 v2 2. Push maximum possible flow 2 through this path3 0 3. Repeat until no path can be Sink found
  • 33. Flow = 6 Augmenting Path Based Algorithms Source 0 5 1. Find path from source to sink with positive capacity 1v1 v2 2. Push maximum possible flow 2 through this path3 0 3. Repeat until no path can be Sink found
  • 34. Flow = 6 Augmenting Path Based Algorithms Source 0 5 1. Find path from source to sink with positive capacity 1v1 v2 2. Push maximum possible flow 2 through this path3 0 3. Repeat until no path can be Sink found
  • 35. Flow = 6 + 2 Augmenting Path Based Algorithms Source 0 3 1. Find path from source to sink with positive capacity 1+2v1 v2 2. Push maximum possible flow 2-2 through this path1 0 3. Repeat until no path can be Sink found
  • 36. Flow = 8 Augmenting Path Based Algorithms Source 0 4 1. Find path from source to sink with positive capacity 3v1 v2 2. Push maximum possible flow 0 through this path2 0 3. Repeat until no path can be Sink found
  • 37. Flow = 8 Augmenting Path Based Algorithms Source 0 4 1. Find path from source to sink with positive capacity 3v1 v2 2. Push maximum possible flow 0 through this path2 0 3. Repeat until no path can be Sink found
  • 38. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 2 9 1 a1 a2 2 5 4 Sink (1)
  • 39. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 2 9 2a1 + 5ā1 1 a1 a2 = 2(a1+ā1) + 3ā1 2 = 2 + 3ā1 5 4 Sink (1)
  • 40. E(a1,a2) = 2 + 3ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 0 9 2a1 + 5ā1 1 a1 a2 = 2(a1+ā1) + 3ā1 2 = 2 + 3ā1 3 4 Sink (1)
  • 41. E(a1,a2) = 2 + 3ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 0 9 9a2 + 4ā2 1 a1 a2 = 4(a2+ā2) + 5ā2 2 = 4 + 5ā2 3 4 Sink (1)
  • 42. E(a1,a2) = 2 + 3ā1+ 5a2 + 4 + 2a1ā2 + ā1a2 Source (0) 0 5 9a2 + 4ā2 1 a1 a2 = 4(a2+ā2) + 5ā2 2 = 4 + 5ā2 3 0 Sink (1)
  • 43. E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2 Source (0) 0 5 1 a1 a2 2 3 0 Sink (1)
  • 44. E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2 Source (0) 0 5 1 a1 a2 2 3 0 Sink (1)
  • 45. E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2 3ā1+ 5a2 + 2a1ā2 = 2(ā1+a2+a1ā2) +ā1+3a2 Source (0) = 2(1+ā1a2) +ā1+3a2 0 5 F1 = ā1+a2+a1ā2 1 a1 a2 F2 = 1+ā1a2 2 a1 a2 F1 F2 3 0 0 0 1 1 0 1 2 2 Sink (1) 1 0 1 1 1 1 1 1
  • 46. E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 3ā1+ 5a2 + 2a1ā2 = 2(ā1+a2+a1ā2) +ā1+3a2 Source (0) = 2(1+ā1a2) +ā1+3a2 0 3 F1 = ā1+a2+a1ā2 3 a1 a2 F2 = 1+ā1a2 0 a1 a2 F1 F2 1 0 0 0 1 1 0 1 2 2 Sink (1) 1 0 1 1 1 1 1 1
  • 47. E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 Source (0) 0 3 3 No more a1 a2 augmenting paths 0 possible 1 0 Sink (1)
  • 48. E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 Residual Graph (positive coefficients) Source (0) Total Flow bound on the 0 3 optimal solution 3 a1 a2 0 1 0 Sink (1)Tight Bound --> Inference of the optimal solution becomes trivial
  • 49. E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 Residual Graph (positive coefficients) Source (0) Total Flow bound on the 0 3 st-mincut cost = 8 optimal solution 3 a1 a2 a1 = 1 a2 = 0 0 E (1,0) = 8 1 0 Sink (1)Tight Bound --> Inference of the optimal solution becomes trivial
  • 50. Augmenting Path and Push-Relabel n: #nodes m: #edges U: maximum edge weight Algorithms assume non- negative edge weights [Slide credit: Andrew Goldberg]
  • 51. Augmenting Path and Push-Relabel n: #nodes m: #edges U: maximum edge weight Algorithms assume non- negative edge weights [Slide credit: Andrew Goldberg]
  • 52. Ford Fulkerson: Choose any augmenting path Source 1000 1000 1 a1 a2 0 1000 1000 Sink
  • 53. Ford Fulkerson: Choose any augmenting path Source 1000 1000 1 a1 a2 0 Good 1000 1000 Augmenting Paths Sink
  • 54. Ford Fulkerson: Choose any augmenting path Source 1000 1000 1 a1 a2 0 Bad 1000 1000 Augmenting Path Sink
  • 55. Ford Fulkerson: Choose any augmenting path Source 999 1000 0 a1 a2 1 1000 999 Sink
  • 56. n: #nodesFord Fulkerson: Choose any augmenting path m: #edges Source 999 1000 0 a1 a2 1 1000 999 Sink We will have to perform 2000 augmentations! Worst case complexity: O (m x Total_Flow) (Pseudo-polynomial bound: depends on flow)
  • 57. n: #nodesDinic: Choose shortest augmenting path m: #edges Source 1000 1000 1 a1 a2 0 1000 1000 Sink Worst case Complexity: O (m n2)
  • 58.  Specialized algorithms for vision problems  Grid graphs  Low connectivity (m ~ O(n))  Dual search tree augmenting path algorithm [Boykov and Kolmogorov PAMI 2004] • Finds approximate shortest augmenting paths efficiently • High worst-case time complexity • Empirically outperforms other algorithms on vision problems Efficient code available on the webhttp://www.adastral.ucl.ac.uk/~vladkolm/software.html
  • 59. E(x) = ∑ ci xi + ∑ dij |xi-xj| E: {0,1}n → R 0 → fg i i,j 1 → bg n = number of pixels x* = arg min E(x) x How to minimize E(x)?
  • 60. Graph *g;For all pixels p Source (0) /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p));endfor all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q));endg->compute_maxflow(); Sink (1)label_p = g->is_connected_to_source(nodeID(p));// is the label of pixel p (0 or 1)
  • 61. Graph *g;For all pixels p Source (0) /* Add a node to the graph */ nodeID(p) = g->add_node(); bgCost(a1) bgCost(a2) /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); a1 a2endfor all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); fgCost(a1) fgCost(a2)endg->compute_maxflow(); Sink (1)label_p = g->is_connected_to_source(nodeID(p));// is the label of pixel p (0 or 1)
  • 62. Graph *g;For all pixels p Source (0) /* Add a node to the graph */ nodeID(p) = g->add_node(); bgCost(a1) bgCost(a2) /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); cost(p,q) a1 a2endfor all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); fgCost(a1) fgCost(a2)endg->compute_maxflow(); Sink (1)label_p = g->is_connected_to_source(nodeID(p));// is the label of pixel p (0 or 1)
  • 63. Graph *g;For all pixels p Source (0) /* Add a node to the graph */ nodeID(p) = g->add_node(); bgCost(a1) bgCost(a2) /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); cost(p,q) a1 a2endfor all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); fgCost(a1) fgCost(a2)endg->compute_maxflow(); Sink (1)label_p = g->is_connected_to_source(nodeID(p));// is the label of pixel p (0 or 1) a1 = bg a2 = fg
  • 64. Lunch
  • 65. Advances in Markov Random Fields for Computer Vision MIT Press, summer 2010 Topics of this course and much, much more Contributors: usual suspects – lecturers on this course + Boykov,Kolmogorov, Weiss, Freeman, .... one for the office and one for home www.research.microsoft.com/vision/MRFbook
  • 66. st-mincut and Pseudo-boolean optimizationMore General Minimization Problems Speed and Efficiency
  • 67.  Non-submodular Energy Functions Mixed (Real-Integer) Problems Higher Order Energy Functions Multi-label Problems  Ordered Labels ▪ Stereo (depth labels)  Unordered Labels ▪ Object segmentation ( ‘car’, `road’, `person’)
  • 68.  Non-submodular Energy Functions Mixed (Real-Integer) Problems Higher Order Energy Functions Multi-label Problems  Ordered Labels ▪ Stereo (depth labels)  Unordered Labels ▪ Object segmentation ( ‘car’, `road’, `person’)
  • 69. E(x) = ∑ θi (xi) + ∑ θij (xi,xj) i i,j θij(0,1) + θij (1,0) ≤θ ij (0,0) + θij (1,1) for some ij  Minimizing general non-submodular functions is NP- hard.  Commonly used method is to solve a relaxation of the problem[Boros and Hammer, ‘02]
  • 70. unary  pq (0,0)   pq (1,1)   pq (0,1)   pq (1,0) pairwise submodular ~ ~ ~ ~  pq (0,0)   pq (1,1)   pq (0,1)   pq (1,0) pairwise nonsubmodular[Boros and Hammer, ‘02]
  • 71. Double number of variables: xp  xp , xp (xp  1 xp )[Boros and Hammer, ‘02]
  • 72. Double number of variables: xp  xp , xp (xp  1 xp ) Ignore constraint and solve[Boros and Hammer, ‘02]
  • 73. Double number of variables: xp  xp , xp (xp  1 xp ) Local Optimality[Boros and Hammer, ‘02]
  • 74. [Rother, Kolmogorov, Lempitsky, Szummer] [CVPR 2007]
  • 75. QPBO: 0 ? ? ? ? ? p q r s t Probe Node p:0 0 0 0 ? ? 0 1 0 1 0 ? p q r s t p q r s tWhat can we say about variables?•r -> is always 0•s -> is always equal to q•t -> is 0 when q = 1
  • 76.  Probe nodes in an order until energy unchanged Simplified energy preserves global optimality and (sometimes) gives the global minimum Result depends slightly on the order
  • 77. 0 0 0 0 0 0 ? 0 ? 1 ? 0 0 0 1 0 0 0 0 0 ? 0 ? 1 ? ? 0 0 1 0 0 0 0 0 ? ? ? ? 0 0 0 0 y (e.g. from BP) x (partial) y’ = FUSE(x,y)• Property: E(y’) ≤ E(y) [autarky property]
  • 78.  Non-submodular Energy Functions Mixed (Real-Integer) Problems Higher Order Energy Functions Multi-label Problems  Ordered Labels ▪ Stereo (image intensity, depth)  Unordered Labels ▪ Object segmentation ( ‘car’, `road’, `person’)
  • 79. Need for a human like segmentation colour appearance based SegmentationImage Segmentation Result [Kumar et al, 05] [Kohli et al, 06,08]
  • 80. x – binary image segmentation (xi ∊ {0,1})ω – non-local parameter (lives in some large set Ω) E(x,ω) = C(ω) + ∑ θi (ω, xi) + ∑ θij (ω,xi,xj) i i,j unary pairwise constant potentials potentials ≥0 ω θi (ω, xi) Pose Shape Prior Stickman Rough Shape Model Prior [Kohli et al, 06,08]
  • 81. x – binary image segmentation (xi ∊ {0,1}) ω – non-local parameter (lives in some large set Ω) E(x,ω) = C(ω) + ∑ θi (ω, xi) + ∑ θij (ω,xi,xj) i i,j unary pairwise constant potentials potentials ≥0 ω Template Position ScaleOrientation [Kohli et al, 06,08] [Lempitsky et al, 08]
  • 82. x – binary image segmentation (xi ∊ {0,1})ω – non-local parameter (lives in some large set Ω) E(x,ω) = C(ω) + ∑ θi (ω, xi) + ∑ θij (ω,xi,xj) i i,j unary pairwise constant potentials potentials ≥0 {x*,ω*} = arg min E(x,ω) x,ω • Standard “graph cut” energy if ω is fixed [Kohli et al, 06,08] [Lempitsky et al, 08]
  • 83. Local Method: Gradient Descent over ω ω * = arg min min E (x,ω) Submodular ω x ω* [Kohli et al, 06,08]
  • 84. Local Method: Gradient Descent over ω ω * = arg min min E (x,ω) Submodular ω x E (x,ω1) Dynamic Graph Similar Energy Cuts Functions 15- 20 time E (x,ω2) speedup! [Kohli et al, 06,08]
  • 85. [Kohli et al, 06,08]
  • 86. Global Method: Branch and Mincut Produces the global optimal solution Exhaustively explores Ω in the worst case [Lempitsky et al, 08]
  • 87. Ω (space of w) is hierarchically clustered Ω0Ω0
  • 88. Standard best-first branch-and-bound search: lowest lower bound B C A Small fraction of nodes is visited
  • 89. 30,000,000 shapes Exhaustive search: 30,000,000 mincuts Branch-and-Mincut: 12,000 mincuts Speed-up: 2500 times (30 seconds per 312x272 image) [Lempitsky et al, 08]
  • 90. Left ventricle epicardium tracking (work in progress) Original sequence Branch & Bound segmentation No shape prior Shape prior from other sequences 5,200,000 templates ≈20 seconds per frame Speed-up 1150Data courtesy: Dr Harald Becher, Department of Cardiovascular Medicine, University of Oxford [Lempitsky et al, 08]
  • 91.  Non-submodular Energy Functions Mixed (Real-Integer) Problems Higher Order Energy Functions Multi-label Problems  Ordered Labels ▪ Stereo (depth labels)  Unordered Labels ▪ Object segmentation ( ‘car’, `road’, `person’)
  • 92.  Pairwise functions have limited expressive power Inability to incorporate region based likelihoods and priorsField of Experts Model[Roth & Black CVPR 2005 ][Potetz, CVPR 2007]Minimize Curvature[Woodford et al. CVPR 2008 ]Other Examples:[Rother, Kolmogorov, Minka & Blake, CVPR 2006][Komodakis and Paragios, CVPR 2009][Rother, Kohli, Feng, Jia, CVPR 2009][Ishikawa, CVPR 2009]And many others ...
  • 93. n = number of pixels E: {0,1}n → R 0 →fg, 1→bg E(X) = ∑ ci xi + ∑ dij |xi-xj| i i,jImage Unary Cost Segmentation [Boykov and Jolly ‘ 01] [Blake et al. ‘04] [Rother, Kolmogorov and Blake `04]
  • 94. [Kohli et al. ‘07] Patch Dictionary (Tree)h(Xp) = { C1 if xi = 0, i ϵ p Cmax otherwise p Cmax  C1
  • 95. n = number of pixels E: {0,1}n → R 0 →fg, 1→bg E(X) = ∑ ci xi + ∑ dij |xi-xj| + ∑ hp (Xp) i i,j ph(Xp) = { C1 if xi = 0, i ϵ p Cmax otherwise p [Kohli et al. ‘07]
  • 96. n = number of pixels E: {0,1}n → R 0 →fg, 1→bg E(X) = ∑ ci xi + ∑ dij |xi-xj| + ∑ hp (Xp) i i,j pImage Pairwise Segmentation Final Segmentation [Kohli et al. ‘07]
  • 97. ExactHigher Order Transformation Pairwise SubmodularSubmodular Function Functions ?Billionnet and M. Minoux [DAM 1985] st-mincutKolmogorov & Zabih [PAMI 2004] SFreedman & Drineas [CVPR2005]Kohli Kumar Torr [CVPR2007, PAMI 2008]Kohli Ladicky Torr [CVPR 2008, IJCV 2009]Ramalingam Kohli Alahari Torr [CVPR 2008]Zivny et al. [CP 2008] T
  • 98. Higher Order Pairwise Function Submodular Function Identified transformable families of higher order function s.t. 1. Constant or polynomial number of auxiliary variables (a) added 2. All pairwise functions (g) are submodular
  • 99. Example: H (X) = F ( ∑ xi ) concave H (X) 0 ∑ xi [Kohli et al. ‘08]
  • 100.  Simple Example using Auxiliary variables f(x) = { 0 if all xi = 0 C1 otherwise x ϵ L = {0,1}n min f(x) = min C1a + C1 ∑ ā xi x x,a ϵ {0,1} Higher Order Quadratic Submodular Submodular Function Function ∑xi < 1 a=0 (ā=1) f(x) = 0 ∑xi ≥ 1 a=1 (ā=0) f(x) = C1
  • 101. min f(x) = min C1a + C1 ∑ ā xi x x,a ϵ {0,1} Higher Order Quadratic SubmodularSubmodular Function Function C1∑xi C1 1 2 3 ∑xi
  • 102. min f(x) = min C1a + C1 ∑ ā xi x x,a ϵ {0,1} Higher Order Quadratic SubmodularSubmodular Function Function C1∑xi a=0 a=1 Lower envelop of concave C1 functions is concave 1 2 3 ∑xi
  • 103. min f(x) x = min f1 (x)a + f2(x)ā x,a ϵ {0,1} Higher Order Quadratic SubmodularSubmodular Function Function f2(x) f1(x) Lower envelop of concave functions is concave 1 2 3 ∑xi
  • 104. min f(x) x = min f1 (x)a + f2(x)ā x,a ϵ {0,1} Higher Order Quadratic SubmodularSubmodular Function Function f2(x) a=0 a=1 f1(x) Lower envelop of concave functions is concave 1 2 3 ∑xi
  • 105.  Transforming Potentials with 3 variables [Woodford, Fitzgibbon, Reid, Torr, CVPR 2008] Transforming general “sparse” higher order functions [Rother, Kohli, Feng, Jia, CVPR 2009] [Ishikawa, CVPR 2009] [Komodakis and Paragios, CVPR 2009]
  • 106. Training Image Pairwise Energy P(x)Test Image Test Image Minimized using Result (60% Noise) st-mincut or max-product message passing
  • 107. Training ImageHigher Order Structure not Preserved Pairwise Energy P(x)Test Image Test Image Minimized using Result (60% Noise) st-mincut or max-product message passing
  • 108. Minimize: E(X) = P(X) + ∑ hc (Xc) c Where: hc: {0,1}|c| → RHigher Order Function (|c| = 10x10 = 100) Assigns cost to 2100 possible labellings!Exploit function structure to transform it to a Pairwise function p1 p2 p 3
  • 109. Training Learned Image PatternsTest Image Test Image Pairwise Higher-Order (60% Noise) Result Result [Joint work with Carsten Rother ]
  • 110.  Non-submodular Energy Functions Mixed (Real-Integer) Problems Higher Order Energy Functions Multi-label Problems  Ordered Labels ▪ Stereo (depth labels)  Unordered Labels ▪ Object segmentation ( ‘car’, `road’, `person’)
  • 111. Min y E(y) = ∑ fi (yi) + ∑ gij (yi,yj) i i,j y ϵ Labels L = {l1, l2, … , lk} Exact Transformation to QPBF [Roy and Cox ’98] [Ishikawa ’03] [Schlesinger & Flach ’06] [Ramalingam, Alahari, Kohli, and Torr ’08] Move making algorithms
  • 112. So what is the problem? Em (y1,y2, ..., yn) Eb (x1,x2, ..., xm) yi ϵ L = {l1, l2, … , lk} xi ϵ L = {0,1} Multi-label Problem Binary label Problem such that:Let Y and X be the set of feasible solutions, then1. One-One encoding function T:X->Y2. arg min Em(y) = T(arg min Eb(x))
  • 113. • Popular encoding scheme [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]# Nodes = n * k# Pairwise = m * k2
  • 114. • Popular encoding scheme [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06] Ishikawa’s result: E(y) = ∑ θi (yi) + ∑ θij (yi,yj) i i,j y ϵ Labels L = {l1, l2, … , lk} Convex θij (yi,yj) = g(|yi-yj|) Function# Nodes = n * k# Pairwise = m * k2 g(|yi-yj|) |yi-yj|
  • 115. • Popular encoding scheme [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06] Schlesinger & Flach ’06: E(y) = ∑ θi (yi) + ∑ θij (yi,yj) i i,j y ϵ Labels L = {l1, l2, … , lk} θij(li+1,lj) + θij (li,lj+1)  θij (li,lj) + θij (li+1,lj+1)# Nodes = n * k lj +1# Pairwise = m * k2 li +1 lj li
  • 116. Image MAP Scanline Solution algorithm [Roy and Cox, 98]
  • 117. θij (yi,yj) = g(|yi-yj|) Applicability  Cannot handle truncated costs (non-robust) |yi-yj| discontinuity preserving potentials Computational Cost Blake&Zisserman’83,87  Very high computational cost  Problem size = |Variables| x |Labels|  Gray level image denoising (1 Mpixel image) (~2.5 x 108 graph nodes)
  • 118. Other “less known” algorithms Unary Pair-wise Complexity Potentials Potentials Ishikawa Arbitrary Convex and T(nk, mk2)Transformation Symmetric [03] Schlesinger Arbitrary Submodular T(nk, mk2)Transformation [06] Hochbaum Linear Convex and T(n, m) + n log k [01] Symmetric Hochbaum Convex Convex and O(mn log n log nk) [01] Symmetric T(a,b) = complexity of maxflow with a nodes and b edges
  • 119. Min y E(y) = ∑ fi (yi) + ∑ gij (yi,yj) i i,j y ϵ Labels L = {l1, l2, … , lk} Exact Transformation to QPBF Move making algorithms [Boykov , Veksler and Zabih 2001] [Woodford, Fitzgibbon, Reid, Torr, 2008] [Lempitsky, Rother, Blake, 2008] [Veksler, 2008] [Kohli, Ladicky, Torr 2008]
  • 120. Energy Solution Space
  • 121. Current Solution Search Neighbourhood Optimal MoveEnergy Solution Space
  • 122. Current Solution Search Neighbourhood Optimal Move xc Key PropertyEnergy (t) Move Space Solution Space Bigger move • Better solutions space • Finding the optimal move hard
  • 123. Minimizing Pairwise Functions[Boykov Veksler and Zabih, PAMI 2001]• Series of locally optimal moves• Each move reduces energy• Optimal move by minimizing submodular function Current Solution Search Neighbourhood Move Space (t) : 2n n Number of Variables L Number of Labels Space of Solutions (x) : Ln Extend to minimize Higher order Functions Kohli et al. ‘07, ‘08, ‘09
  • 124. x = t x1 + (1-t) x2 New Current Second solution Solution solution Em(t) = E(t x1 + (1-t) x2) Minimize over move variables tFor certain x1 and x2, the move energy is sub-modular QPBF [Boykov , Veksler and Zabih 2001]
  • 125. • Variables labeled α, β can swap their labels [Boykov , Veksler and Zabih 2001]
  • 126. • Variables labeled α, β can swap their labels Tree Ground Swap Sky, House House Sky [Boykov , Veksler and Zabih 2001]
  • 127. • Variables labeled α, β can swap their labels  Move energy is submodular if:  Unary Potentials: Arbitrary  Pairwise potentials: Semi-metric θij (la,lb) ≥ 0 θij (la,lb) = 0 a = b Examples: Potts model, Truncated Convex [Boykov , Veksler and Zabih 2001]
  • 128. • Variables take label a or retain current label [Boykov , Veksler and Zabih 2001] [Boykov, Veksler, Zabih]
  • 129. • Variables take label a or retain current label Tree Ground HouseStatus: Expand Ground Initialize with Tree Sky House Sky [Boykov , Veksler and Zabih 2001] [Boykov, Veksler, Zabih]
  • 130. • Variables take label a or retain current label Semi metric Move energy is submodular if: +  Unary Potentials: Arbitrary Triangle Inequality  Pairwise potentials: Metric θij (la,lb) + θij (lb,lc) ≥ θij (la,lc) Examples: Potts model, Truncated linear Cannot solve truncated quadratic [Boykov , Veksler and Zabih 2001] [Boykov, Veksler, Zabih]
  • 131.  Expansion and Swap can be derived as a primal dual scheme Get solution of the dual problem which is a lower bound on the energy of solution Weak guarantee on the solution E(x) < 2(dmax /dmin) E(x*) θij (li,lj) = g(|li-lj|) dmax dmin |yi-yj| [Komodakis et al 05, 07]
  • 132. x = t x1 + (1-t) x2 New First Second solution solution solution Minimize over move variables tMove Type First Second Guarantee Solution SolutionExpansion Old solution All alpha Metric Fusion Any solution Any solution  Move functions can be non-submodular!!
  • 133. x = t x1 + (1-t) x2 x1, x2 can be continuous Optical Flow Example Solution from Method 2 x2Solution from F FinalMethod 1 x1 x Solution [Woodford, Fitzgibbon, Reid, Torr, 2008] [Lempitsky, Rother, Blake, 2008]
  • 134.  Move variables can be multi-label x = (t ==1) x1 + (t==2) x2 +… +(t==k) xk Optimal move found out by using the Ishikawa Transform Useful for minimizing energies with truncated convex pairwise potentials T θij (yi,yj) = min(|yi-yj|2,T) θij (yi,yj) |yi-yj| [Veksler, 2007]
  • 135. Noisy Image Image RangeExpansion Moves Move [Veksler, 2008]
  • 136. Why?
  • 137. 3,600,000,000 PixelsCreated from about 800 8 MegaPixel Images [Kopf et al. (MSR Redmond) SIGGRAPH 2007 ]
  • 138. [Kopf et al. (MSR Redmond) SIGGRAPH 2007 ]
  • 139.  Processing Videos 1 minute video of 1M pixel resolution  3.6 B pixels 3D reconstruction [500 x 500 x 500 = .125B voxels]
  • 140. Segment First Frame Can we do better? SegmentSecond Frame Kohli[Kohli & Torr, ICCV05 PAMI07] & Torr (ICCV05, PAMI07)
  • 141. ImageFlow Segmentation Kohli[Kohli & Torr, ICCV05 PAMI07] & Torr (ICCV05, PAMI07)
  • 142. minimizeFrame 1 EA SA Reuse Computation differences between EB* Simpler A and B fast minimizeFrame 2 EB SB 3–100000 time speedup! Reparametrization [Kohli & Torr, ICCV05 PAMI07] [Komodakis & Paragios, CVPR07]
  • 143. Original EnergyE(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2Reparametrized Energy E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2New EnergyE(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 7a1ā2 + ā1a2New Reparametrized EnergyE(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 + 5a1ā2 [Kohli & Torr, ICCV05 PAMI07]& [Komodakis & Paragios, CVPR07] Kohli Torr (ICCV05, PAMI07)
  • 144. Approximation Original algorithm Approximate Problem (Large) Solution (Slow)Fast partially optimal algorithm [ Alahari Kohli & Torr CVPR ‘08]
  • 145. Approximation Original algorithm Approximate Problem (Large) Solution (Slow) Fast partiallyoptimal algorithm [Kovtun ‘03]Fast partially [ Kohli et al. ‘09] optimal algorithm Reduced Problem Approximation algorithm Solved Problem Approximate (Global Optima) (Fast) Solution [ Alahari Kohli & Torr CVPR ‘08]
  • 146. 3- 100 Times Speed up sky Tree Reweighted Original Building Message Passing Approximate Problem Airplane (Large) Solution (9.89 sec) Grass Fast partiallyoptimal algorithm [Kovtun ‘03]Fast partially [ Kohli et al. ‘09] optimal Tree Reweighted sky algorithm sky Reduced Problem Message Passing Building Building Approximate Solved Problem Airplane Solution Airplane Total Time (Global Optima) (0.30 sec) Grass Grass [ Alahari Kohli & Torr CVPR ‘08]
  • 147.  Minimization with Complex Higher Order Functions  Connectivity  Counting Constraints Hybrid algorithms  Connections between Messages Passing algorithms BP, TRW, and graph cuts
  • 148. Advances in Markov Random Fields for Computer Vision MIT Press, summer 2010 Topics of this course and much, much more Contributors: usual suspects – lecturers on this course + Boykov,Kolmogorov, Weiss, Freeman, .... one for the office and one for home www.research.microsoft.com/vision/MRFbook
  • 149. NP-Hard MAXCUTCSP Space of Problems
  • 150.  Which functions are exactly solvable? Approximate solutions of NP-hard problems Scalability and Efficiency
  • 151.  Which functions are exactly solvable? Boros Hammer [1965], Kolmogorov Zabih [ECCV 2002, PAMI 2004] , Ishikawa [PAMI 2003], Schlesinger [EMMCVPR 2007], Kohli Kumar Torr [CVPR2007, PAMI 2008] , Ramalingam Kohli Alahari Torr [CVPR 2008] , Kohli Ladicky Torr [CVPR 2008, IJCV 2009] , Zivny Jeavons [CP 2008] Approximate solutions of NP-hard problems Schlesinger [76 ], Kleinberg and Tardos [FOCS 99], Chekuri et al. [01], Boykov et al. [PAMI 01], Wainwright et al. [NIPS01], Werner [PAMI 2007], Komodakis et al. [PAMI, 05 07], Lempitsky et al. [ICCV 2007], Kumar et al. [NIPS 2007], Kumar et al. [ICML 2008], Sontag and Jakkola [NIPS 2007], Kohli et al. [ICML 2008], Kohli et al. [CVPR 2008, IJCV 2009], Rother et al. [2009] Scalability and Efficiency Kohli Torr [ICCV 2005, PAMI 2007], Juan and Boykov [CVPR 2006], Alahari Kohli Torr [CVPR 2008] , Delong and Boykov [CVPR 2008]
  • 152.  Iterated Conditional Modes (ICM) Classical Move making Simulated Annealing algorithms Dynamic Programming (DP) Belief Propagtion (BP) Message passing Tree-Reweighted (TRW), Diffusion Graph Cut (GC) Combinatorial Algorithms Branch & Bound Relaxation methods: Convex Optimization (Linear Programming, ...) …