Pushmeet KohliICCV 2009
Course programme9.30-10.00 Introduction (Andrew Blake)10.00-11.00 Discrete Models in Computer Vision (Carsten Rother)15min...
E(x) =   ∑ fi (xi) + ∑ gij (xi,xj) + ∑ hc(xc)         i            ij               c             Unary         Pairwise  ...
n = Number of Variables                                 Segmentation EnergyNP-Hard               MAXCUTCSP     Pair-wise  ...
st-mincut and Pseudo-boolean           optimizationMore General Minimization Problems      Speed and Efficiency
st-mincut and Pseudo-boolean           optimizationMore General Minimization Problems      Speed and Efficiency
Pseudo-boolean function f{0,1}n  ℝ is submodular if        f(A) + f(B)        f(A˅B) + f(A˄B)            for all A,B   ...
   Discrete Analogues of Concave Functions    [Lovasz, ’83]   Widely applied in Operations Research   Applications in M...
   Polynomial time algorithms     Ellipsoid Algorithm: [Grotschel, Lovasz & Schrijver ‘81]     First strongly polynomia...
Source 2            9     Graph (V, E, C)       2           Vertices V = {v1, v2 ... vn}v1            v2    Edges E = {(v1...
What is a st-cut?     Source 2            9       2v1            v2       15             4      Sink
What is a st-cut?                    An st-cut (S,T) divides the nodes                       between source and sink.     ...
What is a st-cut?                    An st-cut (S,T) divides the nodes                       between source and sink.     ...
Construct a graph such that:1. Any st-cut corresponds to an assignment of x2. The cost of the cut is equal to the energy o...
E(x) =       ∑ θi (xi)   +   ∑ θij (xi,xj)                     i              i,jFor all ij       θij(0,1) + θij (1,0)  θ...
E(a1,a2)                Source (0)           a1           a2                 Sink (1)
E(a1,a2) = 2a1                      Source (0)                  2             a1               a2                       Si...
E(a1,a2) = 2a1 + 5ā1                       Source (0)                   2             a1                a2                ...
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2                          Source (0)                  2          9             a1          ...
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2                            Source (0)                  2            9             ...
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2                            Source (0)                   2           9      ...
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2                            Source (0)                   2           9      ...
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2                            Source (0)                   2           9      ...
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2                            Source (0)                   2           9      ...
Solve the dual maximum flow problem                        Compute the maximum flow between     Source                    ...
Flow = 0                     Augmenting Path Based                          Algorithms       Source 2              9      ...
Flow = 0                      Augmenting Path Based                           Algorithms       Source 2              9    ...
Flow = 0 + 2                       Augmenting Path Based                            Algorithms        Source2-2           ...
Flow = 2                      Augmenting Path Based                           Algorithms       Source 0              9    ...
Flow = 2                      Augmenting Path Based                           Algorithms       Source 0              9    ...
Flow = 2                      Augmenting Path Based                           Algorithms       Source 0              9    ...
Flow = 2 + 4                      Augmenting Path Based                           Algorithms       Source 0              5...
Flow = 6                      Augmenting Path Based                           Algorithms       Source 0              5    ...
Flow = 6                      Augmenting Path Based                           Algorithms       Source 0              5    ...
Flow = 6 + 2                      Augmenting Path Based                           Algorithms       Source 0              3...
Flow = 8                      Augmenting Path Based                           Algorithms       Source 0              4    ...
Flow = 8                      Augmenting Path Based                           Algorithms       Source 0              4    ...
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2                            Source (0)                   2           9      ...
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2                            Source (0)                   2           9      ...
E(a1,a2) = 2 + 3ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2                            Source (0)                   0           9        ...
E(a1,a2) = 2 + 3ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2                            Source (0)                   0           9        ...
E(a1,a2) = 2 + 3ā1+ 5a2 + 4 + 2a1ā2 + ā1a2                            Source (0)                   0           5          ...
E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2                            Source (0)                   0           5              ...
E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2                            Source (0)                   0           5              ...
E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2                                         3ā1+ 5a2 + 2a1ā2                           ...
E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2                                          3ā1+ 5a2 + 2a1ā2                                  ...
E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2                            Source (0)                  0            3                      ...
E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2                    Residual Graph                                               (positive c...
E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2                    Residual Graph                                               (positive c...
Augmenting Path and Push-Relabel          n: #nodes                                          m: #edges                    ...
Augmenting Path and Push-Relabel          n: #nodes                                          m: #edges                    ...
Ford Fulkerson: Choose any augmenting path                            Source            1000              1000            ...
Ford Fulkerson: Choose any augmenting path                            Source            1000              1000            ...
Ford Fulkerson: Choose any augmenting path                            Source            1000              1000            ...
Ford Fulkerson: Choose any augmenting path                            Source            999               1000            ...
n: #nodesFord Fulkerson: Choose any augmenting path             m: #edges                             Source            99...
n: #nodesDinic: Choose shortest augmenting path   m: #edges                        Source         1000              1000  ...
   Specialized algorithms for vision problems       Grid graphs       Low connectivity (m ~ O(n))     Dual search tree...
E(x) =   ∑ ci xi + ∑ dij |xi-xj|   E: {0,1}n → R                                          0 → fg         i           i,j  ...
Graph *g;For all pixels p                                                       Source (0)      /* Add a node to the graph...
Graph *g;For all pixels p                                                                    Source (0)      /* Add a node...
Graph *g;For all pixels p                                                                           Source (0)      /* Add...
Graph *g;For all pixels p                                                                            Source (0)      /* Ad...
Lunch
Advances in Markov Random Fields for Computer Vision MIT Press, summer 2010 Topics of this course and much, much more C...
st-mincut and Pseudo-boolean           optimizationMore General Minimization Problems      Speed and Efficiency
   Non-submodular Energy Functions   Mixed (Real-Integer) Problems   Higher Order Energy Functions   Multi-label Probl...
   Non-submodular Energy Functions   Mixed (Real-Integer) Problems   Higher Order Energy Functions   Multi-label Probl...
E(x) =      ∑ θi (xi)   +   ∑ θij (xi,xj)                               i              i,j       θij(0,1) + θij (1,0)   ≤θ...
unary                           pq (0,0)   pq (1,1)   pq (0,1)   pq (1,0)                           pairwise submod...
Double number of variables:   xp  xp , xp (xp  1 xp )[Boros and Hammer, ‘02]
Double number of variables:    xp  xp , xp (xp  1 xp )                   Ignore constraint and solve[Boros and Hammer, ...
Double number of variables:   xp  xp , xp (xp  1 xp )   Local Optimality[Boros and Hammer, ‘02]
[Rother, Kolmogorov, Lempitsky, Szummer] [CVPR 2007]
QPBO:               0               ?       ?       ?       ?       ?                                        p           q...
   Probe nodes in an order until energy    unchanged   Simplified energy preserves global optimality    and (sometimes) ...
0   0    0      0    0    0                              ?        0                                       ?   1           ...
   Non-submodular Energy Functions   Mixed (Real-Integer) Problems   Higher Order Energy Functions   Multi-label Probl...
Need for a human           like segmentation         colour appearance        based SegmentationImage                     ...
x – binary image segmentation (xi ∊ {0,1})ω – non-local parameter (lives in some large set Ω)   E(x,ω) = C(ω) + ∑ θi (ω, x...
x – binary image segmentation (xi ∊ {0,1})     ω – non-local parameter (lives in some large set Ω)        E(x,ω) = C(ω) + ...
x – binary image segmentation (xi ∊ {0,1})ω – non-local parameter (lives in some large set Ω)   E(x,ω) = C(ω) + ∑ θi (ω, x...
Local Method: Gradient Descent over ω    ω * = arg min min E (x,ω)   Submodular               ω    x                      ...
Local Method: Gradient Descent over ω    ω * = arg min min E (x,ω)    Submodular               ω    x  E (x,ω1)           ...
[Kohli et al, 06,08]
Global Method: Branch and Mincut    Produces the global optimal solution  Exhaustively explores Ω in the worst case       ...
Ω (space of w) is hierarchically clustered                                Ω0Ω0
Standard best-first branch-and-bound search:                                               lowest lower bound             ...
30,000,000 shapes                    Exhaustive search: 30,000,000 mincuts                    Branch-and-Mincut: 12,000 mi...
Left ventricle epicardium tracking (work in progress)  Original sequence          Branch & Bound segmentation             ...
   Non-submodular Energy Functions   Mixed (Real-Integer) Problems   Higher Order Energy Functions   Multi-label Probl...
   Pairwise functions have limited expressive power   Inability to incorporate region based likelihoods and priorsField ...
n = number of pixels                                                                 E: {0,1}n → R                        ...
[Kohli et al. ‘07] Patch Dictionary      (Tree)h(Xp) =   {   C1 if xi = 0, i ϵ p              Cmax otherwise              ...
n = number of pixels                                           E: {0,1}n → R                                           0 →...
n = number of pixels                                                       E: {0,1}n → R                                  ...
ExactHigher Order           Transformation       Pairwise SubmodularSubmodular                                            ...
Higher Order                                                               Pairwise  Function                             ...
Example:       H (X) = F ( ∑ xi )                         concave           H (X)                   0           ∑ xi      ...
   Simple Example using Auxiliary variables                f(x) =     {   0 if all xi = 0                               C...
min f(x)             =               min C1a + C1 ∑ ā xi      x                                 x,a ϵ {0,1}   Higher Order...
min f(x)             =               min C1a + C1 ∑ ā xi      x                                 x,a ϵ {0,1}   Higher Order...
min f(x)      x                          =               min f1 (x)a + f2(x)ā                                        x,a ϵ...
min f(x)      x                          =               min f1 (x)a + f2(x)ā                                        x,a ϵ...
   Transforming Potentials with 3 variables    [Woodford, Fitzgibbon, Reid, Torr, CVPR 2008]   Transforming general “spa...
Training                     Image                                  Pairwise                                   Energy     ...
Training                           ImageHigher Order Structure     not Preserved                                        Pa...
Minimize: E(X) = P(X) +                 ∑   hc (Xc)                                         c Where: hc: {0,1}|c| → RHighe...
Training                                   Learned Image                                     PatternsTest Image   Test Ima...
   Non-submodular Energy Functions   Mixed (Real-Integer) Problems   Higher Order Energy Functions   Multi-label Probl...
Min           y                   E(y) =    ∑ fi (yi) + ∑ gij (yi,yj)                             i             i,j       ...
So what is the problem?  Em (y1,y2, ..., yn)                       Eb (x1,x2, ..., xm)   yi ϵ L = {l1, l2, … , lk}        ...
• Popular encoding scheme [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]# Nodes = n * k# Pairwise = m * k2
• Popular encoding scheme [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]                                Ishikawa...
• Popular encoding scheme [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]                                 Schlesi...
Image    MAP        Scanline        Solution   algorithm                     [Roy and Cox, 98]
θij (yi,yj) = g(|yi-yj|)   Applicability     Cannot handle truncated costs     (non-robust)                             ...
Other “less known” algorithms                   Unary        Pair-wise       Complexity                 Potentials     Pot...
Min           y                   E(y) =   ∑ fi (yi) + ∑ gij (yi,yj)                             i             i,j        ...
Energy         Solution Space
Current Solution                          Search                          Neighbourhood                          Optimal M...
Current Solution                                                         Search                                           ...
Minimizing Pairwise Functions[Boykov Veksler and Zabih, PAMI 2001]•   Series of locally optimal moves•   Each move reduces...
x = t x1 + (1-t) x2      New             Current               Second    solution          Solution              solution ...
• Variables labeled α, β can swap their labels                               [Boykov , Veksler and Zabih 2001]
• Variables labeled α, β can swap their labels                                              Tree                          ...
• Variables labeled α, β can swap their labels    Move energy is submodular if:      Unary Potentials: Arbitrary      P...
• Variables take label a or retain current label                                 [Boykov , Veksler and Zabih 2001]        ...
• Variables take label a or retain current label                                                  Tree                    ...
• Variables take label a or retain current label                                                          Semi metric   M...
   Expansion and Swap can be derived as a primal dual scheme   Get solution of the dual problem which is a lower bound o...
x = t x1 + (1-t) x2      New              First                  Second    solution         solution                 solut...
x = t x1 + (1-t) x2    x1, x2 can be continuous                             Optical Flow                                  ...
    Move variables can be multi-label                  x = (t ==1) x1 + (t==2) x2 +… +(t==k) xk    Optimal move found ou...
Noisy Image            Image            RangeExpansion            Moves  Move             [Veksler, 2008]
Why?
3,600,000,000 PixelsCreated from about 800 8 MegaPixel Images          [Kopf et al. (MSR Redmond) SIGGRAPH 2007 ]
[Kopf et al. (MSR Redmond) SIGGRAPH 2007 ]
   Processing Videos    1 minute video of 1M pixel resolution     3.6 B pixels   3D reconstruction    [500 x 500 x 500 ...
Segment First Frame   Can we do                better?               SegmentSecond Frame                         Kohli[Koh...
ImageFlow                              Segmentation               Kohli[Kohli & Torr, ICCV05 PAMI07]                     &...
minimizeFrame 1   EA                                     SA                                                        Reuse  ...
Original EnergyE(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2Reparametrized Energy          E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2...
Approximation            Original                         algorithm           Approximate            Problem            (L...
Approximation              Original                                           algorithm             Approximate           ...
3- 100                                                          Times Speed up                                            ...
   Minimization with Complex Higher Order    Functions     Connectivity     Counting Constraints   Hybrid algorithms  ...
Advances in Markov Random Fields for Computer Vision MIT Press, summer 2010 Topics of this course and much, much more C...
NP-Hard           MAXCUTCSP      Space of Problems
   Which functions are exactly solvable?   Approximate solutions of NP-hard problems   Scalability and Efficiency
   Which functions are exactly solvable?    Boros Hammer [1965], Kolmogorov Zabih [ECCV 2002, PAMI 2004] , Ishikawa [PAMI...
   Iterated Conditional Modes (ICM)                                        Classical Move making   Simulated Annealing  ...
Upcoming SlideShare
Loading in …5
×

ICCV2009: MAP Inference in Discrete Models: Part 4

582 views

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
582
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
21
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

ICCV2009: MAP Inference in Discrete Models: Part 4

  1. 1. Pushmeet KohliICCV 2009
  2. 2. Course programme9.30-10.00 Introduction (Andrew Blake)10.00-11.00 Discrete Models in Computer Vision (Carsten Rother)15min Coffee break11.15-12.30 Message Passing: DP, TRW, LP relaxation (Pawan Kumar)12.30-13.00 Quadratic pseudo-boolean optimization (Pushmeet Kohli)1 hour Lunch break14:00-15.00 Transformation and move-making methods (Pushmeet Kohli)15:00-15.30 Speed and Efficiency (Pushmeet Kohli)15min Coffee break15:45-16.15 Comparison of Methods (Carsten Rother)16:30-17.30 Recent Advances: Dual-decomposition, higher-order, etc. (Carsten Rother + Pawan Kumar) All online material will be online (after conference): http://research.microsoft.com/en-us/um/cambridge/projects/tutorial/
  3. 3. E(x) = ∑ fi (xi) + ∑ gij (xi,xj) + ∑ hc(xc) i ij c Unary Pairwise Higher Order ∑ ci xi + ∑ dij |xi-xj| E: {0,1}n → R i i,j n = number of pixels Image Segmentation
  4. 4. n = Number of Variables Segmentation EnergyNP-Hard MAXCUTCSP Pair-wise O(n3) Submodular Functions Tree Structured O(n6) Space of Problems
  5. 5. st-mincut and Pseudo-boolean optimizationMore General Minimization Problems Speed and Efficiency
  6. 6. st-mincut and Pseudo-boolean optimizationMore General Minimization Problems Speed and Efficiency
  7. 7. Pseudo-boolean function f{0,1}n  ℝ is submodular if f(A) + f(B)  f(A˅B) + f(A˄B) for all A,B ϵ {0,1}n (OR) (AND)Example: n = 2, A = [1,0] , B = [0,1] f([1,0]) + f([0,1])  f([1,1]) + f([0,0]) Property : Sum of submodular functions is submodular Binary Image Segmentation Energy is submodular E(x) = ∑ ci xi + ∑ dij |xi-xj| i i,j
  8. 8.  Discrete Analogues of Concave Functions [Lovasz, ’83] Widely applied in Operations Research Applications in Machine Learning  MAP Inference in Markov Random Fields  Clustering [Narasimhan , Jojic, & Bilmes, NIPS 2005]  Structure Learning [Narasimhan & Bilmes, NIPS 2006] Maximizing the spread of influence through a social network [Kempe, Kleinberg & Tardos, KDD 2003]
  9. 9.  Polynomial time algorithms  Ellipsoid Algorithm: [Grotschel, Lovasz & Schrijver ‘81]  First strongly polynomial algorithm: [Iwata et al. ’00] [A. Schrijver ’00]  Current Best: O(n5 Q + n6) [Q is function evaluation time] [Orlin ‘07] Symmetric functions: E(x) = E(1-x)  Can be minimized in O(n3) Minimizing Pairwise submodular functions  Can be transformed to st-mincut/max-flow [Hammer , 1965]  Very low empirical running time ~ O(n) E(X) = ∑ fi (xi) + ∑ gij (xi,xj) i ij
  10. 10. Source 2 9 Graph (V, E, C) 2 Vertices V = {v1, v2 ... vn}v1 v2 Edges E = {(v1, v2) ....} 1 Costs C = {c(1, 2) ....}5 4 Sink
  11. 11. What is a st-cut? Source 2 9 2v1 v2 15 4 Sink
  12. 12. What is a st-cut? An st-cut (S,T) divides the nodes between source and sink. Source 2 9 What is the cost of a st-cut? 2 Sum of cost of all edges going v1 v2 from S to T 1 5 4 Sink5 + 1 + 9 = 15
  13. 13. What is a st-cut? An st-cut (S,T) divides the nodes between source and sink. Source 2 9 What is the cost of a st-cut? 2 Sum of cost of all edges going v1 v2 from S to T 1 5 4 What is the st-mincut? Sink st-cut with the minimum cost2 + 2 + 4 = 8
  14. 14. Construct a graph such that:1. Any st-cut corresponds to an assignment of x2. The cost of the cut is equal to the energy of x : E(x) S st-mincutE(x) T Solution [Hammer, 1965] [Kolmogorov and Zabih, 2002]
  15. 15. E(x) = ∑ θi (xi) + ∑ θij (xi,xj) i i,jFor all ij θij(0,1) + θij (1,0)  θij (0,0) + θij (1,1) Equivalent (transformable) E(x) = ∑ ci xi + ∑ cij xi(1-xj) cij≥0 i i,j
  16. 16. E(a1,a2) Source (0) a1 a2 Sink (1)
  17. 17. E(a1,a2) = 2a1 Source (0) 2 a1 a2 Sink (1)
  18. 18. E(a1,a2) = 2a1 + 5ā1 Source (0) 2 a1 a2 5 Sink (1)
  19. 19. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 Source (0) 2 9 a1 a2 5 4 Sink (1)
  20. 20. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 Source (0) 2 9 a1 a2 2 5 4 Sink (1)
  21. 21. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 2 9 1 a1 a2 2 5 4 Sink (1)
  22. 22. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 2 9 1 a1 a2 2 5 4 Sink (1)
  23. 23. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 2 9 Cost of cut = 11 1 a1 a2 a1 = 1 a2 = 1 2 E (1,1) = 11 5 4 Sink (1)
  24. 24. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 2 9 st-mincut cost = 8 1 a1 a2 a1 = 1 a2 = 0 2 E (1,0) = 8 5 4 Sink (1)
  25. 25. Solve the dual maximum flow problem Compute the maximum flow between Source Source and Sink s.t. 2 9 Edges: Flow < Capacity 1 Nodes: Flow in = Flow outv1 v2 25 4 Min-cutMax-flow Theorem Sink In every network, the maximum flow equals the cost of the st-mincut Assuming non-negative capacity
  26. 26. Flow = 0 Augmenting Path Based Algorithms Source 2 9 1v1 v2 25 4 Sink
  27. 27. Flow = 0 Augmenting Path Based Algorithms Source 2 9 1. Find path from source to sink with positive capacity 1v1 v2 25 4 Sink
  28. 28. Flow = 0 + 2 Augmenting Path Based Algorithms Source2-2 9 1. Find path from source to sink with positive capacity 1 v1 v2 2. Push maximum possible flow 2 through this path 45-2 Sink
  29. 29. Flow = 2 Augmenting Path Based Algorithms Source 0 9 1. Find path from source to sink with positive capacity 1v1 v2 2. Push maximum possible flow 2 through this path3 4 Sink
  30. 30. Flow = 2 Augmenting Path Based Algorithms Source 0 9 1. Find path from source to sink with positive capacity 1v1 v2 2. Push maximum possible flow 2 through this path3 4 3. Repeat until no path can be Sink found
  31. 31. Flow = 2 Augmenting Path Based Algorithms Source 0 9 1. Find path from source to sink with positive capacity 1v1 v2 2. Push maximum possible flow 2 through this path3 4 3. Repeat until no path can be Sink found
  32. 32. Flow = 2 + 4 Augmenting Path Based Algorithms Source 0 5 1. Find path from source to sink with positive capacity 1v1 v2 2. Push maximum possible flow 2 through this path3 0 3. Repeat until no path can be Sink found
  33. 33. Flow = 6 Augmenting Path Based Algorithms Source 0 5 1. Find path from source to sink with positive capacity 1v1 v2 2. Push maximum possible flow 2 through this path3 0 3. Repeat until no path can be Sink found
  34. 34. Flow = 6 Augmenting Path Based Algorithms Source 0 5 1. Find path from source to sink with positive capacity 1v1 v2 2. Push maximum possible flow 2 through this path3 0 3. Repeat until no path can be Sink found
  35. 35. Flow = 6 + 2 Augmenting Path Based Algorithms Source 0 3 1. Find path from source to sink with positive capacity 1+2v1 v2 2. Push maximum possible flow 2-2 through this path1 0 3. Repeat until no path can be Sink found
  36. 36. Flow = 8 Augmenting Path Based Algorithms Source 0 4 1. Find path from source to sink with positive capacity 3v1 v2 2. Push maximum possible flow 0 through this path2 0 3. Repeat until no path can be Sink found
  37. 37. Flow = 8 Augmenting Path Based Algorithms Source 0 4 1. Find path from source to sink with positive capacity 3v1 v2 2. Push maximum possible flow 0 through this path2 0 3. Repeat until no path can be Sink found
  38. 38. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 2 9 1 a1 a2 2 5 4 Sink (1)
  39. 39. E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 2 9 2a1 + 5ā1 1 a1 a2 = 2(a1+ā1) + 3ā1 2 = 2 + 3ā1 5 4 Sink (1)
  40. 40. E(a1,a2) = 2 + 3ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 0 9 2a1 + 5ā1 1 a1 a2 = 2(a1+ā1) + 3ā1 2 = 2 + 3ā1 3 4 Sink (1)
  41. 41. E(a1,a2) = 2 + 3ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 Source (0) 0 9 9a2 + 4ā2 1 a1 a2 = 4(a2+ā2) + 5ā2 2 = 4 + 5ā2 3 4 Sink (1)
  42. 42. E(a1,a2) = 2 + 3ā1+ 5a2 + 4 + 2a1ā2 + ā1a2 Source (0) 0 5 9a2 + 4ā2 1 a1 a2 = 4(a2+ā2) + 5ā2 2 = 4 + 5ā2 3 0 Sink (1)
  43. 43. E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2 Source (0) 0 5 1 a1 a2 2 3 0 Sink (1)
  44. 44. E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2 Source (0) 0 5 1 a1 a2 2 3 0 Sink (1)
  45. 45. E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2 3ā1+ 5a2 + 2a1ā2 = 2(ā1+a2+a1ā2) +ā1+3a2 Source (0) = 2(1+ā1a2) +ā1+3a2 0 5 F1 = ā1+a2+a1ā2 1 a1 a2 F2 = 1+ā1a2 2 a1 a2 F1 F2 3 0 0 0 1 1 0 1 2 2 Sink (1) 1 0 1 1 1 1 1 1
  46. 46. E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 3ā1+ 5a2 + 2a1ā2 = 2(ā1+a2+a1ā2) +ā1+3a2 Source (0) = 2(1+ā1a2) +ā1+3a2 0 3 F1 = ā1+a2+a1ā2 3 a1 a2 F2 = 1+ā1a2 0 a1 a2 F1 F2 1 0 0 0 1 1 0 1 2 2 Sink (1) 1 0 1 1 1 1 1 1
  47. 47. E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 Source (0) 0 3 3 No more a1 a2 augmenting paths 0 possible 1 0 Sink (1)
  48. 48. E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 Residual Graph (positive coefficients) Source (0) Total Flow bound on the 0 3 optimal solution 3 a1 a2 0 1 0 Sink (1)Tight Bound --> Inference of the optimal solution becomes trivial
  49. 49. E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 Residual Graph (positive coefficients) Source (0) Total Flow bound on the 0 3 st-mincut cost = 8 optimal solution 3 a1 a2 a1 = 1 a2 = 0 0 E (1,0) = 8 1 0 Sink (1)Tight Bound --> Inference of the optimal solution becomes trivial
  50. 50. Augmenting Path and Push-Relabel n: #nodes m: #edges U: maximum edge weight Algorithms assume non- negative edge weights [Slide credit: Andrew Goldberg]
  51. 51. Augmenting Path and Push-Relabel n: #nodes m: #edges U: maximum edge weight Algorithms assume non- negative edge weights [Slide credit: Andrew Goldberg]
  52. 52. Ford Fulkerson: Choose any augmenting path Source 1000 1000 1 a1 a2 0 1000 1000 Sink
  53. 53. Ford Fulkerson: Choose any augmenting path Source 1000 1000 1 a1 a2 0 Good 1000 1000 Augmenting Paths Sink
  54. 54. Ford Fulkerson: Choose any augmenting path Source 1000 1000 1 a1 a2 0 Bad 1000 1000 Augmenting Path Sink
  55. 55. Ford Fulkerson: Choose any augmenting path Source 999 1000 0 a1 a2 1 1000 999 Sink
  56. 56. n: #nodesFord Fulkerson: Choose any augmenting path m: #edges Source 999 1000 0 a1 a2 1 1000 999 Sink We will have to perform 2000 augmentations! Worst case complexity: O (m x Total_Flow) (Pseudo-polynomial bound: depends on flow)
  57. 57. n: #nodesDinic: Choose shortest augmenting path m: #edges Source 1000 1000 1 a1 a2 0 1000 1000 Sink Worst case Complexity: O (m n2)
  58. 58.  Specialized algorithms for vision problems  Grid graphs  Low connectivity (m ~ O(n))  Dual search tree augmenting path algorithm [Boykov and Kolmogorov PAMI 2004] • Finds approximate shortest augmenting paths efficiently • High worst-case time complexity • Empirically outperforms other algorithms on vision problems Efficient code available on the webhttp://www.adastral.ucl.ac.uk/~vladkolm/software.html
  59. 59. E(x) = ∑ ci xi + ∑ dij |xi-xj| E: {0,1}n → R 0 → fg i i,j 1 → bg n = number of pixels x* = arg min E(x) x How to minimize E(x)?
  60. 60. Graph *g;For all pixels p Source (0) /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p));endfor all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q));endg->compute_maxflow(); Sink (1)label_p = g->is_connected_to_source(nodeID(p));// is the label of pixel p (0 or 1)
  61. 61. Graph *g;For all pixels p Source (0) /* Add a node to the graph */ nodeID(p) = g->add_node(); bgCost(a1) bgCost(a2) /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); a1 a2endfor all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); fgCost(a1) fgCost(a2)endg->compute_maxflow(); Sink (1)label_p = g->is_connected_to_source(nodeID(p));// is the label of pixel p (0 or 1)
  62. 62. Graph *g;For all pixels p Source (0) /* Add a node to the graph */ nodeID(p) = g->add_node(); bgCost(a1) bgCost(a2) /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); cost(p,q) a1 a2endfor all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); fgCost(a1) fgCost(a2)endg->compute_maxflow(); Sink (1)label_p = g->is_connected_to_source(nodeID(p));// is the label of pixel p (0 or 1)
  63. 63. Graph *g;For all pixels p Source (0) /* Add a node to the graph */ nodeID(p) = g->add_node(); bgCost(a1) bgCost(a2) /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); cost(p,q) a1 a2endfor all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost(p,q)); fgCost(a1) fgCost(a2)endg->compute_maxflow(); Sink (1)label_p = g->is_connected_to_source(nodeID(p));// is the label of pixel p (0 or 1) a1 = bg a2 = fg
  64. 64. Lunch
  65. 65. Advances in Markov Random Fields for Computer Vision MIT Press, summer 2010 Topics of this course and much, much more Contributors: usual suspects – lecturers on this course + Boykov,Kolmogorov, Weiss, Freeman, .... one for the office and one for home www.research.microsoft.com/vision/MRFbook
  66. 66. st-mincut and Pseudo-boolean optimizationMore General Minimization Problems Speed and Efficiency
  67. 67.  Non-submodular Energy Functions Mixed (Real-Integer) Problems Higher Order Energy Functions Multi-label Problems  Ordered Labels ▪ Stereo (depth labels)  Unordered Labels ▪ Object segmentation ( ‘car’, `road’, `person’)
  68. 68.  Non-submodular Energy Functions Mixed (Real-Integer) Problems Higher Order Energy Functions Multi-label Problems  Ordered Labels ▪ Stereo (depth labels)  Unordered Labels ▪ Object segmentation ( ‘car’, `road’, `person’)
  69. 69. E(x) = ∑ θi (xi) + ∑ θij (xi,xj) i i,j θij(0,1) + θij (1,0) ≤θ ij (0,0) + θij (1,1) for some ij  Minimizing general non-submodular functions is NP- hard.  Commonly used method is to solve a relaxation of the problem[Boros and Hammer, ‘02]
  70. 70. unary  pq (0,0)   pq (1,1)   pq (0,1)   pq (1,0) pairwise submodular ~ ~ ~ ~  pq (0,0)   pq (1,1)   pq (0,1)   pq (1,0) pairwise nonsubmodular[Boros and Hammer, ‘02]
  71. 71. Double number of variables: xp  xp , xp (xp  1 xp )[Boros and Hammer, ‘02]
  72. 72. Double number of variables: xp  xp , xp (xp  1 xp ) Ignore constraint and solve[Boros and Hammer, ‘02]
  73. 73. Double number of variables: xp  xp , xp (xp  1 xp ) Local Optimality[Boros and Hammer, ‘02]
  74. 74. [Rother, Kolmogorov, Lempitsky, Szummer] [CVPR 2007]
  75. 75. QPBO: 0 ? ? ? ? ? p q r s t Probe Node p:0 0 0 0 ? ? 0 1 0 1 0 ? p q r s t p q r s tWhat can we say about variables?•r -> is always 0•s -> is always equal to q•t -> is 0 when q = 1
  76. 76.  Probe nodes in an order until energy unchanged Simplified energy preserves global optimality and (sometimes) gives the global minimum Result depends slightly on the order
  77. 77. 0 0 0 0 0 0 ? 0 ? 1 ? 0 0 0 1 0 0 0 0 0 ? 0 ? 1 ? ? 0 0 1 0 0 0 0 0 ? ? ? ? 0 0 0 0 y (e.g. from BP) x (partial) y’ = FUSE(x,y)• Property: E(y’) ≤ E(y) [autarky property]
  78. 78.  Non-submodular Energy Functions Mixed (Real-Integer) Problems Higher Order Energy Functions Multi-label Problems  Ordered Labels ▪ Stereo (image intensity, depth)  Unordered Labels ▪ Object segmentation ( ‘car’, `road’, `person’)
  79. 79. Need for a human like segmentation colour appearance based SegmentationImage Segmentation Result [Kumar et al, 05] [Kohli et al, 06,08]
  80. 80. x – binary image segmentation (xi ∊ {0,1})ω – non-local parameter (lives in some large set Ω) E(x,ω) = C(ω) + ∑ θi (ω, xi) + ∑ θij (ω,xi,xj) i i,j unary pairwise constant potentials potentials ≥0 ω θi (ω, xi) Pose Shape Prior Stickman Rough Shape Model Prior [Kohli et al, 06,08]
  81. 81. x – binary image segmentation (xi ∊ {0,1}) ω – non-local parameter (lives in some large set Ω) E(x,ω) = C(ω) + ∑ θi (ω, xi) + ∑ θij (ω,xi,xj) i i,j unary pairwise constant potentials potentials ≥0 ω Template Position ScaleOrientation [Kohli et al, 06,08] [Lempitsky et al, 08]
  82. 82. x – binary image segmentation (xi ∊ {0,1})ω – non-local parameter (lives in some large set Ω) E(x,ω) = C(ω) + ∑ θi (ω, xi) + ∑ θij (ω,xi,xj) i i,j unary pairwise constant potentials potentials ≥0 {x*,ω*} = arg min E(x,ω) x,ω • Standard “graph cut” energy if ω is fixed [Kohli et al, 06,08] [Lempitsky et al, 08]
  83. 83. Local Method: Gradient Descent over ω ω * = arg min min E (x,ω) Submodular ω x ω* [Kohli et al, 06,08]
  84. 84. Local Method: Gradient Descent over ω ω * = arg min min E (x,ω) Submodular ω x E (x,ω1) Dynamic Graph Similar Energy Cuts Functions 15- 20 time E (x,ω2) speedup! [Kohli et al, 06,08]
  85. 85. [Kohli et al, 06,08]
  86. 86. Global Method: Branch and Mincut Produces the global optimal solution Exhaustively explores Ω in the worst case [Lempitsky et al, 08]
  87. 87. Ω (space of w) is hierarchically clustered Ω0Ω0
  88. 88. Standard best-first branch-and-bound search: lowest lower bound B C A Small fraction of nodes is visited
  89. 89. 30,000,000 shapes Exhaustive search: 30,000,000 mincuts Branch-and-Mincut: 12,000 mincuts Speed-up: 2500 times (30 seconds per 312x272 image) [Lempitsky et al, 08]
  90. 90. Left ventricle epicardium tracking (work in progress) Original sequence Branch & Bound segmentation No shape prior Shape prior from other sequences 5,200,000 templates ≈20 seconds per frame Speed-up 1150Data courtesy: Dr Harald Becher, Department of Cardiovascular Medicine, University of Oxford [Lempitsky et al, 08]
  91. 91.  Non-submodular Energy Functions Mixed (Real-Integer) Problems Higher Order Energy Functions Multi-label Problems  Ordered Labels ▪ Stereo (depth labels)  Unordered Labels ▪ Object segmentation ( ‘car’, `road’, `person’)
  92. 92.  Pairwise functions have limited expressive power Inability to incorporate region based likelihoods and priorsField of Experts Model[Roth & Black CVPR 2005 ][Potetz, CVPR 2007]Minimize Curvature[Woodford et al. CVPR 2008 ]Other Examples:[Rother, Kolmogorov, Minka & Blake, CVPR 2006][Komodakis and Paragios, CVPR 2009][Rother, Kohli, Feng, Jia, CVPR 2009][Ishikawa, CVPR 2009]And many others ...
  93. 93. n = number of pixels E: {0,1}n → R 0 →fg, 1→bg E(X) = ∑ ci xi + ∑ dij |xi-xj| i i,jImage Unary Cost Segmentation [Boykov and Jolly ‘ 01] [Blake et al. ‘04] [Rother, Kolmogorov and Blake `04]
  94. 94. [Kohli et al. ‘07] Patch Dictionary (Tree)h(Xp) = { C1 if xi = 0, i ϵ p Cmax otherwise p Cmax  C1
  95. 95. n = number of pixels E: {0,1}n → R 0 →fg, 1→bg E(X) = ∑ ci xi + ∑ dij |xi-xj| + ∑ hp (Xp) i i,j ph(Xp) = { C1 if xi = 0, i ϵ p Cmax otherwise p [Kohli et al. ‘07]
  96. 96. n = number of pixels E: {0,1}n → R 0 →fg, 1→bg E(X) = ∑ ci xi + ∑ dij |xi-xj| + ∑ hp (Xp) i i,j pImage Pairwise Segmentation Final Segmentation [Kohli et al. ‘07]
  97. 97. ExactHigher Order Transformation Pairwise SubmodularSubmodular Function Functions ?Billionnet and M. Minoux [DAM 1985] st-mincutKolmogorov & Zabih [PAMI 2004] SFreedman & Drineas [CVPR2005]Kohli Kumar Torr [CVPR2007, PAMI 2008]Kohli Ladicky Torr [CVPR 2008, IJCV 2009]Ramalingam Kohli Alahari Torr [CVPR 2008]Zivny et al. [CP 2008] T
  98. 98. Higher Order Pairwise Function Submodular Function Identified transformable families of higher order function s.t. 1. Constant or polynomial number of auxiliary variables (a) added 2. All pairwise functions (g) are submodular
  99. 99. Example: H (X) = F ( ∑ xi ) concave H (X) 0 ∑ xi [Kohli et al. ‘08]
  100. 100.  Simple Example using Auxiliary variables f(x) = { 0 if all xi = 0 C1 otherwise x ϵ L = {0,1}n min f(x) = min C1a + C1 ∑ ā xi x x,a ϵ {0,1} Higher Order Quadratic Submodular Submodular Function Function ∑xi < 1 a=0 (ā=1) f(x) = 0 ∑xi ≥ 1 a=1 (ā=0) f(x) = C1
  101. 101. min f(x) = min C1a + C1 ∑ ā xi x x,a ϵ {0,1} Higher Order Quadratic SubmodularSubmodular Function Function C1∑xi C1 1 2 3 ∑xi
  102. 102. min f(x) = min C1a + C1 ∑ ā xi x x,a ϵ {0,1} Higher Order Quadratic SubmodularSubmodular Function Function C1∑xi a=0 a=1 Lower envelop of concave C1 functions is concave 1 2 3 ∑xi
  103. 103. min f(x) x = min f1 (x)a + f2(x)ā x,a ϵ {0,1} Higher Order Quadratic SubmodularSubmodular Function Function f2(x) f1(x) Lower envelop of concave functions is concave 1 2 3 ∑xi
  104. 104. min f(x) x = min f1 (x)a + f2(x)ā x,a ϵ {0,1} Higher Order Quadratic SubmodularSubmodular Function Function f2(x) a=0 a=1 f1(x) Lower envelop of concave functions is concave 1 2 3 ∑xi
  105. 105.  Transforming Potentials with 3 variables [Woodford, Fitzgibbon, Reid, Torr, CVPR 2008] Transforming general “sparse” higher order functions [Rother, Kohli, Feng, Jia, CVPR 2009] [Ishikawa, CVPR 2009] [Komodakis and Paragios, CVPR 2009]
  106. 106. Training Image Pairwise Energy P(x)Test Image Test Image Minimized using Result (60% Noise) st-mincut or max-product message passing
  107. 107. Training ImageHigher Order Structure not Preserved Pairwise Energy P(x)Test Image Test Image Minimized using Result (60% Noise) st-mincut or max-product message passing
  108. 108. Minimize: E(X) = P(X) + ∑ hc (Xc) c Where: hc: {0,1}|c| → RHigher Order Function (|c| = 10x10 = 100) Assigns cost to 2100 possible labellings!Exploit function structure to transform it to a Pairwise function p1 p2 p 3
  109. 109. Training Learned Image PatternsTest Image Test Image Pairwise Higher-Order (60% Noise) Result Result [Joint work with Carsten Rother ]
  110. 110.  Non-submodular Energy Functions Mixed (Real-Integer) Problems Higher Order Energy Functions Multi-label Problems  Ordered Labels ▪ Stereo (depth labels)  Unordered Labels ▪ Object segmentation ( ‘car’, `road’, `person’)
  111. 111. Min y E(y) = ∑ fi (yi) + ∑ gij (yi,yj) i i,j y ϵ Labels L = {l1, l2, … , lk} Exact Transformation to QPBF [Roy and Cox ’98] [Ishikawa ’03] [Schlesinger & Flach ’06] [Ramalingam, Alahari, Kohli, and Torr ’08] Move making algorithms
  112. 112. So what is the problem? Em (y1,y2, ..., yn) Eb (x1,x2, ..., xm) yi ϵ L = {l1, l2, … , lk} xi ϵ L = {0,1} Multi-label Problem Binary label Problem such that:Let Y and X be the set of feasible solutions, then1. One-One encoding function T:X->Y2. arg min Em(y) = T(arg min Eb(x))
  113. 113. • Popular encoding scheme [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]# Nodes = n * k# Pairwise = m * k2
  114. 114. • Popular encoding scheme [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06] Ishikawa’s result: E(y) = ∑ θi (yi) + ∑ θij (yi,yj) i i,j y ϵ Labels L = {l1, l2, … , lk} Convex θij (yi,yj) = g(|yi-yj|) Function# Nodes = n * k# Pairwise = m * k2 g(|yi-yj|) |yi-yj|
  115. 115. • Popular encoding scheme [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06] Schlesinger & Flach ’06: E(y) = ∑ θi (yi) + ∑ θij (yi,yj) i i,j y ϵ Labels L = {l1, l2, … , lk} θij(li+1,lj) + θij (li,lj+1)  θij (li,lj) + θij (li+1,lj+1)# Nodes = n * k lj +1# Pairwise = m * k2 li +1 lj li
  116. 116. Image MAP Scanline Solution algorithm [Roy and Cox, 98]
  117. 117. θij (yi,yj) = g(|yi-yj|) Applicability  Cannot handle truncated costs (non-robust) |yi-yj| discontinuity preserving potentials Computational Cost Blake&Zisserman’83,87  Very high computational cost  Problem size = |Variables| x |Labels|  Gray level image denoising (1 Mpixel image) (~2.5 x 108 graph nodes)
  118. 118. Other “less known” algorithms Unary Pair-wise Complexity Potentials Potentials Ishikawa Arbitrary Convex and T(nk, mk2)Transformation Symmetric [03] Schlesinger Arbitrary Submodular T(nk, mk2)Transformation [06] Hochbaum Linear Convex and T(n, m) + n log k [01] Symmetric Hochbaum Convex Convex and O(mn log n log nk) [01] Symmetric T(a,b) = complexity of maxflow with a nodes and b edges
  119. 119. Min y E(y) = ∑ fi (yi) + ∑ gij (yi,yj) i i,j y ϵ Labels L = {l1, l2, … , lk} Exact Transformation to QPBF Move making algorithms [Boykov , Veksler and Zabih 2001] [Woodford, Fitzgibbon, Reid, Torr, 2008] [Lempitsky, Rother, Blake, 2008] [Veksler, 2008] [Kohli, Ladicky, Torr 2008]
  120. 120. Energy Solution Space
  121. 121. Current Solution Search Neighbourhood Optimal MoveEnergy Solution Space
  122. 122. Current Solution Search Neighbourhood Optimal Move xc Key PropertyEnergy (t) Move Space Solution Space Bigger move • Better solutions space • Finding the optimal move hard
  123. 123. Minimizing Pairwise Functions[Boykov Veksler and Zabih, PAMI 2001]• Series of locally optimal moves• Each move reduces energy• Optimal move by minimizing submodular function Current Solution Search Neighbourhood Move Space (t) : 2n n Number of Variables L Number of Labels Space of Solutions (x) : Ln Extend to minimize Higher order Functions Kohli et al. ‘07, ‘08, ‘09
  124. 124. x = t x1 + (1-t) x2 New Current Second solution Solution solution Em(t) = E(t x1 + (1-t) x2) Minimize over move variables tFor certain x1 and x2, the move energy is sub-modular QPBF [Boykov , Veksler and Zabih 2001]
  125. 125. • Variables labeled α, β can swap their labels [Boykov , Veksler and Zabih 2001]
  126. 126. • Variables labeled α, β can swap their labels Tree Ground Swap Sky, House House Sky [Boykov , Veksler and Zabih 2001]
  127. 127. • Variables labeled α, β can swap their labels  Move energy is submodular if:  Unary Potentials: Arbitrary  Pairwise potentials: Semi-metric θij (la,lb) ≥ 0 θij (la,lb) = 0 a = b Examples: Potts model, Truncated Convex [Boykov , Veksler and Zabih 2001]
  128. 128. • Variables take label a or retain current label [Boykov , Veksler and Zabih 2001] [Boykov, Veksler, Zabih]
  129. 129. • Variables take label a or retain current label Tree Ground HouseStatus: Expand Ground Initialize with Tree Sky House Sky [Boykov , Veksler and Zabih 2001] [Boykov, Veksler, Zabih]
  130. 130. • Variables take label a or retain current label Semi metric Move energy is submodular if: +  Unary Potentials: Arbitrary Triangle Inequality  Pairwise potentials: Metric θij (la,lb) + θij (lb,lc) ≥ θij (la,lc) Examples: Potts model, Truncated linear Cannot solve truncated quadratic [Boykov , Veksler and Zabih 2001] [Boykov, Veksler, Zabih]
  131. 131.  Expansion and Swap can be derived as a primal dual scheme Get solution of the dual problem which is a lower bound on the energy of solution Weak guarantee on the solution E(x) < 2(dmax /dmin) E(x*) θij (li,lj) = g(|li-lj|) dmax dmin |yi-yj| [Komodakis et al 05, 07]
  132. 132. x = t x1 + (1-t) x2 New First Second solution solution solution Minimize over move variables tMove Type First Second Guarantee Solution SolutionExpansion Old solution All alpha Metric Fusion Any solution Any solution  Move functions can be non-submodular!!
  133. 133. x = t x1 + (1-t) x2 x1, x2 can be continuous Optical Flow Example Solution from Method 2 x2Solution from F FinalMethod 1 x1 x Solution [Woodford, Fitzgibbon, Reid, Torr, 2008] [Lempitsky, Rother, Blake, 2008]
  134. 134.  Move variables can be multi-label x = (t ==1) x1 + (t==2) x2 +… +(t==k) xk Optimal move found out by using the Ishikawa Transform Useful for minimizing energies with truncated convex pairwise potentials T θij (yi,yj) = min(|yi-yj|2,T) θij (yi,yj) |yi-yj| [Veksler, 2007]
  135. 135. Noisy Image Image RangeExpansion Moves Move [Veksler, 2008]
  136. 136. Why?
  137. 137. 3,600,000,000 PixelsCreated from about 800 8 MegaPixel Images [Kopf et al. (MSR Redmond) SIGGRAPH 2007 ]
  138. 138. [Kopf et al. (MSR Redmond) SIGGRAPH 2007 ]
  139. 139.  Processing Videos 1 minute video of 1M pixel resolution  3.6 B pixels 3D reconstruction [500 x 500 x 500 = .125B voxels]
  140. 140. Segment First Frame Can we do better? SegmentSecond Frame Kohli[Kohli & Torr, ICCV05 PAMI07] & Torr (ICCV05, PAMI07)
  141. 141. ImageFlow Segmentation Kohli[Kohli & Torr, ICCV05 PAMI07] & Torr (ICCV05, PAMI07)
  142. 142. minimizeFrame 1 EA SA Reuse Computation differences between EB* Simpler A and B fast minimizeFrame 2 EB SB 3–100000 time speedup! Reparametrization [Kohli & Torr, ICCV05 PAMI07] [Komodakis & Paragios, CVPR07]
  143. 143. Original EnergyE(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2Reparametrized Energy E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2New EnergyE(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 7a1ā2 + ā1a2New Reparametrized EnergyE(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 + 5a1ā2 [Kohli & Torr, ICCV05 PAMI07]& [Komodakis & Paragios, CVPR07] Kohli Torr (ICCV05, PAMI07)
  144. 144. Approximation Original algorithm Approximate Problem (Large) Solution (Slow)Fast partially optimal algorithm [ Alahari Kohli & Torr CVPR ‘08]
  145. 145. Approximation Original algorithm Approximate Problem (Large) Solution (Slow) Fast partiallyoptimal algorithm [Kovtun ‘03]Fast partially [ Kohli et al. ‘09] optimal algorithm Reduced Problem Approximation algorithm Solved Problem Approximate (Global Optima) (Fast) Solution [ Alahari Kohli & Torr CVPR ‘08]
  146. 146. 3- 100 Times Speed up sky Tree Reweighted Original Building Message Passing Approximate Problem Airplane (Large) Solution (9.89 sec) Grass Fast partiallyoptimal algorithm [Kovtun ‘03]Fast partially [ Kohli et al. ‘09] optimal Tree Reweighted sky algorithm sky Reduced Problem Message Passing Building Building Approximate Solved Problem Airplane Solution Airplane Total Time (Global Optima) (0.30 sec) Grass Grass [ Alahari Kohli & Torr CVPR ‘08]
  147. 147.  Minimization with Complex Higher Order Functions  Connectivity  Counting Constraints Hybrid algorithms  Connections between Messages Passing algorithms BP, TRW, and graph cuts
  148. 148. Advances in Markov Random Fields for Computer Vision MIT Press, summer 2010 Topics of this course and much, much more Contributors: usual suspects – lecturers on this course + Boykov,Kolmogorov, Weiss, Freeman, .... one for the office and one for home www.research.microsoft.com/vision/MRFbook
  149. 149. NP-Hard MAXCUTCSP Space of Problems
  150. 150.  Which functions are exactly solvable? Approximate solutions of NP-hard problems Scalability and Efficiency
  151. 151.  Which functions are exactly solvable? Boros Hammer [1965], Kolmogorov Zabih [ECCV 2002, PAMI 2004] , Ishikawa [PAMI 2003], Schlesinger [EMMCVPR 2007], Kohli Kumar Torr [CVPR2007, PAMI 2008] , Ramalingam Kohli Alahari Torr [CVPR 2008] , Kohli Ladicky Torr [CVPR 2008, IJCV 2009] , Zivny Jeavons [CP 2008] Approximate solutions of NP-hard problems Schlesinger [76 ], Kleinberg and Tardos [FOCS 99], Chekuri et al. [01], Boykov et al. [PAMI 01], Wainwright et al. [NIPS01], Werner [PAMI 2007], Komodakis et al. [PAMI, 05 07], Lempitsky et al. [ICCV 2007], Kumar et al. [NIPS 2007], Kumar et al. [ICML 2008], Sontag and Jakkola [NIPS 2007], Kohli et al. [ICML 2008], Kohli et al. [CVPR 2008, IJCV 2009], Rother et al. [2009] Scalability and Efficiency Kohli Torr [ICCV 2005, PAMI 2007], Juan and Boykov [CVPR 2006], Alahari Kohli Torr [CVPR 2008] , Delong and Boykov [CVPR 2008]
  152. 152.  Iterated Conditional Modes (ICM) Classical Move making Simulated Annealing algorithms Dynamic Programming (DP) Belief Propagtion (BP) Message passing Tree-Reweighted (TRW), Diffusion Graph Cut (GC) Combinatorial Algorithms Branch & Bound Relaxation methods: Convex Optimization (Linear Programming, ...) …

×