CVPR2012: Tutorial: Graphcut-based Optimisation for Computer Vision

3,555 views

Published on

Published in: Education, Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,555
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
162
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Many different approaches have been proposed for dealing with co-occurrence . In order to justify our approach, I'll run through some basic criteria that we think a co-occurrence potential should satisfy. Firstly, text...
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • There are 3 existing approaches to co-occurrence. Describe None Of them, exhibit invariance, and there is a direct trade off between the use of a global or dynamic cost function, that adapts to the current labelling and computational efficiency.
  • There are 3 existing approaches to co-occurrence. Describe None Of them, exhibit invariance, and there is a direct trade off between the use of a global or dynamic cost function, that adapts to the current labelling and computational efficiency.
  • There are 3 existing approaches to co-occurrence. Describe None Of them, exhibit invariance, and there is a direct trade off between the use of a global or dynamic cost function, that adapts to the current labelling and computational efficiency.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • CVPR2012: Tutorial: Graphcut-based Optimisation for Computer Vision

    1. 1. GraphCut-based Optimisation for Computer Vision Ľubor Ladický
    2. 2. Overview• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm• Markov and Conditional Random Fields• Random Field Optimisation using Graph Cuts • Submodular vs. Non-Submodular Problems • Pairwise vs. Higher Order Problems • 2-Label vs. Multi-Label Problems• Recent Advances in Random Field Optimisation• Conclusions
    3. 3. Overview• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm• Markov and Conditional Random Fields• Random Field Optimisation using Graph Cuts • Submodular vs. Non-Submodular Problems • Pairwise vs. Higher Order Problems • 2-Label vs. Multi-Label Problems• Recent Advances in Random Field Optimisation• Conclusions
    4. 4. Image Labelling Problems Assign a label to each image pixelGeometry Estimation Image Denoising Object Segmentation Depth Estimation Sky Building Tree Grass
    5. 5. Image Labelling Problems• Labellings highly structured Possible labelling Unprobable labelling Impossible labelling
    6. 6. Image Labelling Problems• Labellings highly structured• Labels highly correlated with very complex dependencies • Neighbouring pixels tend to take the same label • Low number of connected components • Classes present may be seen in one image • Geometric / Location consistency • Planarity in depth estimation • … many others (task dependent)
    7. 7. Image Labelling Problems• Labelling highly structured• Labels highly correlated with very complex dependencies• Independent label estimation too hard
    8. 8. Image Labelling Problems• Labelling highly structured• Labels highly correlated with very complex dependencies• Independent label estimation too hard• Whole labelling should be formulated as one optimisation problem
    9. 9. Image Labelling Problems• Labelling highly structured• Labels highly correlated with very complex dependencies• Independent label estimation too hard• Whole labelling should be formulated as one optimisation problem• Number of pixels up to millions • Hard to train complex dependencies • Optimisation problem is hard to infer
    10. 10. Image Labelling Problems• Labelling highly structured• Labels highly correlated with very complex dependencies• Independent label estimation too hard• Whole labelling should be formulated as one optimisation problem• Number of pixels up to millions • Hard to train complex dependencies • Optimisation problem is hard to infer Vision is hard !
    11. 11. Image Labelling Problems• You can either • Change the subject from the Computer Vision to the History of Renaissance Art Vision is hard !
    12. 12. Image Labelling Problems• You can either • Change the subject from the Computer Vision to the History of Renaissance Art or • Learn everything about Random Fields and Graph Cuts Vision is hard !
    13. 13. Foreground / Background Estimation Rother et al. SIGGRAPH04
    14. 14. Foreground / Background Estimation Data term Smoothness termData term Estimated using FG / BG colour modelsSmoothness term where Intensity dependent smoothness
    15. 15. Foreground / Background EstimationData term Smoothness term
    16. 16. Foreground / Background Estimation Data term Smoothness termHow to solve this optimisation problem?
    17. 17. Foreground / Background Estimation Data term Smoothness term How to solve this optimisation problem?• Transform into min-cut / max-flow problem• Solve it using min-cut / max-flow algorithm
    18. 18. Overview• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm• Markov and Conditional Random Fields• Random Field Optimisation using Graph Cuts • Submodular vs. Non-Submodular Problems • Pairwise vs. Higher Order Problems • 2-Label vs. Multi-Label Problems• Applications• Conclusions
    19. 19. Max-Flow Problem source 5 9 4 2 2 3 5 2 3 1 1 6 5 3 2 6 8 5Task : sink Maximize the flow from the sink to the source such that 1) The flow it conserved for each node 2) The flow for each pipe does not exceed the capacity
    20. 20. Max-Flow Problem source 5 9 4 2 2 3 52 3 1 1 6 5 3 2 6 8 5 sink
    21. 21. Max-Flow Problem source flow from the flow from node i source 5 9 to node j capacity 4 2 2 set of edges 3 52 3 1 1 6 5 3 2 conservation of flow set of nodes 6 8 5 sink
    22. 22. Max-Flow Problem source 9 Ford & Fulkerson algorithm (1956) 5 4 2 2 Find the path from source to sink 3 52 3 While (path exists) 1 1 6 5 flow += maximum capacity in the path 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 8 5 End sink
    23. 23. Max-Flow Problem source 9 Ford & Fulkerson algorithm (1956) 5 4 2 2 Find the path from source to sink 3 52 3 While (path exists) 1 1 6 5 flow += maximum capacity in the path 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 8 5 End sink
    24. 24. Max-Flow Problem source 9 Ford & Fulkerson algorithm (1956) 5 4 2 2 Find the path from source to sink 3 52 3 While (path exists) 1 1 6 5 flow += maximum capacity in the path 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 8 5 End sink flow = 3
    25. 25. Max-Flow Problem source 9 Ford & Fulkerson algorithm (1956) 5-3 4 2+ 2 Find the path from source to sink 3 5 3-32 3 While (path exists) 1 6-3 1 flow += maximum capacity in the path +3 5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 8-3 5 End sink flow = 3
    26. 26. Max-Flow Problem source 9 Ford & Fulkerson algorithm (1956) 2 4 5 2 Find the path from source to sink 52 3 While (path exists) 1 1 3 flow += maximum capacity in the path 3 5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 5 5 End sink flow = 3
    27. 27. Max-Flow Problem source 9 Ford & Fulkerson algorithm (1956) 2 4 5 2 Find the path from source to sink 52 3 While (path exists) 1 1 3 flow += maximum capacity in the path 3 5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 5 5 End sink flow = 3
    28. 28. Max-Flow Problem source 9 Ford & Fulkerson algorithm (1956) 2 4 5 2 Find the path from source to sink 52 3 While (path exists) 1 1 3 flow += maximum capacity in the path 3 5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 5 5 End sink flow = 6
    29. 29. Max-Flow Problem source 9-3 Ford & Fulkerson algorithm (1956) 2 4 5 2 Find the path from source to sink 52 +3 3-3 While (path exists) 1 1 3 5 flow += maximum capacity in the path 3 3-3 Build the residual graph (“subtract” the flow) 2 +3 Find the path in the residual graph 6 5-3 5 End sink flow = 6
    30. 30. Max-Flow Problem source 6 Ford & Fulkerson algorithm (1956) 2 4 5 2 Find the path from source to sink 52 3 While (path exists) 1 1 3 flow += maximum capacity in the path 3 5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 2 5 End sink flow = 6
    31. 31. Max-Flow Problem source 6 Ford & Fulkerson algorithm (1956) 2 4 5 2 Find the path from source to sink 52 3 While (path exists) 1 1 3 flow += maximum capacity in the path 3 5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 2 5 End sink flow = 6
    32. 32. Max-Flow Problem source 6 Ford & Fulkerson algorithm (1956) 2 4 5 2 Find the path from source to sink 52 3 While (path exists) 1 1 3 flow += maximum capacity in the path 3 5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 2 5 End sink flow = 11
    33. 33. Max-Flow Problem source 6-5 Ford & Fulkerson algorithm (1956) 2 4 5 2+5 Find the path from source to sink 5-52 3 While (path exists) 1 1+5 3 flow += maximum capacity in the path 3 5-5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 2 5-5 End sink flow = 11
    34. 34. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 2 4 5 Find the path from source to sink 72 3 While (path exists) 1 6 3 flow += maximum capacity in the path 3 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 2 End sink flow = 11
    35. 35. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 2 4 5 Find the path from source to sink 72 3 While (path exists) 1 6 3 flow += maximum capacity in the path 3 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 2 End sink flow = 11
    36. 36. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 2 4 5 Find the path from source to sink 72 3 While (path exists) 1 6 3 flow += maximum capacity in the path 3 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 2 End sink flow = 13
    37. 37. Max-Flow Problem source 2-2 1 Ford & Fulkerson algorithm (1956) 4 5 Find the path from source to sink 72-2 While (path exists) +2 3 1 6 3 flow += maximum capacity in the path +2 3 3 Build the residual graph (“subtract” the flow) 2-2 Find the path in the residual graph 6 2-2 End sink flow = 13
    38. 38. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 4 5 Find the path from source to sink 72 3 While (path exists) 1 6 3 flow += maximum capacity in the path 2 3 3 Build the residual graph (“subtract” the flow) Find the path in the residual graph 6 End sink flow = 13
    39. 39. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 4 5 Find the path from source to sink 72 3 While (path exists) 1 6 3 flow += maximum capacity in the path 2 3 3 Build the residual graph (“subtract” the flow) Find the path in the residual graph 6 End sink flow = 13
    40. 40. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 4 5 Find the path from source to sink 72 3 While (path exists) 1 6 3 flow += maximum capacity in the path 2 3 3 Build the residual graph (“subtract” the flow) Find the path in the residual graph 6 End sink flow = 15
    41. 41. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 4-2 5 Find the path from source to sink 72 3 While (path exists) 1 3-2 6 flow += maximum capacity in the path 2-2 3+ 32 Build the residual graph (“subtract” the flow) +2 Find the path in the residual graph 6-2 End sink flow = 15
    42. 42. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 2 5 Find the path from source to sink 72 3 While (path exists) 1 6 1 flow += maximum capacity in the path 5 3 Build the residual graph (“subtract” the flow) 2 4 Find the path in the residual graph End sink flow = 15
    43. 43. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 2 5 Find the path from source to sink 72 3 While (path exists) 1 6 1 flow += maximum capacity in the path 5 3 Build the residual graph (“subtract” the flow) 2 4 Find the path in the residual graph End sink flow = 15
    44. 44. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 2 5 Find the path from source to sink 72 3 While (path exists) 1 6 1 flow += maximum capacity in the path 5 3 Build the residual graph (“subtract” the flow) 2 4 Find the path in the residual graph End sink flow = 15
    45. 45. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 2 5 Why is the solution globally optimal ? 72 3 1 6 1 5 2 3 4 sink flow = 15
    46. 46. Max-Flow ProblemS source 1 Ford & Fulkerson algorithm (1956) 2 5 Why is the solution globally optimal ? 72 3 1 6 1 1. Let S be the set of reachable nodes in the 5 3 residual graph 2 4 sink flow = 15
    47. 47. Max-Flow ProblemS5 source 9 Ford & Fulkerson algorithm (1956) 4 2 2 Why is the solution globally optimal ? 3 52 3 1 1 6 5 1. Let S be the set of reachable nodes in the 3 residual graph 2 2. The flow from S to V - S equals to the sum 6 8 5 of capacities from S to V – S sink
    48. 48. Max-Flow ProblemA5 source 9 Ford & Fulkerson algorithm (1956) 4 2 2 Why is the solution globally optimal ? 3 52 3 1 1 6 5 1. Let S be the set of reachable nodes in the residual graph 3 2 2. The flow from S to V - S equals to the sum of 6 5 capacities from S to V – S 8 3. The flow from any A to V - A is upper bounded sink by the sum of capacities from A to V – A
    49. 49. Max-Flow Problem source 5/5 8/9 Ford & Fulkerson algorithm (1956) 2/4 0/2 0/2 Why is the solution globally optimal ? 5/5 2/2 3/3 0/1 0/1 3/3 5/6 5/5 1. Let S be the set of reachable nodes in the residual graph 3/3 0/2 2. The flow from S to V - S equals to the sum of 2/6 5/5 capacities from S to V – S 8/8 3. The flow from any A to V - A is upper bounded sink by the sum of capacities from A to V – A 4. The solution is globally optimal flow = 15Individual flows obtained by summing up all paths
    50. 50. Max-Flow Problem source 5/5 8/9 Ford & Fulkerson algorithm (1956) 2/4 0/2 0/2 Why is the solution globally optimal ? 3/3 5/52/2 0/1 0/1 3/3 5/6 5/5 1. Let S be the set of reachable nodes in the residual graph 3/3 0/2 2. The flow from S to V - S equals to the sum of 2/6 5/5 capacities from S to V – S 8/8 3. The flow from any A to V - A is upper bounded sink by the sum of capacities from A to V – A 4. The solution is globally optimal flow = 15
    51. 51. Max-Flow Problem source Ford & Fulkerson algorithm (1956)1000 1000 Order does matter 11000 1000 sink
    52. 52. Max-Flow Problem source Ford & Fulkerson algorithm (1956)1000 1000 Order does matter 11000 1000 sink
    53. 53. Max-Flow Problem source Ford & Fulkerson algorithm (1956)1000 999 Order does matter 1999 1000 sink
    54. 54. Max-Flow Problem source Ford & Fulkerson algorithm (1956)1000 999 Order does matter 1999 1000 sink
    55. 55. Max-Flow Problem source Ford & Fulkerson algorithm (1956)999 999 Order does matter 1999 999 sink
    56. 56. Max-Flow Problem source Ford & Fulkerson algorithm (1956)999 999 Order does matter • Standard algorithm not polynomial 1 • Breath first leads to O(VE2) • Path found in O(E)999 999 • At least one edge gets saturated • The saturated edge distance to the sink source has to increase and is at most V leading to O(VE) (Edmonds & Karp, Dinic)
    57. 57. Max-Flow Problem source Ford & Fulkerson algorithm (1956)999 999 Order does matter • Standard algorithm not polynomial 1 • Breath first leads to O(VE2) (Edmonds & Karp)999 999 • Various methods use different algorithm to find the path and vary in complexity sink
    58. 58. Min-Cut Problem source 5 9 4 2 2 3 5 2 3 1 1 6 5 3 2 6 8 5Task : sink Minimize the cost of the cut 1) Each node is either assigned to the source S or sink T 2) The cost of the edge (i, j) is taken if (i∈S) and (j∈T)
    59. 59. Min-Cut Problem source 5 9 4 2 2 3 52 3 1 1 6 5 3 2 6 8 5 sink
    60. 60. Min-Cut Problem source 5 9 edge costs 4 2 2 3 52 3 1 1 6 5 source set sink set 3 2 6 8 5 sink
    61. 61. Min-Cut ProblemS5 source 9 edge costs 4 2 2 3 5 cost = 182 3 1 1 6 5 source set sink set 3 2 6 8 5 sink T
    62. 62. Min-Cut ProblemS5 source 9 cost = 25 edge costs 4 2 2 3 52 3 1 1 6 5 source set sink set 3 2 6 8 5 sink T
    63. 63. Min-Cut ProblemS5 source 9 edge costs 4 2 2 3 5 cost = 232 3 1 1 6 5 source set sink set 3 2 6 8 5 sink T
    64. 64. Min-Cut Problem source 5 9 x1 4 x2 2 2 3 52 x3 3 1 1 x4 6 5 x6 3 2 x5 6 8 5 sink
    65. 65. Min-Cut Problem source 5 9 x1 4 x2 2 2 3 52 x3 transformable into Linear program (LP) 3 1 1 x4 6 5 x6 3 2 x5 6 8 5 sink
    66. 66. Min-Cut Problem source 5 9 x1 4 x2 2 2 3 52 x3 transformable into Linear program (LP) 3 1 1 x4 6 5 x6 3 2 x5 6 8 5 sink Dual to max-flow problem
    67. 67. Min-Cut Problem source After the substitution of the constraints : 5 9 x1 4 x2 2 2 3 52 x3 3 1 1 x4 6 5 x6 3 2 x5 6 8 5 sink
    68. 68. Min-Cut Problem source edges sink edges source 5 9 x1 4 x2 2 2 3 52 x3 3 1 1 x4 6 5 x6 3 2 x5 xi = 0 ⇒ xi ∈ S xi = 1 ⇒ xi ∈ T 6 8 5 sink Edges between variables
    69. 69. Min-Cut Problem source C(x) = 5x1 + 9x2 + 4x3 + 3x3(1-x1) + 2x1(1-x3) 5 9 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 6x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 2 2 3 5 + 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)2 x3 3 1 1 + 8(1-x5) + 5(1-x6) x4 6 5 x6 3 2 x5 6 8 5 sink
    70. 70. Min-Cut Problem source C(x) = 5x1 + 9x2 + 4x3 + 3x3(1-x1) + 2x1(1-x3) 5 9 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 6x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 2 2 3 5 + 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)2 x3 3 1 1 + 8(1-x5) + 5(1-x6) x4 6 5 x6 3 2 x5 6 8 5 sink
    71. 71. Min-Cut Problem source C(x) = 2x1 + 9x2 + 4x3 + 2x1(1-x3) 5 9 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 2 2 3 5 + 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)2 x3 3 1 1 + 5(1-x5) + 5(1-x6) x4 6 5 x6 + 3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5) 3 2 x5 6 8 5 sink
    72. 72. Min-Cut Problem source C(x) = 2x1 + 9x2 + 4x3 + 2x1(1-x3) 5 9 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 2 2 3 5 + 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)2 x3 3 1 1 + 5(1-x5) + 5(1-x6) x4 6 5 x6 + 3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5) 3 2 x5 6 8 5 3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5) sink = 3 + 3x1(1-x3) + 3x3(1-x5)
    73. 73. Min-Cut Problem source C(x) = 3 + 2x1 + 9x2 + 4x3 + 5x1(1-x3) 2 9 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 5 2 5 + 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)2 x3 3 1 1 + 5(1-x5) + 5(1-x6) + 3x3(1-x5) 3 x4 3 5 x6 3 2 x5 6 5 5 sink
    74. 74. Min-Cut Problem source C(x) = 3 + 2x1 + 9x2 + 4x3 + 5x1(1-x3) 2 9 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 5 2 5 + 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)2 x3 3 1 1 + 5(1-x5) + 5(1-x6) + 3x3(1-x5) 3 x4 3 5 x6 3 2 x5 6 5 5 sink
    75. 75. Min-Cut Problem source C(x) = 3 + 2x1 + 6x2 + 4x3 + 5x1(1-x3) 2 9 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 5 2 5 + 2x5(1-x4) + 6(1-x4)2 x3 3 1 1 + 2(1-x5) + 5(1-x6) + 3x3(1-x5) 3 x4 3 5 x6 + 3x2 + 3x6(1-x2) + 3x5(1-x6) + 3(1-x5) 3 2 x5 6 5 5 3x2 + 3x6(1-x2) + 3x5(1-x6) + 3(1-x5) sink = 3 + 3x2(1-x6) + 3x6(1-x5)
    76. 76. Min-Cut Problem source C(x) = 6 + 2x1 + 6x2 + 4x3 + 5x1(1-x3) 2 6 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 5 2 5 + 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)2 x3 3 1 1 + 2(1-x5) + 5(1-x6) + 3x3(1-x5) 3 x4 3 5 x6 3 2 x5 6 2 5 sink
    77. 77. Min-Cut Problem source C(x) = 6 + 2x1 + 6x2 + 4x3 + 5x1(1-x3) 2 6 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 5 2 5 + 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)2 x3 3 1 1 + 2(1-x5) + 5(1-x6) + 3x3(1-x5) 3 x4 3 5 x6 3 2 x5 6 2 5 sink
    78. 78. Min-Cut Problem source C(x) = 11 + 2x1 + 1x2 + 4x3 + 5x1(1-x3) 2 1 + 3x3(1-x1) + 7x2(1-x3) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 6x3(1-x6) 5 7 + 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)2 x3 3 1 6 + 2(1-x5) + 3x3(1-x5) 3 x4 3 x6 3 2 x5 6 2 sink
    79. 79. Min-Cut Problem source C(x) = 11 + 2x1 + 1x2 + 4x3 + 5x1(1-x3) 2 1 + 3x3(1-x1) + 7x2(1-x3) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 6x3(1-x6) 5 7 + 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)2 x3 3 1 6 + 2(1-x5) + 3x3(1-x5) 3 x4 3 x6 3 2 x5 6 2 sink
    80. 80. Min-Cut Problem source C(x) = 13 + 1x2 + 4x3 + 5x1(1-x3) 1 + 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 6x3(1-x6) 5 7 + 2x4(1-x5) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5) x3 32 1 6 + 3x3(1-x5) 3x4 2 3 x6 3 x5 6 sink
    81. 81. Min-Cut Problem source C(x) = 13 + 1x2 + 2x3 + 5x1(1-x3) 1 + 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 6x3(1-x6) 5 7 + 2x4(1-x5) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5) x3 32 1 6 + 3x3(1-x5) 3x4 2 3 x6 3 x5 6 sink
    82. 82. Min-Cut Problem source C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3) 1 + 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)x1 2 x2 + 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5) 5 7 + 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5) x3 32 1 6 1x4 5 x6 2 3 x5 4 sink
    83. 83. Min-Cut Problem source C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3) 1 + 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)x1 2 x2 + 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5) 5 7 + 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5) x3 32 1 6 1x4 5 x6 2 3 x5 4 sink
    84. 84. Min-Cut Problem min cut C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3) S source 1 + 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4) x1 2 x2 + 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5) 5 7 + 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5) x3 3 2 1 6 1 cost = 0 x4 5 x6 cost = 0 2 3 x5 4 • All coefficients positive sink T • Must be global minimumS – set of reachable nodes from s
    85. 85. Min-Cut Problem min cut C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3) S source 1 + 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4) x1 2 x2 + 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5) 5 7 + 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5) x3 3 2 1 6 1 x4 5 x6 cost = 0 2 3 x5 4 • All coefficients positive sink T • Must be global minimumT – set of nodes that can reach t(not necesarly the same)
    86. 86. Overview• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm• Markov and Conditional Random Fields• Random Field Optimisation using Graph Cuts • Submodular vs. Non-Submodular Problems • Pairwise vs. Higher Order Problems • 2-Label vs. Multi-Label Problems• Recent Advances in Random Field Optimisation• Conclusions
    87. 87. Markov and Conditional RF• Markov / Conditional Random fields model probabilistic dependencies of the set of random variables
    88. 88. Markov and Conditional RF• Markov / Conditional Random fields model conditional dependencies between random variables• Each variable is conditionally independent of all other variables given its neighbours
    89. 89. Markov and Conditional RF• Markov / Conditional Random fields model conditional dependencies between random variables• Each variable is conditionally independent of all other variables given its neighbours• Posterior probability of the labelling x given data D is : partition function cliques potential functions
    90. 90. Markov and Conditional RF• Markov / Conditional Random fields model conditional dependencies between random variables• Each variable is conditionally independent of all other variables given its neighbours• Posterior probability of the labelling x given data D is : partition function cliques potential functions• Energy of the labelling is defined as :
    91. 91. Markov and Conditional RF• The most probable (Max a Posteriori (MAP)) labelling is defined as:
    92. 92. Markov and Conditional RF• The most probable (Max a Posteriori (MAP)) labelling is defined as:• The only distinction (MRF vs. CRF) is that in the CRF the conditional dependencies between variables depend also on the data
    93. 93. Markov and Conditional RF• The most probable (Max a Posteriori (MAP)) labelling is defined as:• The only distinction (MRF vs. CRF) is that in the CRF the conditional dependencies between variables depend also on the data• Typically we define an energy first and then pretend there is an underlying probabilistic distribution there, but there isn’t really (Pssssst, don’t tell anyone)
    94. 94. Overview• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm• Markov and Conditional Random Fields• Random Field Optimisation using Graph Cuts • Submodular vs. Non-Submodular Problems • Pairwise vs. Higher Order Problems • 2-Label vs. Multi-Label Problems• Recent Advances in Random Field Optimisation• Conclusions
    95. 95. Pairwise CRF modelsStandard CRF Energy Data term Smoothness term
    96. 96. Graph Cut based InferenceThe energy / potential is called submodular (only) if for everypair of variables :
    97. 97. Graph Cut based InferenceFor 2-label problems x∈{0, 1} :
    98. 98. Graph Cut based Inference For 2-label problems x∈{0, 1} :Pairwise potential can be transformed into :where
    99. 99. Graph Cut based Inference For 2-label problems x∈{0, 1} :Pairwise potential can be transformed into :where
    100. 100. Graph Cut based Inference For 2-label problems x∈{0, 1} :Pairwise potential can be transformed into :where
    101. 101. Graph Cut based InferenceAfter summing up :
    102. 102. Graph Cut based InferenceAfter summing up :Let :
    103. 103. Graph Cut based InferenceAfter summing up :Let : Then :
    104. 104. Graph Cut based Inference After summing up : Let : Then :Equivalent st-mincut problem is :
    105. 105. Foreground / Background Estimation Data term Smoothness termData term Estimated using FG / BG colour modelsSmoothness term where Intensity dependent smoothness
    106. 106. Foreground / Background Estimationsourcesink Rother et al. SIGGRAPH04
    107. 107. Graph Cut based InferenceExtendable to (some) Multi-label CRFs • each state of multi-label variable encoded using multiple binary variables • the cost of every possible cut must be the same as the associated energy • the solution obtained by inverting the encoding Encoding multi-label energy Build Obtain solution Graph Cut Graph solution
    108. 108. Dense Stereo Estimation• For each pixel assigns a disparity label : z• Disparities from the discrete set {0, 1, .. D} Left Camera Image Right Camera Image Dense Stereo Result
    109. 109. Dense Stereo EstimationData term Left image Shifted right feature image feature Left Camera Image Right Camera Image
    110. 110. Dense Stereo EstimationData term Left image Shifted right feature image featureSmoothness term
    111. 111. Dense Stereo Estimation source sour Encoding cexi0 xk0 xk1 xk2 . . . .xiD xkD sink Ishikawa PAMI03
    112. 112. Dense Stereo Estimation source sour ∞ ce ∞ Unary Potentialxi0 xk0 ∞ ∞ xk1 xk2 . . ∞ ∞ The cost of every cut should be equal to the corresponding . . energy under the encoding ∞ ∞xiD xkD cost = + sink Ishikawa PAMI03
    113. 113. Dense Stereo Estimation source sour ∞ ce ∞ Unary Potentialxi0 xk0 ∞ ∞ xk1 xk2 cost = + . . ∞ ∞ The cost of every cut should be equal to the corresponding . . energy under the encoding ∞ ∞xiD xkD sink Ishikawa PAMI03
    114. 114. Dense Stereo Estimation source sour ∞ ce ∞ Unary Potentialxi0 xk0 ∞ ∞ cost = ∞ xk1 xk2 . . ∞ ∞ The cost of every cut should be equal to the corresponding . . energy under the encoding ∞ ∞xiD xkD sink Ishikawa PAMI03
    115. 115. Dense Stereo Estimation source sour ∞ ce ∞ Pairwise Potential Kxi0 xk0 ∞ ∞ xk1 xk2 cost = + +K K . . ∞ ∞ The cost of every cut should be K equal to the corresponding . . energy under the encoding ∞ ∞ KxiD xkD sink Ishikawa PAMI03
    116. 116. Dense Stereo Estimation source sour ∞ ce ∞ Pairwise Potential Kxi0 xk0 ∞ ∞ xk1 xk2 K . . ∞ ∞ The cost of every cut should be K equal to the corresponding . . energy under the encoding ∞ ∞ KxiD xkD cost = + +DK sink Ishikawa PAMI03
    117. 117. Dense Stereo Estimation source sour ce ∞ Extendable to any convex cost ∞xi0 xk0 ∞ ∞ xk1 xk2 Convex function . . ∞ ∞ . . ∞ ∞xiD xkD See [Ishikawa PAMI03] for more details sink
    118. 118. Graph Cut based InferenceMove making algorithms• Original problem decomposed into a series of subproblems solvable with graph cut• In each subproblem we find the optimal move from the current solution in a restricted search space EncodingInitial solution Propose Build Update solution Graph Cut move Graph solution
    119. 119. Graph Cut based InferenceExample : [Boykov et al. 01] αβ-swap – each variable taking label α or β can change its label to α or β – all αβ-moves are iteratively performed till convergence α-expansion – each variable either keeps its old label or changes to α – all α-moves are iteratively performed till convergence Transformation function :
    120. 120. Graph Cut based InferenceSufficient conditions for move making algorithms : (all possible moves are submodular) αβ-swap : semi-metricity Proof:
    121. 121. Graph Cut based InferenceSufficient conditions for move making algorithms : (all possible moves are submodular) α-expansion : metricity Proof:
    122. 122. Object-class Segmentation Data term Smoothness termData term Discriminatively trained classifierSmoothness term where
    123. 123. Object-class Segmentation grassOriginal Image Initial solution
    124. 124. Object-class Segmentation grass building grassOriginal Image Initial solution Building expansion
    125. 125. Object-class Segmentation grass building grass Original Image Initial solution Building expansion sky buildinggrass Sky expansion
    126. 126. Object-class Segmentation grass building grass Original Image Initial solution Building expansion sky sky tree building buildinggrass grass Sky expansion Tree expansion
    127. 127. Object-class Segmentation grass building grass Original Image Initial solution Building expansion sky sky sky tree tree building building building aeroplanegrass grass grass Sky expansion Tree expansion Final Solution
    128. 128. Graph Cut based InferenceRange-(swap) moves [Veksler 07] – Each variable in the convex range can change its label to any other label in the convex range – all range-moves are iteratively performed till convergenceRange-expansion [Kumar, Veksler & Torr 1] – Expansion version of range swap moves – Each varibale can change its label to any label in a convex range or keep its old label
    129. 129. Dense Stereo Reconstruction Left Camera Image Right Camera Image Dense Stereo ResultData term Same as beforeSmoothness term
    130. 130. Dense Stereo Reconstruction Left Camera Image Right Camera Image Dense Stereo ResultData term Same as beforeSmoothness term Convex range Truncation
    131. 131. Graph Cut based InferenceOriginal Image Initial Solution
    132. 132. Graph Cut based InferenceOriginal Image Initial Solution After 1st expansion
    133. 133. Graph Cut based Inference Original Image Initial Solution After 1st expansionAfter 2nd expansion
    134. 134. Graph Cut based Inference Original Image Initial Solution After 1st expansionAfter 2nd expansion After 3rd expansion
    135. 135. Graph Cut based Inference Original Image Initial Solution After 1st expansionAfter 2nd expansion After 3rd expansion Final solution
    136. 136. Graph Cut based Inference• Extendible to certain classes of binary higher order energies• Higher order terms have to be transformable to the pairwise submodular energy with binary auxiliary variables zC Higher order term Pairwise term EncodingInitial solution Propose Transf. to Update solution Graph Cut move pairwise subm. solution
    137. 137. Graph Cut based Inference• Extendible to certain classes of binary higher order energies• Higher order terms have to be transformable to the pairwise submodular energy with binary auxiliary variables zC Higher order term Pairwise termExample : Kolmogorov ECCV06, Ramalingam et al. DAM12
    138. 138. Graph Cut based InferenceEnforces label consistency of the clique as a weak constraint Input image Pairwise CRF Higher order CRF Kohli et al. CVPR07
    139. 139. Graph Cut based Inference If the energy is not submodular / graph-representable – Overestimation by a submodular energy – Quadratic Pseudo-Boolean Optimisation (QPBO) EncodingInitial solution solution Propose Transf. to Update Graph Cut move pairwise subm. solution
    140. 140. Graph Cut based Inference Overestimation by a submodular energy (convex / concave) – We find an energy E’(t) s.t. – it is tight in the current solution E’(t0) = E(t0) – Overestimates E(t) for all t E’(t) ≥ E(t) – We replace E(t) by E’(t) – The moves are not optimal, but quaranteed to converge – The tighter over-estimation, the better Example : EncodingInitial solution Propose Overestim. by Update solution Graph Cut move pairwise subm. solution
    141. 141. Graph Cut based InferenceQuadratic Pseudo-Boolean Optimisation– Each binary variable is encoded using two binary _ _ variables xi and xi s.t. xi = 1 - xi
    142. 142. Graph Cut based Inference Quadratic Pseudo-Boolean Optimisation sourcexi xj_ _ _xi xj – Energy submodular in xi and xi _ – Solved by dropping the constraint xi = 1 - xi _ – If xi = 1 - xi the solution for xi is optimal sink
    143. 143. Overview• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm• Markov and Conditional Random Fields• Random Field Optimisation using Graph Cuts • Submodular vs. Non-Submodular Problems • Pairwise vs. Higher Order Problems • 2-Label vs. Multi-Label Problems• Recent Advances in Random Field Optimisation• Conclusions
    144. 144. Motivation• To propose formulations enforcing certain structure properties of the output useful in computer vision – Label Consistency in segments as a weak constraint – Label agreement over different scales – Label-set consistency – Multiple domains consistency• To propose feasible Graph-Cut based inference method
    145. 145. Structures in CRF• Taskar et al. 02 – associative potentials• Woodford et al. 08 – planarity constraint• Vicente et al. 08 – connectivity constraint• Nowozin & Lampert 09 – connectivity constraint• Roth & Black 09 – field of experts• Woodford et al. 09 – marginal probability• Delong et al. 10 – label occurrence costs
    146. 146. Object-class Segmentation Problem• Aims to assign a class label for each pixel of an image• Classifier trained on the training set• Evaluated on never seen test images
    147. 147. Pairwise CRF modelsStandard CRF Energy for Object Segmentation Local context Cannot encode global consistency of labels!! Ladický, Russell, Kohli, Torr ECCV10
    148. 148. Encoding Co-occurrence [Heitz et al. 08]Co-occurrence is a powerful cue [Rabinovich et al. ‘07] • Thing – Thing • Stuff - Stuff • Stuff - Thing[ Images from Rabinovich et al. 07 ] Ladický, Russell, Kohli, Torr ECCV10
    149. 149. Encoding Co-occurrence [Heitz et al. 08] Co-occurrence is a powerful cue [Rabinovich et al. ‘07] • Thing – Thing • Stuff - Stuff • Stuff - ThingProposed solutions :1. Csurka et al. 08 - Hard decision for label estimation2. Torralba et al. 03 - GIST based unary potential3. Rabinovich et al. 07 - Full-connected CRF [ Images from Rabinovich et al. 07 ] Ladický, Russell, Kohli, Torr ECCV10
    150. 150. So...What properties should these global co-occurence potentials have ? Ladický, Russell, Kohli, Torr ECCV10
    151. 151. Desired properties1. No hard decisions Ladický, Russell, Kohli, Torr ECCV10
    152. 152. Desired properties1. No hard decisions Incorporation in probabilistic framework Unlikely possibilities are not completely ruled out Ladický, Russell, Kohli, Torr ECCV10
    153. 153. Desired properties1. No hard decisions2. Invariance to region size Ladický, Russell, Kohli, Torr ECCV10
    154. 154. Desired properties1. No hard decisions2. Invariance to region size Cost for occurrence of {people, house, road etc .. } invariant to image area Ladický, Russell, Kohli, Torr ECCV10
    155. 155. Desired properties1. No hard decisions2. Invariance to region sizeThe only possible solution : L(x)={ , , } Local context Global context Cost defined over the assigned labels L(x) Ladický, Russell, Kohli, Torr ECCV10
    156. 156. Desired properties1. No hard decisions2. Invariance to region size3. Parsimony – simple solutions preferred L(x)={ aeroplane, tree, flower, building, boat, grass, sky }   L(x)={ building, tree, grass, sky } Ladický, Russell, Kohli, Torr ECCV10
    157. 157. Desired properties1. No hard decisions2. Invariance to region size3. Parsimony – simple solutions preferred4. Efficiency Ladický, Russell, Kohli, Torr ECCV10
    158. 158. Desired properties1. No hard decisions2. Invariance to region size3. Parsimony – simple solutions preferred4. Efficiency a) Memory requirements as O(n) with the image size and number or labels b) Inference tractable Ladický, Russell, Kohli, Torr ECCV10
    159. 159. Previous work• Torralba et al.(2003) – Gist-based unary potentials• Rabinovich et al.(2007) - complete pairwise graphs• Csurka et al.(2008) - hard estimation of labels present Ladický, Russell, Kohli, Torr ECCV10
    160. 160. Related work• Zhu & Yuille 1996 – MDL prior• Bleyer et al. 2010 – Surface Stereo MDL prior• Hoiem et al. 2007 – 3D Layout CRF MDL Prior C(x) = K |L(x)|• Delong et al. 2010 – label occurence cost C(x) = ΣLKLδL(x) Ladický, Russell, Kohli, Torr ECCV10
    161. 161. Related work• Zhu & Yuille 1996 – MDL prior• Bleyer et al. 2010 – Surface Stereo MDL prior• Hoiem et al. 2007 – 3D Layout CRF MDL Prior C(x) = K |L(x)|• Delong et al. 2010 – label occurence cost C(x) = ΣLKLδL(x) All special cases of our model Ladický, Russell, Kohli, Torr ECCV10
    162. 162. Inference for Co-occurenceCo-occurence representationLabel indicator functions Ladický, Russell, Kohli, Torr ECCV10
    163. 163. Inference for Co-occurence Cost of currentMove Energy label set Ladický, Russell, Kohli, Torr ECCV10
    164. 164. Inference for Co-occurenceMove EnergyDecomposition to α-dependent and α-independent part α-independent α-dependent Ladický, Russell, Kohli, Torr ECCV10
    165. 165. Inference for Co-occurenceMove EnergyDecomposition to α-dependent and α-independent partEither α or all labels in the image after the move Ladický, Russell, Kohli, Torr ECCV10
    166. 166. Inference for Co-occurenceMove Energy submodular non-submodular Ladický, Russell, Kohli, Torr ECCV10
    167. 167. Inference for Co-occurenceMove Energy non-submodularNon-submodular energy overestimated by E(t) • E(t) = E(t) for current solution • E(t) ≥ E(t) for any other labelling Ladický, Russell, Kohli, Torr ECCV10
    168. 168. Inference for Co-occurenceMove Energy non-submodularNon-submodular energy overestimated by E(t) • E(t) = E(t) for current solution • E(t) ≥ E(t) for any other labellingOccurrence - tight Ladický, Russell, Kohli, Torr ECCV10
    169. 169. Inference for Co-occurenceMove Energy non-submodularNon-submodular energy overestimated by E(t) • E(t) = E(t) for current solution • E(t) ≥ E(t) for any other labellingCo-occurrence overestimation Ladický, Russell, Kohli, Torr ECCV10
    170. 170. Inference for Co-occurenceMove Energy non-submodularNon-submodular energy overestimated by E(t) • E(t) = E(t) for current solution • E(t) ≥ E(t) for any other labellingGeneral case [See the paper] Ladický, Russell, Kohli, Torr ECCV10
    171. 171. Inference for Co-occurenceMove Energy non-submodularNon-submodular energy overestimated by E(t) • E(t) = E(t) for current solution • E(t) ≥ E(t) for any other labellingQuadratic representation Ladický, Russell, Kohli, Torr ECCV10
    172. 172. Potential Training for Co-occurenceGeneratively trained Co-occurence potential Approximated by 2nd order representation Ladický, Russell, Kohli, Torr ICCV09
    173. 173. Results on MSRC data set Comparisons of results with and without co-occurenceInput Image Without With Input Image Without With Co-occurence Co-occurence Co-occurence Co-occurence Ladický, Russell, Kohli, Torr ICCV09
    174. 174. Pairwise CRF modelsPixel CRF Energy for Object-class Segmentation Data term Smoothness term Input Image Pairwise CRF Result Shotton et al. ECCV06
    175. 175. Pairwise CRF modelsPixel CRF Energy for Object-class Segmentation Data term Smoothness term • Lacks long range interactions • Results oversmoothed • Data term information may be insufficient
    176. 176. Pairwise CRF modelsSegment CRF Energy for Object-class Segmentation Data term Smoothness term Input Image Unsupervised Shi, Malik PAMI2000, Comaniciu, Meer PAMI2002, Segmentation Felzenschwalb, Huttenlocher, IJCV2004,
    177. 177. Pairwise CRF modelsSegment CRF Energy for Object-class Segmentation Data term Smoothness term Input Image Pairwise CRF Result Batra et al. CVPR08, Yang et al. CVPR07, Zitnick et al. CVPR08, Rabinovich et al. ICCV07, Boix et al. CVPR10
    178. 178. Pairwise CRF modelsSegment CRF Energy for Object-class Segmentation Data term Smoothness term • Allows long range interactions • Does not distinguish between instances of objects • Cannot recover from incorrect segmentation
    179. 179. Pairwise CRF modelsSegment CRF Energy for Object-class Segmentation Data term Smoothness term • Allows long range interactions • Does not distinguish between instances of objects • Cannot recover from incorrect segmentation
    180. 180. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency term Input Image Higher order CRF Kohli, Ladický, Torr CVPR08
    181. 181. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency term Kohli, Ladický, Torr CVPR08
    182. 182. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency term No dominant label Kohli, Ladický, Torr CVPR08
    183. 183. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency term Dominant label l Kohli, Ladický, Torr CVPR08
    184. 184. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency term • Enforces consistency as a weak constraint • Can recover from incorrect segmentation • Can combine multiple segmentations • Does not use features at segment level Kohli, Ladický, Torr CVPR08
    185. 185. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency term Transformable to pairwise graph with auxiliary variable Kohli, Ladický, Torr CVPR08
    186. 186. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency termwhere Ladický, Russell, Kohli, Torr ICCV09
    187. 187. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency term Input Image Higher order CRF Ladický, Russell, Kohli, Torr ICCV09
    188. 188. Associative Hierarchical CRFs Can be generalized to Hierarchical model• Allows unary potentials for region variables• Allows pairwise potentials for region variables• Allows multiple layers and multiple hierarchies Ladický, Russell, Kohli, Torr ICCV09
    189. 189. Associative Hierarchical CRFsAH CRF Energy for Object-class Segmentation Pairwise CRF Higher order term Ladický, Russell, Kohli, Torr ICCV09
    190. 190. Associative Hierarchical CRFsAH CRF Energy for Object-class Segmentation Pairwise CRF Higher order termHigher order term recursively defined as Segment Segment Segment higher unary term pairwise term order term Ladický, Russell, Kohli, Torr ICCV09
    191. 191. Associative Hierarchical CRFsAH CRF Energy for Object-class Segmentation Pairwise CRF Higher order termHigher order term recursively defined as Segment Segment Segment higher unary term pairwise term order term Why is this generalisation useful? Ladický, Russell, Kohli, Torr ICCV09
    192. 192. Associative Hierarchical CRFsLet us analyze the case with• one segmentation• potentials only over segment level Ladický, Russell, Kohli, Torr ICCV09
    193. 193. Associative Hierarchical CRFsLet us analyze the case with• one segmentation• potentials only over segment level 1) Minimum is segment-consistent Ladický, Russell, Kohli, Torr ICCV09
    194. 194. Associative Hierarchical CRFsLet us analyze the case with• one segmentation• potentials only over segment level 2) Minimum is segment-consistent 3) Cost of every segment-consistent CRF equal to the cost of pairwise CRF over segments Ladický, Russell, Kohli, Torr ICCV09
    195. 195. Associative Hierarchical CRFsLet us analyze the case with• one segmentation• potentials only over segment level Equivalent to pairwise CRF over segments! Ladický, Russell, Kohli, Torr ICCV09
    196. 196. Associative Hierarchical CRFs• Merges information over multiple scales• Easy to train potentials and learn parameters• Allows multiple segmentations and hierarchies• Allows long range interactions• Limited (?) to associative interlayer connections Ladický, Russell, Kohli, Torr ICCV09
    197. 197. Inference for AH-CRFGraph Cut based move making algorithms [Boykov et al. 01]α-expansion transformation function for base layer Ladický (thesis) 2011
    198. 198. Inference for AH-CRFGraph Cut based move making algorithms [Boykov et al. 01]α-expansion transformation function for base layerα-expansion transformation function for auxiliary layer(2 binary variables per node) Ladický (thesis) 2011
    199. 199. Inference for AH-CRFInterlayer connection between the base layer and the first auxiliary layer: Ladický (thesis) 2011
    200. 200. Inference for AH-CRFInterlayer connection between the base layer and the first auxiliary layer:The move energy is:where Ladický (thesis) 2011
    201. 201. Inference for AH-CRFThe move energy for interlayer connection between the base and auxiliary layer is:Can be transformed to binary submodular pairwise function: Ladický (thesis) 2011
    202. 202. Inference for AH-CRFThe move energy for interlayer connection between auxiliary layers is:Can be transformed to binary submodular pairwise function: Ladický (thesis) 2011
    203. 203. Inference for AH-CRFGraph constructions Ladický (thesis) 2011
    204. 204. Inference for AH-CRFPairwise potential between auxiliary variables : Ladický (thesis) 2011
    205. 205. Inference for AH-CRFMove energy if x c = xd :Can be transformed to binary submodular pairwise function: Ladický (thesis) 2011
    206. 206. Inference for AH-CRFMove energy if x c ≠ xd :Can be transformed to binary submodular pairwise function: Ladický (thesis) 2011
    207. 207. Inference for AH-CRFGraph constructions Ladický (thesis) 2011
    208. 208. Potential training for Object class Segmentation Pixel unary potentialUnary likelihoods based on spatial configuration (Shotton et al. ECCV06) Classifier trained using boosting Ladický, Russell, Kohli, Torr ICCV09
    209. 209. Potential training for Object class SegmentationSegment unary potentialClassifier trained using boosting (resp. Kernel SVMs) Ladický, Russell, Kohli, Torr ICCV09
    210. 210. Results on MSRC dataset
    211. 211. Results on Corel dataset
    212. 212. Results on VOC 2010 datasetInput Image Result Input Image Result
    213. 213. VOC 2010-QualitativeInput Image Result Input Image Result
    214. 214. CamVid-QualitativeBrostow et al. Our Result Ground TruthInput Image
    215. 215. Quantitative comparison MSRC datasetVOC2009 dataset Corel dataset
    216. 216. Quantitative comparisonMSRC datasetVOC2009 dataset Ladický, Russell, Kohli, Torr ICCV09
    217. 217. Dense Stereo Reconstruction Unary Potential Disparity = 0Unary Cost dependent on the similarity of patches, e.g.cross correlation
    218. 218. Dense Stereo Reconstruction Unary Potential Disparity = 5Unary Cost dependent on the similarity of patches, e.g.cross correlation
    219. 219. Dense Stereo Reconstruction Unary Potential Disparity = 10Unary Cost dependent on the similarity of patches, e.g.cross correlation
    220. 220. Dense Stereo Reconstruction Pairwise Potential• Encourages label consistency in adjacent pixels• Cost based on the distance of labels Linear Truncated Quadratic Truncated
    221. 221. Dense Stereo ReconstructionDoes not work for Road Scenes !Original Image Dense Stereo Reconstruction
    222. 222. Dense Stereo ReconstructionDoes not work for Road Scenes !Different brightness Patches can be matched to any in cameras other patch for flat surfices
    223. 223. Dense Stereo Reconstruction Does not work for Road Scenes ! Different brightness Patches can be matched to any in cameras other patch for flat surficesCould object recognition for road scenes help? Recognition of road scenes is relatively easy
    224. 224. Joint Dense Stereo Reconstruction and Object class Segmentation• Object class and 3D location are mutually informative ∞ • Sky always in infinity (disparity = 0) sky
    225. 225. Joint Dense Stereo Reconstruction and Object class Segmentation• Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buses & pedestrians have their typical height sky building car
    226. 226. Joint Dense Stereo Reconstruction and Object class Segmentation• Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buses & pedestrians have their typical height • Road and pavement on the ground plane sky building car road
    227. 227. Joint Dense Stereo Reconstruction and Object class Segmentation• Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buses & pedestrians have their typical height • Road and pavement on the ground plane • Buildings and pavement on the sides sky building car road
    228. 228. Joint Dense Stereo Reconstruction and Object class Segmentation• Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buses & pedestrians have their typical height • Road and pavement on the ground plane • Buildings and pavement on the sides• Both problems formulated as CRF • Joint approach possible? sky building car road
    229. 229. Joint Formulation• Each pixels takes label zi = [ xi yi ] ∈ L1 × L2• Dependency of xi and yi encoded as a unary and pairwise potential, e.g. • strong correlation between x = road, y = near ground plane • strong correlation between x = sky, y = 0 • Correlation of edge in object class and disparity domain
    230. 230. Joint Formulation Unary Potential Object layer Joint unary links Disparity layer• Weighted sum of object class, depth and joint potential• Joint unary potential based on histograms of height
    231. 231. Joint Formulation Pairwise Potential Object layer Joint pairwise links Disparity layer• Object class and depth edges correlated• Transitions in depth occur often at the object boundaries
    232. 232. Joint Formulation
    233. 233. Inference• Standard α-expansion • Each node in each expansion move keeps its old label or takes a new label [xL1, yL2], • Possible in case of metric pairwise potentials
    234. 234. Inference• Standard α-expansion • Each node in each expansion move keeps its old label or takes a new label [xL1, yL2], • Possible in case of metric pairwise potentials Too many moves! ( |L1| |L2| ) Impractical !

    ×