CVPR2012: Tutorial: Graphcut-based Optimisation for Computer Vision
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,582
On Slideshare
2,582
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
102
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Many different approaches have been proposed for dealing with co-occurrence . In order to justify our approach, I'll run through some basic criteria that we think a co-occurrence potential should satisfy. Firstly, text...
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • Read slide Discuss how tenis courts make it less likely for lemons to appear in the image.
  • There are 3 existing approaches to co-occurrence. Describe None Of them, exhibit invariance, and there is a direct trade off between the use of a global or dynamic cost function, that adapts to the current labelling and computational efficiency.
  • There are 3 existing approaches to co-occurrence. Describe None Of them, exhibit invariance, and there is a direct trade off between the use of a global or dynamic cost function, that adapts to the current labelling and computational efficiency.
  • There are 3 existing approaches to co-occurrence. Describe None Of them, exhibit invariance, and there is a direct trade off between the use of a global or dynamic cost function, that adapts to the current labelling and computational efficiency.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.
  • Qualitative results on the MSRC challenge . These examples show co-occurrence potentials performing as expected both removing small classes and prohibiting unlikely combinations of class such as cow and sheep from occurring together.

Transcript

  • 1. GraphCut-based Optimisation for Computer Vision Ľubor Ladický
  • 2. Overview• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm• Markov and Conditional Random Fields• Random Field Optimisation using Graph Cuts • Submodular vs. Non-Submodular Problems • Pairwise vs. Higher Order Problems • 2-Label vs. Multi-Label Problems• Recent Advances in Random Field Optimisation• Conclusions
  • 3. Overview• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm• Markov and Conditional Random Fields• Random Field Optimisation using Graph Cuts • Submodular vs. Non-Submodular Problems • Pairwise vs. Higher Order Problems • 2-Label vs. Multi-Label Problems• Recent Advances in Random Field Optimisation• Conclusions
  • 4. Image Labelling Problems Assign a label to each image pixelGeometry Estimation Image Denoising Object Segmentation Depth Estimation Sky Building Tree Grass
  • 5. Image Labelling Problems• Labellings highly structured Possible labelling Unprobable labelling Impossible labelling
  • 6. Image Labelling Problems• Labellings highly structured• Labels highly correlated with very complex dependencies • Neighbouring pixels tend to take the same label • Low number of connected components • Classes present may be seen in one image • Geometric / Location consistency • Planarity in depth estimation • … many others (task dependent)
  • 7. Image Labelling Problems• Labelling highly structured• Labels highly correlated with very complex dependencies• Independent label estimation too hard
  • 8. Image Labelling Problems• Labelling highly structured• Labels highly correlated with very complex dependencies• Independent label estimation too hard• Whole labelling should be formulated as one optimisation problem
  • 9. Image Labelling Problems• Labelling highly structured• Labels highly correlated with very complex dependencies• Independent label estimation too hard• Whole labelling should be formulated as one optimisation problem• Number of pixels up to millions • Hard to train complex dependencies • Optimisation problem is hard to infer
  • 10. Image Labelling Problems• Labelling highly structured• Labels highly correlated with very complex dependencies• Independent label estimation too hard• Whole labelling should be formulated as one optimisation problem• Number of pixels up to millions • Hard to train complex dependencies • Optimisation problem is hard to infer Vision is hard !
  • 11. Image Labelling Problems• You can either • Change the subject from the Computer Vision to the History of Renaissance Art Vision is hard !
  • 12. Image Labelling Problems• You can either • Change the subject from the Computer Vision to the History of Renaissance Art or • Learn everything about Random Fields and Graph Cuts Vision is hard !
  • 13. Foreground / Background Estimation Rother et al. SIGGRAPH04
  • 14. Foreground / Background Estimation Data term Smoothness termData term Estimated using FG / BG colour modelsSmoothness term where Intensity dependent smoothness
  • 15. Foreground / Background EstimationData term Smoothness term
  • 16. Foreground / Background Estimation Data term Smoothness termHow to solve this optimisation problem?
  • 17. Foreground / Background Estimation Data term Smoothness term How to solve this optimisation problem?• Transform into min-cut / max-flow problem• Solve it using min-cut / max-flow algorithm
  • 18. Overview• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm• Markov and Conditional Random Fields• Random Field Optimisation using Graph Cuts • Submodular vs. Non-Submodular Problems • Pairwise vs. Higher Order Problems • 2-Label vs. Multi-Label Problems• Applications• Conclusions
  • 19. Max-Flow Problem source 5 9 4 2 2 3 5 2 3 1 1 6 5 3 2 6 8 5Task : sink Maximize the flow from the sink to the source such that 1) The flow it conserved for each node 2) The flow for each pipe does not exceed the capacity
  • 20. Max-Flow Problem source 5 9 4 2 2 3 52 3 1 1 6 5 3 2 6 8 5 sink
  • 21. Max-Flow Problem source flow from the flow from node i source 5 9 to node j capacity 4 2 2 set of edges 3 52 3 1 1 6 5 3 2 conservation of flow set of nodes 6 8 5 sink
  • 22. Max-Flow Problem source 9 Ford & Fulkerson algorithm (1956) 5 4 2 2 Find the path from source to sink 3 52 3 While (path exists) 1 1 6 5 flow += maximum capacity in the path 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 8 5 End sink
  • 23. Max-Flow Problem source 9 Ford & Fulkerson algorithm (1956) 5 4 2 2 Find the path from source to sink 3 52 3 While (path exists) 1 1 6 5 flow += maximum capacity in the path 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 8 5 End sink
  • 24. Max-Flow Problem source 9 Ford & Fulkerson algorithm (1956) 5 4 2 2 Find the path from source to sink 3 52 3 While (path exists) 1 1 6 5 flow += maximum capacity in the path 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 8 5 End sink flow = 3
  • 25. Max-Flow Problem source 9 Ford & Fulkerson algorithm (1956) 5-3 4 2+ 2 Find the path from source to sink 3 5 3-32 3 While (path exists) 1 6-3 1 flow += maximum capacity in the path +3 5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 8-3 5 End sink flow = 3
  • 26. Max-Flow Problem source 9 Ford & Fulkerson algorithm (1956) 2 4 5 2 Find the path from source to sink 52 3 While (path exists) 1 1 3 flow += maximum capacity in the path 3 5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 5 5 End sink flow = 3
  • 27. Max-Flow Problem source 9 Ford & Fulkerson algorithm (1956) 2 4 5 2 Find the path from source to sink 52 3 While (path exists) 1 1 3 flow += maximum capacity in the path 3 5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 5 5 End sink flow = 3
  • 28. Max-Flow Problem source 9 Ford & Fulkerson algorithm (1956) 2 4 5 2 Find the path from source to sink 52 3 While (path exists) 1 1 3 flow += maximum capacity in the path 3 5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 5 5 End sink flow = 6
  • 29. Max-Flow Problem source 9-3 Ford & Fulkerson algorithm (1956) 2 4 5 2 Find the path from source to sink 52 +3 3-3 While (path exists) 1 1 3 5 flow += maximum capacity in the path 3 3-3 Build the residual graph (“subtract” the flow) 2 +3 Find the path in the residual graph 6 5-3 5 End sink flow = 6
  • 30. Max-Flow Problem source 6 Ford & Fulkerson algorithm (1956) 2 4 5 2 Find the path from source to sink 52 3 While (path exists) 1 1 3 flow += maximum capacity in the path 3 5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 2 5 End sink flow = 6
  • 31. Max-Flow Problem source 6 Ford & Fulkerson algorithm (1956) 2 4 5 2 Find the path from source to sink 52 3 While (path exists) 1 1 3 flow += maximum capacity in the path 3 5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 2 5 End sink flow = 6
  • 32. Max-Flow Problem source 6 Ford & Fulkerson algorithm (1956) 2 4 5 2 Find the path from source to sink 52 3 While (path exists) 1 1 3 flow += maximum capacity in the path 3 5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 2 5 End sink flow = 11
  • 33. Max-Flow Problem source 6-5 Ford & Fulkerson algorithm (1956) 2 4 5 2+5 Find the path from source to sink 5-52 3 While (path exists) 1 1+5 3 flow += maximum capacity in the path 3 5-5 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 2 5-5 End sink flow = 11
  • 34. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 2 4 5 Find the path from source to sink 72 3 While (path exists) 1 6 3 flow += maximum capacity in the path 3 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 2 End sink flow = 11
  • 35. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 2 4 5 Find the path from source to sink 72 3 While (path exists) 1 6 3 flow += maximum capacity in the path 3 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 2 End sink flow = 11
  • 36. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 2 4 5 Find the path from source to sink 72 3 While (path exists) 1 6 3 flow += maximum capacity in the path 3 3 Build the residual graph (“subtract” the flow) 2 Find the path in the residual graph 6 2 End sink flow = 13
  • 37. Max-Flow Problem source 2-2 1 Ford & Fulkerson algorithm (1956) 4 5 Find the path from source to sink 72-2 While (path exists) +2 3 1 6 3 flow += maximum capacity in the path +2 3 3 Build the residual graph (“subtract” the flow) 2-2 Find the path in the residual graph 6 2-2 End sink flow = 13
  • 38. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 4 5 Find the path from source to sink 72 3 While (path exists) 1 6 3 flow += maximum capacity in the path 2 3 3 Build the residual graph (“subtract” the flow) Find the path in the residual graph 6 End sink flow = 13
  • 39. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 4 5 Find the path from source to sink 72 3 While (path exists) 1 6 3 flow += maximum capacity in the path 2 3 3 Build the residual graph (“subtract” the flow) Find the path in the residual graph 6 End sink flow = 13
  • 40. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 4 5 Find the path from source to sink 72 3 While (path exists) 1 6 3 flow += maximum capacity in the path 2 3 3 Build the residual graph (“subtract” the flow) Find the path in the residual graph 6 End sink flow = 15
  • 41. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 4-2 5 Find the path from source to sink 72 3 While (path exists) 1 3-2 6 flow += maximum capacity in the path 2-2 3+ 32 Build the residual graph (“subtract” the flow) +2 Find the path in the residual graph 6-2 End sink flow = 15
  • 42. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 2 5 Find the path from source to sink 72 3 While (path exists) 1 6 1 flow += maximum capacity in the path 5 3 Build the residual graph (“subtract” the flow) 2 4 Find the path in the residual graph End sink flow = 15
  • 43. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 2 5 Find the path from source to sink 72 3 While (path exists) 1 6 1 flow += maximum capacity in the path 5 3 Build the residual graph (“subtract” the flow) 2 4 Find the path in the residual graph End sink flow = 15
  • 44. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 2 5 Find the path from source to sink 72 3 While (path exists) 1 6 1 flow += maximum capacity in the path 5 3 Build the residual graph (“subtract” the flow) 2 4 Find the path in the residual graph End sink flow = 15
  • 45. Max-Flow Problem source 1 Ford & Fulkerson algorithm (1956) 2 5 Why is the solution globally optimal ? 72 3 1 6 1 5 2 3 4 sink flow = 15
  • 46. Max-Flow ProblemS source 1 Ford & Fulkerson algorithm (1956) 2 5 Why is the solution globally optimal ? 72 3 1 6 1 1. Let S be the set of reachable nodes in the 5 3 residual graph 2 4 sink flow = 15
  • 47. Max-Flow ProblemS5 source 9 Ford & Fulkerson algorithm (1956) 4 2 2 Why is the solution globally optimal ? 3 52 3 1 1 6 5 1. Let S be the set of reachable nodes in the 3 residual graph 2 2. The flow from S to V - S equals to the sum 6 8 5 of capacities from S to V – S sink
  • 48. Max-Flow ProblemA5 source 9 Ford & Fulkerson algorithm (1956) 4 2 2 Why is the solution globally optimal ? 3 52 3 1 1 6 5 1. Let S be the set of reachable nodes in the residual graph 3 2 2. The flow from S to V - S equals to the sum of 6 5 capacities from S to V – S 8 3. The flow from any A to V - A is upper bounded sink by the sum of capacities from A to V – A
  • 49. Max-Flow Problem source 5/5 8/9 Ford & Fulkerson algorithm (1956) 2/4 0/2 0/2 Why is the solution globally optimal ? 5/5 2/2 3/3 0/1 0/1 3/3 5/6 5/5 1. Let S be the set of reachable nodes in the residual graph 3/3 0/2 2. The flow from S to V - S equals to the sum of 2/6 5/5 capacities from S to V – S 8/8 3. The flow from any A to V - A is upper bounded sink by the sum of capacities from A to V – A 4. The solution is globally optimal flow = 15Individual flows obtained by summing up all paths
  • 50. Max-Flow Problem source 5/5 8/9 Ford & Fulkerson algorithm (1956) 2/4 0/2 0/2 Why is the solution globally optimal ? 3/3 5/52/2 0/1 0/1 3/3 5/6 5/5 1. Let S be the set of reachable nodes in the residual graph 3/3 0/2 2. The flow from S to V - S equals to the sum of 2/6 5/5 capacities from S to V – S 8/8 3. The flow from any A to V - A is upper bounded sink by the sum of capacities from A to V – A 4. The solution is globally optimal flow = 15
  • 51. Max-Flow Problem source Ford & Fulkerson algorithm (1956)1000 1000 Order does matter 11000 1000 sink
  • 52. Max-Flow Problem source Ford & Fulkerson algorithm (1956)1000 1000 Order does matter 11000 1000 sink
  • 53. Max-Flow Problem source Ford & Fulkerson algorithm (1956)1000 999 Order does matter 1999 1000 sink
  • 54. Max-Flow Problem source Ford & Fulkerson algorithm (1956)1000 999 Order does matter 1999 1000 sink
  • 55. Max-Flow Problem source Ford & Fulkerson algorithm (1956)999 999 Order does matter 1999 999 sink
  • 56. Max-Flow Problem source Ford & Fulkerson algorithm (1956)999 999 Order does matter • Standard algorithm not polynomial 1 • Breath first leads to O(VE2) • Path found in O(E)999 999 • At least one edge gets saturated • The saturated edge distance to the sink source has to increase and is at most V leading to O(VE) (Edmonds & Karp, Dinic)
  • 57. Max-Flow Problem source Ford & Fulkerson algorithm (1956)999 999 Order does matter • Standard algorithm not polynomial 1 • Breath first leads to O(VE2) (Edmonds & Karp)999 999 • Various methods use different algorithm to find the path and vary in complexity sink
  • 58. Min-Cut Problem source 5 9 4 2 2 3 5 2 3 1 1 6 5 3 2 6 8 5Task : sink Minimize the cost of the cut 1) Each node is either assigned to the source S or sink T 2) The cost of the edge (i, j) is taken if (i∈S) and (j∈T)
  • 59. Min-Cut Problem source 5 9 4 2 2 3 52 3 1 1 6 5 3 2 6 8 5 sink
  • 60. Min-Cut Problem source 5 9 edge costs 4 2 2 3 52 3 1 1 6 5 source set sink set 3 2 6 8 5 sink
  • 61. Min-Cut ProblemS5 source 9 edge costs 4 2 2 3 5 cost = 182 3 1 1 6 5 source set sink set 3 2 6 8 5 sink T
  • 62. Min-Cut ProblemS5 source 9 cost = 25 edge costs 4 2 2 3 52 3 1 1 6 5 source set sink set 3 2 6 8 5 sink T
  • 63. Min-Cut ProblemS5 source 9 edge costs 4 2 2 3 5 cost = 232 3 1 1 6 5 source set sink set 3 2 6 8 5 sink T
  • 64. Min-Cut Problem source 5 9 x1 4 x2 2 2 3 52 x3 3 1 1 x4 6 5 x6 3 2 x5 6 8 5 sink
  • 65. Min-Cut Problem source 5 9 x1 4 x2 2 2 3 52 x3 transformable into Linear program (LP) 3 1 1 x4 6 5 x6 3 2 x5 6 8 5 sink
  • 66. Min-Cut Problem source 5 9 x1 4 x2 2 2 3 52 x3 transformable into Linear program (LP) 3 1 1 x4 6 5 x6 3 2 x5 6 8 5 sink Dual to max-flow problem
  • 67. Min-Cut Problem source After the substitution of the constraints : 5 9 x1 4 x2 2 2 3 52 x3 3 1 1 x4 6 5 x6 3 2 x5 6 8 5 sink
  • 68. Min-Cut Problem source edges sink edges source 5 9 x1 4 x2 2 2 3 52 x3 3 1 1 x4 6 5 x6 3 2 x5 xi = 0 ⇒ xi ∈ S xi = 1 ⇒ xi ∈ T 6 8 5 sink Edges between variables
  • 69. Min-Cut Problem source C(x) = 5x1 + 9x2 + 4x3 + 3x3(1-x1) + 2x1(1-x3) 5 9 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 6x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 2 2 3 5 + 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)2 x3 3 1 1 + 8(1-x5) + 5(1-x6) x4 6 5 x6 3 2 x5 6 8 5 sink
  • 70. Min-Cut Problem source C(x) = 5x1 + 9x2 + 4x3 + 3x3(1-x1) + 2x1(1-x3) 5 9 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 6x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 2 2 3 5 + 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)2 x3 3 1 1 + 8(1-x5) + 5(1-x6) x4 6 5 x6 3 2 x5 6 8 5 sink
  • 71. Min-Cut Problem source C(x) = 2x1 + 9x2 + 4x3 + 2x1(1-x3) 5 9 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 2 2 3 5 + 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)2 x3 3 1 1 + 5(1-x5) + 5(1-x6) x4 6 5 x6 + 3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5) 3 2 x5 6 8 5 sink
  • 72. Min-Cut Problem source C(x) = 2x1 + 9x2 + 4x3 + 2x1(1-x3) 5 9 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 2 2 3 5 + 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)2 x3 3 1 1 + 5(1-x5) + 5(1-x6) x4 6 5 x6 + 3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5) 3 2 x5 6 8 5 3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5) sink = 3 + 3x1(1-x3) + 3x3(1-x5)
  • 73. Min-Cut Problem source C(x) = 3 + 2x1 + 9x2 + 4x3 + 5x1(1-x3) 2 9 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 5 2 5 + 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)2 x3 3 1 1 + 5(1-x5) + 5(1-x6) + 3x3(1-x5) 3 x4 3 5 x6 3 2 x5 6 5 5 sink
  • 74. Min-Cut Problem source C(x) = 3 + 2x1 + 9x2 + 4x3 + 5x1(1-x3) 2 9 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 5 2 5 + 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)2 x3 3 1 1 + 5(1-x5) + 5(1-x6) + 3x3(1-x5) 3 x4 3 5 x6 3 2 x5 6 5 5 sink
  • 75. Min-Cut Problem source C(x) = 3 + 2x1 + 6x2 + 4x3 + 5x1(1-x3) 2 9 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 5 2 5 + 2x5(1-x4) + 6(1-x4)2 x3 3 1 1 + 2(1-x5) + 5(1-x6) + 3x3(1-x5) 3 x4 3 5 x6 + 3x2 + 3x6(1-x2) + 3x5(1-x6) + 3(1-x5) 3 2 x5 6 5 5 3x2 + 3x6(1-x2) + 3x5(1-x6) + 3(1-x5) sink = 3 + 3x2(1-x6) + 3x6(1-x5)
  • 76. Min-Cut Problem source C(x) = 6 + 2x1 + 6x2 + 4x3 + 5x1(1-x3) 2 6 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 5 2 5 + 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)2 x3 3 1 1 + 2(1-x5) + 5(1-x6) + 3x3(1-x5) 3 x4 3 5 x6 3 2 x5 6 2 5 sink
  • 77. Min-Cut Problem source C(x) = 6 + 2x1 + 6x2 + 4x3 + 5x1(1-x3) 2 6 + 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6) 5 2 5 + 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)2 x3 3 1 1 + 2(1-x5) + 5(1-x6) + 3x3(1-x5) 3 x4 3 5 x6 3 2 x5 6 2 5 sink
  • 78. Min-Cut Problem source C(x) = 11 + 2x1 + 1x2 + 4x3 + 5x1(1-x3) 2 1 + 3x3(1-x1) + 7x2(1-x3) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 6x3(1-x6) 5 7 + 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)2 x3 3 1 6 + 2(1-x5) + 3x3(1-x5) 3 x4 3 x6 3 2 x5 6 2 sink
  • 79. Min-Cut Problem source C(x) = 11 + 2x1 + 1x2 + 4x3 + 5x1(1-x3) 2 1 + 3x3(1-x1) + 7x2(1-x3) + 2x4(1-x1) x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 6x3(1-x6) 5 7 + 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)2 x3 3 1 6 + 2(1-x5) + 3x3(1-x5) 3 x4 3 x6 3 2 x5 6 2 sink
  • 80. Min-Cut Problem source C(x) = 13 + 1x2 + 4x3 + 5x1(1-x3) 1 + 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 6x3(1-x6) 5 7 + 2x4(1-x5) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5) x3 32 1 6 + 3x3(1-x5) 3x4 2 3 x6 3 x5 6 sink
  • 81. Min-Cut Problem source C(x) = 13 + 1x2 + 2x3 + 5x1(1-x3) 1 + 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)x1 4 x2 + 1x5(1-x1) + 3x5(1-x3) + 6x3(1-x6) 5 7 + 2x4(1-x5) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5) x3 32 1 6 + 3x3(1-x5) 3x4 2 3 x6 3 x5 6 sink
  • 82. Min-Cut Problem source C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3) 1 + 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)x1 2 x2 + 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5) 5 7 + 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5) x3 32 1 6 1x4 5 x6 2 3 x5 4 sink
  • 83. Min-Cut Problem source C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3) 1 + 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)x1 2 x2 + 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5) 5 7 + 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5) x3 32 1 6 1x4 5 x6 2 3 x5 4 sink
  • 84. Min-Cut Problem min cut C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3) S source 1 + 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4) x1 2 x2 + 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5) 5 7 + 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5) x3 3 2 1 6 1 cost = 0 x4 5 x6 cost = 0 2 3 x5 4 • All coefficients positive sink T • Must be global minimumS – set of reachable nodes from s
  • 85. Min-Cut Problem min cut C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3) S source 1 + 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4) x1 2 x2 + 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5) 5 7 + 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5) x3 3 2 1 6 1 x4 5 x6 cost = 0 2 3 x5 4 • All coefficients positive sink T • Must be global minimumT – set of nodes that can reach t(not necesarly the same)
  • 86. Overview• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm• Markov and Conditional Random Fields• Random Field Optimisation using Graph Cuts • Submodular vs. Non-Submodular Problems • Pairwise vs. Higher Order Problems • 2-Label vs. Multi-Label Problems• Recent Advances in Random Field Optimisation• Conclusions
  • 87. Markov and Conditional RF• Markov / Conditional Random fields model probabilistic dependencies of the set of random variables
  • 88. Markov and Conditional RF• Markov / Conditional Random fields model conditional dependencies between random variables• Each variable is conditionally independent of all other variables given its neighbours
  • 89. Markov and Conditional RF• Markov / Conditional Random fields model conditional dependencies between random variables• Each variable is conditionally independent of all other variables given its neighbours• Posterior probability of the labelling x given data D is : partition function cliques potential functions
  • 90. Markov and Conditional RF• Markov / Conditional Random fields model conditional dependencies between random variables• Each variable is conditionally independent of all other variables given its neighbours• Posterior probability of the labelling x given data D is : partition function cliques potential functions• Energy of the labelling is defined as :
  • 91. Markov and Conditional RF• The most probable (Max a Posteriori (MAP)) labelling is defined as:
  • 92. Markov and Conditional RF• The most probable (Max a Posteriori (MAP)) labelling is defined as:• The only distinction (MRF vs. CRF) is that in the CRF the conditional dependencies between variables depend also on the data
  • 93. Markov and Conditional RF• The most probable (Max a Posteriori (MAP)) labelling is defined as:• The only distinction (MRF vs. CRF) is that in the CRF the conditional dependencies between variables depend also on the data• Typically we define an energy first and then pretend there is an underlying probabilistic distribution there, but there isn’t really (Pssssst, don’t tell anyone)
  • 94. Overview• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm• Markov and Conditional Random Fields• Random Field Optimisation using Graph Cuts • Submodular vs. Non-Submodular Problems • Pairwise vs. Higher Order Problems • 2-Label vs. Multi-Label Problems• Recent Advances in Random Field Optimisation• Conclusions
  • 95. Pairwise CRF modelsStandard CRF Energy Data term Smoothness term
  • 96. Graph Cut based InferenceThe energy / potential is called submodular (only) if for everypair of variables :
  • 97. Graph Cut based InferenceFor 2-label problems x∈{0, 1} :
  • 98. Graph Cut based Inference For 2-label problems x∈{0, 1} :Pairwise potential can be transformed into :where
  • 99. Graph Cut based Inference For 2-label problems x∈{0, 1} :Pairwise potential can be transformed into :where
  • 100. Graph Cut based Inference For 2-label problems x∈{0, 1} :Pairwise potential can be transformed into :where
  • 101. Graph Cut based InferenceAfter summing up :
  • 102. Graph Cut based InferenceAfter summing up :Let :
  • 103. Graph Cut based InferenceAfter summing up :Let : Then :
  • 104. Graph Cut based Inference After summing up : Let : Then :Equivalent st-mincut problem is :
  • 105. Foreground / Background Estimation Data term Smoothness termData term Estimated using FG / BG colour modelsSmoothness term where Intensity dependent smoothness
  • 106. Foreground / Background Estimationsourcesink Rother et al. SIGGRAPH04
  • 107. Graph Cut based InferenceExtendable to (some) Multi-label CRFs • each state of multi-label variable encoded using multiple binary variables • the cost of every possible cut must be the same as the associated energy • the solution obtained by inverting the encoding Encoding multi-label energy Build Obtain solution Graph Cut Graph solution
  • 108. Dense Stereo Estimation• For each pixel assigns a disparity label : z• Disparities from the discrete set {0, 1, .. D} Left Camera Image Right Camera Image Dense Stereo Result
  • 109. Dense Stereo EstimationData term Left image Shifted right feature image feature Left Camera Image Right Camera Image
  • 110. Dense Stereo EstimationData term Left image Shifted right feature image featureSmoothness term
  • 111. Dense Stereo Estimation source sour Encoding cexi0 xk0 xk1 xk2 . . . .xiD xkD sink Ishikawa PAMI03
  • 112. Dense Stereo Estimation source sour ∞ ce ∞ Unary Potentialxi0 xk0 ∞ ∞ xk1 xk2 . . ∞ ∞ The cost of every cut should be equal to the corresponding . . energy under the encoding ∞ ∞xiD xkD cost = + sink Ishikawa PAMI03
  • 113. Dense Stereo Estimation source sour ∞ ce ∞ Unary Potentialxi0 xk0 ∞ ∞ xk1 xk2 cost = + . . ∞ ∞ The cost of every cut should be equal to the corresponding . . energy under the encoding ∞ ∞xiD xkD sink Ishikawa PAMI03
  • 114. Dense Stereo Estimation source sour ∞ ce ∞ Unary Potentialxi0 xk0 ∞ ∞ cost = ∞ xk1 xk2 . . ∞ ∞ The cost of every cut should be equal to the corresponding . . energy under the encoding ∞ ∞xiD xkD sink Ishikawa PAMI03
  • 115. Dense Stereo Estimation source sour ∞ ce ∞ Pairwise Potential Kxi0 xk0 ∞ ∞ xk1 xk2 cost = + +K K . . ∞ ∞ The cost of every cut should be K equal to the corresponding . . energy under the encoding ∞ ∞ KxiD xkD sink Ishikawa PAMI03
  • 116. Dense Stereo Estimation source sour ∞ ce ∞ Pairwise Potential Kxi0 xk0 ∞ ∞ xk1 xk2 K . . ∞ ∞ The cost of every cut should be K equal to the corresponding . . energy under the encoding ∞ ∞ KxiD xkD cost = + +DK sink Ishikawa PAMI03
  • 117. Dense Stereo Estimation source sour ce ∞ Extendable to any convex cost ∞xi0 xk0 ∞ ∞ xk1 xk2 Convex function . . ∞ ∞ . . ∞ ∞xiD xkD See [Ishikawa PAMI03] for more details sink
  • 118. Graph Cut based InferenceMove making algorithms• Original problem decomposed into a series of subproblems solvable with graph cut• In each subproblem we find the optimal move from the current solution in a restricted search space EncodingInitial solution Propose Build Update solution Graph Cut move Graph solution
  • 119. Graph Cut based InferenceExample : [Boykov et al. 01] αβ-swap – each variable taking label α or β can change its label to α or β – all αβ-moves are iteratively performed till convergence α-expansion – each variable either keeps its old label or changes to α – all α-moves are iteratively performed till convergence Transformation function :
  • 120. Graph Cut based InferenceSufficient conditions for move making algorithms : (all possible moves are submodular) αβ-swap : semi-metricity Proof:
  • 121. Graph Cut based InferenceSufficient conditions for move making algorithms : (all possible moves are submodular) α-expansion : metricity Proof:
  • 122. Object-class Segmentation Data term Smoothness termData term Discriminatively trained classifierSmoothness term where
  • 123. Object-class Segmentation grassOriginal Image Initial solution
  • 124. Object-class Segmentation grass building grassOriginal Image Initial solution Building expansion
  • 125. Object-class Segmentation grass building grass Original Image Initial solution Building expansion sky buildinggrass Sky expansion
  • 126. Object-class Segmentation grass building grass Original Image Initial solution Building expansion sky sky tree building buildinggrass grass Sky expansion Tree expansion
  • 127. Object-class Segmentation grass building grass Original Image Initial solution Building expansion sky sky sky tree tree building building building aeroplanegrass grass grass Sky expansion Tree expansion Final Solution
  • 128. Graph Cut based InferenceRange-(swap) moves [Veksler 07] – Each variable in the convex range can change its label to any other label in the convex range – all range-moves are iteratively performed till convergenceRange-expansion [Kumar, Veksler & Torr 1] – Expansion version of range swap moves – Each varibale can change its label to any label in a convex range or keep its old label
  • 129. Dense Stereo Reconstruction Left Camera Image Right Camera Image Dense Stereo ResultData term Same as beforeSmoothness term
  • 130. Dense Stereo Reconstruction Left Camera Image Right Camera Image Dense Stereo ResultData term Same as beforeSmoothness term Convex range Truncation
  • 131. Graph Cut based InferenceOriginal Image Initial Solution
  • 132. Graph Cut based InferenceOriginal Image Initial Solution After 1st expansion
  • 133. Graph Cut based Inference Original Image Initial Solution After 1st expansionAfter 2nd expansion
  • 134. Graph Cut based Inference Original Image Initial Solution After 1st expansionAfter 2nd expansion After 3rd expansion
  • 135. Graph Cut based Inference Original Image Initial Solution After 1st expansionAfter 2nd expansion After 3rd expansion Final solution
  • 136. Graph Cut based Inference• Extendible to certain classes of binary higher order energies• Higher order terms have to be transformable to the pairwise submodular energy with binary auxiliary variables zC Higher order term Pairwise term EncodingInitial solution Propose Transf. to Update solution Graph Cut move pairwise subm. solution
  • 137. Graph Cut based Inference• Extendible to certain classes of binary higher order energies• Higher order terms have to be transformable to the pairwise submodular energy with binary auxiliary variables zC Higher order term Pairwise termExample : Kolmogorov ECCV06, Ramalingam et al. DAM12
  • 138. Graph Cut based InferenceEnforces label consistency of the clique as a weak constraint Input image Pairwise CRF Higher order CRF Kohli et al. CVPR07
  • 139. Graph Cut based Inference If the energy is not submodular / graph-representable – Overestimation by a submodular energy – Quadratic Pseudo-Boolean Optimisation (QPBO) EncodingInitial solution solution Propose Transf. to Update Graph Cut move pairwise subm. solution
  • 140. Graph Cut based Inference Overestimation by a submodular energy (convex / concave) – We find an energy E’(t) s.t. – it is tight in the current solution E’(t0) = E(t0) – Overestimates E(t) for all t E’(t) ≥ E(t) – We replace E(t) by E’(t) – The moves are not optimal, but quaranteed to converge – The tighter over-estimation, the better Example : EncodingInitial solution Propose Overestim. by Update solution Graph Cut move pairwise subm. solution
  • 141. Graph Cut based InferenceQuadratic Pseudo-Boolean Optimisation– Each binary variable is encoded using two binary _ _ variables xi and xi s.t. xi = 1 - xi
  • 142. Graph Cut based Inference Quadratic Pseudo-Boolean Optimisation sourcexi xj_ _ _xi xj – Energy submodular in xi and xi _ – Solved by dropping the constraint xi = 1 - xi _ – If xi = 1 - xi the solution for xi is optimal sink
  • 143. Overview• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm• Markov and Conditional Random Fields• Random Field Optimisation using Graph Cuts • Submodular vs. Non-Submodular Problems • Pairwise vs. Higher Order Problems • 2-Label vs. Multi-Label Problems• Recent Advances in Random Field Optimisation• Conclusions
  • 144. Motivation• To propose formulations enforcing certain structure properties of the output useful in computer vision – Label Consistency in segments as a weak constraint – Label agreement over different scales – Label-set consistency – Multiple domains consistency• To propose feasible Graph-Cut based inference method
  • 145. Structures in CRF• Taskar et al. 02 – associative potentials• Woodford et al. 08 – planarity constraint• Vicente et al. 08 – connectivity constraint• Nowozin & Lampert 09 – connectivity constraint• Roth & Black 09 – field of experts• Woodford et al. 09 – marginal probability• Delong et al. 10 – label occurrence costs
  • 146. Object-class Segmentation Problem• Aims to assign a class label for each pixel of an image• Classifier trained on the training set• Evaluated on never seen test images
  • 147. Pairwise CRF modelsStandard CRF Energy for Object Segmentation Local context Cannot encode global consistency of labels!! Ladický, Russell, Kohli, Torr ECCV10
  • 148. Encoding Co-occurrence [Heitz et al. 08]Co-occurrence is a powerful cue [Rabinovich et al. ‘07] • Thing – Thing • Stuff - Stuff • Stuff - Thing[ Images from Rabinovich et al. 07 ] Ladický, Russell, Kohli, Torr ECCV10
  • 149. Encoding Co-occurrence [Heitz et al. 08] Co-occurrence is a powerful cue [Rabinovich et al. ‘07] • Thing – Thing • Stuff - Stuff • Stuff - ThingProposed solutions :1. Csurka et al. 08 - Hard decision for label estimation2. Torralba et al. 03 - GIST based unary potential3. Rabinovich et al. 07 - Full-connected CRF [ Images from Rabinovich et al. 07 ] Ladický, Russell, Kohli, Torr ECCV10
  • 150. So...What properties should these global co-occurence potentials have ? Ladický, Russell, Kohli, Torr ECCV10
  • 151. Desired properties1. No hard decisions Ladický, Russell, Kohli, Torr ECCV10
  • 152. Desired properties1. No hard decisions Incorporation in probabilistic framework Unlikely possibilities are not completely ruled out Ladický, Russell, Kohli, Torr ECCV10
  • 153. Desired properties1. No hard decisions2. Invariance to region size Ladický, Russell, Kohli, Torr ECCV10
  • 154. Desired properties1. No hard decisions2. Invariance to region size Cost for occurrence of {people, house, road etc .. } invariant to image area Ladický, Russell, Kohli, Torr ECCV10
  • 155. Desired properties1. No hard decisions2. Invariance to region sizeThe only possible solution : L(x)={ , , } Local context Global context Cost defined over the assigned labels L(x) Ladický, Russell, Kohli, Torr ECCV10
  • 156. Desired properties1. No hard decisions2. Invariance to region size3. Parsimony – simple solutions preferred L(x)={ aeroplane, tree, flower, building, boat, grass, sky }   L(x)={ building, tree, grass, sky } Ladický, Russell, Kohli, Torr ECCV10
  • 157. Desired properties1. No hard decisions2. Invariance to region size3. Parsimony – simple solutions preferred4. Efficiency Ladický, Russell, Kohli, Torr ECCV10
  • 158. Desired properties1. No hard decisions2. Invariance to region size3. Parsimony – simple solutions preferred4. Efficiency a) Memory requirements as O(n) with the image size and number or labels b) Inference tractable Ladický, Russell, Kohli, Torr ECCV10
  • 159. Previous work• Torralba et al.(2003) – Gist-based unary potentials• Rabinovich et al.(2007) - complete pairwise graphs• Csurka et al.(2008) - hard estimation of labels present Ladický, Russell, Kohli, Torr ECCV10
  • 160. Related work• Zhu & Yuille 1996 – MDL prior• Bleyer et al. 2010 – Surface Stereo MDL prior• Hoiem et al. 2007 – 3D Layout CRF MDL Prior C(x) = K |L(x)|• Delong et al. 2010 – label occurence cost C(x) = ΣLKLδL(x) Ladický, Russell, Kohli, Torr ECCV10
  • 161. Related work• Zhu & Yuille 1996 – MDL prior• Bleyer et al. 2010 – Surface Stereo MDL prior• Hoiem et al. 2007 – 3D Layout CRF MDL Prior C(x) = K |L(x)|• Delong et al. 2010 – label occurence cost C(x) = ΣLKLδL(x) All special cases of our model Ladický, Russell, Kohli, Torr ECCV10
  • 162. Inference for Co-occurenceCo-occurence representationLabel indicator functions Ladický, Russell, Kohli, Torr ECCV10
  • 163. Inference for Co-occurence Cost of currentMove Energy label set Ladický, Russell, Kohli, Torr ECCV10
  • 164. Inference for Co-occurenceMove EnergyDecomposition to α-dependent and α-independent part α-independent α-dependent Ladický, Russell, Kohli, Torr ECCV10
  • 165. Inference for Co-occurenceMove EnergyDecomposition to α-dependent and α-independent partEither α or all labels in the image after the move Ladický, Russell, Kohli, Torr ECCV10
  • 166. Inference for Co-occurenceMove Energy submodular non-submodular Ladický, Russell, Kohli, Torr ECCV10
  • 167. Inference for Co-occurenceMove Energy non-submodularNon-submodular energy overestimated by E(t) • E(t) = E(t) for current solution • E(t) ≥ E(t) for any other labelling Ladický, Russell, Kohli, Torr ECCV10
  • 168. Inference for Co-occurenceMove Energy non-submodularNon-submodular energy overestimated by E(t) • E(t) = E(t) for current solution • E(t) ≥ E(t) for any other labellingOccurrence - tight Ladický, Russell, Kohli, Torr ECCV10
  • 169. Inference for Co-occurenceMove Energy non-submodularNon-submodular energy overestimated by E(t) • E(t) = E(t) for current solution • E(t) ≥ E(t) for any other labellingCo-occurrence overestimation Ladický, Russell, Kohli, Torr ECCV10
  • 170. Inference for Co-occurenceMove Energy non-submodularNon-submodular energy overestimated by E(t) • E(t) = E(t) for current solution • E(t) ≥ E(t) for any other labellingGeneral case [See the paper] Ladický, Russell, Kohli, Torr ECCV10
  • 171. Inference for Co-occurenceMove Energy non-submodularNon-submodular energy overestimated by E(t) • E(t) = E(t) for current solution • E(t) ≥ E(t) for any other labellingQuadratic representation Ladický, Russell, Kohli, Torr ECCV10
  • 172. Potential Training for Co-occurenceGeneratively trained Co-occurence potential Approximated by 2nd order representation Ladický, Russell, Kohli, Torr ICCV09
  • 173. Results on MSRC data set Comparisons of results with and without co-occurenceInput Image Without With Input Image Without With Co-occurence Co-occurence Co-occurence Co-occurence Ladický, Russell, Kohli, Torr ICCV09
  • 174. Pairwise CRF modelsPixel CRF Energy for Object-class Segmentation Data term Smoothness term Input Image Pairwise CRF Result Shotton et al. ECCV06
  • 175. Pairwise CRF modelsPixel CRF Energy for Object-class Segmentation Data term Smoothness term • Lacks long range interactions • Results oversmoothed • Data term information may be insufficient
  • 176. Pairwise CRF modelsSegment CRF Energy for Object-class Segmentation Data term Smoothness term Input Image Unsupervised Shi, Malik PAMI2000, Comaniciu, Meer PAMI2002, Segmentation Felzenschwalb, Huttenlocher, IJCV2004,
  • 177. Pairwise CRF modelsSegment CRF Energy for Object-class Segmentation Data term Smoothness term Input Image Pairwise CRF Result Batra et al. CVPR08, Yang et al. CVPR07, Zitnick et al. CVPR08, Rabinovich et al. ICCV07, Boix et al. CVPR10
  • 178. Pairwise CRF modelsSegment CRF Energy for Object-class Segmentation Data term Smoothness term • Allows long range interactions • Does not distinguish between instances of objects • Cannot recover from incorrect segmentation
  • 179. Pairwise CRF modelsSegment CRF Energy for Object-class Segmentation Data term Smoothness term • Allows long range interactions • Does not distinguish between instances of objects • Cannot recover from incorrect segmentation
  • 180. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency term Input Image Higher order CRF Kohli, Ladický, Torr CVPR08
  • 181. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency term Kohli, Ladický, Torr CVPR08
  • 182. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency term No dominant label Kohli, Ladický, Torr CVPR08
  • 183. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency term Dominant label l Kohli, Ladický, Torr CVPR08
  • 184. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency term • Enforces consistency as a weak constraint • Can recover from incorrect segmentation • Can combine multiple segmentations • Does not use features at segment level Kohli, Ladický, Torr CVPR08
  • 185. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency term Transformable to pairwise graph with auxiliary variable Kohli, Ladický, Torr CVPR08
  • 186. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency termwhere Ladický, Russell, Kohli, Torr ICCV09
  • 187. Robust PN approachSegment CRF Energy for Object-class Segmentation Pairwise CRF Segment consistency term Input Image Higher order CRF Ladický, Russell, Kohli, Torr ICCV09
  • 188. Associative Hierarchical CRFs Can be generalized to Hierarchical model• Allows unary potentials for region variables• Allows pairwise potentials for region variables• Allows multiple layers and multiple hierarchies Ladický, Russell, Kohli, Torr ICCV09
  • 189. Associative Hierarchical CRFsAH CRF Energy for Object-class Segmentation Pairwise CRF Higher order term Ladický, Russell, Kohli, Torr ICCV09
  • 190. Associative Hierarchical CRFsAH CRF Energy for Object-class Segmentation Pairwise CRF Higher order termHigher order term recursively defined as Segment Segment Segment higher unary term pairwise term order term Ladický, Russell, Kohli, Torr ICCV09
  • 191. Associative Hierarchical CRFsAH CRF Energy for Object-class Segmentation Pairwise CRF Higher order termHigher order term recursively defined as Segment Segment Segment higher unary term pairwise term order term Why is this generalisation useful? Ladický, Russell, Kohli, Torr ICCV09
  • 192. Associative Hierarchical CRFsLet us analyze the case with• one segmentation• potentials only over segment level Ladický, Russell, Kohli, Torr ICCV09
  • 193. Associative Hierarchical CRFsLet us analyze the case with• one segmentation• potentials only over segment level 1) Minimum is segment-consistent Ladický, Russell, Kohli, Torr ICCV09
  • 194. Associative Hierarchical CRFsLet us analyze the case with• one segmentation• potentials only over segment level 2) Minimum is segment-consistent 3) Cost of every segment-consistent CRF equal to the cost of pairwise CRF over segments Ladický, Russell, Kohli, Torr ICCV09
  • 195. Associative Hierarchical CRFsLet us analyze the case with• one segmentation• potentials only over segment level Equivalent to pairwise CRF over segments! Ladický, Russell, Kohli, Torr ICCV09
  • 196. Associative Hierarchical CRFs• Merges information over multiple scales• Easy to train potentials and learn parameters• Allows multiple segmentations and hierarchies• Allows long range interactions• Limited (?) to associative interlayer connections Ladický, Russell, Kohli, Torr ICCV09
  • 197. Inference for AH-CRFGraph Cut based move making algorithms [Boykov et al. 01]α-expansion transformation function for base layer Ladický (thesis) 2011
  • 198. Inference for AH-CRFGraph Cut based move making algorithms [Boykov et al. 01]α-expansion transformation function for base layerα-expansion transformation function for auxiliary layer(2 binary variables per node) Ladický (thesis) 2011
  • 199. Inference for AH-CRFInterlayer connection between the base layer and the first auxiliary layer: Ladický (thesis) 2011
  • 200. Inference for AH-CRFInterlayer connection between the base layer and the first auxiliary layer:The move energy is:where Ladický (thesis) 2011
  • 201. Inference for AH-CRFThe move energy for interlayer connection between the base and auxiliary layer is:Can be transformed to binary submodular pairwise function: Ladický (thesis) 2011
  • 202. Inference for AH-CRFThe move energy for interlayer connection between auxiliary layers is:Can be transformed to binary submodular pairwise function: Ladický (thesis) 2011
  • 203. Inference for AH-CRFGraph constructions Ladický (thesis) 2011
  • 204. Inference for AH-CRFPairwise potential between auxiliary variables : Ladický (thesis) 2011
  • 205. Inference for AH-CRFMove energy if x c = xd :Can be transformed to binary submodular pairwise function: Ladický (thesis) 2011
  • 206. Inference for AH-CRFMove energy if x c ≠ xd :Can be transformed to binary submodular pairwise function: Ladický (thesis) 2011
  • 207. Inference for AH-CRFGraph constructions Ladický (thesis) 2011
  • 208. Potential training for Object class Segmentation Pixel unary potentialUnary likelihoods based on spatial configuration (Shotton et al. ECCV06) Classifier trained using boosting Ladický, Russell, Kohli, Torr ICCV09
  • 209. Potential training for Object class SegmentationSegment unary potentialClassifier trained using boosting (resp. Kernel SVMs) Ladický, Russell, Kohli, Torr ICCV09
  • 210. Results on MSRC dataset
  • 211. Results on Corel dataset
  • 212. Results on VOC 2010 datasetInput Image Result Input Image Result
  • 213. VOC 2010-QualitativeInput Image Result Input Image Result
  • 214. CamVid-QualitativeBrostow et al. Our Result Ground TruthInput Image
  • 215. Quantitative comparison MSRC datasetVOC2009 dataset Corel dataset
  • 216. Quantitative comparisonMSRC datasetVOC2009 dataset Ladický, Russell, Kohli, Torr ICCV09
  • 217. Dense Stereo Reconstruction Unary Potential Disparity = 0Unary Cost dependent on the similarity of patches, e.g.cross correlation
  • 218. Dense Stereo Reconstruction Unary Potential Disparity = 5Unary Cost dependent on the similarity of patches, e.g.cross correlation
  • 219. Dense Stereo Reconstruction Unary Potential Disparity = 10Unary Cost dependent on the similarity of patches, e.g.cross correlation
  • 220. Dense Stereo Reconstruction Pairwise Potential• Encourages label consistency in adjacent pixels• Cost based on the distance of labels Linear Truncated Quadratic Truncated
  • 221. Dense Stereo ReconstructionDoes not work for Road Scenes !Original Image Dense Stereo Reconstruction
  • 222. Dense Stereo ReconstructionDoes not work for Road Scenes !Different brightness Patches can be matched to any in cameras other patch for flat surfices
  • 223. Dense Stereo Reconstruction Does not work for Road Scenes ! Different brightness Patches can be matched to any in cameras other patch for flat surficesCould object recognition for road scenes help? Recognition of road scenes is relatively easy
  • 224. Joint Dense Stereo Reconstruction and Object class Segmentation• Object class and 3D location are mutually informative ∞ • Sky always in infinity (disparity = 0) sky
  • 225. Joint Dense Stereo Reconstruction and Object class Segmentation• Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buses & pedestrians have their typical height sky building car
  • 226. Joint Dense Stereo Reconstruction and Object class Segmentation• Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buses & pedestrians have their typical height • Road and pavement on the ground plane sky building car road
  • 227. Joint Dense Stereo Reconstruction and Object class Segmentation• Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buses & pedestrians have their typical height • Road and pavement on the ground plane • Buildings and pavement on the sides sky building car road
  • 228. Joint Dense Stereo Reconstruction and Object class Segmentation• Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buses & pedestrians have their typical height • Road and pavement on the ground plane • Buildings and pavement on the sides• Both problems formulated as CRF • Joint approach possible? sky building car road
  • 229. Joint Formulation• Each pixels takes label zi = [ xi yi ] ∈ L1 × L2• Dependency of xi and yi encoded as a unary and pairwise potential, e.g. • strong correlation between x = road, y = near ground plane • strong correlation between x = sky, y = 0 • Correlation of edge in object class and disparity domain
  • 230. Joint Formulation Unary Potential Object layer Joint unary links Disparity layer• Weighted sum of object class, depth and joint potential• Joint unary potential based on histograms of height
  • 231. Joint Formulation Pairwise Potential Object layer Joint pairwise links Disparity layer• Object class and depth edges correlated• Transitions in depth occur often at the object boundaries
  • 232. Joint Formulation
  • 233. Inference• Standard α-expansion • Each node in each expansion move keeps its old label or takes a new label [xL1, yL2], • Possible in case of metric pairwise potentials
  • 234. Inference• Standard α-expansion • Each node in each expansion move keeps its old label or takes a new label [xL1, yL2], • Possible in case of metric pairwise potentials Too many moves! ( |L1| |L2| ) Impractical !
  • 235. Inference• Projected move for product label space • One / Some of the label components remain(s) constant after the move• Set of projected moves • α-expansion in the object class projection • Range-expansion in the depth projection
  • 236. Dataset• Leuven Road Scene dataset • Contained Left camera Right camera Object GT Disparity GT • 3 sequences • 643 pairs of images • We labelled • 50 training + 20 test images • Object class (7 labels) • Disparity (100 labels) • Publicly available • http://cms.brookes.ac.uk/research/visiongroup/files/Leuven.zip
  • 237. Qualitative Results• Large improvement for dense stereo estimation• Minor improvement in object class segmentation
  • 238. Quantitative ResultsDependency of the ratio of correctly labelled pixels within the maximum allowed error delta
  • 239. Summary• We proposed : • New models enforcing higher order structure • Label-consistency in segments • Consistency between scale • Label-set consistency • Consistency between different domains • Graph-Cut based inference methods • Source code http://cms.brookes.ac.uk/staff/PhilipTorr/ale.htm There is more to come !
  • 240. Thank you Questions?