Computer vision:models, learning and inference              Chapter 10           Graphical Models      Please send errata ...
Models for grids•   Consider models where one unknown world state    at each pixel in the image – takes the form of a grid...
Binary Denoising            Before                                                   AfterImage represented as binary disc...
Multi-label Denoising            Before                                                   After   Image represented as dis...
Denoising GoalObserved Data                                            Uncorrupted Image                                  ...
Denoising Goal         Observed Data                               Uncorrupted Image• Most of the pixels stay the same• Ob...
Markov random fieldsThis is just the typical property of an undirected model.We’ll continue the discussion in terms of und...
Markov random fieldsNormalizing constant (partition function)                    Potential function                       ...
Markov random fieldsNormalizing constant                              Cost function (partition function)                  ...
Smoothing exampleComputer vision: models, learning and inference. ©2011 Simon J.D. Prince   10
Smoothing ExampleSmooth solutions (e.g. 0000,1111) have high probabilityZ was computed by summing the 16 un-normalized pro...
Smoothing Example    Samples from larger grid -- mostly smoothCannot compute partition function Z here - intractable      ...
Denoising GoalObserved Data                                            Uncorrupted Image                                  ...
Denoising overviewBayes’ rule:Likelihoods:                                                                       Probabili...
Denoising with MRFs                                                                                        MRF Prior (pair...
MAP Inference         Unary terms                                          Pairwise terms(compatability of data with label...
Graph Cuts Overview Graph cuts used to optimise this cost function:              Unary terms                              ...
Graph Cuts OverviewGraph cuts used to optimise this cost function:             Unary terms                                ...
Max-Flow ProblemGoal:To push as much ‘flow’ as possible through the directed graph from the sourceto the sink.Cannot excee...
Saturated EdgesWhen we are pushing the maximum amount of flow:•   There must be at least one saturated edge on any path fr...
Augmenting PathsTwo numbers represent: current flow / total capacity                                                      ...
Augmenting PathsChoose any route from source to sink with spare capacity and push as much flow asyou can. One edge (here 6...
Augmenting PathsChoose another route, respecting remaining capacity. This time edge 5-6 saturates.                        ...
Augmenting PathsA third route. Edge 1-4 saturates                                                                         ...
Augmenting PathsA fourth route. Edge 2-5 saturates                                                                        ...
Augmenting PathsA fifth route. Edge 2-4 saturates                                                                         ...
Augmenting PathsThere is now no further route from source to sink – there is a saturated edge alongevery possible route (h...
Augmenting PathsThe saturated edges separate the source from the sink and form the min-cut solution.Nodes either connect t...
Graph Cuts: Binary MRFGraph cuts used to optimise this cost function:               Unary terms                           ...
Graph Construction•   One node per pixel (here a 3x3 image)•   Edge from source to every pixel node•   Edge from every pix...
Graph Construction                                          Now add capacities so that                                    ...
Example 1                                                                           32Computer vision: models, learning an...
Example 2                                                                           33Computer vision: models, learning an...
Example 3                                                                           34Computer vision: models, learning an...
35Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
Graph Cuts: Binary MRFGraph cuts used to optimise this cost function:               Unary terms                           ...
General Pairwise costsModify graph to•   Add P(0,0) to edge s-b     • Implies that solutions 0,0 and        1,0 also pay t...
ReparameterizationThe max-flow / min-cut algorithmsrequire that all of the capacities arenon-negative.However, because we ...
Reparameterization 1The minimum cut chooses the same links in these two graphs Computer vision: models, learning and infer...
Reparameterization 2The minimum cut chooses the same links in these two graphs                  40 Computer vision: models...
Submodularity                                                       Subtract constant                                     ...
SubmodularityIf this condition is obeyed, it is said that the problem is “submodular” and it canbe solved in polynomial ti...
Denoising Results                                       Pairwise costs increasingOriginal                     Pairwise cos...
Plan of Talk•   Denoising problem•   Markov random fields (MRFs)•   Max-flow / min-cut•   Binary MRFs – submodular (exact ...
Multiple Labels                                          Construction for two pixels                                      ...
Constraint Edges                                                              The edges with infinite                     ...
Multiple Labels                                   Inter-pixel edges have costs                                   defined a...
Example CutsMust cut links from before cut on pixel a to after cut on pixel b.            48   Computer vision: models, le...
Pairwise Costs  Must cut links from before cut on  pixel a to after cut on pixel b.  Costs were carefully chosen so  that ...
Reparameterization                                                                           50Computer vision: models, le...
SubmodularityWe require the remaining inter-pixel links to bepositive so thatorBy mathematical induction we can get the mo...
SubmodularityIf not submodular then the problem is NP hard.                               52  Computer vision: models, lea...
Convex vs. non-convex costsQuadratic                       Truncated Quadratic                                Potts Model•...
What is wrong with convex costs?     Observed noisy image                                        Denoised result• Pay lowe...
Plan of Talk•   Denoising problem•   Markov random fields (MRFs)•   Max-flow / min-cut•   Binary MRFs - submodular (exact ...
Alpha Expansion Algorithm• break multilabel problem into a series of binary problems• at each iteration, pick label and ex...
Alpha Expansion Ideas• For every iteration   – For every label   – Expand label using optimal graph cut solutionCo-ordinat...
Alpha Expansion Construction                          Binary graph cut – either cut link to source                        ...
Alpha Expansion Construction                                  Graph is dynamic. Structure of inter-pixel links            ...
Alpha Expansion Construction                                  Case 1:                                  Adjacent pixels bot...
Alpha Expansion Construction                                  Case 2: Adjacent pixels are     .                           ...
Alpha Expansion Construction                                  Case 3: Adjacent pixels are   . Result either               ...
Alpha Expansion Construction                                   Case 4: Adjacent pixels are . Result either                ...
Example Cut 1                                                                           64Computer vision: models, learnin...
Example Cut 1                                                                       Important!                            ...
Example Cut 2                                                                           66Computer vision: models, learnin...
Example Cut 3                                                                           67Computer vision: models, learnin...
Denoising Results                                                                                 68      Computer vision:...
Conditional Random Fields  Computer vision: models, learning and inference. ©2011 Simon J.D. Prince   69
Directed model for gridsCannot use graph cuts as three-wise term. Easy to draw samples.     Computer vision: models, learn...
Applications• Background subtraction        Computer vision: models, learning and inference. ©2011 Simon J.D. Prince   71
Applications• Grab cut        Computer vision: models, learning and inference. ©2011 Simon J.D. Prince   72
Applications• Stereo vision         Computer vision: models, learning and inference. ©2011 Simon J.D. Prince   73
Applications• Shift-map image editing        Computer vision: models, learning and inference. ©2011 Simon J.D. Prince   74
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince   75
Applications• Shift-map image editing        Computer vision: models, learning and inference. ©2011 Simon J.D. Prince   76
Applications• Super-resolution        Computer vision: models, learning and inference. ©2011 Simon J.D. Prince   77
Applications• Texture synthesis         Computer vision: models, learning and inference. ©2011 Simon J.D. Prince   78
Image QuiltingComputer vision: models, learning and inference. ©2011 Simon J.D. Prince   79
Applications• Synthesizing faces         Computer vision: models, learning and inference. ©2011 Simon J.D. Prince   80
Upcoming SlideShare
Loading in...5
×

12 cv mil_models_for_grids

352

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
352
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

12 cv mil_models_for_grids

  1. 1. Computer vision:models, learning and inference Chapter 10 Graphical Models Please send errata to s.prince@cs.ucl.ac.uk
  2. 2. Models for grids• Consider models where one unknown world state at each pixel in the image – takes the form of a grid.• Loops in the graphical model so cannot use dynamic programming or belief propagation• Define probability distributions that favour certain configurations of world states – Called Markov random fields – Inference using a set of techniques called graph cuts 2 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  3. 3. Binary Denoising Before AfterImage represented as binary discrete variables. Some proportion of pixels randomly changed polarity. 3 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  4. 4. Multi-label Denoising Before After Image represented as discrete variables representing intensity. Someproportion of pixels randomly changed according to a uniform distribution. 4 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  5. 5. Denoising GoalObserved Data Uncorrupted Image 5Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  6. 6. Denoising Goal Observed Data Uncorrupted Image• Most of the pixels stay the same• Observed image is not as smooth as originalNow consider pdf over binary images that encourages smoothness – Markov random field 6 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  7. 7. Markov random fieldsThis is just the typical property of an undirected model.We’ll continue the discussion in terms of undirected models 7 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  8. 8. Markov random fieldsNormalizing constant (partition function) Potential function Returns positive number Subset of variables (clique) 8 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  9. 9. Markov random fieldsNormalizing constant Cost function (partition function) Returns any number Subset of variables Relationship (clique) 9 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  10. 10. Smoothing exampleComputer vision: models, learning and inference. ©2011 Simon J.D. Prince 10
  11. 11. Smoothing ExampleSmooth solutions (e.g. 0000,1111) have high probabilityZ was computed by summing the 16 un-normalized probabilities Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 11
  12. 12. Smoothing Example Samples from larger grid -- mostly smoothCannot compute partition function Z here - intractable Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 12
  13. 13. Denoising GoalObserved Data Uncorrupted Image 13Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  14. 14. Denoising overviewBayes’ rule:Likelihoods: Probability of flipping polarityPrior: Markov random field (smoothness)MAP Inference: Graph cuts 14 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  15. 15. Denoising with MRFs MRF Prior (pairwise cliques)Original image, w LikelihoodsObserved image, x Inference : 15 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  16. 16. MAP Inference Unary terms Pairwise terms(compatability of data with label y) (compatability of neighboring labels) 16 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  17. 17. Graph Cuts Overview Graph cuts used to optimise this cost function: Unary terms Pairwise terms (compatability of data with label y) (compatability of neighboring labels)Three main cases: 17 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  18. 18. Graph Cuts OverviewGraph cuts used to optimise this cost function: Unary terms Pairwise terms (compatability of data with label y) (compatability of neighboring labels)Approach:Convert minimization into the form of a standard CS problem, MAXIMUM FLOW or MINIMUM CUT ON A GRAPHPolynomial-time methods for solving this problem are known 18 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  19. 19. Max-Flow ProblemGoal:To push as much ‘flow’ as possible through the directed graph from the sourceto the sink.Cannot exceed the (non-negative) capacities cij associated with each edge. 19 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  20. 20. Saturated EdgesWhen we are pushing the maximum amount of flow:• There must be at least one saturated edge on any path from source to sink (otherwise we could push more flow)• The set of saturated edges hence separate the source and sink 20 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  21. 21. Augmenting PathsTwo numbers represent: current flow / total capacity 21 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  22. 22. Augmenting PathsChoose any route from source to sink with spare capacity and push as much flow asyou can. One edge (here 6-t) will saturate. 22 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  23. 23. Augmenting PathsChoose another route, respecting remaining capacity. This time edge 5-6 saturates. 23 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  24. 24. Augmenting PathsA third route. Edge 1-4 saturates 24 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  25. 25. Augmenting PathsA fourth route. Edge 2-5 saturates 25 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  26. 26. Augmenting PathsA fifth route. Edge 2-4 saturates 26 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  27. 27. Augmenting PathsThere is now no further route from source to sink – there is a saturated edge alongevery possible route (highlighted arrows) 27 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  28. 28. Augmenting PathsThe saturated edges separate the source from the sink and form the min-cut solution.Nodes either connect to the source or connect to the sink. 28 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  29. 29. Graph Cuts: Binary MRFGraph cuts used to optimise this cost function: Unary terms Pairwise terms (compatability of data with label w) (compatability of neighboring labels) First work with binary case (i.e. True label w is 0 or 1) Constrain pairwise costs so that they are “zero-diagonal” 29 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  30. 30. Graph Construction• One node per pixel (here a 3x3 image)• Edge from source to every pixel node• Edge from every pixel node to sink• Reciprocal edges between neighbours Note that in the minimum cut EITHER the edge connecting to the source will be cut, OR the edge connecting to the sink, but NOT BOTH (unnecessary). Which determines whether we give that pixel label 1 or label 0. Now a 1 to 1 mapping between possible labelling and possible minimum cuts 30 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  31. 31. Graph Construction Now add capacities so that minimum cut, minimizes our cost function Unary costs U(0), U(1) attached to links to source and sink. • Either one or the other is paid. Pairwise costs between pixel nodes as shown. • Why? Easiest to understand with some worked examples. 31Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  32. 32. Example 1 32Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  33. 33. Example 2 33Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  34. 34. Example 3 34Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  35. 35. 35Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  36. 36. Graph Cuts: Binary MRFGraph cuts used to optimise this cost function: Unary terms Pairwise terms (compatability of data with label w) (compatability of neighboring labels) Summary of approach • Associate each possible solution with a minimum cut on a graph • Set capacities on graph, so cost of cut matches the cost function • Use augmenting paths to find minimum cut • This minimizes the cost function and finds the MAP solution 36 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  37. 37. General Pairwise costsModify graph to• Add P(0,0) to edge s-b • Implies that solutions 0,0 and 1,0 also pay this cost• Subtract P(0,0) from edge b-a • Solution 1,0 has this cost removed againSimilar approach for P(1,1) 37 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  38. 38. ReparameterizationThe max-flow / min-cut algorithmsrequire that all of the capacities arenon-negative.However, because we have asubtraction on edge a-b we cannotguarantee that this will be thecase, even if all the original unary andpairwise costs were positive.The solution to this problem isreparamaterization: find new graphwhere costs (capacities) are differentbut choice of minimum solution is thesame (usually just by adding a constantto each solution) 38 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  39. 39. Reparameterization 1The minimum cut chooses the same links in these two graphs Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  40. 40. Reparameterization 2The minimum cut chooses the same links in these two graphs 40 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  41. 41. Submodularity Subtract constant Add constant, Adding together implies 41Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  42. 42. SubmodularityIf this condition is obeyed, it is said that the problem is “submodular” and it canbe solved in polynomial time.If it is not obeyed then the problem is NP hard.Usually it is not a problem as we tend to favour smooth solutions. 42 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  43. 43. Denoising Results Pairwise costs increasingOriginal Pairwise costs increasing 43 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  44. 44. Plan of Talk• Denoising problem• Markov random fields (MRFs)• Max-flow / min-cut• Binary MRFs – submodular (exact solution)• Multi-label MRFs – submodular (exact solution)• Multi-label MRFs - non-submodular (approximate) 44 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  45. 45. Multiple Labels Construction for two pixels (a and b) and four labels (1,2,3,4) There are 5 nodes for each pixel and 4 edges between them have unary costs for the 4 labels. One of these edges must be cut in the min-cut solution and the choice will determine which label we assign. 45Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  46. 46. Constraint Edges The edges with infinite capacity pointing upwards are called constraint edges. They prevent solutions that cut the chain of edges associated with a pixel more than once (and hence given an ambiguous labelling) 46Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  47. 47. Multiple Labels Inter-pixel edges have costs defined as: Superfluous terms : For all i,j where K is number of labels47Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  48. 48. Example CutsMust cut links from before cut on pixel a to after cut on pixel b. 48 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  49. 49. Pairwise Costs Must cut links from before cut on pixel a to after cut on pixel b. Costs were carefully chosen so that sum of these links gives appropriate pairwise term.If pixel a takes label I and pixel b takes label J 49 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  50. 50. Reparameterization 50Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  51. 51. SubmodularityWe require the remaining inter-pixel links to bepositive so thatorBy mathematical induction we can get the moregeneral result 51 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  52. 52. SubmodularityIf not submodular then the problem is NP hard. 52 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  53. 53. Convex vs. non-convex costsQuadratic Truncated Quadratic Potts Model• Convex • Not Convex • Not Convex• Submodular • Not Submodular • Not Submodular 53 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  54. 54. What is wrong with convex costs? Observed noisy image Denoised result• Pay lower price for many small changes than one large one• Result: blurring at large changes in intensity 54 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  55. 55. Plan of Talk• Denoising problem• Markov random fields (MRFs)• Max-flow / min-cut• Binary MRFs - submodular (exact solution)• Multi-label MRFs – submodular (exact solution)• Multi-label MRFs - non-submodular (approximate) 55 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  56. 56. Alpha Expansion Algorithm• break multilabel problem into a series of binary problems• at each iteration, pick label and expand (retain original or change to ) Initial Iteration 1 labelling (orange) Iteration 2 Iteration 3 (yellow) (red) 56 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  57. 57. Alpha Expansion Ideas• For every iteration – For every label – Expand label using optimal graph cut solutionCo-ordinate descent in label space.Each step optimal, but overall global maximum not guaranteedProved to be within a factor of 2 of global optimum.Requires that pairwise costs form a metric: 57 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  58. 58. Alpha Expansion Construction Binary graph cut – either cut link to source (assigned to ) or to sink (retain current label) Unary costs attached to links between 58 source, sink andSimon J.D.nodes appropriately. Computer vision: models, learning and inference. ©2011 pixel Prince
  59. 59. Alpha Expansion Construction Graph is dynamic. Structure of inter-pixel links depends on and the choice of labels. There are four cases. 59 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  60. 60. Alpha Expansion Construction Case 1: Adjacent pixels both have label already. Pairwise cost is zero – no need for extra edges. 60 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  61. 61. Alpha Expansion Construction Case 2: Adjacent pixels are . Result either • (no cost and no new edge). • (P( ), add new edge). 61 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  62. 62. Alpha Expansion Construction Case 3: Adjacent pixels are . Result either • (no cost and no new edge). • (P( ), add new edge). • (P( ), add new edge). 62 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  63. 63. Alpha Expansion Construction Case 4: Adjacent pixels are . Result either • (P( ), add new edge). • (P( ), add new edge). • (P( ), add new edge). 63 • (no cost Prince no new edge). Computer vision: models, learning and inference. ©2011 Simon J.D. and
  64. 64. Example Cut 1 64Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  65. 65. Example Cut 1 Important! 65Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  66. 66. Example Cut 2 66Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  67. 67. Example Cut 3 67Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  68. 68. Denoising Results 68 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  69. 69. Conditional Random Fields Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 69
  70. 70. Directed model for gridsCannot use graph cuts as three-wise term. Easy to draw samples. Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 70
  71. 71. Applications• Background subtraction Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 71
  72. 72. Applications• Grab cut Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 72
  73. 73. Applications• Stereo vision Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 73
  74. 74. Applications• Shift-map image editing Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 74
  75. 75. Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 75
  76. 76. Applications• Shift-map image editing Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 76
  77. 77. Applications• Super-resolution Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 77
  78. 78. Applications• Texture synthesis Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 78
  79. 79. Image QuiltingComputer vision: models, learning and inference. ©2011 Simon J.D. Prince 79
  80. 80. Applications• Synthesizing faces Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 80
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×