Your SlideShare is downloading. ×
  • Like
12 cv mil_models_for_grids
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

12 cv mil_models_for_grids

  • 332 views
Published

 

Published in Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
332
On SlideShare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
3
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Computer vision:models, learning and inference Chapter 10 Graphical Models Please send errata to s.prince@cs.ucl.ac.uk
  • 2. Models for grids• Consider models where one unknown world state at each pixel in the image – takes the form of a grid.• Loops in the graphical model so cannot use dynamic programming or belief propagation• Define probability distributions that favour certain configurations of world states – Called Markov random fields – Inference using a set of techniques called graph cuts 2 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 3. Binary Denoising Before AfterImage represented as binary discrete variables. Some proportion of pixels randomly changed polarity. 3 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 4. Multi-label Denoising Before After Image represented as discrete variables representing intensity. Someproportion of pixels randomly changed according to a uniform distribution. 4 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 5. Denoising GoalObserved Data Uncorrupted Image 5Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 6. Denoising Goal Observed Data Uncorrupted Image• Most of the pixels stay the same• Observed image is not as smooth as originalNow consider pdf over binary images that encourages smoothness – Markov random field 6 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 7. Markov random fieldsThis is just the typical property of an undirected model.We’ll continue the discussion in terms of undirected models 7 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 8. Markov random fieldsNormalizing constant (partition function) Potential function Returns positive number Subset of variables (clique) 8 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 9. Markov random fieldsNormalizing constant Cost function (partition function) Returns any number Subset of variables Relationship (clique) 9 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 10. Smoothing exampleComputer vision: models, learning and inference. ©2011 Simon J.D. Prince 10
  • 11. Smoothing ExampleSmooth solutions (e.g. 0000,1111) have high probabilityZ was computed by summing the 16 un-normalized probabilities Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 11
  • 12. Smoothing Example Samples from larger grid -- mostly smoothCannot compute partition function Z here - intractable Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 12
  • 13. Denoising GoalObserved Data Uncorrupted Image 13Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 14. Denoising overviewBayes’ rule:Likelihoods: Probability of flipping polarityPrior: Markov random field (smoothness)MAP Inference: Graph cuts 14 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 15. Denoising with MRFs MRF Prior (pairwise cliques)Original image, w LikelihoodsObserved image, x Inference : 15 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 16. MAP Inference Unary terms Pairwise terms(compatability of data with label y) (compatability of neighboring labels) 16 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 17. Graph Cuts Overview Graph cuts used to optimise this cost function: Unary terms Pairwise terms (compatability of data with label y) (compatability of neighboring labels)Three main cases: 17 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 18. Graph Cuts OverviewGraph cuts used to optimise this cost function: Unary terms Pairwise terms (compatability of data with label y) (compatability of neighboring labels)Approach:Convert minimization into the form of a standard CS problem, MAXIMUM FLOW or MINIMUM CUT ON A GRAPHPolynomial-time methods for solving this problem are known 18 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 19. Max-Flow ProblemGoal:To push as much ‘flow’ as possible through the directed graph from the sourceto the sink.Cannot exceed the (non-negative) capacities cij associated with each edge. 19 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 20. Saturated EdgesWhen we are pushing the maximum amount of flow:• There must be at least one saturated edge on any path from source to sink (otherwise we could push more flow)• The set of saturated edges hence separate the source and sink 20 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 21. Augmenting PathsTwo numbers represent: current flow / total capacity 21 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 22. Augmenting PathsChoose any route from source to sink with spare capacity and push as much flow asyou can. One edge (here 6-t) will saturate. 22 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 23. Augmenting PathsChoose another route, respecting remaining capacity. This time edge 5-6 saturates. 23 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 24. Augmenting PathsA third route. Edge 1-4 saturates 24 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 25. Augmenting PathsA fourth route. Edge 2-5 saturates 25 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 26. Augmenting PathsA fifth route. Edge 2-4 saturates 26 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 27. Augmenting PathsThere is now no further route from source to sink – there is a saturated edge alongevery possible route (highlighted arrows) 27 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 28. Augmenting PathsThe saturated edges separate the source from the sink and form the min-cut solution.Nodes either connect to the source or connect to the sink. 28 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 29. Graph Cuts: Binary MRFGraph cuts used to optimise this cost function: Unary terms Pairwise terms (compatability of data with label w) (compatability of neighboring labels) First work with binary case (i.e. True label w is 0 or 1) Constrain pairwise costs so that they are “zero-diagonal” 29 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 30. Graph Construction• One node per pixel (here a 3x3 image)• Edge from source to every pixel node• Edge from every pixel node to sink• Reciprocal edges between neighbours Note that in the minimum cut EITHER the edge connecting to the source will be cut, OR the edge connecting to the sink, but NOT BOTH (unnecessary). Which determines whether we give that pixel label 1 or label 0. Now a 1 to 1 mapping between possible labelling and possible minimum cuts 30 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 31. Graph Construction Now add capacities so that minimum cut, minimizes our cost function Unary costs U(0), U(1) attached to links to source and sink. • Either one or the other is paid. Pairwise costs between pixel nodes as shown. • Why? Easiest to understand with some worked examples. 31Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 32. Example 1 32Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 33. Example 2 33Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 34. Example 3 34Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 35. 35Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 36. Graph Cuts: Binary MRFGraph cuts used to optimise this cost function: Unary terms Pairwise terms (compatability of data with label w) (compatability of neighboring labels) Summary of approach • Associate each possible solution with a minimum cut on a graph • Set capacities on graph, so cost of cut matches the cost function • Use augmenting paths to find minimum cut • This minimizes the cost function and finds the MAP solution 36 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 37. General Pairwise costsModify graph to• Add P(0,0) to edge s-b • Implies that solutions 0,0 and 1,0 also pay this cost• Subtract P(0,0) from edge b-a • Solution 1,0 has this cost removed againSimilar approach for P(1,1) 37 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 38. ReparameterizationThe max-flow / min-cut algorithmsrequire that all of the capacities arenon-negative.However, because we have asubtraction on edge a-b we cannotguarantee that this will be thecase, even if all the original unary andpairwise costs were positive.The solution to this problem isreparamaterization: find new graphwhere costs (capacities) are differentbut choice of minimum solution is thesame (usually just by adding a constantto each solution) 38 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 39. Reparameterization 1The minimum cut chooses the same links in these two graphs Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 40. Reparameterization 2The minimum cut chooses the same links in these two graphs 40 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 41. Submodularity Subtract constant Add constant, Adding together implies 41Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 42. SubmodularityIf this condition is obeyed, it is said that the problem is “submodular” and it canbe solved in polynomial time.If it is not obeyed then the problem is NP hard.Usually it is not a problem as we tend to favour smooth solutions. 42 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 43. Denoising Results Pairwise costs increasingOriginal Pairwise costs increasing 43 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 44. Plan of Talk• Denoising problem• Markov random fields (MRFs)• Max-flow / min-cut• Binary MRFs – submodular (exact solution)• Multi-label MRFs – submodular (exact solution)• Multi-label MRFs - non-submodular (approximate) 44 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 45. Multiple Labels Construction for two pixels (a and b) and four labels (1,2,3,4) There are 5 nodes for each pixel and 4 edges between them have unary costs for the 4 labels. One of these edges must be cut in the min-cut solution and the choice will determine which label we assign. 45Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 46. Constraint Edges The edges with infinite capacity pointing upwards are called constraint edges. They prevent solutions that cut the chain of edges associated with a pixel more than once (and hence given an ambiguous labelling) 46Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 47. Multiple Labels Inter-pixel edges have costs defined as: Superfluous terms : For all i,j where K is number of labels47Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 48. Example CutsMust cut links from before cut on pixel a to after cut on pixel b. 48 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 49. Pairwise Costs Must cut links from before cut on pixel a to after cut on pixel b. Costs were carefully chosen so that sum of these links gives appropriate pairwise term.If pixel a takes label I and pixel b takes label J 49 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 50. Reparameterization 50Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 51. SubmodularityWe require the remaining inter-pixel links to bepositive so thatorBy mathematical induction we can get the moregeneral result 51 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 52. SubmodularityIf not submodular then the problem is NP hard. 52 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 53. Convex vs. non-convex costsQuadratic Truncated Quadratic Potts Model• Convex • Not Convex • Not Convex• Submodular • Not Submodular • Not Submodular 53 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 54. What is wrong with convex costs? Observed noisy image Denoised result• Pay lower price for many small changes than one large one• Result: blurring at large changes in intensity 54 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 55. Plan of Talk• Denoising problem• Markov random fields (MRFs)• Max-flow / min-cut• Binary MRFs - submodular (exact solution)• Multi-label MRFs – submodular (exact solution)• Multi-label MRFs - non-submodular (approximate) 55 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 56. Alpha Expansion Algorithm• break multilabel problem into a series of binary problems• at each iteration, pick label and expand (retain original or change to ) Initial Iteration 1 labelling (orange) Iteration 2 Iteration 3 (yellow) (red) 56 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 57. Alpha Expansion Ideas• For every iteration – For every label – Expand label using optimal graph cut solutionCo-ordinate descent in label space.Each step optimal, but overall global maximum not guaranteedProved to be within a factor of 2 of global optimum.Requires that pairwise costs form a metric: 57 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 58. Alpha Expansion Construction Binary graph cut – either cut link to source (assigned to ) or to sink (retain current label) Unary costs attached to links between 58 source, sink andSimon J.D.nodes appropriately. Computer vision: models, learning and inference. ©2011 pixel Prince
  • 59. Alpha Expansion Construction Graph is dynamic. Structure of inter-pixel links depends on and the choice of labels. There are four cases. 59 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 60. Alpha Expansion Construction Case 1: Adjacent pixels both have label already. Pairwise cost is zero – no need for extra edges. 60 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 61. Alpha Expansion Construction Case 2: Adjacent pixels are . Result either • (no cost and no new edge). • (P( ), add new edge). 61 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 62. Alpha Expansion Construction Case 3: Adjacent pixels are . Result either • (no cost and no new edge). • (P( ), add new edge). • (P( ), add new edge). 62 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 63. Alpha Expansion Construction Case 4: Adjacent pixels are . Result either • (P( ), add new edge). • (P( ), add new edge). • (P( ), add new edge). 63 • (no cost Prince no new edge). Computer vision: models, learning and inference. ©2011 Simon J.D. and
  • 64. Example Cut 1 64Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 65. Example Cut 1 Important! 65Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 66. Example Cut 2 66Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 67. Example Cut 3 67Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 68. Denoising Results 68 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
  • 69. Conditional Random Fields Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 69
  • 70. Directed model for gridsCannot use graph cuts as three-wise term. Easy to draw samples. Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 70
  • 71. Applications• Background subtraction Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 71
  • 72. Applications• Grab cut Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 72
  • 73. Applications• Stereo vision Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 73
  • 74. Applications• Shift-map image editing Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 74
  • 75. Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 75
  • 76. Applications• Shift-map image editing Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 76
  • 77. Applications• Super-resolution Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 77
  • 78. Applications• Texture synthesis Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 78
  • 79. Image QuiltingComputer vision: models, learning and inference. ©2011 Simon J.D. Prince 79
  • 80. Applications• Synthesizing faces Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 80