Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Image Segmentation Using Gm Ms

3,540 views

Published on

Shows how Gaussian Mixture Models can be used in image segmentation

  • Login to see the comments

Image Segmentation Using Gm Ms

  1. 1. Allan caine<br />adcaine@cs.uwaterloo.ca<br />http://www.cs.uwaterloo.ca/~adcaine/<br />May 28, 2009<br />Chroma Keying using Gaussian Mixture Models (GMMs)<br />
  2. 2. Expectations<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />2<br />You should take notes in case this material appears on an exam or assignment, possibly in a different form<br />It is expected that you will use the Matlab software found at http://www.cs.uwaterloo.ca/~adcaine/ to generate a toy problem and solve it<br />These lecture notes are on-line at the above address<br />
  3. 3. Inspiration<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />3<br />Have you ever wondered how is this done? Can we emulate this process on a computer?<br />
  4. 4. Co-ordination<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />4<br />By viewing an off-camera monitor, theweather announcer can co-ordinate herhand movements with the background.The result is a seemingly realistic televisionbroadcast, in spite of the fact that theannouncer is not actually in front of anyweather map.<br />
  5. 5. Teaching Objectives<br />Describe how chroma keying (blue screening) works<br />Theorize how classification might solve the chroma keying problem on a computer<br />Enumerate the axioms of probability <br />Explain the concept behind digital images<br />Express the intuition behind the Expectation-Maximization algorithm in terms of a Gaussian Mixture Model (GMM), using the definition of a Gaussian.<br />Generate a toy GMM problem, and solve the toy problem, recovering the hidden variables<br />5<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />
  6. 6. Outline<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />6<br />Chroma Keying and our Assumptions<br />RBG Color Model<br />Axioms of Probability<br />Gaussian Distributions<br />Gaussian Mixture Model (GMM)<br />Bayes’ Rule and the Expectation Step<br />Update Rules and the Maximization Step<br />Solving a Toy Problem<br />Segmentation using a GMM and Masks<br />Further Work<br />
  7. 7. Chroma Keying and our Assumptions<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />7<br />Chroma Keying and our Assumptions<br />RBG Color Model<br />Axioms of Probability<br />Gaussian Distributions<br />Gaussian Mixture Model (GMM)<br />Bayes’ Rule and the Expectation Step<br />Update Rules and the Maximization Step<br />Solving a Toy Problem<br />Segmentation using a GMM and Masks<br />Further Work<br />
  8. 8. Basic Idea behind Chroma Keying<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />8<br />If the background is of one color, saygreen, and the foreground person is notof that color, then the person can besegmented from the background.The person can then be superimposed on abackground of our choosing.<br />Normally, the chroma key is assumed to beknown. To make this problem moreinteresting, we will assume that the chromakey is unknown, and must be learned. So,it’s an AI problem.<br />
  9. 9. Assumptions for the Segmentation Problem<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />9<br />We want to segment the foreground from the background<br />There is a background of a single color<br />However, the background color is unknown; the background color must be learned<br />No part of the person or object in the foreground has the background color<br />The foreground person or object can be in any position in the image; we cannot deduce the background color simply by examining a pixel in a certain fixed position<br />
  10. 10. RBG Color Model<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />10<br />Chroma Keying and our Assumptions<br />RBG Color Model<br />Axioms of Probability<br />Gaussian Distributions<br />Gaussian Mixture Model (GMM)<br />Bayes’ Rule and the Expectation Step<br />Update Rules and the Maximization Step<br />Solving a Toy Problem<br />Segmentation using a GMM and Masks<br />Further Work<br />
  11. 11. Explanation of RGB Color Model<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />11<br />Every color is expressed as an ordered triple<br />Each value can be between 0 and 255; i.e. 256 possibilities  8 bits can store 256 possibilities<br />The numbers represent the amounts of red, green, and blue in that order  (R,G,B)<br />If R = G = B, then the pixel is a shade of gray<br />
  12. 12. RGB as a Data Structure in Matlab<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />12<br />
  13. 13. Scatter Diagrams<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />13<br />If only we could determine which class a pixel belonged to: foreground orbackground; the purple sphere or the green background, respectively.<br />
  14. 14. Make your own Scatter Plot<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />14<br />
  15. 15. Actual Scatter Plot<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />15<br />
  16. 16. Axioms of Probability<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />16<br />Chroma Keying and our Assumptions<br />RBG Color Model<br />Axioms of Probability<br />Gaussian Distributions<br />Gaussian Mixture Model (GMM)<br />Bayes’ Rule and the Expectation Step<br />Update Rules and the Maximization Step<br />Solving a Toy Problem<br />Segmentation using a GMM and Masks<br />Further Work<br />
  17. 17. (Some) Axioms of Probability<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />17<br />
  18. 18. Bayesian Philosophy of Choosing<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />18<br />Suppose two events, A and B, occur with probability P(A) and P(B) such that P(A) + P(B) = 1.<br />If you had to place a bet on which event would occur and if P(A) ≠ P(B), what event would you bet on?<br />Suppose P(A) = P(B). What event would you bet on?<br />
  19. 19. Gaussian Distributions<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />19<br />Chroma Keying and our Assumptions<br />RBG Color Model<br />Axioms of Probability<br />Gaussian Distributions<br />Gaussian Mixture Model (GMM)<br />Bayes’ Rule and the Expectation Step<br />Update Rules and the Maximization Step<br />Solving a Toy Problem<br />Segmentation using a GMM and Masks<br />Further Work<br />
  20. 20. Gaussian Distributions<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />20<br />
  21. 21. One-D Gaussian<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />21<br />
  22. 22. Variations in μ<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />22<br />
  23. 23. Variations in σ<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />23<br />
  24. 24. 2D Gaussian<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />24<br />
  25. 25. Comparison of two 2D Gaussians<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />25<br />
  26. 26. 2D Gaussian with σx ≠ σy<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />26<br />
  27. 27. 3D Gaussian<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />27<br />
  28. 28. Gaussian Mixture Model (GMM)<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />28<br />Chroma Keying and our Assumptions<br />RBG Color Model<br />Axioms of Probability<br />Gaussian Distributions<br />Gaussian Mixture Model (GMM)<br />Bayes’ Rule and the Expectation Step<br />Update Rules and the Maximization Step<br />Solving a Toy Problem<br />Segmentation using a GMM and Masks<br />Further Work<br />
  29. 29. GMM Model<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />29<br />The number of classes is known<br />Each class has a particular mean and variance<br />Each class has a mixing (or membership) weight. All mixing weights add up to one<br />
  30. 30. Toy Problem Parameters<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />30<br />
  31. 31. Generating the Toy Problem’s DataSet<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />31<br />20%<br />40%<br />Start<br />40%<br />
  32. 32. The GMM Problem<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />32<br />??%<br />??%<br />Start<br />??%<br />
  33. 33. Bayes’ Rule and the Expectation Step<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />33<br />Chroma Keying and our Assumptions<br />RBG Color Model<br />Axioms of Probability<br />Gaussian Distributions<br />Gaussian Mixture Model (GMM)<br />Bayes’ Rule and the Expectation Step<br />Update Rules and the Maximization Step<br />Solving a Toy Problem<br />Segmentation using a GMM and Masks<br />Further Work<br />
  34. 34. Bayes’ Rule<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />34<br />The likelihood of A given B<br />A prior probability on B<br />The likelihood of B given A<br />A normalization constant<br />
  35. 35. The Main of Idea of GMMs: E-Step<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />35<br />The probability of position j havinga pixel with values x given class i. Basically, evaluate a Gaussian at xusing μi and Σi as parameters<br />The probability that position j is generatedby class 1 or 2: foreground or background<br />The prior probability ofclass i; i.e. the mixing<br />weights<br />Essentially, a normalization constantto ensure that P(C=1|xj) +P(C=2|xj) = 1for j fixed.<br />
  36. 36. Update Rules and the Maximization Step<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />36<br />Chroma Keying and our Assumptions<br />RBG Color Model<br />Axioms of Probability<br />Gaussian Distributions<br />Gaussian Mixture Model (GMM)<br />Bayes’ Rule and the Expectation Step<br />Update Rules and the Maximization Step<br />Solving a Toy Problem<br />Segmentation using a GMM and Masks<br />Further Work<br />
  37. 37. Update Rules: M-Step<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />37<br />In words, define what each ofthese mathematical statementsmean.<br />What do you think is the intuitionbehind these three update rules?<br />
  38. 38. Solving a Toy Problem<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />38<br />Chroma Keying and our Assumptions<br />RBG Color Model<br />Axioms of Probability<br />Gaussian Distributions<br />Gaussian Mixture Model (GMM)<br />Bayes’ Rule and the Expectation Step<br />Update Rules and the Maximization Step<br />Solving a Toy Problem<br />Segmentation using a GMM and Masks<br />Further Work<br />
  39. 39. Toy Problem Visually<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />39<br />
  40. 40. Typical Output<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />40<br />Class 2<br />Class 3<br />Class 1<br />
  41. 41. Solution to the Toy Problem<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />41<br />
  42. 42. Comparison of Statistics<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />42<br />
  43. 43. Recovery of the Model<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />43<br />
  44. 44. 3D GMM Problem<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />44<br />
  45. 45. Segmentation using a GMM and Masks<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />45<br />Chroma Keying and our Assumptions<br />RBG Color Model<br />Axioms of Probability<br />Gaussian Distributions<br />Gaussian Mixture Model (GMM)<br />Bayes’ Rule and the Expectation Step<br />Update Rules and the Maximization Step<br />Solving a Toy Problem<br />Segmentation using a GMM and Masks<br />Further Work<br />
  46. 46. Segmentation Example<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />46<br />
  47. 47. Segmentation<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />47<br />The upshot of all of this is that the foreground has a parameterization of:<br />The novelty of this result is that we have a complete and numericalcharacterization of the image without knowing beforehand thecolor of the background. These parameters have been learned.<br />
  48. 48. Producing the Mask<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />48<br />A Gaussian computed using themean and variance of the foregroundclass.<br />The mixing weight 0.9695<br />A Gaussian computed using themean and variance of the backgroundclass.<br />The mixing weight 0.0305<br />How would we use a Bayesian philosophy to decide ifa pixel represents the foreground or background?<br />
  49. 49. Masks<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />49<br />This mask permits the sphereto show through while excludingthe background. It’s used tosegment the sphere; that is,segment out the foreground.<br />This mask is the logical complementof the first mask. It indicates wherethe background pixels can be found.This mask is used to prepare the newbackground so that it may becombined with the foreground.<br />
  50. 50. Using the Masks for the Foreground and Background<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />50<br />
  51. 51. Adding the Images Together<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />51<br />
  52. 52. Before and After<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />52<br />Before<br />After<br />
  53. 53. Further Work<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />53<br />Chroma Keying and our Assumptions<br />RBG Color Model<br />Axioms of Probability<br />Gaussian Distributions<br />Gaussian Mixture Model (GMM)<br />Bayes’ Rule and the Expectation Step<br />Update Rules and the Maximization Step<br />Solving a Toy Problem<br />Segmentation using a GMM and Masks<br />Further Work<br />
  54. 54. Further Work<br />Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/<br />54<br />Complex foregrounds and backgrounds<br />Tracking objects<br />Speeding up processing<br />Sampling<br />Smoothing the transition from background to foreground to avoid ragged edges<br />Using alternative color spaces such as Hue-Saturation-Value (HSV) for better classification<br />

×