Shows how Gaussian Mixture Models can be used in image segmentation

1. Allan caine adcaine@cs.uwaterloo.ca http://www.cs.uwaterloo.ca/~adcaine/ May 28, 2009 Chroma Keying using Gaussian Mixture Models (GMMs)

2. Expectations Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 2 You should take notes in case this material appears on an exam or assignment, possibly in a different form It is expected that you will use the Matlab software found at http://www.cs.uwaterloo.ca/~adcaine/ to generate a toy problem and solve it These lecture notes are on-line at the above address

3. Inspiration Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 3 Have you ever wondered how is this done? Can we emulate this process on a computer?

4. Co-ordination Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 4 By viewing an off-camera monitor, theweather announcer can co-ordinate herhand movements with the background.The result is a seemingly realistic televisionbroadcast, in spite of the fact that theannouncer is not actually in front of anyweather map.

5. Teaching Objectives Describe how chroma keying (blue screening) works Theorize how classification might solve the chroma keying problem on a computer Enumerate the axioms of probability Explain the concept behind digital images Express the intuition behind the Expectation-Maximization algorithm in terms of a Gaussian Mixture Model (GMM), using the definition of a Gaussian. Generate a toy GMM problem, and solve the toy problem, recovering the hidden variables 5 Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/

6. Outline Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 6 Chroma Keying and our Assumptions RBG Color Model Axioms of Probability Gaussian Distributions Gaussian Mixture Model (GMM) Bayes’ Rule and the Expectation Step Update Rules and the Maximization Step Solving a Toy Problem Segmentation using a GMM and Masks Further Work

7. Chroma Keying and our Assumptions Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 7 Chroma Keying and our Assumptions RBG Color Model Axioms of Probability Gaussian Distributions Gaussian Mixture Model (GMM) Bayes’ Rule and the Expectation Step Update Rules and the Maximization Step Solving a Toy Problem Segmentation using a GMM and Masks Further Work

8. Basic Idea behind Chroma Keying Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 8 If the background is of one color, saygreen, and the foreground person is notof that color, then the person can besegmented from the background.The person can then be superimposed on abackground of our choosing. Normally, the chroma key is assumed to beknown. To make this problem moreinteresting, we will assume that the chromakey is unknown, and must be learned. So,it’s an AI problem.

9. Assumptions for the Segmentation Problem Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 9 We want to segment the foreground from the background There is a background of a single color However, the background color is unknown; the background color must be learned No part of the person or object in the foreground has the background color The foreground person or object can be in any position in the image; we cannot deduce the background color simply by examining a pixel in a certain fixed position

10. RBG Color Model Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 10 Chroma Keying and our Assumptions RBG Color Model Axioms of Probability Gaussian Distributions Gaussian Mixture Model (GMM) Bayes’ Rule and the Expectation Step Update Rules and the Maximization Step Solving a Toy Problem Segmentation using a GMM and Masks Further Work

11. Explanation of RGB Color Model Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 11 Every color is expressed as an ordered triple Each value can be between 0 and 255; i.e. 256 possibilities  8 bits can store 256 possibilities The numbers represent the amounts of red, green, and blue in that order  (R,G,B) If R = G = B, then the pixel is a shade of gray

12. RGB as a Data Structure in Matlab Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 12

13. Scatter Diagrams Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 13 If only we could determine which class a pixel belonged to: foreground orbackground; the purple sphere or the green background, respectively.

14. Make your own Scatter Plot Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 14

15. Actual Scatter Plot Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 15

16. Axioms of Probability Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 16 Chroma Keying and our Assumptions RBG Color Model Axioms of Probability Gaussian Distributions Gaussian Mixture Model (GMM) Bayes’ Rule and the Expectation Step Update Rules and the Maximization Step Solving a Toy Problem Segmentation using a GMM and Masks Further Work

17. (Some) Axioms of Probability Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 17

18. Bayesian Philosophy of Choosing Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 18 Suppose two events, A and B, occur with probability P(A) and P(B) such that P(A) + P(B) = 1. If you had to place a bet on which event would occur and if P(A) ≠ P(B), what event would you bet on? Suppose P(A) = P(B). What event would you bet on?

19. Gaussian Distributions Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 19 Chroma Keying and our Assumptions RBG Color Model Axioms of Probability Gaussian Distributions Gaussian Mixture Model (GMM) Bayes’ Rule and the Expectation Step Update Rules and the Maximization Step Solving a Toy Problem Segmentation using a GMM and Masks Further Work

20. Gaussian Distributions Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 20

21. One-D Gaussian Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 21

22. Variations in μ Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 22

23. Variations in σ Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 23

24. 2D Gaussian Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 24

25. Comparison of two 2D Gaussians Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 25

26. 2D Gaussian with σx ≠ σy Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 26

27. 3D Gaussian Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 27

28. Gaussian Mixture Model (GMM) Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 28 Chroma Keying and our Assumptions RBG Color Model Axioms of Probability Gaussian Distributions Gaussian Mixture Model (GMM) Bayes’ Rule and the Expectation Step Update Rules and the Maximization Step Solving a Toy Problem Segmentation using a GMM and Masks Further Work

29. GMM Model Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 29 The number of classes is known Each class has a particular mean and variance Each class has a mixing (or membership) weight. All mixing weights add up to one

30. Toy Problem Parameters Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 30

31. Generating the Toy Problem’s DataSet Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 31 20% 40% Start 40%

32. The GMM Problem Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 32 ??% ??% Start ??%

33. Bayes’ Rule and the Expectation Step Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 33 Chroma Keying and our Assumptions RBG Color Model Axioms of Probability Gaussian Distributions Gaussian Mixture Model (GMM) Bayes’ Rule and the Expectation Step Update Rules and the Maximization Step Solving a Toy Problem Segmentation using a GMM and Masks Further Work

34. Bayes’ Rule Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 34 The likelihood of A given B A prior probability on B The likelihood of B given A A normalization constant

35. The Main of Idea of GMMs: E-Step Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 35 The probability of position j havinga pixel with values x given class i. Basically, evaluate a Gaussian at xusing μi and Σi as parameters The probability that position j is generatedby class 1 or 2: foreground or background The prior probability ofclass i; i.e. the mixing weights Essentially, a normalization constantto ensure that P(C=1|xj) +P(C=2|xj) = 1for j fixed.

36. Update Rules and the Maximization Step Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 36 Chroma Keying and our Assumptions RBG Color Model Axioms of Probability Gaussian Distributions Gaussian Mixture Model (GMM) Bayes’ Rule and the Expectation Step Update Rules and the Maximization Step Solving a Toy Problem Segmentation using a GMM and Masks Further Work

37. Update Rules: M-Step Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 37 In words, define what each ofthese mathematical statementsmean. What do you think is the intuitionbehind these three update rules?

38. Solving a Toy Problem Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 38 Chroma Keying and our Assumptions RBG Color Model Axioms of Probability Gaussian Distributions Gaussian Mixture Model (GMM) Bayes’ Rule and the Expectation Step Update Rules and the Maximization Step Solving a Toy Problem Segmentation using a GMM and Masks Further Work

39. Toy Problem Visually Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 39

40. Typical Output Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 40 Class 2 Class 3 Class 1

41. Solution to the Toy Problem Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 41

42. Comparison of Statistics Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 42

43. Recovery of the Model Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 43

44. 3D GMM Problem Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 44

45. Segmentation using a GMM and Masks Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 45 Chroma Keying and our Assumptions RBG Color Model Axioms of Probability Gaussian Distributions Gaussian Mixture Model (GMM) Bayes’ Rule and the Expectation Step Update Rules and the Maximization Step Solving a Toy Problem Segmentation using a GMM and Masks Further Work

46. Segmentation Example Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 46

47. Segmentation Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 47 The upshot of all of this is that the foreground has a parameterization of: The novelty of this result is that we have a complete and numericalcharacterization of the image without knowing beforehand thecolor of the background. These parameters have been learned.

48. Producing the Mask Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 48 A Gaussian computed using themean and variance of the foregroundclass. The mixing weight 0.9695 A Gaussian computed using themean and variance of the backgroundclass. The mixing weight 0.0305 How would we use a Bayesian philosophy to decide ifa pixel represents the foreground or background?

49. Masks Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 49 This mask permits the sphereto show through while excludingthe background. It’s used tosegment the sphere; that is,segment out the foreground. This mask is the logical complementof the first mask. It indicates wherethe background pixels can be found.This mask is used to prepare the newbackground so that it may becombined with the foreground.

50. Using the Masks for the Foreground and Background Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 50

51. Adding the Images Together Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 51

52. Before and After Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 52 Before After

53. Further Work Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 53 Chroma Keying and our Assumptions RBG Color Model Axioms of Probability Gaussian Distributions Gaussian Mixture Model (GMM) Bayes’ Rule and the Expectation Step Update Rules and the Maximization Step Solving a Toy Problem Segmentation using a GMM and Masks Further Work

54. Further Work Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/ 54 Complex foregrounds and backgrounds Tracking objects Speeding up processing Sampling Smoothing the transition from background to foreground to avoid ragged edges Using alternative color spaces such as Hue-Saturation-Value (HSV) for better classification