Parametric Density Estimation using Gaussian Mixture Models

1. Parametric Density Estimation using Gaussian Mixture Models Parametric Density Estimation using Gaussian Mixture Models An Application of the EM Algorithm Based on tutorials by Je A. Bilmes and by Ludwig Schwardt Amirkabir University of Technology Bahman 12, 1389

2. Parametric Density Estimation using Gaussian Mixture Models Outline 1 Introduction 2 Density Estimation 3 Gaussian Mixture Model 4 Some Results 5 Other Applications of EM

7. Parametric Density Estimation using Gaussian Mixture Models Introduction Outline 1 Introduction 2 Density Estimation 3 Gaussian Mixture Model 4 Some Results 5 Other Applications of EM

8. Parametric Density Estimation using Gaussian Mixture Models Introduction What does EM do? iterative maximization of the likelihood function when data has missing values or when maximization of the likelihood is dicult data augmentation X is incomplete data Z = (X; Y) is the complete data set

15. Parametric Density Estimation using Gaussian Mixture Models Introduction EM and MLE maximum likelihood estimation estimates density parameters =) EM app: parametric density estimation when density is a mixture of Gaussians

20. Parametric Density Estimation using Gaussian Mixture Models Density Estimation Outline 1 Introduction 2 Density Estimation 3 Gaussian Mixture Model 4 Some Results 5 Other Applications of EM

21. Parametric Density Estimation using Gaussian Mixture Models Density Estimation The density estimation problem Denition Our parametric density estimation problem: X=f xi Ng i=1 , xi P R D f (xi jΘ), i.i.d. assumption € f (xi jΘ) = K=1 k p(xi jk ) k K components (given) Θ = (1 ; : : : ; K ; 1 ; : : : ; K )

27. Parametric Density Estimation using Gaussian Mixture Models Density Estimation Constraint on mixing probabilities i Sum to unity p(xi jk )dxi = 1 RD 1= f (xi jΘ)dxi RD K ˆ = k p(xi jk )dxi RD k=1 K ˆ = k : k=1

28. Parametric Density Estimation using Gaussian Mixture Models Density Estimation Constraint on mixing probabilities i Sum to unity p(xi jk )dxi = 1 RD 1= f (xi jΘ)dxi RD K ˆ = k p(xi jk )dxi RD k=1 K ˆ = k : k=1

29. Parametric Density Estimation using Gaussian Mixture Models Density Estimation The maximum likelihood problem Denition Our MLE problem: N p(XjΘ) = i=1 p(xi jΘ) = v(ΘjX) Θ£ = arg maxΘ v(ΘjX) the log-likelihood: log(v(ΘjX)) €N €K ¡ log(v(ΘjX)) = i=1 log k=1 k p(xi ; k ) dicult to optimize b/c it contains log of sum

35. Parametric Density Estimation using Gaussian Mixture Models Density Estimation The EM formulation Denition Our EM problem: Z = (X; Y) v(ΘjZ) = v(ΘjX; Y) = p(X; YjΘ) yi P 1 : : : K yi = k if the ith sample was generated by the kth component Y is a random vector €N log(v(ΘjX; Y)) = i=1 log( yi p(xi jyi ))

42. Parametric Density Estimation using Gaussian Mixture Models Density Estimation The EM formulation Continued Denition The E- and M-steps with the auxiliary function Q: E-step: Q(Θ; Θ(i 1) ) = E [log(p(X; YjΘ))jX; Θ(i 1) ] M-step: Θ(i) = arg maxΘ Q(Θ; Θ(i 1) )

45. Parametric Density Estimation using Gaussian Mixture Models Density Estimation The EM formulation The Q function Q(Θ; Θ(t) ) = E [log(p(X; YjΘ))jX; Θ(t) ] K N ˆˆ = log(k p(xi jk ))p(kjxi ; Θ(t) ) k=1 i=1 K N ˆˆ = log(k )p(kjxi ; Θ(t) ) k=1 i=1 K N ˆˆ + log(p(xi jk ))p(kjxi ; Θ(t) ) k=1 i=1

46. Parametric Density Estimation using Gaussian Mixture Models Density Estimation The EM formulation Solving for k € We use the Lagrange multiplier to enforce k k = 1. @ ˆˆ ˆ [ log(k )p(kjxi ; Θ(t) ) + ( k 1)] = 0 @k k i k ˆ 1 p(kjxi ; Θ(t) ) + = 0 i k Summing both sides over k, we get = N . 1 ˆ k = p(kjxi ; Θ(t) ): N i

51. Parametric Density Estimation using Gaussian Mixture Models Gaussian Mixture Model Outline 1 Introduction 2 Density Estimation 3 Gaussian Mixture Model 4 Some Results 5 Other Applications of EM

52. Parametric Density Estimation using Gaussian Mixture Models Gaussian Mixture Model GMM Solving for mk and Σk Gaussian components D p(xi jxk ; Σk ) = (2) 2 jΣk j 2 exp( 1 (xi mk )T Σ 1 (xi mk )) 1 2 k € i xi p(kjxi ; Θ(t) ) mk = € i p(kjxi ; Θ ) (t) p(kjxi ; Θ(t) )(xi mk )(xi mk )T Σk = p(kjxi ; Θ(t) )

53. Parametric Density Estimation using Gaussian Mixture Models Gaussian Mixture Model GMM Solving for mk and Σk Gaussian components D p(xi jxk ; Σk ) = (2) 2 jΣk j 2 exp( 1 (xi mk )T Σ 1 (xi mk )) 1 2 k € i xi p(kjxi ; Θ(t) ) mk = € i p(kjxi ; Θ ) (t) p(kjxi ; Θ(t) )(xi mk )(xi mk )T Σk = p(kjxi ; Θ(t) )

54. Parametric Density Estimation using Gaussian Mixture Models Gaussian Mixture Model Choice of Σk Full covariance Fits data best, but costly in high-dimensional space. Figure: full covariance

55. Parametric Density Estimation using Gaussian Mixture Models Gaussian Mixture Model Choice of Σk Diagonal covariance A tradeo between cost and quality. Figure: diagonal covariance

56. Parametric Density Estimation using Gaussian Mixture Models Gaussian Mixture Model Choice of Σk Spherical covariance, Σk = k I 2 Needs many components to cover data. Figure: spherical covariance Decision should be based on the size of training data set so that all parameters may be tuned

59. Parametric Density Estimation using Gaussian Mixture Models Some Results Outline 1 Introduction 2 Density Estimation 3 Gaussian Mixture Model 4 Some Results 5 Other Applications of EM

60. Parametric Density Estimation using Gaussian Mixture Models Some Results Data data generated from a mixture of three bivariate Gaussians

61. Parametric Density Estimation using Gaussian Mixture Models Some Results PDFs estimated pdf contours (after 43 iterations)

62. Parametric Density Estimation using Gaussian Mixture Models Some Results Clusters xi assigned to the cluster k corresponding to the highest p(kjxi ; Θ)

63. Parametric Density Estimation using Gaussian Mixture Models Other Applications of EM Outline 1 Introduction 2 Density Estimation 3 Gaussian Mixture Model 4 Some Results 5 Other Applications of EM

64. Parametric Density Estimation using Gaussian Mixture Models Other Applications of EM A problem with missing data Life-testing experiment lifetime of light bulbs follows an exponential distribution with mean two separate experiments N bulbs tested until failed failure times recorded as x1 ; : : : ; x N M bulbs tested failure times not recorded only number of bulbs , that failed at time r t missing data are failure times 1 M u ;:::;u

73. Parametric Density Estimation using Gaussian Mixture Models Other Applications of EM EM formulation Life-testing experiment €M log(v(jx; u)) = N (log + x=) i (log + ui =) expected value for bulb still burning: t + t k one that burned out: 1tee = (k) = th(k) ( ) t = E-step: Q(; (k) ) = (N + M ) log 1 (N x + (M r)(t + k ) + r( k th k )) ( ) ( ) ( ) M-step: 1 N +M ( N x + (M r)(t + (k) ) + r((k) th(k) ))

79. Parametric Density Estimation using Gaussian Mixture Models Thank you for your attention. Any questions?

Parametric Density Estimation using Gaussian Mixture Models

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Parametric Density Estimation using Gaussian Mixture Models

Similar to Parametric Density Estimation using Gaussian Mixture Models (20)

Recently uploaded

Recently uploaded (20)

Parametric Density Estimation using Gaussian Mixture Models