+ Follow

Parametric Density Estimation using Gaussian Mixture Models

by Pardis N

384 views

Based on tutorials by Jeff A. Bilmes and by Ludwig Schwardt

Based on tutorials by Jeff A. Bilmes and by Ludwig Schwardt

Slides by Pardis Noorzad

Accessibility

View text version

Upload Details

Uploaded via SlideShare as Adobe PDF

Usage Rights

File a copyright complaint

Statistics

Likes: 0
Downloads: 12
Comments: 0
Embed Views: 0
Views on SlideShare: 384
Total Views: 384

Parametric Density Estimation using Gaussian Mixture Models Presentation Transcript

Parametric Density Estimation using Gaussian Mixture Models Parametric Density Estimation using Gaussian Mixture Models An Application of the EM Algorithm Based on tutorials by Je A. Bilmes and by Ludwig Schwardt Amirkabir University of Technology Bahman 12, 1389

Parametric Density Estimation using Gaussian Mixture Models Outline 1 Introduction 2 Density Estimation 3 Gaussian Mixture Model 4 Some Results 5 Other Applications of EM

Parametric Density Estimation using Gaussian Mixture Models Introduction Outline 1 Introduction 2 Density Estimation 3 Gaussian Mixture Model 4 Some Results 5 Other Applications of EM

Parametric Density Estimation using Gaussian Mixture Models Introduction What does EM do? iterative maximization of the likelihood function when data has missing values or when maximization of the likelihood is dicult data augmentation X is incomplete data Z = (X; Y) is the complete data set

Parametric Density Estimation using Gaussian Mixture Models Introduction EM and MLE maximum likelihood estimation estimates density parameters =) EM app: parametric density estimation when density is a mixture of Gaussians

Parametric Density Estimation using Gaussian Mixture Models Density Estimation Outline 1 Introduction 2 Density Estimation 3 Gaussian Mixture Model 4 Some Results 5 Other Applications of EM

Parametric Density Estimation using Gaussian Mixture Models Density Estimation The density estimation problem Denition Our parametric density estimation problem: X=f xi Ng i=1 , xi P R D f (xi jΘ), i.i.d. assumption f (xi jΘ) = K=1 k p(xi jk ) k K components (given) Θ = (1 ; : : : ; K ; 1 ; : : : ; K )

Parametric Density Estimation using Gaussian Mixture Models Density Estimation Constraint on mixing probabilities i Sum to unity p(xi jk )dxi = 1 RD 1= f (xi jΘ)dxi RD K = k p(xi jk )dxi RD k=1 K = k : k=1

Parametric Density Estimation using Gaussian Mixture Models Density Estimation The maximum likelihood problem Denition Our MLE problem: N p(XjΘ) = i=1 p(xi jΘ) = v(ΘjX) Θ£ = arg maxΘ v(ΘjX) the log-likelihood: log(v(ΘjX)) N K ¡ log(v(ΘjX)) = i=1 log k=1 k p(xi ; k ) dicult to optimize b/c it contains log of sum

Parametric Density Estimation using Gaussian Mixture Models Density Estimation The EM formulation Denition Our EM problem: Z = (X; Y) v(ΘjZ) = v(ΘjX; Y) = p(X; YjΘ) yi P 1 : : : K yi = k if the ith sample was generated by the kth component Y is a random vector N log(v(ΘjX; Y)) = i=1 log( yi p(xi jyi ))

Parametric Density Estimation using Gaussian Mixture Models Density Estimation The EM formulation Continued Denition The E- and M-steps with the auxiliary function Q: E-step: Q(Θ; Θ(i 1) ) = E [log(p(X; YjΘ))jX; Θ(i 1) ] M-step: Θ(i) = arg maxΘ Q(Θ; Θ(i 1) )

Parametric Density Estimation using Gaussian Mixture Models Density Estimation The EM formulation The Q function Q(Θ; Θ(t) ) = E [log(p(X; YjΘ))jX; Θ(t) ] K N = log(k p(xi jk ))p(kjxi ; Θ(t) ) k=1 i=1 K N = log(k )p(kjxi ; Θ(t) ) k=1 i=1 K N + log(p(xi jk ))p(kjxi ; Θ(t) ) k=1 i=1

Parametric Density Estimation using Gaussian Mixture Models Density Estimation The EM formulation Solving for k We use the Lagrange multiplier to enforce k k = 1. @ [ log(k )p(kjxi ; Θ(t) ) + ( k 1)] = 0 @k k i k 1 p(kjxi ; Θ(t) ) + = 0 i k Summing both sides over k, we get = N . 1 k = p(kjxi ; Θ(t) ): N i

Parametric Density Estimation using Gaussian Mixture Models Gaussian Mixture Model Outline 1 Introduction 2 Density Estimation 3 Gaussian Mixture Model 4 Some Results 5 Other Applications of EM

Parametric Density Estimation using Gaussian Mixture Models Gaussian Mixture Model GMM Solving for mk and Σk Gaussian components D p(xi jxk ; Σk ) = (2) 2 jΣk j 2 exp( 1 (xi mk )T Σ 1 (xi mk )) 1 2 k i xi p(kjxi ; Θ(t) ) mk = i p(kjxi ; Θ ) (t) p(kjxi ; Θ(t) )(xi mk )(xi mk )T Σk = p(kjxi ; Θ(t) )

Parametric Density Estimation using Gaussian Mixture Models Gaussian Mixture Model Choice of Σk Full covariance Fits data best, but costly in high-dimensional space. Figure: full covariance

Parametric Density Estimation using Gaussian Mixture Models Gaussian Mixture Model Choice of Σk Diagonal covariance A tradeo between cost and quality. Figure: diagonal covariance

Parametric Density Estimation using Gaussian Mixture Models Gaussian Mixture Model Choice of Σk Spherical covariance, Σk = k I 2 Needs many components to cover data. Figure: spherical covariance Decision should be based on the size of training data set so that all parameters may be tuned

Parametric Density Estimation using Gaussian Mixture Models Some Results Outline 1 Introduction 2 Density Estimation 3 Gaussian Mixture Model 4 Some Results 5 Other Applications of EM

Parametric Density Estimation using Gaussian Mixture Models Some Results Data data generated from a mixture of three bivariate Gaussians

Parametric Density Estimation using Gaussian Mixture Models Some Results PDFs estimated pdf contours (after 43 iterations)

Parametric Density Estimation using Gaussian Mixture Models Some Results Clusters xi assigned to the cluster k corresponding to the highest p(kjxi ; Θ)

Parametric Density Estimation using Gaussian Mixture Models Other Applications of EM Outline 1 Introduction 2 Density Estimation 3 Gaussian Mixture Model 4 Some Results 5 Other Applications of EM

Parametric Density Estimation using Gaussian Mixture Models Other Applications of EM A problem with missing data Life-testing experiment lifetime of light bulbs follows an exponential distribution with mean two separate experiments N bulbs tested until failed failure times recorded as x1 ; : : : ; x N M bulbs tested failure times not recorded only number of bulbs , that failed at time r t missing data are failure times 1 M u ;:::;u

Parametric Density Estimation using Gaussian Mixture Models Other Applications of EM EM formulation Life-testing experiment M log(v(jx; u)) = N (log + x=) i (log + ui =) expected value for bulb still burning: t + t k one that burned out: 1tee = (k) = th(k) ( ) t = E-step: Q(; (k) ) = (N + M ) log 1 (N x + (M r)(t + k ) + r( k th k )) ( ) ( ) ( ) M-step: 1 N +M ( N x + (M r)(t + (k) ) + r((k) th(k) ))

Parametric Density Estimation using Gaussian Mixture Models Thank you for your attention. Any questions?