Azure Monitor & Application Insight to monitor Infrastructure & Application
Slides: Simplifying Gaussian Mixture Models Via Entropic Quantization (EUSIPCO 2009)
1. Simplifying Gaussian Mixture Models
Via Entropic Quantization
Frank Nielsen1 2 , Vincent Garcia1 , and Richard Nock3
1 Ecole Polytechnique (Paris, France)
Sony Computer Science Laboratories (Tokyo, Japan)
Universit´ des Antilles et de la Guyane (Guadeloupe, France)
e
2
3
28th august 2009
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
1 / 23
2. Introduction
Plan
1
Introduction
Mixture models
Problem
Mixture model simplification
2
Mixture model simplification
KLD and Bregman divergence
Sided BKMC
Symmetric BKMC
jMEF
3
Experiments
Quality measure and initialization
Sided BKMC
BKMC vs UTAC
4
Conclusion
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
2 / 23
3. Introduction
Mixture models
Mixture models
Mixture model is a powerful framework to estimate PDF
Mixture model f
n
f (x) =
αi fi (x)
i=1
where αi ≥ 0 denotes a weight with
n
i=1 αi
=1
If f is a Gaussian mixture model (GMM),
(x − µi )T Σ−1 (x − µi )
1
i
fi (x) =
exp −
2
(2π)d/2 |Σi |1/2
with µi mean and Σi covariance matrix
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
3 / 23
4. Introduction
Problem
Problem
2.5
2
1.5
1
0.5
0
−0.5
0
0.5
1
1.5
Density estimation using kernel-based Parzen estimator
Mixture models usually contain a lot of components
Estimation of statistical measures is computationally expensive
Need to reduce the number of components
Re-lear a simpler mixture model from dataset
Simplify the mixture model f
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
4 / 23
5. Introduction
Mixture model simplification
Mixture model simplification
Given a mixture model f of n components
n
f (x) =
αi fi (x)
i=1
Compute a mixture model g of m components
m
αj gj (x)
g (x) =
j=1
such as g is the best approximation of f
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
5 / 23
6. Mixture model simplification
Plan
1
Introduction
Mixture models
Problem
Mixture model simplification
2
Mixture model simplification
KLD and Bregman divergence
Sided BKMC
Symmetric BKMC
jMEF
3
Experiments
Quality measure and initialization
Sided BKMC
BKMC vs UTAC
4
Conclusion
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
6 / 23
7. Mixture model simplification
KLD and Bregman divergence
Relative entropy and Bregman divergence
The fundamental measure between statistical distributions is the
relative entropy, also called the Kullback-Leibler divergence
Given fi and fj two distributions, the KLD is given by
KLD(fi ||fj ) =
fi (x) log
fi (x)
dx
fj (x)
In the case of normal distriubtions
det Σj
1
1
KLD(fi ||fj ) = log
+ tr Σ−1 Σi
j
2
det Σi
2
1
d
+ (µj − µi )T Σ−1 (µj − µi ) −
j
2
2
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
7 / 23
8. Mixture model simplification
KLD and Bregman divergence
Relative entropy and Bregman divergence
Nomral distributions belong to the class of exponential families
Canonical form of exponential families
f (x) = exp
˜
˜
Θ, t(x) − F (Θ) + C (x)
Estimation of the KLD by computing the Bregman divergence defined
for the log normalizer F
˜ ˜
KLD(fi ||fj ) = DF (Θj ||Θi )
where
˜ ˜
˜
˜
˜
˜
˜
DF (Θj ||Θi ) = F (Θj ) − F (Θi ) − Θj − Θi , F (Θi )
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
8 / 23
9. Mixture model simplification
KLD and Bregman divergence
Relative entropy and Bregman divergence
For multivariate normal distributions
Sufficient statistics
1
t(x) = (x, − xx T )
2
Natural parameters
1
˜
Θ = (θ, Θ) = (Σ−1 µ, Σ−1 )
2
Log normalizer
1
d
1
˜
F (Θ) = tr(Θ−1 θθT ) − log det Θ + log π
4
2
2
˜
F (Θ) =
V. Garcia (X, Paris, France)
1 −1
1
1
Θ θ , − Θ−1 − (Θ−1 θ)(Θ−1 θ)T
2
2
4
Simplifying GMMs
28th august 2009
9 / 23
10. Mixture model simplification
Sided BKMC
Bregman k-means clustering
K-means clustering
Set of points
Initialize k centroids = k classes
Repetition until convergence
Repartition step (distance)
Computation of centroids (centers of mass)
Bregman K-means clustering
Set of distributions
Initialize k centroids (αi , gi ) = GMM with k components
Repetition until convergence
Repartition step (sided Bregman divergence)
Computation of centroids (sided centroids)
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
10 / 23
11. Mixture model simplification
Sided BKMC
Sided centroids
5 multivariate Gaussians
Right-centroid
Left-centroid
http://www.sonycsl.co.jp/person/nielsen/BNCj/
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
11 / 23
12. Mixture model simplification
Sided BKMC
Right-sided BKMC algorithm
1: Initialize the GMM g
2: repeat
3:
Compute the cluster C : the Gaussian fi belongs to cluster Cj if and only if
˜ ˜
˜ ˜
DF (Θi Θj ) < DF (Θi Θl ), ∀l ∈ [1, m] {j}
4:
Compute the centroids: the weight and the natural parameters of the j-th
centroid (i.e. Gaussian gj ) are given by:
αj =
αi ,
i
The sum
i
θj =
i
αi θi
,
i αi
Θj =
i
αi Θi
i αi
is performed on i ∈ [1, m] such as fi ∈ Cj
5: until the cluster does not change between two iterations
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
12 / 23
13. Mixture model simplification
Sided BKMC
Left-sided BKMC algorithm
1: Initialize the GMM g
2: repeat
3:
Compute the cluster C : the Gaussian fi belongs to cluster Cj if and only if
˜ ˜
˜ ˜
DF (Θj Θi ) < DF (Θl Θi ), ∀l ∈ [1, m] {j}
4:
Compute the centroids: the weight and the natural parameters of the j-th
centroid (i.e. Gaussian gj ) are given by:
αj =
αi ,
˜
Θj =
F −1
i
i
where
˜
F −1 (Θ) =
− Θ + θθT
−1
θ, −
αi
αj
˜
F Θi
1
Θ + θθT
2
−1
5: until the cluster does not change between two iterations
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
13 / 23
14. Mixture model simplification
Symmetric BKMC
Symmetric BKMC algorithm
Symmetric similarity measure can be required (e.g. CBIR)
Repartition step: Symmetric Bregman divergence
˜ ˜
SDF (Θp , Θq ) =
˜ ˜
˜ ˜
DF (Θq ||Θp ) + DF (Θp ||Θq )
2
Computation of symmetric centroid:
Compute right and left centroids (cr and cl )
The symmetric centroid cs belongs to the geodesic link joining cr and cl
cλ =
F −1 (λ F (cr ) + (1 − λ) F (cl ))
The symmetric centroid cs = cλ verifies
SDF (cλ , cr ) = SDF (cλ , cl ).
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
14 / 23
15. Mixture model simplification
jMEF
jMEF
jMEF : Java library for Mixture of Exponential Families
Create and manage MEF
Simplify MEF using BKMC
Available on line at www.lix.polytechnique.fr/∼nielsen/MEF
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
15 / 23
16. Experiments
Plan
1
Introduction
Mixture models
Problem
Mixture model simplification
2
Mixture model simplification
KLD and Bregman divergence
Sided BKMC
Symmetric BKMC
jMEF
3
Experiments
Quality measure and initialization
Sided BKMC
BKMC vs UTAC
4
Conclusion
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
16 / 23
17. Experiments
Quality measure and initialization
Quality measure and initialization
Simplification quality measure
KLD(f g ) (right-sided)
No closed-form expression
Draw 10,000 points to estimate this KLD (Monte-Carlo)
Initial GMM f
Learnt from an image
K-means on RGB pixels ⇒ 32 classes
EM algorithm ⇒ fi
Weights αi : proportion of pixels in each class
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
17 / 23
18. Experiments
Sided BKMC
Sided BKMC
Evolution of KLD(f g ) as a function of m
The simplification quality increases with m
Left-sided BKMC provides the best results
Right-sided BKMC provides the worst results
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
18 / 23
19. Experiments
BKMC vs UTAC
BKMC vs UTAC
UTAC algorithm based on sigma points + EM algorithm
BKMC provides better results than UTAC
BKMC is faster than UTAC: 20ms vs 100ms
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
19 / 23
20. Experiments
BKMC vs UTAC
Clustering-based image segmentation
Image
UTAC
BKMC
KLD=0.23
KLD=0.11
KLD=0.16
V. Garcia (X, Paris, France)
f
KLD=0.13
Simplifying GMMs
28th august 2009
20 / 23
21. Experiments
BKMC vs UTAC
Clustering-based image segmentation
Image
UTAC
BKMC
KLD=0.69
KLD=0.53
KLD=0.36
V. Garcia (X, Paris, France)
f
KLD=0.18
Simplifying GMMs
28th august 2009
21 / 23
22. Conclusion
Plan
1
Introduction
Mixture models
Problem
Mixture model simplification
2
Mixture model simplification
KLD and Bregman divergence
Sided BKMC
Symmetric BKMC
jMEF
3
Experiments
Quality measure and initialization
Sided BKMC
BKMC vs UTAC
4
Conclusion
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
22 / 23
23. Conclusion
Conclusion
GMM simplification algorithm based on k-means and Bregman
divergence
BKMC is faster and provides better results than UTAC algorithm
BKMC extends to mixtures of exponential families
jMEF available on line at www.lix.polytechnique.fr/∼nielsen/MEF
Included features:
Create/manage mixtures of exponential families
BKMC algorithm
Hierarchical GMM (ACCV 2009)
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
23 / 23