Shinichi Tamura, profile picture
Uploaded byShinichi Tamura
PDF, PPTX3,446 views

PRML 9.1-9.2: K-means Clustering & Mixtures of Gaussians

The document discusses k-means clustering and Gaussian mixtures, focusing on the clustering problem, its application in image compression, and the EM algorithm for Gaussian mixtures. It explains the iterative method for k-means, involving steps to assign data to the nearest cluster and update cluster means. Variations such as the online version of k-means and strategies like the Robbins-Monro algorithm are also introduced.

Technology◦
Related topics:
K-means•

Embed presentation

Download as PDF, PPTX
1 / 136
2 / 136
3 / 136
4 / 136
5 / 136
6 / 136
7 / 136
8 / 136
9 / 136
10 / 136
11 / 136
12 / 136
13 / 136
14 / 136
15 / 136
16 / 136
17 / 136
18 / 136
19 / 136
20 / 136
21 / 136
22 / 136
23 / 136
24 / 136
25 / 136
26 / 136
27 / 136
28 / 136
29 / 136
30 / 136
31 / 136
32 / 136
33 / 136
34 / 136
35 / 136
36 / 136
37 / 136
38 / 136
39 / 136
40 / 136
41 / 136
42 / 136
43 / 136
44 / 136
45 / 136
46 / 136
47 / 136
48 / 136
49 / 136
50 / 136
51 / 136
52 / 136
53 / 136
54 / 136
55 / 136
56 / 136
57 / 136
58 / 136
59 / 136
60 / 136
61 / 136
62 / 136
63 / 136
64 / 136
65 / 136
66 / 136
67 / 136
68 / 136
69 / 136
70 / 136
71 / 136
72 / 136
73 / 136
74 / 136
75 / 136
76 / 136
77 / 136
78 / 136
79 / 136
80 / 136
81 / 136
82 / 136
83 / 136
84 / 136
85 / 136
86 / 136
87 / 136
88 / 136
89 / 136
90 / 136
91 / 136
92 / 136
93 / 136
94 / 136
95 / 136
96 / 136
97 / 136
98 / 136
99 / 136
100 / 136
101 / 136
102 / 136
103 / 136
104 / 136
105 / 136
106 / 136
107 / 136
108 / 136
109 / 136
110 / 136
111 / 136
112 / 136
113 / 136
114 / 136
115 / 136
116 / 136
117 / 136
118 / 136
119 / 136
120 / 136
121 / 136
122 / 136
123 / 136
124 / 136
125 / 136
126 / 136
127 / 136
128 / 136
129 / 136
130 / 136
131 / 136
132 / 136
133 / 136
134 / 136
135 / 136
136 / 136
PRML 9.1-9.2 K-means Clustering & Mixtures of Gaussians July 16, 2014 by Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Clustering Problem An unsupervised machine learning problem Divide data in some group (=cluster) where ü  similar data > same group ü  dissimilar data > different group July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Clustering Problem Divide data in some group (=cluster) where ü  similar data > same group ü  dissimilar data > different group July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Clustering Problem Divide data in some group (=cluster) where ü  similar data > same group ü  dissimilar data > different group Minimize N n=1 xn − µk(n) 2 July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Clustering Problem Divide data in some group (=cluster) where ü  similar data > same group ü  dissimilar data > different group Minimize N n=1 xn − µk(n) 2 Center of the cluster July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Clustering Problem Given data set and # of cluster K Let be cluster representative and be assignment indicator ( ), Here, J is called “distortion measure”. X = {x1, . . . , xN } µk rnk rnk = 1 if x ∈ Ck Minimize J = N n=1 K k=1 rnk xn − µk 2 July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering How to solve that? July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering How to solve that? and are dependent each other > No closed form solution µk rnk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering How to solve that? and are dependent each other > No closed form solution Use iterative algorithm ! µk rnk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Strategy and can't be updated simultaneously µk rnk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Strategy and can't be updated simultaneously > Update them one by one µk rnk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Update of (assignment) Since each can be determined independently, J will be minimum if they are assigned to the nearest . rnk xn µk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Update of (assignment) Since each can be determined independently, J will be minimum if they are assigned to the nearest . Therefore, rnk xn µk rnk = 1 if k = arg minj xn − µj 2 , 0 otherwise. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Update of (parameter estimation) Optimal is obtained by setting derivative 0. µk µk ∂ ∂µk N n=1 K k =1 rnk xn − µk 2 = 0. ⇐⇒ 2 N n=1 rnk(xn − µk) = 0. ∴ µk = N n=1 rnkxn N n=1 rnk = 1 Nk xn∈Ck xn. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Update of (parameter estimation) Optimal is obtained by setting derivative 0. µk µk ∂ ∂µk N n=1 K k =1 rnk xn − µk 2 = 0. ⇐⇒ 2 N n=1 rnk(xn − µk) = 0. ∴ µk = N n=1 rnkxn N n=1 rnk = 1 Nk xn∈Ck xn. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Update of (parameter estimation) Optimal is obtained by setting derivative 0. µk µk ∂ ∂µk N n=1 K k =1 rnk xn − µk 2 = 0. ⇐⇒ 2 N n=1 rnk(xn − µk) = 0. ∴ µk = N n=1 rnkxn N n=1 rnk = 1 Nk xn∈Ck xn. Mean of the cluster July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Update of (parameter estimation) Optimal is obtained by setting derivative 0. µk µk ∂ ∂µk N n=1 K k =1 rnk xn − µk 2 = 0. ⇐⇒ 2 N n=1 rnk(xn − µk) = 0. ∴ µk = N n=1 rnkxn N n=1 rnk = 1 Nk xn∈Ck xn. Mean of the cluster July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Update of (parameter estimation) Optimal is obtained by setting derivative 0. µk µk ∂ ∂µk N n=1 K k =1 rnk xn − µk 2 = 0. ⇐⇒ 2 N n=1 rnk(xn − µk) = 0. ∴ µk = N n=1 rnkxn N n=1 rnk = 1 Nk xn∈Ck xn. Mean of the cluster is the mean of the cluster Cost function J corresponds to the sum of inner-class variance! µk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Update of (parameter estimation) Optimal is obtained by setting derivative 0. µk µk ∂ ∂µk N n=1 K k =1 rnk xn − µk 2 = 0. ⇐⇒ 2 N n=1 rnk(xn − µk) = 0. ∴ µk = N n=1 rnkxn N n=1 rnk = 1 Nk xn∈Ck xn. Mean of the cluster is the mean of the cluster Cost function J corresponds to the sum of inner-class variance! µk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering K-means algorithm 1. Initialize , 2. Repeat following two steps until converge i) Assign each to closest ii) Update to the mean of the cluster µk rnk xn µk µk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering K-means algorithm 1. Initialize , 2. Repeat following two steps until converge i) Assign each to closest ii) Update to the mean of the cluster µk rnk xn µk µk E step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering K-means algorithm 1. Initialize , 2. Repeat following two steps until converge i) Assign each to closest ii) Update to the mean of the cluster µk rnk xn µk µk M step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Convergence property Both steps never increase J, so we can obtain better result in every iteration. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Convergence property Both steps never increase J, so we can obtain better result in every iteration. Since is finite, algorithm converge after finite iterations. rnk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm E step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm M step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm E step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm M step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm E step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm M step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm E step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm M step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Calculation performance E step ... Comparison of every data point and every cluster mean > O(KN) µk xn July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Calculation performance E step ... Comparison of every data point and every cluster mean > O(KN) µk xn Not good July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Calculation performance E step ... Comparison of every data point and every cluster mean > O(KN) µk xn Not good Improve with kd-tree, triangle inequality...etc July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Calculation performance E step ... Comparison of every data point and every cluster mean > O(KN) µk xn July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Calculation performance E step ... Comparison of every data point and every cluster mean > O(KN) M step ... Calculation of mean for every cluster > O(N) µk xn July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Here, two variation will be introduced: 1.  On-line version 2.  General dissimilarity July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Here, two variation will be introduced: 1.  On-line version 2.  General dissimilarity July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 1. On-line version The case where one datum is observed at once. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 1. On-line version The case where one datum is observed at once. > Apply Robbins-Monro algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 1. On-line version The case where one datum is observed at once. > Apply Robbins-Monro algorithm µnew k = µold k + ηn(xn − µold k ). July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 1. On-line version The case where one datum is observed at once. > Apply Robbins-Monro algorithm µnew k = µold k + ηn(xn − µold k ). Learning rate July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 1. On-line version The case where one datum is observed at once. > Apply Robbins-Monro algorithm µnew k = µold k + ηn(xn − µold k ). Learning rate Decrease with iteration July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering Here, two variation will be introduced: 1.  On-line version 2.  General dissimilarity July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 2. General dissimilarity Euclidian distance is not ü  appropriate to categorical data, etc. ü  robust to outlier. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 2. General dissimilarity Euclidian distance is not ü  appropriate to categorical data, etc. ü  robust to outlier. > Use general dissimilarity measure V(x, x ) July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 2. General dissimilarity Euclidian distance is not ü  appropriate to categorical data, etc. ü  robust to outlier. > Use general dissimilarity measure V(x, x ) E step ... No difference July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 2. General dissimilarity Euclidian distance is not ü  appropriate to categorical data, etc. ü  robust to outlier. > Use general dissimilarity measure V(x, x ) M step ... Not assured J is easy to minimize July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 2. General dissimilarity To make M-step easy, restrict to the vector chosen from > A solution can be obtained by finite number of comparison µk {xn} July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 2. General dissimilarity To make M-step easy, restrict to the vector chosen from > A solution can be obtained by finite number of comparison µk {xn} µk = arg min xn xn ∈Ck V(xn, xn ) July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Application for Image Compression K-means algorithm can be applied to Image Compression and Segmentation July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Application for Image Compression K-means algorithm can be applied to Image Compression and Segmentation Basic Idea Treat similar pixel as same one July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Application for Image Compression K-means algorithm can be applied to Image Compression and Segmentation Basic Idea Treat similar pixel as same one Original data July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Application for Image Compression K-means algorithm can be applied to Image Compression and Segmentation Basic Idea Treat similar pixel as same one Cluster center July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Application for Image Compression K-means algorithm can be applied to Image Compression and Segmentation Basic Idea Treat similar pixel as same one Cluster center (pallet / code-book vector) July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Application for Image Compression K-means algorithm can be applied to Image Compression and Segmentation Basic Idea Treat similar pixel as same one = so called “vector quantization” July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Application for Image Compression Demo July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Application for Image Compression Demo July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Application for Image Compression Demo July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Application for Image Compression Compression rate Original image...24N bits (N=# of pixels) July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Application for Image Compression Compression rate Original image...24N bits (N=# of pixels) Compressed image... 24K+N log2K bits (K=# of pallet) July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Application for Image Compression Compression rate Original image...24N bits (N=# of pixels) Compressed image... 24K+N log2K bits (K=# of pallet) 16.7% if N~1M, K=10 July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable In K-means, all assignments are equal, “all or nothing”. Treated same July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable In K-means, all assignments are equal, “all or nothing”. Is these “hard” assignment appropriate? Treated same July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable In K-means, all assignments are equal, “all or nothing”. Is these “hard” assignment appropriate? > Want introduce "soft" assignment Treated same Probabilistic July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Introduce random variable z, having 1-of-K representation > Control unobserved “states” July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Introduce random variable z, having 1-of-K representation > Control unobserved “states” Once state is determined, July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Introduce random variable z, having 1-of-K representation > Control unobserved “states” Once state is determined, x is drawn from Gaussian of the state p(x|zk = 1) = N(x|µk, Σk). July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Introduce random variable z, having 1-of-K representation > Control unobserved “states” Once state is determined, x is drawn from Gaussian of the state p(x|zk = 1) = N(x|µk, Σk). x z Graphical representation July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Here the distribution over x is p(x) = z p(z)p(x|z) = K k=1 p(zk = 1)p(x|zk = 1) = K k=1 πkN(x|µk, Σk). July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Here the distribution over x is p(x) = z p(z)p(x|z) = K k=1 p(zk = 1)p(x|zk = 1) = K k=1 πkN(x|µk, Σk). z is 1-of-K rep. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Here the distribution over x is p(x) = z p(z)p(x|z) = K k=1 p(zk = 1)p(x|zk = 1) = K k=1 πkN(x|µk, Σk). July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Here the distribution over x is p(x) = z p(z)p(x|z) = K k=1 p(zk = 1)p(x|zk = 1) = K k=1 πkN(x|µk, Σk). Gaussian Mixtures ! July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Estimate (or “explain”) x came from which state γ(zk) ≡ p(zk = 1|x) = p(zk = 1)p(x|zk = 1) j p(zj = 1)p(x|zj = 1) = πkN(x|µk, Σk) j πjN(x|µj, Σj) . July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Estimate (or “explain”) x came from which state γ(zk) ≡ p(zk = 1|x) = p(zk = 1)p(x|zk = 1) j p(zj = 1)p(x|zj = 1) = πkN(x|µk, Σk) j πjN(x|µj, Σj) . July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Estimate (or “explain”) x came from which state γ(zk) ≡ p(zk = 1|x) = p(zk = 1)p(x|zk = 1) j p(zj = 1)p(x|zj = 1) = πkN(x|µk, Σk) j πjN(x|µj, Σj) . July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Estimate (or “explain”) x came from which state γ(zk) ≡ p(zk = 1|x) = p(zk = 1)p(x|zk = 1) j p(zj = 1)p(x|zj = 1) = πkN(x|µk, Σk) j πjN(x|µj, Σj) . Posteriors
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Estimate (or “explain”) x came from which state γ(zk) ≡ p(zk = 1|x) = p(zk = 1)p(x|zk = 1) j p(zj = 1)p(x|zj = 1) = πkN(x|µk, Σk) j πjN(x|µj, Σj) . Posteriors Priors July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Estimate (or “explain”) x came from which state γ(zk) ≡ p(zk = 1|x) = p(zk = 1)p(x|zk = 1) j p(zj = 1)p(x|zj = 1) = πkN(x|µk, Σk) j πjN(x|µj, Σj) . Posteriors Priors Likelihood July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Estimate (or “explain”) x came from which state This value is also called “responsibilities” γ(zk) ≡ p(zk = 1|x) = p(zk = 1)p(x|zk = 1) j p(zj = 1)p(x|zj = 1) = πkN(x|µk, Σk) j πjN(x|µj, Σj) . July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Example of Gaussian Mixtures (a) 0 0.5 1 0 0.5 1 (b) 0 0.5 1 0 0.5 1 (c) 0 0.5 1 0 0.5 1 No state info Coloured by true state Coloured by responsibility July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates ML estimates of mixtures of Gaussians have two problems: i.  Presence of Singularities ii.  Identifiability July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates ML estimates of mixtures of Gaussians have two problems: i.  Presence of Singularities ii.  Identifiability July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities What if a mean collides with a data point? ∃j, m µj = xm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities What if a mean collides with a data point? Likelihood can be however large by ∃j, m µj = xm σj → 0 L ∝   1 σj + k=j pk,m   n=m   1 σj exp − (xn − µj)2 2σ2 j + k=j pk,n   →∞. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities What if a mean collides with a data point? Likelihood can be however large by ∃j, m µj = xm σj → 0 L ∝   1 σj + k=j pk,m   n=m   1 σj exp − (xn − µj)2 2σ2 j + k=j pk,n   →∞.→ ∞ July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities What if a mean collides with a data point? Likelihood can be however large by ∃j, m µj = xm σj → 0 L ∝   1 σj + k=j pk,m   n=m   1 σj exp − (xn − µj)2 2σ2 j + k=j pk,n   →∞. → ∞ July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities What if a mean collides with a data point? Likelihood can be however large by ∃j, m µj = xm σj → 0 L ∝   1 σj + k=j pk,m   n=m   1 σj exp − (xn − µj)2 2σ2 j + k=j pk,n   →∞. → ∞ → 0 July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities What if a mean collides with a data point? Likelihood can be however large by ∃j, m µj = xm σj → 0 L ∝   1 σj + k=j pk,m   n=m   1 σj exp − (xn − µj)2 2σ2 j + k=j pk,n   →∞. → ∞ > 0 July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities What if a mean collides with a data point? Likelihood can be however large by ∃j, m µj = xm σj → 0 L ∝   1 σj + k=j pk,m   n=m   1 σj exp − (xn − µj)2 2σ2 j + k=j pk,n   →∞. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities It doesn't occur in single Gaussian. L ∝ 1 σN j n=m exp − (xn − µj)2 2σ2 j →0. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities It doesn't occur in single Gaussian. L ∝ 1 σN j n=m exp − (xn − µj)2 2σ2 j →0.→ ∞ July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities It doesn't occur in single Gaussian. L ∝ 1 σN j n=m exp − (xn − µj)2 2σ2 j →0.→ ∞ → 0 July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities It doesn't occur in single Gaussian. L ∝ 1 σN j n=m exp − (xn − µj)2 2σ2 j →0. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities It doesn't occur in single Gaussian. It doesn't occur in Bayesian approach either. L ∝ 1 σN j n=m exp − (xn − µj)2 2σ2 j →0. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates ML estimates of mixtures of Gaussians have two problems: i.  Presence of Singularities ii.  Identifiability July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates ii) Identifiability Optimal solutions are not unique: If we have a solution, there are (K!-1) other equivalent solution. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Problems of ML estimates ii) Identifiability Optimal solutions are not unique: If we have a solution, there are (K!-1) other equivalent solution. Matters when interpret, but does not matter when model only July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures The conditions of ML are obtained by where ∂ ∂µk L = 0, ∂ ∂Σk L = 0, ∂ ∂πk L + λ j πj − 1 = 0. L(π, µ, Σ) = N n=1 ln K k=1 πkN(xn|µk, Σk) July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures The conditions of ML where µk = 1 Nk N n=1 γn(zk)xn, Σk = 1 Nk N n=1 γn(zk)(xn − µj)(xn − µj)T , πk = Nk N , Nk = N n=1 γn(zk) July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures The conditions of ML where µk = 1 Nk N n=1 γn(zk)xn, Σk = 1 Nk N n=1 γn(zk)(xn − µj)(xn − µj)T , πk = Nk N , Nk = N n=1 γn(zk) γn(zk) appeared July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Recall that γn(zk) = πkN(xn|µk, Σk) j πjN(xn|µj, Σj) . July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Recall that γn(zk) = πkN(xn|µk, Σk) j πjN(xn|µj, Σj) . Parameters appeared July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Recall that γn(zk) = πkN(xn|µk, Σk) j πjN(xn|µj, Σj) . Parameters appeared = No closed form solution July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Recall that Again, use iterative algorithm! γn(zk) = πkN(xn|µk, Σk) j πjN(xn|µj, Σj) . Parameters appeared = No closed form solution July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures EM algorithm for Gaussian Mixtures 1. Initialize parameters 2. Repeat following two steps until converge i) Calculate ii) Update parameters γn(zk) = πkN(xn|µk, Σk) j πjN(xn|µj, Σj) . July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures EM algorithm for Gaussian Mixtures 1. Initialize parameters 2. Repeat following two steps until converge i) Calculate ii) Update parameters γn(zk) = πkN(xn|µk, Σk) j πjN(xn|µj, Σj) . E step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures EM algorithm for Gaussian Mixtures 1. Initialize parameters 2. Repeat following two steps until converge i) Calculate ii) Update parameters γn(zk) = πkN(xn|µk, Σk) j πjN(xn|µj, Σj) . M step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Comparison with K-means EM for Gaussian Mixtures K-means Clustering July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA

More Related Content

PDF
[3D勉強会@関東] Deep Reinforcement Learning of Volume-guided Progressive View Inpa...
bySeiya Ito
 
PPTX
An introduction to Prolog language slide
by2021uam4641
 
PDF
線形？非線形？
bynishio
 
PDF
「3.1.2最小二乗法の幾何学」PRML勉強会4 @筑波大学 #prml学ぼう
byJunpei Tsuji
 
PDF
PRML8章
by弘毅 露崎
 
PDF
Prml11 4
by正志 坪坂
 
PDF
やさしく音声分析法を学ぶ： ケプストラム分析とLPC分析
byShinnosuke Takamichi
 
PDF
パターン認識と機械学習 13章 系列データ
byemonosuke
 
[3D勉強会@関東] Deep Reinforcement Learning of Volume-guided Progressive View Inpa...
bySeiya Ito
 
An introduction to Prolog language slide
by2021uam4641
 
線形？非線形？
bynishio
 
「3.1.2最小二乗法の幾何学」PRML勉強会4 @筑波大学 #prml学ぼう
byJunpei Tsuji
 
PRML8章
by弘毅 露崎
 
Prml11 4
by正志 坪坂
 
やさしく音声分析法を学ぶ： ケプストラム分析とLPC分析
byShinnosuke Takamichi
 
パターン認識と機械学習 13章 系列データ
byemonosuke
 

What's hot

PDF
PRML輪読#4
bymatsuolab
 
PDF
What multimodal foundation models cannot perceive
byUniversity of Amsterdam
 
PDF
Quantum neural network
byVijayananda Mohire
 
PDF
20180118 一般化線形モデル(glm)
byMasakazu Shinoda
 
PDF
高次アンビソニックスのための マイクロホンアレイ設計の研究
byhorieyuto
 
PDF
幾何を使った統計のはなし
byToru Imai
 
PDF
PRML復々習レーン#3 3.1.3-3.1.5
bysleepy_yoshi
 
PDF
Bishop prml 11.5-11.6_wk77_100606-1152(発表に使った資料)
byWataru Kishimoto
 
PDF
J-KAC：日本語オーディオブック・紙芝居朗読音声コーパス
byShinnosuke Takamichi
 
PDF
因果推論の奥へ: "What works" meets "why it works"
bytakehikoihayashi
 
PDF
PCAの最終形態GPLVMの解説
by弘毅 露崎
 
PPTX
ベイズ統計学の概論的紹介
byNaoki Hayashi
 
PPTX
冗長変換とその画像復元応用
byShogo Muramatsu
 
PDF
2 4.devianceと尤度比検定
bylogics-of-blue
 
PPTX
音響メディア信号処理における独立成分分析の発展と応用, History of independent component analysis for sou...
byDaichi Kitamura
 
PPTX
大富豪に対する機械学習の適用 + α
byKatsuki Ohto
 
PDF
【第30回人工知能学会全国大会 発表資料】ストーリー展開と一貫性を同時に考慮した歌詞生成モデル【JSAI30th】
byKento Watanabe
 
PDF
Pythonによる機械学習入門 ～Deep Learningに挑戦～
byYasutomo Kawanishi
 
PPT
Opintojen kuormittavuus ja opintojen mitoittaminen
byVesa Linja-aho
 
PDF
ベイズ推論による機械学習入門　第４章
byYosukeAkasaka
 
PRML輪読#4
bymatsuolab
 
What multimodal foundation models cannot perceive
byUniversity of Amsterdam
 
Quantum neural network
byVijayananda Mohire
 
20180118 一般化線形モデル(glm)
byMasakazu Shinoda
 
高次アンビソニックスのための マイクロホンアレイ設計の研究
byhorieyuto
 
幾何を使った統計のはなし
byToru Imai
 
PRML復々習レーン#3 3.1.3-3.1.5
bysleepy_yoshi
 
Bishop prml 11.5-11.6_wk77_100606-1152(発表に使った資料)
byWataru Kishimoto
 
J-KAC：日本語オーディオブック・紙芝居朗読音声コーパス
byShinnosuke Takamichi
 
因果推論の奥へ: "What works" meets "why it works"
bytakehikoihayashi
 
PCAの最終形態GPLVMの解説
by弘毅 露崎
 
ベイズ統計学の概論的紹介
byNaoki Hayashi
 
冗長変換とその画像復元応用
byShogo Muramatsu
 
2 4.devianceと尤度比検定
bylogics-of-blue
 
音響メディア信号処理における独立成分分析の発展と応用, History of independent component analysis for sou...
byDaichi Kitamura
 
大富豪に対する機械学習の適用 + α
byKatsuki Ohto
 
【第30回人工知能学会全国大会 発表資料】ストーリー展開と一貫性を同時に考慮した歌詞生成モデル【JSAI30th】
byKento Watanabe
 
Pythonによる機械学習入門 ～Deep Learningに挑戦～
byYasutomo Kawanishi
 
Opintojen kuormittavuus ja opintojen mitoittaminen
byVesa Linja-aho
 
ベイズ推論による機械学習入門　第４章
byYosukeAkasaka
 

Viewers also liked

PPT
K mean-clustering algorithm
byparry prabhu
 
PDF
PRML 2.4-2.5: The Exponential Family & Nonparametric Methods
byShinichi Tamura
 
DOCX
Capítol 1
byLaura Esquembri Torrentbó
 
PPTX
Pattern recognition binoy k means clustering
by108kaushik
 
PPTX
Fast Single-pass K-means Clusterting at Oxford
byMapR Technologies
 
PDF
ESL 17.3.2-17.4: Graphical Lasso and Boltzmann Machines
byShinichi Tamura
 
PDF
MLaPP 2章 「確率」（前編）
byShinichi Tamura
 
PDF
NIPS 2016 輪読: Supervised Word Movers Distance
byShinichi Tamura
 
PDF
05. k means clustering ( k-means 클러스터링)
byJeonghun Yoon
 
PDF
PRML 13.2.2: The Forward-Backward Algorithm
byShinichi Tamura
 
PDF
ESL 4.4.3-4.5: Logistic Reression (contd.) and Separating Hyperplane
byShinichi Tamura
 
DOCX
Layer 3 messages
byJohn Samir
 
PDF
Kmeans initialization
bydjempol
 
PPT
Simple Linier Regression
bydessybudiyanti
 
PDF
Clustering：k-means, expect-maximization and gaussian mixture model
byjins0618
 
PPT
Tems layer3_messages
bybadgirl3086
 
PDF
如何用十分鐘快速瞭解一個程式語言 《以JavaScript和C語言為例》
by鍾誠 陳鍾誠
 
PDF
K means Clustering
byEdureka!
 
PPTX
K-Means Clustering Algorithm - Cluster Analysis | Machine Learning Algorithm ...
byEdureka!
 
PPTX
Application of Clustering in Data Science using Real-life Examples
byEdureka!
 
K mean-clustering algorithm
byparry prabhu
 
PRML 2.4-2.5: The Exponential Family & Nonparametric Methods
byShinichi Tamura
 
Capítol 1
byLaura Esquembri Torrentbó
 
Pattern recognition binoy k means clustering
by108kaushik
 
Fast Single-pass K-means Clusterting at Oxford
byMapR Technologies
 
ESL 17.3.2-17.4: Graphical Lasso and Boltzmann Machines
byShinichi Tamura
 
MLaPP 2章 「確率」（前編）
byShinichi Tamura
 
NIPS 2016 輪読: Supervised Word Movers Distance
byShinichi Tamura
 
05. k means clustering ( k-means 클러스터링)
byJeonghun Yoon
 
PRML 13.2.2: The Forward-Backward Algorithm
byShinichi Tamura
 
ESL 4.4.3-4.5: Logistic Reression (contd.) and Separating Hyperplane
byShinichi Tamura
 
Layer 3 messages
byJohn Samir
 
Kmeans initialization
bydjempol
 
Simple Linier Regression
bydessybudiyanti
 
Clustering：k-means, expect-maximization and gaussian mixture model
byjins0618
 
Tems layer3_messages
bybadgirl3086
 
如何用十分鐘快速瞭解一個程式語言 《以JavaScript和C語言為例》
by鍾誠 陳鍾誠
 
K means Clustering
byEdureka!
 
K-Means Clustering Algorithm - Cluster Analysis | Machine Learning Algorithm ...
byEdureka!
 
Application of Clustering in Data Science using Real-life Examples
byEdureka!
 

Similar to PRML 9.1-9.2: K-means Clustering & Mixtures of Gaussians

PDF
13_Unsupervised Learning.pdf
byEmanAsem4
 
DOCX
Neural nw k means
byEng. Dr. Dennis N. Mwighusa
 
PDF
Unsupervised Learning Clustering - Mathematcis
byigfreeze7
 
PPTX
AI-Lec20 Clustering I - Kmean.pptx
bySyed Ejaz
 
PPT
K mean-clustering
byAfzaal Subhani
 
PPTX
K-Means Clustering Presentation Slides for Machine Learning Course
byssuserfece35
 
DOCX
8.clustering algorithm.k means.em algorithm
byLaura Petrosanu
 
PPTX
Unsupervised Learning: Clustering
byExperfy
 
PPTX
"k-means-clustering" presentation @ Papers We Love Bucharest
byAdrian Florea
 
PPT
Unsupervised Machine Learning, Clustering, K-Means
byMomonLuffy
 
PPT
k-mean-Clustering impact on AI using DSS
byMarkNaguibElAbd
 
PPT
k-mean-clustering algorithm with example.ppt
bygeethar79
 
PPT
k-mean-clustering.ppt
byRanimeLoutar
 
PPT
k-mean-clustering algorithm for machine learning
byvipulkondekar
 
PDF
20 k-means, k-center, k-meoids and variations
byAndres Mendez-Vazquez
 
PPT
k-mean-clustering (1) clustering topic explanation
bymy123lapto
 
PPT
K mean-clustering
byPVP College
 
PDF
Noura2
byDr-mahmoud Algamel
 
PDF
Parametric Comparison of K-means and Adaptive K-means Clustering Performance ...
byIJECEIAES
 
PPTX
K means clustering
bykeshav goyal
 
13_Unsupervised Learning.pdf
byEmanAsem4
 
Neural nw k means
byEng. Dr. Dennis N. Mwighusa
 
Unsupervised Learning Clustering - Mathematcis
byigfreeze7
 
AI-Lec20 Clustering I - Kmean.pptx
bySyed Ejaz
 
K mean-clustering
byAfzaal Subhani
 
K-Means Clustering Presentation Slides for Machine Learning Course
byssuserfece35
 
8.clustering algorithm.k means.em algorithm
byLaura Petrosanu
 
Unsupervised Learning: Clustering
byExperfy
 
"k-means-clustering" presentation @ Papers We Love Bucharest
byAdrian Florea
 
Unsupervised Machine Learning, Clustering, K-Means
byMomonLuffy
 
k-mean-Clustering impact on AI using DSS
byMarkNaguibElAbd
 
k-mean-clustering algorithm with example.ppt
bygeethar79
 
k-mean-clustering.ppt
byRanimeLoutar
 
k-mean-clustering algorithm for machine learning
byvipulkondekar
 
20 k-means, k-center, k-meoids and variations
byAndres Mendez-Vazquez
 
k-mean-clustering (1) clustering topic explanation
bymy123lapto
 
K mean-clustering
byPVP College
 
Noura2
byDr-mahmoud Algamel
 
Parametric Comparison of K-means and Adaptive K-means Clustering Performance ...
byIJECEIAES
 
K means clustering
bykeshav goyal
 

Recently uploaded

PDF
A Brief Introduction About Christopher Elwell Woburn
byChristopher Elwell Woburn
 
PDF
Scott M. Graffius' Phases of Team Development - Applied to Human Teams and Hu...
byScott M. Graffius
 
PPTX
Large Language Model. LLM Audit Artificial Intelligence
byMANASCHOUDHARY22
 
PDF
Top Benefits of Using KVM VPS Hosting for Growing Businesses
byEthernet Servers
 
PPTX
IBM_NEXA_DAC2021 NEXA, a cloud-native platform for collaborative hardware log...
byArun Joseph
 
PDF
DataLiva-LivaGRC- Business Software for Governance, Risk, and Compliance
byMustafa Kuğu
 
PPTX
Search Engine Responses to Conspiratorial Search Practices AANZCA 2025 Presen...
bykkasianenko
 
PDF
IAC 500 Sensor - Humidity Measurement Device
byCS Instruments
 
PDF
Xemelgo - RFID Industry Predictions for 2026
byRich Rogers
 
PPTX
SRWE_Module_14_SRWE_ccna_basics_routing_concepts.pptx
byahmedmahmoudece
 
PPTX
Scrape YouTube Video Data for Influencer Analytics.pptx
byWeb Data Crawler
 
PPT
Information and Communication Technologies and Power
byJeffrey Hart
 
PPTX
Route_Inspection_Problem_Presentation.pptx
byIqraHanif27
 
PPTX
ICT Entrepreneur Quiz for grade 7...........
bykayramos6
 
PDF
Top 10 API Automation Testing Tools: Features, Pros & Cons
byhenrycavill30521
 
PDF
Achieving Interoperability in Healthcare IT: A Strategic Guide
byHart, Inc.
 
PDF
threat-hunters-cookbook for cybersecurity
bylobster6719
 
PDF
Benefits of Using the IAC 500 Sensor for Ambient Conditions
byCS Instruments
 
PDF
Artificial Intelligence(AI) full Book.pdf
bydeyanimesh299
 
PDF
Strengthening cybern resilience for a leading indian Financial Services group
bypandeydevika621
 
A Brief Introduction About Christopher Elwell Woburn
byChristopher Elwell Woburn
 
Scott M. Graffius' Phases of Team Development - Applied to Human Teams and Hu...
byScott M. Graffius
 
Large Language Model. LLM Audit Artificial Intelligence
byMANASCHOUDHARY22
 
Top Benefits of Using KVM VPS Hosting for Growing Businesses
byEthernet Servers
 
IBM_NEXA_DAC2021 NEXA, a cloud-native platform for collaborative hardware log...
byArun Joseph
 
DataLiva-LivaGRC- Business Software for Governance, Risk, and Compliance
byMustafa Kuğu
 
Search Engine Responses to Conspiratorial Search Practices AANZCA 2025 Presen...
bykkasianenko
 
IAC 500 Sensor - Humidity Measurement Device
byCS Instruments
 
Xemelgo - RFID Industry Predictions for 2026
byRich Rogers
 
SRWE_Module_14_SRWE_ccna_basics_routing_concepts.pptx
byahmedmahmoudece
 
Scrape YouTube Video Data for Influencer Analytics.pptx
byWeb Data Crawler
 
Information and Communication Technologies and Power
byJeffrey Hart
 
Route_Inspection_Problem_Presentation.pptx
byIqraHanif27
 
ICT Entrepreneur Quiz for grade 7...........
bykayramos6
 
Top 10 API Automation Testing Tools: Features, Pros & Cons
byhenrycavill30521
 
Achieving Interoperability in Healthcare IT: A Strategic Guide
byHart, Inc.
 
threat-hunters-cookbook for cybersecurity
bylobster6719
 
Benefits of Using the IAC 500 Sensor for Ambient Conditions
byCS Instruments
 
Artificial Intelligence(AI) full Book.pdf
bydeyanimesh299
 
Strengthening cybern resilience for a leading indian Financial Services group
bypandeydevika621
 

PRML 9.1-9.2: K-means Clustering & Mixtures of Gaussians

  • 1.
    PRML 9.1-9.2 K-means Clustering & Mixturesof Gaussians July 16, 2014 by Shinichi TAMURA
  • 2.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 3.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 4.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 5.
    Mixtures of Gaussians K-means Clustering Clustering Problem An unsupervised machine learning problem Divide data in some group (=cluster) where ü  similar data > same group ü  dissimilar data > different group July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 6.
    Mixtures of Gaussians K-means Clustering Clustering Problem Divide data in some group (=cluster) where ü  similar data > same group ü  dissimilar data > different group July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 7.
    Mixtures of Gaussians K-means Clustering Clustering Problem Divide data in some group (=cluster) where ü  similar data > same group ü  dissimilar data > different group Minimize N n=1 xn − µk(n) 2 July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 8.
    Mixtures of Gaussians K-means Clustering Clustering Problem Divide data in some group (=cluster) where ü  similar data > same group ü  dissimilar data > different group Minimize N n=1 xn − µk(n) 2 Center of the cluster July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 9.
    Mixtures of Gaussians K-means Clustering Clustering Problem Given data set and # of cluster K Let be cluster representative and be assignment indicator ( ), Here, J is called “distortion measure”. X = {x1, . . . , xN } µk rnk rnk = 1 if x ∈ Ck Minimize J = N n=1 K k=1 rnk xn − µk 2 July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 10.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 11.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 12.
    Mixtures of Gaussians K-means Clustering K-means Clustering How to solve that? July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 13.
    Mixtures of Gaussians K-means Clustering K-means Clustering How to solve that? and are dependent each other > No closed form solution µk rnk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 14.
    Mixtures of Gaussians K-means Clustering K-means Clustering How to solve that? and are dependent each other > No closed form solution Use iterative algorithm ! µk rnk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 15.
    Mixtures of Gaussians K-means Clustering K-means Clustering Strategy and can't be updated simultaneously µk rnk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 16.
    Mixtures of Gaussians K-means Clustering K-means Clustering Strategy and can't be updated simultaneously > Update them one by one µk rnk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 17.
    Mixtures of Gaussians K-means Clustering K-means Clustering Update of (assignment) Since each can be determined independently, J will be minimum if they are assigned to the nearest . rnk xn µk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 18.
    Mixtures of Gaussians K-means Clustering K-means Clustering Update of (assignment) Since each can be determined independently, J will be minimum if they are assigned to the nearest . Therefore, rnk xn µk rnk = 1 if k = arg minj xn − µj 2 , 0 otherwise. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 19.
    Mixtures of Gaussians K-means Clustering K-means Clustering Update of (parameter estimation) Optimal is obtained by setting derivative 0. µk µk ∂ ∂µk N n=1 K k =1 rnk xn − µk 2 = 0. ⇐⇒ 2 N n=1 rnk(xn − µk) = 0. ∴ µk = N n=1 rnkxn N n=1 rnk = 1 Nk xn∈Ck xn. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 20.
    Mixtures of Gaussians K-means Clustering K-means Clustering Update of (parameter estimation) Optimal is obtained by setting derivative 0. µk µk ∂ ∂µk N n=1 K k =1 rnk xn − µk 2 = 0. ⇐⇒ 2 N n=1 rnk(xn − µk) = 0. ∴ µk = N n=1 rnkxn N n=1 rnk = 1 Nk xn∈Ck xn. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 21.
    Mixtures of Gaussians K-means Clustering K-means Clustering Update of (parameter estimation) Optimal is obtained by setting derivative 0. µk µk ∂ ∂µk N n=1 K k =1 rnk xn − µk 2 = 0. ⇐⇒ 2 N n=1 rnk(xn − µk) = 0. ∴ µk = N n=1 rnkxn N n=1 rnk = 1 Nk xn∈Ck xn. Mean of the cluster July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 22.
    Mixtures of Gaussians K-means Clustering K-means Clustering Update of (parameter estimation) Optimal is obtained by setting derivative 0. µk µk ∂ ∂µk N n=1 K k =1 rnk xn − µk 2 = 0. ⇐⇒ 2 N n=1 rnk(xn − µk) = 0. ∴ µk = N n=1 rnkxn N n=1 rnk = 1 Nk xn∈Ck xn. Mean of the cluster July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 23.
    Mixtures of Gaussians K-means Clustering K-means Clustering Update of (parameter estimation) Optimal is obtained by setting derivative 0. µk µk ∂ ∂µk N n=1 K k =1 rnk xn − µk 2 = 0. ⇐⇒ 2 N n=1 rnk(xn − µk) = 0. ∴ µk = N n=1 rnkxn N n=1 rnk = 1 Nk xn∈Ck xn. Mean of the cluster is the mean of the cluster Cost function J corresponds to the sum of inner-class variance! µk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 24.
    Mixtures of Gaussians K-means Clustering K-means Clustering Update of (parameter estimation) Optimal is obtained by setting derivative 0. µk µk ∂ ∂µk N n=1 K k =1 rnk xn − µk 2 = 0. ⇐⇒ 2 N n=1 rnk(xn − µk) = 0. ∴ µk = N n=1 rnkxn N n=1 rnk = 1 Nk xn∈Ck xn. Mean of the cluster is the mean of the cluster Cost function J corresponds to the sum of inner-class variance! µk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 25.
    Mixtures of Gaussians K-means Clustering K-means Clustering K-means algorithm 1. Initialize , 2. Repeat following two steps until converge i) Assign each to closest ii) Update to the mean of the cluster µk rnk xn µk µk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 26.
    Mixtures of Gaussians K-means Clustering K-means Clustering K-means algorithm 1. Initialize , 2. Repeat following two steps until converge i) Assign each to closest ii) Update to the mean of the cluster µk rnk xn µk µk E step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 27.
    Mixtures of Gaussians K-means Clustering K-means Clustering K-means algorithm 1. Initialize , 2. Repeat following two steps until converge i) Assign each to closest ii) Update to the mean of the cluster µk rnk xn µk µk M step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 28.
    Mixtures of Gaussians K-means Clustering K-means Clustering Convergence property Both steps never increase J, so we can obtain better result in every iteration. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 29.
    Mixtures of Gaussians K-means Clustering K-means Clustering Convergence property Both steps never increase J, so we can obtain better result in every iteration. Since is finite, algorithm converge after finite iterations. rnk July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 30.
    Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 31.
    Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm E step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 32.
    Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm M step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 33.
    Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm E step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 34.
    Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm M step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 35.
    Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm E step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 36.
    Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm M step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 37.
    Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm E step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 38.
    Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm M step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 39.
    Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 40.
    Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 41.
    Mixtures of Gaussians K-means Clustering K-means Clustering Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 42.
    Mixtures of Gaussians K-means Clustering K-means Clustering Calculation performance E step ... Comparison of every data point and every cluster mean > O(KN) µk xn July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 43.
    Mixtures of Gaussians K-means Clustering K-means Clustering Calculation performance E step ... Comparison of every data point and every cluster mean > O(KN) µk xn Not good July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 44.
    Mixtures of Gaussians K-means Clustering K-means Clustering Calculation performance E step ... Comparison of every data point and every cluster mean > O(KN) µk xn Not good Improve with kd-tree, triangle inequality...etc July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 45.
    Mixtures of Gaussians K-means Clustering K-means Clustering Calculation performance E step ... Comparison of every data point and every cluster mean > O(KN) µk xn July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 46.
    Mixtures of Gaussians K-means Clustering K-means Clustering Calculation performance E step ... Comparison of every data point and every cluster mean > O(KN) M step ... Calculation of mean for every cluster > O(N) µk xn July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 47.
    Mixtures of Gaussians K-means Clustering K-means Clustering Here, two variation will be introduced: 1.  On-line version 2.  General dissimilarity July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 48.
    Mixtures of Gaussians K-means Clustering K-means Clustering Here, two variation will be introduced: 1.  On-line version 2.  General dissimilarity July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 49.
    Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 1. On-line version The case where one datum is observed at once. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 50.
    Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 1. On-line version The case where one datum is observed at once. > Apply Robbins-Monro algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 51.
    Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 1. On-line version The case where one datum is observed at once. > Apply Robbins-Monro algorithm µnew k = µold k + ηn(xn − µold k ). July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 52.
    Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 1. On-line version The case where one datum is observed at once. > Apply Robbins-Monro algorithm µnew k = µold k + ηn(xn − µold k ). Learning rate July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 53.
    Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 1. On-line version The case where one datum is observed at once. > Apply Robbins-Monro algorithm µnew k = µold k + ηn(xn − µold k ). Learning rate Decrease with iteration July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 54.
    Mixtures of Gaussians K-means Clustering K-means Clustering Here, two variation will be introduced: 1.  On-line version 2.  General dissimilarity July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 55.
    Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 2. General dissimilarity Euclidian distance is not ü  appropriate to categorical data, etc. ü  robust to outlier. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 56.
    Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 2. General dissimilarity Euclidian distance is not ü  appropriate to categorical data, etc. ü  robust to outlier. > Use general dissimilarity measure V(x, x ) July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 57.
    Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 2. General dissimilarity Euclidian distance is not ü  appropriate to categorical data, etc. ü  robust to outlier. > Use general dissimilarity measure V(x, x ) E step ... No difference July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 58.
    Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 2. General dissimilarity Euclidian distance is not ü  appropriate to categorical data, etc. ü  robust to outlier. > Use general dissimilarity measure V(x, x ) M step ... Not assured J is easy to minimize July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 59.
    Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 2. General dissimilarity To make M-step easy, restrict to the vector chosen from > A solution can be obtained by finite number of comparison µk {xn} July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 60.
    Mixtures of Gaussians K-means Clustering K-means Clustering [Variation] 2. General dissimilarity To make M-step easy, restrict to the vector chosen from > A solution can be obtained by finite number of comparison µk {xn} µk = arg min xn xn ∈Ck V(xn, xn ) July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 61.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 62.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 63.
    Mixtures of Gaussians K-means Clustering Application for Image Compression K-means algorithm can be applied to Image Compression and Segmentation July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 64.
    Mixtures of Gaussians K-means Clustering Application for Image Compression K-means algorithm can be applied to Image Compression and Segmentation Basic Idea Treat similar pixel as same one July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 65.
    Mixtures of Gaussians K-means Clustering Application for Image Compression K-means algorithm can be applied to Image Compression and Segmentation Basic Idea Treat similar pixel as same one Original data July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 66.
    Mixtures of Gaussians K-means Clustering Application for Image Compression K-means algorithm can be applied to Image Compression and Segmentation Basic Idea Treat similar pixel as same one Cluster center July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 67.
    Mixtures of Gaussians K-means Clustering Application for Image Compression K-means algorithm can be applied to Image Compression and Segmentation Basic Idea Treat similar pixel as same one Cluster center (pallet / code-book vector) July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 68.
    Mixtures of Gaussians K-means Clustering Application for Image Compression K-means algorithm can be applied to Image Compression and Segmentation Basic Idea Treat similar pixel as same one = so called “vector quantization” July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 69.
    Mixtures of Gaussians K-means Clustering Application for Image Compression Demo July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 70.
    Mixtures of Gaussians K-means Clustering Application for Image Compression Demo July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 71.
    Mixtures of Gaussians K-means Clustering Application for Image Compression Demo July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 72.
    Mixtures of Gaussians K-means Clustering Application for Image Compression Compression rate Original image...24N bits (N=# of pixels) July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 73.
    Mixtures of Gaussians K-means Clustering Application for Image Compression Compression rate Original image...24N bits (N=# of pixels) Compressed image... 24K+N log2K bits (K=# of pallet) July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 74.
    Mixtures of Gaussians K-means Clustering Application for Image Compression Compression rate Original image...24N bits (N=# of pixels) Compressed image... 24K+N log2K bits (K=# of pallet) 16.7% if N~1M, K=10 July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 75.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 76.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 77.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 78.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable In K-means, all assignments are equal, “all or nothing”. Treated same July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 79.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable In K-means, all assignments are equal, “all or nothing”. Is these “hard” assignment appropriate? Treated same July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 80.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable In K-means, all assignments are equal, “all or nothing”. Is these “hard” assignment appropriate? > Want introduce "soft" assignment Treated same Probabilistic July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 81.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Introduce random variable z, having 1-of-K representation > Control unobserved “states” July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 82.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Introduce random variable z, having 1-of-K representation > Control unobserved “states” Once state is determined, July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 83.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Introduce random variable z, having 1-of-K representation > Control unobserved “states” Once state is determined, x is drawn from Gaussian of the state p(x|zk = 1) = N(x|µk, Σk). July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 84.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Introduce random variable z, having 1-of-K representation > Control unobserved “states” Once state is determined, x is drawn from Gaussian of the state p(x|zk = 1) = N(x|µk, Σk). x z Graphical representation July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 85.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Here the distribution over x is p(x) = z p(z)p(x|z) = K k=1 p(zk = 1)p(x|zk = 1) = K k=1 πkN(x|µk, Σk). July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 86.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Here the distribution over x is p(x) = z p(z)p(x|z) = K k=1 p(zk = 1)p(x|zk = 1) = K k=1 πkN(x|µk, Σk). z is 1-of-K rep. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 87.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Here the distribution over x is p(x) = z p(z)p(x|z) = K k=1 p(zk = 1)p(x|zk = 1) = K k=1 πkN(x|µk, Σk). July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 88.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Here the distribution over x is p(x) = z p(z)p(x|z) = K k=1 p(zk = 1)p(x|zk = 1) = K k=1 πkN(x|µk, Σk). Gaussian Mixtures ! July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 89.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Estimate (or “explain”) x came from which state γ(zk) ≡ p(zk = 1|x) = p(zk = 1)p(x|zk = 1) j p(zj = 1)p(x|zj = 1) = πkN(x|µk, Σk) j πjN(x|µj, Σj) . July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 90.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Estimate (or “explain”) x came from which state γ(zk) ≡ p(zk = 1|x) = p(zk = 1)p(x|zk = 1) j p(zj = 1)p(x|zj = 1) = πkN(x|µk, Σk) j πjN(x|µj, Σj) . July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 91.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Estimate (or “explain”) x came from which state γ(zk) ≡ p(zk = 1|x) = p(zk = 1)p(x|zk = 1) j p(zj = 1)p(x|zj = 1) = πkN(x|µk, Σk) j πjN(x|µj, Σj) . July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 92.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Estimate (or “explain”) x came from which state γ(zk) ≡ p(zk = 1|x) = p(zk = 1)p(x|zk = 1) j p(zj = 1)p(x|zj = 1) = πkN(x|µk, Σk) j πjN(x|µj, Σj) . Posteriors
  • 93.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Estimate (or “explain”) x came from which state γ(zk) ≡ p(zk = 1|x) = p(zk = 1)p(x|zk = 1) j p(zj = 1)p(x|zj = 1) = πkN(x|µk, Σk) j πjN(x|µj, Σj) . Posteriors Priors July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 94.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Estimate (or “explain”) x came from which state γ(zk) ≡ p(zk = 1|x) = p(zk = 1)p(x|zk = 1) j p(zj = 1)p(x|zj = 1) = πkN(x|µk, Σk) j πjN(x|µj, Σj) . Posteriors Priors Likelihood July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 95.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Estimate (or “explain”) x came from which state This value is also called “responsibilities” γ(zk) ≡ p(zk = 1|x) = p(zk = 1)p(x|zk = 1) j p(zj = 1)p(x|zj = 1) = πkN(x|µk, Σk) j πjN(x|µj, Σj) . July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 96.
    Mixtures of Gaussians K-means Clustering Introduction of Latent Variable Example of Gaussian Mixtures (a) 0 0.5 1 0 0.5 1 (b) 0 0.5 1 0 0.5 1 (c) 0 0.5 1 0 0.5 1 No state info Coloured by true state Coloured by responsibility July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 97.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 98.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 99.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates ML estimates of mixtures of Gaussians have two problems: i.  Presence of Singularities ii.  Identifiability July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 100.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates ML estimates of mixtures of Gaussians have two problems: i.  Presence of Singularities ii.  Identifiability July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 101.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities What if a mean collides with a data point? ∃j, m µj = xm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 102.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities What if a mean collides with a data point? Likelihood can be however large by ∃j, m µj = xm σj → 0 L ∝   1 σj + k=j pk,m   n=m   1 σj exp − (xn − µj)2 2σ2 j + k=j pk,n   →∞. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 103.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities What if a mean collides with a data point? Likelihood can be however large by ∃j, m µj = xm σj → 0 L ∝   1 σj + k=j pk,m   n=m   1 σj exp − (xn − µj)2 2σ2 j + k=j pk,n   →∞.→ ∞ July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 104.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities What if a mean collides with a data point? Likelihood can be however large by ∃j, m µj = xm σj → 0 L ∝   1 σj + k=j pk,m   n=m   1 σj exp − (xn − µj)2 2σ2 j + k=j pk,n   →∞. → ∞ July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 105.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities What if a mean collides with a data point? Likelihood can be however large by ∃j, m µj = xm σj → 0 L ∝   1 σj + k=j pk,m   n=m   1 σj exp − (xn − µj)2 2σ2 j + k=j pk,n   →∞. → ∞ → 0 July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 106.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities What if a mean collides with a data point? Likelihood can be however large by ∃j, m µj = xm σj → 0 L ∝   1 σj + k=j pk,m   n=m   1 σj exp − (xn − µj)2 2σ2 j + k=j pk,n   →∞. → ∞ > 0 July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 107.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities What if a mean collides with a data point? Likelihood can be however large by ∃j, m µj = xm σj → 0 L ∝   1 σj + k=j pk,m   n=m   1 σj exp − (xn − µj)2 2σ2 j + k=j pk,n   →∞. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 108.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities It doesn't occur in single Gaussian. L ∝ 1 σN j n=m exp − (xn − µj)2 2σ2 j →0. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 109.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities It doesn't occur in single Gaussian. L ∝ 1 σN j n=m exp − (xn − µj)2 2σ2 j →0.→ ∞ July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 110.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities It doesn't occur in single Gaussian. L ∝ 1 σN j n=m exp − (xn − µj)2 2σ2 j →0.→ ∞ → 0 July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 111.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities It doesn't occur in single Gaussian. L ∝ 1 σN j n=m exp − (xn − µj)2 2σ2 j →0. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 112.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates i) Presence of Singularities It doesn't occur in single Gaussian. It doesn't occur in Bayesian approach either. L ∝ 1 σN j n=m exp − (xn − µj)2 2σ2 j →0. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 113.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates ML estimates of mixtures of Gaussians have two problems: i.  Presence of Singularities ii.  Identifiability July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 114.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates ii) Identifiability Optimal solutions are not unique: If we have a solution, there are (K!-1) other equivalent solution. July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 115.
    Mixtures of Gaussians K-means Clustering Problems of ML estimates ii) Identifiability Optimal solutions are not unique: If we have a solution, there are (K!-1) other equivalent solution. Matters when interpret, but does not matter when model only July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 116.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 117.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 118.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures The conditions of ML are obtained by where ∂ ∂µk L = 0, ∂ ∂Σk L = 0, ∂ ∂πk L + λ j πj − 1 = 0. L(π, µ, Σ) = N n=1 ln K k=1 πkN(xn|µk, Σk) July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 119.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures The conditions of ML where µk = 1 Nk N n=1 γn(zk)xn, Σk = 1 Nk N n=1 γn(zk)(xn − µj)(xn − µj)T , πk = Nk N , Nk = N n=1 γn(zk) July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 120.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures The conditions of ML where µk = 1 Nk N n=1 γn(zk)xn, Σk = 1 Nk N n=1 γn(zk)(xn − µj)(xn − µj)T , πk = Nk N , Nk = N n=1 γn(zk) γn(zk) appeared July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 121.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Recall that γn(zk) = πkN(xn|µk, Σk) j πjN(xn|µj, Σj) . July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 122.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Recall that γn(zk) = πkN(xn|µk, Σk) j πjN(xn|µj, Σj) . Parameters appeared July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 123.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Recall that γn(zk) = πkN(xn|µk, Σk) j πjN(xn|µj, Σj) . Parameters appeared = No closed form solution July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 124.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Recall that Again, use iterative algorithm! γn(zk) = πkN(xn|µk, Σk) j πjN(xn|µj, Σj) . Parameters appeared = No closed form solution July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 125.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures EM algorithm for Gaussian Mixtures 1. Initialize parameters 2. Repeat following two steps until converge i) Calculate ii) Update parameters γn(zk) = πkN(xn|µk, Σk) j πjN(xn|µj, Σj) . July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 126.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures EM algorithm for Gaussian Mixtures 1. Initialize parameters 2. Repeat following two steps until converge i) Calculate ii) Update parameters γn(zk) = πkN(xn|µk, Σk) j πjN(xn|µj, Σj) . E step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 127.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures EM algorithm for Gaussian Mixtures 1. Initialize parameters 2. Repeat following two steps until converge i) Calculate ii) Update parameters γn(zk) = πkN(xn|µk, Σk) j πjN(xn|µj, Σj) . M step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 128.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 129.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 130.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 131.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 132.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Demo of algorithm July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 133.
    Mixtures of Gaussians K-means Clustering EM-algorithm for Gaussian Mixtures Comparison with K-means EM for Gaussian Mixtures K-means Clustering July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 134.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 135.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA
  • 136.
    Mixtures of Gaussians K-means Clustering Today's topics 1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression 2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA