This document discusses machine learning techniques for modeling data as mixtures of distributions. It introduces mixture models, likelihood functions, and maximum likelihood estimation. The EM algorithm is presented as a method for estimating parameters in mixture models with hidden variables. Mixture models of Gaussians and applying the EM algorithm to estimate parameters for a mixture of Gaussians model are specifically discussed.

1. Machine Learning Interviews –
Day 4
Arpit Agarwal

2. Meta-Idea
Probability
Model Data
Inference
(Likelihood)
A model of the data generating process gives rise to data.
Model estimation from data is most commonly through Likelihood estimation

3. Likelihood Function
P Data Model P Model
( | ) ( )
( )
( | )
P Data
P Model Data 
Likelihood Function
Find the “best” model which has generated the data. In a likelihood function
the data is considered fixed and one searches for the best model over the
different choices available.

4. Maximum Likelihood Estimation
• We want to select a model which will
maximize the probability that the data was
generated from the model
maxlog P(Data|Model)

5. Examples
• Suppose we have the following data
– 0,1,1,0,0,1,1,0
• In this case it is sensible to choose the
Bernoulli distribution (B(p)) as the model
space.
• Now we want to choose the best p, i.e.,

6. Examples
Suppose the following are marks in a course
55.5, 67, 87, 48, 63
Marks typically follow a Normal distribution
whose density function is
Now, we want to find the best , such that

7. Examples – Mixture of Gaussian
• Suppose we have data about heights of
people (in cm)
– 185,140,134,150,170
• Heights follow a normal (log normal)
distribution but men on average are taller
than women. This suggests a mixture of two
distributions

8. Mixture of Gaussians
• The density function is given by
where = probability that a random point is
generated from k-th gaussian

9. 0.5
0.4
0.3
0.2
0.1
0
Component 1 Component 2
-5 0 5 10
p(x)
0.5
0.4
0.3
0.2
0.1
0
Mixture Model
-5 0 5 10
x
p(x)

10. Mixture of Gaussian
• Let D = {x(1),x(2),…x(n)} be a set of n observations
points generated from a mixture of k gaussians.
• Let H = {z(1),z(2),..z(n)} be a set of n values of a hidden
variable Z.
– z(i) corresponds to the gaussian to which x(i) belongs
• Our goal is to find out the best , and
• For a new data point find out which gaussian it belongs
to.

11. Mixture of Gaussian – ML approach
• Maximize the log-likelihood
• Difficult to solve in general.
• Idea: Introduce z(i)’s in the optimization problem

12. Solution – EM Algorithm
• We use EM when we want to do maximum likelihood
parameter estimation but we have hidden data in our model.
• The log-likelihood of the observed data is
 
log p(D | ) log p(D,H | )
H
• As the likelihood might also depend on the values of the
hidden data, not only do we have to estimate  but also H

13. EM Algorithm - Outline
• start with initial guess of parameters
• E step: based on the current parameters and
observer variables find the probability
distribution over hidden variables
• M step: with respect to the probability
distribution over hidden variables, maximize
the joint log-likelihood.
• Repeat until convergence

14. EM Algorithm

15. Back to Mixture of Gaussians
• Let D = {x(1),x(2),…x(n)} be a set of n observations
points generated from a mixture of k gaussians.
• Let H = {z(1),z(2),..z(n)} be a set of n values of a hidden
variable Z.
– z(i) corresponds to the gaussian to which x(i) belongs
• Our goal is to find out the best , and
• For a new data point find out which gaussian it belongs
to.

16. EM Algorithm for Mixture of Normals

17. K-Means