The Gaussian distribution is an important probability distribution that is commonly used in statistics and machine learning. It has several key properties: (1) it is defined by a mean and covariance matrix, (2) the expectation of values drawn from it is the mean, and (3) the contours of constant probability density form ellipses. The conditional and marginal distributions of multivariate Gaussians can also be Gaussian. Bayes' theorem and maximum likelihood estimation can be applied to Gaussian distributions.

1. The Gaussian Distribution
CSC 411/2515
Adopted from PRML 2.3

2. The Gaussian Distribution
 For a 𝐷-dimensional vector 𝑥, the multivariate Gaussian
distribution takes the form:
𝒩 𝑥|𝜇, Σ =
1
2𝜋
𝐷
2 Σ
1
2
exp −
1
2
𝑥 − 𝜇 ⊤
Σ−1
𝑥 − 𝜇
 Motivations:
 Maximum of the entropy
 Central limit theorem

3. The Gaussian Distribution: Properties
 The law is a function of the Mahalanobis distance from 𝑥 to 𝜇:
Δ2 = 𝑥 − 𝜇 ⊤Σ−1 𝑥 − 𝜇
 The expectation of 𝑥 under the Gaussian distribution is:
𝔼 𝑥 = 𝜇
 The covariance matrix of 𝑥 is:
cov 𝑥 = Σ

4. The Gaussian Distribution: Properties
 The law (quadratic function) is constant on elliptical surfaces:
 𝜆𝑖 are the eigenvalues of Σ
 𝑢𝑖 are the associated eigenvectors

5. The Gaussian Distribution: more examples
 Contours of constant probability density:
in which the covariance matrix is:
a) of general form
b) diagonal
c) proportional to the identity matrix

6. Conditional Law
 Given a Gaussian distribution 𝒩 𝑥|𝜇, Σ with
𝑥 = 𝑥𝑎, 𝑥𝑏
⊤, 𝜇 = 𝜇𝑎, 𝜇𝑏
⊤
Σ =
Σ𝑎𝑎 Σ𝑎𝑏
Σ𝑏𝑎 Σ𝑏𝑏
=
Λ𝑎𝑎 Λ𝑎𝑏
Λ𝑏𝑎 Λ𝑏𝑏
−1
 What’s the conditional distribution 𝑝(𝑥𝑎|𝑥𝑏)?

7. Conditional Law
 Given a Gaussian distribution 𝒩 𝑥|𝜇, Σ with
𝑥 = 𝑥𝑎, 𝑥𝑏
⊤, 𝜇 = 𝜇𝑎, 𝜇𝑏
⊤
Σ =
Σ𝑎𝑎 Σ𝑎𝑏
Σ𝑏𝑎 Σ𝑏𝑏
=
Λ𝑎𝑎 Λ𝑎𝑏
Λ𝑏𝑎 Λ𝑏𝑏
−1
 What’s the conditional distribution 𝑝(𝑥𝑎|𝑥𝑏)?
−
1
2
𝑥 − 𝜇 ⊤Σ−1 𝑥 − 𝜇 =
−
1
2
𝑥𝑎 − 𝜇𝑎
⊤Λ𝑎𝑎 𝑥𝑎 − 𝜇𝑎 −
1
2
𝑥𝑎 − 𝜇𝑎
⊤Λ𝑎𝑏 𝑥𝑏 − 𝜇𝑏
−
1
2
𝑥𝑏 − 𝜇𝑏
⊤
Λ𝑏𝑎 𝑥𝑎 − 𝜇𝑎 −
1
2
𝑥𝑏 − 𝜇𝑏
⊤
Λ𝑏𝑏 𝑥𝑏 − 𝜇𝑏

8. Conditional Law
 Using the definition:
−
1
2
𝑥 − 𝜇 ⊤
Σ−1
𝑥 − 𝜇 = −
1
2
𝑥⊤
Σ−1
𝑥 + 𝑥⊤
Σ−1
𝜇 + const
 What’s the conditional distribution 𝑝(𝑥𝑎|𝑥𝑏)?
 Σ𝑎|𝑏
−1
= Λ𝑎𝑎
 Σ𝑎|𝑏
−1
𝜇𝑎|𝑏 = Λ𝑎𝑎𝜇𝑎 − Λ𝑎𝑏 𝑥𝑏 − 𝜇𝑏
Σ =
Σ𝑎𝑎 Σ𝑎𝑏
Σ𝑏𝑎 Σ𝑏𝑏
=
Λ𝑎𝑎 Λ𝑎𝑏
Λ𝑏𝑎 Λ𝑏𝑏
−1
 Λ𝑎𝑎 = Σ𝑎𝑎 − Σ𝑎𝑏Σ𝑏𝑏
−1
Σ𝑏𝑎
−1
 Λ𝑎𝑏 = − Σ𝑎𝑎 − Σ𝑎𝑏Σ𝑏𝑏
−1
Σ𝑏𝑎
−1
Σ𝑎𝑏Σ𝑏𝑏
−1
Inverse partition identity:
𝐴 𝐵
𝐶 𝐷
−1
= 𝑀 −𝑀𝐵𝐷−1
−𝐷−1
𝐶𝑀 𝐷−1
+ 𝐷−1
𝐶𝑀𝐵𝐷−1
𝑀 = 𝐴 − 𝐵𝐷−1
𝐶 −1

9. Conditional Law
𝜇𝑎|𝑏 = 𝜇𝑎 + Λ𝑎𝑎Λ𝑎𝑏
−1
𝑥𝑏 − 𝜇𝑏
 The form using precision matrix:
Λ𝑎|𝑏 = Λ𝑎𝑎
𝜇𝑎|𝑏 = 𝜇𝑎 + Σ𝑎𝑏Σ𝑏𝑏
−1
𝑥𝑏 − 𝜇𝑏
 The conditional distribution 𝑝 𝑥𝑎 𝑥𝑏 is a Gaussian with:
Σ𝑎|𝑏 = Σ𝑎𝑎 − Σ𝑎𝑏Σ𝑏𝑏
−1
Σ𝑏𝑎

10. Marginal Law
 The marginal distribution is given by:
𝑝 𝑥𝑎 = 𝑝 𝑥𝑎, 𝑥𝑏 𝑑𝑥𝑏
 Picking out those terms that involve 𝑥𝑏, we have
−
1
2
𝑥𝑏
⊤
Λ𝑏𝑏𝑥𝑏 + 𝑥𝑏
⊤
𝑚 = −
1
2
𝑥𝑏 − Λ𝑏𝑏
−1
𝑚
⊤
Λ𝑏𝑏 𝑥𝑏 − Λ𝑏𝑏
−1
𝑚 +
1
2
𝑚⊤Λ𝑏𝑏
−1
𝑚
𝑚 = Λ𝑏𝑏𝜇𝑏 − Λ𝑏𝑎 𝑥𝑎 − 𝜇𝑎
 Integrate over 𝑥𝑏 (unnormalized Gaussian)
exp −
1
2
𝑥𝑏 − Λ𝑏𝑏
−1
𝑚
⊤
Λ𝑏𝑏 𝑥𝑏 − Λ𝑏𝑏
−1
𝑚 𝑑𝑥𝑏
 The integral is equal to the normalization term

11. Marginal Law
 After integrating over 𝑥𝑏, we pick out the remaining terms:
 The marginal distribution is a Gaussian with
𝔼 𝑥𝑎 = 𝜇𝑎 cov 𝑥𝑎 = Σ𝑎𝑎
−
1
2
𝑥𝑎
⊤Λ𝑎𝑎𝑥𝑎 + 𝑥𝑎
⊤ Λ𝑎𝑎𝜇𝑎 + Λ𝑎𝑏𝜇𝑏 +
1
2
𝑚⊤Λ𝑏𝑏
−1
𝑚 + const
𝑚 = Λ𝑏𝑏𝜇𝑏 − Λ𝑏𝑎 𝑥𝑎 − 𝜇𝑎

12. Short Summary
 Conditional distribution:
Σ =
Σ𝑎𝑎 Σ𝑎𝑏
Σ𝑏𝑎 Σ𝑏𝑏
=
Λ𝑎𝑎 Λ𝑎𝑏
Λ𝑏𝑎 Λ𝑏𝑏
−1
𝑝 𝑥𝑎 𝑥𝑏 = 𝒩 𝑥𝑎|𝜇𝑎|𝑏, Λ𝑎𝑎
−1
𝜇𝑎|𝑏 = 𝜇𝑎 − Λ𝑎𝑎
−1
Λ𝑎𝑏(𝑥𝑏 − 𝜇𝑏)
 Marginal distribution:
𝑝 𝑥𝑎 = 𝒩 𝑥𝑎|𝜇𝑎, Σ𝑎𝑎

13. Bayes’ theorem for Gaussian variables
 Setup:
 What’s the marginal distribution 𝑝 𝑦 and conditional
distribution 𝑝 𝑥|𝑦 ?
𝑝 𝑥 = 𝒩 𝑥|𝜇, Λ−1
𝑝 𝑦|𝑥 = 𝒩 𝑦|A𝑥 + 𝑏, L−1
 How about first compute 𝑝 𝑧 , where 𝑧 = 𝑥, 𝑦 ⊤
 𝑝 𝑧 is a Gaussian distribution, consider the log of the
joint distribution
ln 𝑝 𝑧 = ln 𝑝 𝑥 + ln 𝑝 𝑦 𝑥
= −
1
2
𝑥 − 𝜇 ⊤
Λ 𝑥 − 𝜇
−
1
2
𝑦 − A𝑥 + 𝑏 ⊤L 𝑦 − A𝑥 + 𝑏 + const

14. Bayes’ theorem for Gaussian variables
 The same trick (consider the second order terms), we get
𝔼 𝑧 =
𝜇
A𝜇 + 𝑏
cov 𝑧 = Λ−1
Λ−1
A
AΛ−1 𝐿−1 + AΛ−1A
 We can then get 𝑝 𝑦 and 𝑝 𝑥|𝑦 by marginal and conditional
laws!

15. Maximum likelihood for the Gaussian
 Assume we have X = 𝑥1, … , 𝑥𝑁
⊤
in which the observation
𝑥𝑛 are assumed to be drawn independently from a
multivariate Gaussian, the log likelihood function is given by
ln 𝑝 X 𝜇, Σ = −
𝑁𝐷
2
ln 2𝜋 −
𝑁
2
ln Σ −
1
2
𝑛=1
𝑁
𝑥𝑛 − 𝜇 ⊤
Σ−1
𝑥𝑛 − 𝜇
 Setting the derivative to zero, we obtain the solution for the
maximum likelihood estimator:
𝜕
𝜕𝜇
ln 𝑝 X 𝜇, Σ =
𝑛=1
𝑁
Σ−1 𝑥𝑛 − 𝜇 = 0
ΣML =
1
𝑁
𝑛=1
𝑁
𝑥𝑛 − 𝜇ML 𝑥𝑛 − 𝜇ML
⊤
𝜇ML =
1
𝑁
𝑛=1
𝑁
𝑥𝑛

16. Maximum likelihood for the Gaussian
 The empirical mean is unbiased in the sense
𝔼 𝜇ML = 𝜇
 However, the maximum likelihood estimate for the covariance
has an expectation that is less that the true value:
𝔼 ΣML =
𝑁 − 1
𝑁
Σ
 We can correct it by multiplying ΣML by the factor
𝑁
𝑁−1

17. Conjugate prior for the Gaussian
 Introducing prior distributions over the parameters of the
Gaussian
 The maximum likelihood framework only gives point estimates
for the parameters, we would like to have uncertainty
estimation (confidence interval) for the estimation
 The conjugate prior for 𝜇 is a Gaussian
 The conjugate prior for precision Λ is a Gamma distribution
 We would like the posterior 𝑝 𝜃 𝐷 ∝ 𝑝 𝜃 𝑝(𝐷|𝜃) has the
same form as the prior (Conjugate prior!)

18. The Gaussian Distribution: limitations
 A lot of parameters to estimate 𝐷 +
𝐷2+𝐷
2
: structured
approximation (e.g., diagonal variance matrix)
 Maximum likelihood estimators are not robust to outliers:
Student’s t-distribution (bottom left)
 Not able to describe periodic data: von Mises distribution
 Unimodel distribution: Mixture of Gaussian (bottom right)

19. The Gaussian Distribution: frontiers
 Gaussian Process
 Bayesian Neural Networks
 Generative modeling (Variational Autoencoder)
 ……