This document discusses whitening transformations, which transform a multivariate normal distribution with an arbitrary covariance matrix into a spherical normal distribution with an identity covariance matrix. It provides background on normal distributions and their properties. Key points covered include how the shape of a normal distribution's support region is defined by its covariance matrix, how to perform a coordinate transformation using the eigenvectors and eigenvalues of the covariance matrix to whiten the data, and how this results in the Mahalanobis distance becoming a standardized Euclidean distance.

1. • Objectives:
Normal Random Variables
Support Regions
Whitening Transformations
• Resources:
DHS – Chap. 2 (Part 2)
K.F. – Intro to PR
X. Z. – PR Course
S.B. – Statistical Norm.
Java Applet
• URL: .../publications/courses/ece_8443/lectures/current/lecture_07.ppt
ECE 8443 – Pattern Recognition
LECTURE 07: WHITENING TRANSFORMATIONS

2. • A normal or Gaussian density is a powerful model for
modeling continuous-valued feature vectors corrupted by
noise due to its analytical tractability.
• Univariate normal distribution:
07: WHITENING TRANSFORMATIONS
UNIVARIATE NORMAL DISTRIBUTION
]
x
exp[
)
x
(
p
2
2
1
2
1












 
















dx
)
x
(
p
)
x
(
)
x
[(
E
dx
)
x
(
xp
]
x
[
E 2
2
2
where the mean and covariance are defined by:
and the entropy is given by:
 







)
e
log(
dx
)
x
(
p
ln
)
x
(
p
))
x
(
p
(
H 2
2
2
1

3. 07: WHITENING TRANSFORMATIONS
MEAN AND VARIANCE
• A normal distribution is completely specified by its mean
and variance:
• A normal distribution achieves the maximum entropy of all
distributions having a given mean and variance.
• Central Limit Theorem: The sum of a large number of
small, independent random variables will lead to a
Gaussian distribution.
• The peak is at:
• 66% of the area is within
one ; 95% is within two ;
99% is within three .




2
1
)
(
p

4. 07: WHITENING TRANSFORMATIONS
MULTIVARIATE NORMAL DISTRIBUTIONS
• A multivariate distribution is defined as:
)]
(
)
(
exp[
)
(
)
(
p t
/
/
d
μ
μ 





 
x
x
x 1
2
1
2 2
1
2
1
where  represents the mean (vector) and  represents
the covariance (matrix).
• Note the exponent term is really a dot product or
weighted Euclidean distance.
• The covariance is always
symmetric and positive
semidefinite.
• How does the shape vary as
a function of the
covariance?

5. 07: WHITENING TRANSFORMATIONS
SUPPORT REGIONS
• A support region is the
obtained by the intersection
of a Gaussian distribution
with a plane.
• For a horizontal plane, this
generates an ellipse whose
points are of equal
probability density.
• The shape of the support
region is defined by the
covariance matrix.

6. 07: WHITENING TRANSFORMATIONS
DERIVATION

7. 07: WHITENING TRANSFORMATIONS
IDENTITY COVARIANCE

8. 07: WHITENING TRANSFORMATIONS
UNEQUAL VARIANCES

9. 07: WHITENING TRANSFORMATIONS
OFF-DIAGONAL ELEMENTS

10. 07: WHITENING TRANSFORMATIONS
UNCONSTRAINED ELEMENTS

11. 07: WHITENING TRANSFORMATIONS
COORDINATE TRANSFORMATIONS
• Why is it convenient to convert an arbitrary distribution
into a spherical one? (Hint: Euclidean distance)
• Consider the transformation: Aw=  -1/2
where  is the matrix whose columns are the orthonormal
eigenvectors of  and  is a diagonal matrix of
eigenvalues (=   t). Note that  is unitary.
• What is the covariance of y=Awx?
E[yyt] = (Awx)(Awx)t =( -1/2x) ( -1/2x)t
=  -1/2x xt -1/2 t =  -1/2  -1/2 t
=  -1/2   t -1/2 t
=  t -1/2  -1/2 (t)
= I

12. 07: WHITENING TRANSFORMATIONS
MAHALANOBIS DISTANCE
)
(
)
( t
μ
μ 

 
x
x 1
• The weighted Euclidean distance:
is known as the Mahalanobis distance, and represents a
statistically normalized distance calculation that results
from our whitening transformation.
• Consider an example using our Java Applet.