This document provides an overview of Linear Discriminant Analysis (LDA) for dimensionality reduction. LDA seeks to perform dimensionality reduction while preserving class discriminatory information as much as possible, unlike PCA which does not consider class labels. LDA finds a linear combination of features that separates classes best by maximizing the between-class variance while minimizing the within-class variance. This is achieved by solving the generalized eigenvalue problem involving the within-class and between-class scatter matrices. The document provides mathematical details and an example to illustrate LDA for a two-class problem.

1. A Tutorial on Data
Reduction
Linear Discriminant
Analysis (LDA)
Shireen Elhabian and Aly A. Farag
University of Louisville, CVIP Lab
September 2009

2. Outline
• LDA objective
• Recall … PCA
• Now … LDA
• LDA … Two Classes
– Counter example
• LDA … C Classes
– Illustrative Example
• LDA vs PCA Example
• Limitations of LDA

3. LDA Objective
• The objective of LDA is to perform
dimensionality reduction …
– So what, PCA does this L…
• However, we want to preserve as much of the
class discriminatory information as possible.
– OK, that’s new, let dwell deeper J …

4. Recall … PCA
• In PCA, the main idea to re-express the available dataset to
extract the relevant information by reducing the redundancy
and minimize the noise.
• We didn’t care about whether this dataset represent features from one
or more classes, i.e. the discrimination power was not taken into
consideration while we were talking about PCA.
• In PCA, we had a dataset matrix X with dimensions mxn, where
columns represent different data samples.
• We first started by subtracting the mean to have a zero mean dataset,
then we computed the covariance matrix Sx = XXT.
• Eigen values and eigen vectors were then computed for Sx. Hence the
new basis vectors are those eigen vectors with highest eigen values,
where the number of those vectors was our choice.
• Thus, using the new basis, we can project the dataset onto a less
dimensional space with more powerful data representation.
n – feature vectors
(data samples)
m-dimensionaldatavector

5. Now … LDA
• Consider a pattern classification problem, where we have C-
classes, e.g. seabass, tuna, salmon …
• Each class has Ni m-dimensional samples, where i = 1,2, …, C.
• Hence we have a set of m-dimensional samples {x1, x2,…, xNi}
belong to class ωi.
• Stacking these samples from different classes into one big fat
matrix X such that each column represents one sample.
• We seek to obtain a transformation of X to Y through
projecting the samples in X onto a hyperplane with
dimension C-1.
• Let’s see what does this mean?

6. LDA … Two Classes
• Assume we have m-dimensional samples {x1,
x2,…, xN}, N1 of which belong to ω1 and
N2 belong to ω2.
• We seek to obtain a scalar y by projecting
the samples x onto a line (C-1 space, C = 2).
– where w is the projection vectors used to project x
to y.
• Of all the possible lines we would like to
select the one that maximizes the
separability of the scalars.
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ë
é
=
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ë
é
==
mm
T
w
w
wand
x
x
xwherexwy
.
.
.
.
11
The two classes are not well
separated when projected onto
this line
This line succeeded in separating
the two classes and in the
meantime reducing the
dimensionality of our problem
from two features (x1,x2) to only a
scalar value y.

7. LDA … Two Classes
• In order to find a good projection vector, we need to define a
measure of separation between the projections.
• The mean vector of each class in x and y feature space is:
– i.e. projecting x to y will lead to projecting the mean of x to the mean of
y.
• We could then choose the distance between the projected means
as our objective function
i
i
ii
i
i
i
T
xi
T
x
T
iyixi
wx
N
w
xw
N
y
N
andx
N
µ
µµ
w
www
==
===
å
ååå
Î
ÎÎÎ
1
11~1
( )222 111
~~)( µµµµµµ -=-=-= TTT
wwwwJ

8. LDA … Two Classes
• However, the distance between the projected means is not a very
good measure since it does not take into account the standard
deviation within the classes.
This axis has a larger distance between means
This axis yields better class separability

9. LDA … Two Classes
• The solution proposed by Fisher is to maximize a function that
represents the difference between the means, normalized by a
measure of the within-class variability, or the so-called scatter.
• For each class we define the scatter, an equivalent of the
variance, as; (sum of square differences between the projected samples and their class
mean).
• measures the variability within class ωi after projecting it on
the y-space.
• Thus measures the variability within the two
classes at hand after projection, hence it is called within-class scatter
of the projected samples.
( )åÎ
-=
iy
ii ys
w
µ 22 ~~
2~
is
2
2
2
1
~~ ss +

10. LDA … Two Classes
• The Fisher linear discriminant is defined as
the linear function wTx that maximizes the
criterion function: (the distance between the
projected means normalized by the within-
class scatter of the projected samples.
• Therefore, we will be looking for a projection
where examples from the same class are
projected very close to each other and, at the
same time, the projected means are as farther
apart as possible
2
2
2
1
2
2
~~
~~
)( 1
ss
wJ
+
-
=
µµ

11. LDA … Two Classes
• In order to find the optimum projection w*, we need to express
J(w) as an explicit function of w.
• We will define a measure of the scatter in multivariate feature
space x which are denoted as scatter matrices;
• Where Si is the covariance matrix of class ωi, and Sw is called the
within-class scatter matrix.
( )( )
21 SSS
xxS
w
T
i
x
ii
i
+=
--= åÎ
µµ
w
2
2
2
1
2
2
~~
~~
)( 1
ss
wJ
+
-
=
µµ

12. LDA … Two Classes
• Now, the scatter of the projection y can then be expressed as a function of
the scatter matrix in feature space x.
Where is the within-class scatter matrix of the projected samples y.
( ) ( )
( )( )
( )( )
( ) WW
TTTT
i
T
x
T
ii
T
x
T
ii
T
x
i
TT
y
ii
SwSwwSSwwSwwSwss
wSwwxxw
wxxw
wxwys
i
i
ii
~~~
~~
2121
2
2
2
1
222
==+=+=+
=÷
÷
ø
ö
ç
ç
è
æ
--=
--=
-=-=
å
å
åå
Î
Î
ÎÎ
w
w
ww
µµ
µµ
µµ
WS
~
2
2
2
1
2
2
~~
~~
)( 1
ss
wJ
+
-
=
µµ

13. LDA … Two Classes
• Similarly, the difference between the projected means (in y-space) can be
expressed in terms of the means in the original feature space (x-space).
• The matrix SB is called the between-class scatter of the original samples/feature
vectors, while is the between-class scatter of the projected samples y.
• Since SB is the outer product of two vectors, its rank is at most one.
( ) ( )
( )( )
BB
T
S
TT
TT
SwSw
ww
ww
B
~
~~
2121
2
21
2
21
==
--=
-=-
!!! "!!! #$
µµµµ
µµµµ
BS
~
2
2
2
1
2
2
~~
~~
)( 1
ss
wJ
+
-
=
µµ

14. LDA … Two Classes
• We can finally express the Fisher criterion in terms of SW and SB
as:
• Hence J(w) is a measure of the difference between class
means (encoded in the between-class scatter matrix)
normalized by a measure of the within-class scatter matrix.
wSw
wSw
ss
wJ
W
T
B
T
=
+
-
= 2
2
2
1
2
2
~~
~~
)( 1
µµ

15. LDA … Two Classes
• To find the maximum of J(w), we differentiate and equate to
zero.
( ) ( ) ( ) ( )
( ) ( )
0)(
0)(
0
:2
022
0
0)(
1
=-Þ
=-Þ
=÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
Þ
=-Þ
=-Þ
=÷÷
ø
ö
çç
è
æ
=
-
wwJwSS
wSwJwS
wS
wSw
wSw
wS
wSw
wSw
wSwbyDividing
wSwSwwSwSw
wSw
dw
d
wSwwSw
dw
d
wSw
wSw
wSw
dw
d
wJ
dw
d
BW
WB
W
W
T
B
T
B
W
T
W
T
W
T
WB
T
BW
T
W
T
B
T
B
T
W
T
W
T
B
T

16. LDA … Two Classes
• Solving the generalized eigen value problem
yields
• This is known as Fisher’s Linear Discriminant, although it is not a
discriminant but rather a specific choice of direction for the projection
of the data down to one dimension.
• Using the same notation as PCA, the solution will be the eigen
vector(s) of
scalarwJwherewwSS BW ===-
)(1
ll
( )21
1*
maxarg)(maxarg µµ -=÷÷
ø
ö
çç
è
æ
== -
W
W
T
B
T
ww
S
wSw
wSw
wJw
BWX SSS 1-
=

17. LDA … Two Classes - Example
• Compute the Linear Discriminant projection for the following two-
dimensional dataset.
– Samples for class ω1 : X1=(x1,x2)={(4,2),(2,4),(2,3),(3,6),(4,4)}
– Sample for class ω2 : X2=(x1,x2)={(9,10),(6,8),(9,5),(8,7),(10,8)}
0 1 2 3 4 5 6 7 8 9 10
0
1
2
3
4
5
6
7
8
9
10
x1
x2

18. LDA … Two Classes - Example
• The classes mean are :
÷÷
ø
ö
çç
è
æ
=ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
+÷÷
ø
ö
çç
è
æ
+÷÷
ø
ö
çç
è
æ
+÷÷
ø
ö
çç
è
æ
+÷÷
ø
ö
çç
è
æ
==
÷÷
ø
ö
çç
è
æ
=ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
+÷÷
ø
ö
çç
è
æ
+÷÷
ø
ö
çç
è
æ
+÷÷
ø
ö
çç
è
æ
+÷÷
ø
ö
çç
è
æ
==
å
å
Î
Î
6.7
4.8
8
10
7
8
5
9
8
6
10
9
5
11
8.3
3
4
4
6
3
3
2
4
2
2
4
5
11
2
2
1
1
2
1
w
w
µ
µ
x
x
x
N
x
N

19. LDA … Two Classes - Example
• Covariance matrix of the first class:
( )( )
÷÷
ø
ö
çç
è
æ
-
-
=
ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
+ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
+ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
+
ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
+ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
=--= åÎ
2.225.0
25.01
8.3
3
4
4
8.3
3
6
3
8.3
3
3
2
8.3
3
4
2
8.3
3
2
4
222
22
111
1
T
x
xxS µµ
w

20. LDA … Two Classes - Example
• Covariance matrix of the second class:
( )( )
÷÷
ø
ö
çç
è
æ
-
-
=
ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
+ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
+ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
+
ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
+ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
=--= åÎ
3.305.0
05.03.2
6.7
4.8
8
10
6.7
4.8
7
8
6.7
4.8
5
9
6.7
4.8
8
6
6.7
4.8
10
9
222
22
222
2
T
x
xxS µµ
w

21. LDA … Two Classes - Example
• Within-class scatter matrix:
÷÷
ø
ö
çç
è
æ
-
-
=
÷÷
ø
ö
çç
è
æ
-
-
+÷÷
ø
ö
çç
è
æ
-
-
=+=
5.53.0
3.03.3
3.305.0
05.03.2
2.225.0
25.01
21 SSSw

22. LDA … Two Classes - Example
• Between-class scatter matrix:
( )( )
( )
÷÷
ø
ö
çç
è
æ
=
--÷÷
ø
ö
çç
è
æ
-
-
=
ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
=
--=
44.1452.20
52.2016.29
8.34.5
8.3
4.5
6.7
4.8
8.3
3
6.7
4.8
8.3
3
2121
T
T
BS µµµµ

23. LDA … Two Classes - Example
• The LDA projection is then obtained as the solution of the generalized eigen
value problem
( )( )
( )
2007.12,0
02007.1202007.12
02339.4489.69794.22213.9
9794.22339.4
489.62213.9
0
10
01
44.1452.20
52.2016.29
1827.00166.0
0166.03045.0
0
10
01
44.1452.20
52.2016.29
5.53.0
3.03.3
0
21
2
1
1
1
==Þ
=-Þ=-Þ
=´---=
÷÷
ø
ö
çç
è
æ
-
-
Þ
=÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
÷÷
ø
ö
çç
è
æ
Þ
=÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
÷÷
ø
ö
çç
è
æ
-
-
Þ
=-Þ
=
-
-
-
ll
llll
ll
l
l
l
l
l
l
ISS
wwSS
BW
BW

24. LDA … Two Classes - Example
• Hence
• The optimal projection is the one that given maximum λ = J(w)
!
*
21
2
1
2
2
1
1
4173.0
9088.0
8178.0
5755.0
;
2007.12
9794.22339.4
489.62213.9
0
9794.22339.4
489.62213.9
2
1
wwandw
Thus
w
w
w
and
w
w
w
=÷÷
ø
ö
çç
è
æ
=÷÷
ø
ö
çç
è
æ-
=
÷÷
ø
ö
çç
è
æ
=÷÷
ø
ö
çç
è
æ
÷÷
ø
ö
çç
è
æ
=÷÷
ø
ö
çç
è
æ
"#"$%
l
l

25. LDA … Two Classes - Example
( )
÷÷
ø
ö
çç
è
æ
=
÷÷
ø
ö
çç
è
æ
-
-
÷÷
ø
ö
çç
è
æ
=
ú
û
ù
ê
ë
é
÷÷
ø
ö
çç
è
æ
-÷÷
ø
ö
çç
è
æ
÷÷
ø
ö
çç
è
æ
-
-
=-=
-
-
4173.0
9088.0
8.3
4.5
1827.00166.0
0166.03045.0
6.7
4.8
8.3
3
5.53.0
3.03.3
1
21
1*
µµWSw
Or directly;

26. LDA - Projection
-4 -3 -2 -1 0 1 2 3 4 5 6
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
y
p(y|wi
)
Classes PDF : using the LDA projection vector with the other eigen value = 8.8818e-016
-7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10
0
1
2
3
4
5
6
7
8
9
10
x1
x
2
LDA projection vector with the other eigen value = 8.8818e-016
The projection vector
corresponding to the
smallest eigen value
Using this vector leads to
bad separability
between the two classes

27. LDA - Projection
0 5 10 15
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
y
p(y|wi
)
Classes PDF : using the LDA projection vector with highest eigen value = 12.2007
0 1 2 3 4 5 6 7 8 9 10
0
1
2
3
4
5
6
7
8
9
10
x1
x
2
LDA projection vector with the highest eigen value = 12.2007
The projection vector
corresponding to the
highest eigen value
Using this vector leads to
good separability
between the two classes

28. LDA … C-Classes
• Now, we have C-classes instead of just two.
• We are now seeking (C-1) projections [y1, y2, …, yC-1] by means
of (C-1) projection vectors wi.
• wi can be arranged by columns into a projection matrix W =
[w1|w2|…|wC-1] such that:
[ ]1211
1
1
11
1
1
|...||
.
.
,
.
.
--´
-
´-´
=
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ë
é
=
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ë
é
=
CCm
C
C
m
m
wwwWand
y
y
y
x
x
xwhere
xWyxwy TT
ii =Þ=

29. LDA … C-Classes
• If we have n-feature vectors, we can stack them into one matrix
as follows;
[ ]1211
1
2
1
1
1
1
2
1
1
1
1
21
1
2
1
1
1
|...||
.
....
....
.
,
.
....
....
.
--´
---
´-´
=
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ë
é
=
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ë
é
=
CCm
n
CCC
n
nC
n
mmm
n
nm
wwwWand
yyy
yyy
Y
xxx
xxx
Xwhere
XWY T
=

30. LDA – C-Classes
• Recall the two classes case, the
within-class scatter was computed as:
• This can be generalized in the C-
classes case as:
21 SSSw +=
( )( )
å
å
å
Î
Î
=
=
--=
=
i
i
xi
i
T
i
x
ii
C
i
iW
x
N
and
xxSwhere
SS
w
w
µ
µµ
1
1
x1
x2
µ1
µ2
µ3
Sw2
Example of two-dimensional
features (m = 2), with three
classes C = 3.
Ni : number of data samples
in class ωi.

31. LDA – C-Classes
• Recall the two classes case, the between-
class scatter was computed as:
• For C-classes case, we will measure the
between-class scatter with respect to the
mean of all classes as follows:
( )( )
å
åå
å
Î
""
=
=
==
--=
ixi
i
x
ii
x
C
i
T
iiiB
x
N
and
N
N
x
N
where
NS
w
µ
µµ
µµµµ
1
11
1
( )( )T
BS 2121 µµµµ --=
x1
x2
µ1
µ2
µ3
Sw2
µ
Example of two-dimensional
features (m = 2), with three
classes C = 3.
Ni : number of data samples
in class ωi.
N: number of all data .

32. LDA – C-Classes
• Similarly,
– We can define the mean vectors for the projected samples y as:
– While the scatter matrices for the projected samples y will be:
åå "Î
==
yyi
i y
N
andy
N i
1~1~ µµ
w
( )( )ååå = Î=
--==
C
i
T
i
y
i
C
i
iW yySS
i11
~~~~
µµ
w
( )( )å=
--=
C
i
T
iiiB NS
1
~~~~~
µµµµ

33. LDA – C-Classes
• Recall in two-classes case, we have expressed the scatter matrices of the
projected samples in terms of those of the original samples as:
This still hold in C-classes case.
• Recall that we are looking for a projection that maximizes the ratio of
between-class to within-class scatter.
• Since the projection is no longer a scalar (it has C-1 dimensions), we then use
the determinant of the scatter matrices to obtain a scalar objective function:
• And we will seek the projection W* that maximizes this ratio.
WSWS
WSWS
B
T
B
W
T
W
=
=
~
~
WSW
WSW
S
S
WJ
W
T
B
T
W
B
== ~
~
)(

34. LDA – C-Classes
• To find the maximum of J(W), we differentiate with respect to W and equate
to zero.
• Recall in two-classes case, we solved the eigen value problem.
• For C-classes case, we have C-1 projection vectors, hence the eigen value
problem can be generalized to the C-classes case as:
• Thus, It can be shown that the optimal projection matrix W* is the one whose
columns are the eigenvectors corresponding to the largest eigen values of the
following generalized eigen value problem:
1,...2,1)(1
-====-
CiandscalarwJwherewwSS iiiiiBW ll
scalarwJwherewwSS BW ===-
)(1
ll
[ ]*
1
*
2
*
1
**
**1
|...||)( -
-
===
=
C
BW
wwwWandscalarWJwhere
WWSS
l
l

35. Illustration – 3 Classes
• Let’s generate a dataset for each
class to simulate the three classes
shown
• For each class do the following,
– Use the random number generator
to generate a uniform stream of
500 samples that follows U(0,1).
– Using the Box-Muller approach,
convert the generated uniform
stream to N(0,1).
x1
x2
µ1
µ2
µ3
µ
– Then use the method of eigen values
and eigen vectors to manipulate the
standard normal to have the required
mean vector and covariance matrix .
– Estimate the mean and covariance
matrix of the resulted dataset.

36. • By visual inspection of the figure,
classes parameters (means and
covariance matrices can be given as
follows:
Dataset Generation
x1
x2
µ1
µ2
µ3
µ
÷÷
ø
ö
çç
è
æ
=
÷÷
ø
ö
çç
è
æ
=
÷÷
ø
ö
çç
è
æ
-
-
=
ú
û
ù
ê
ë
é
+=ú
û
ù
ê
ë
é
-
-
+=ú
û
ù
ê
ë
é-
+=
ú
û
ù
ê
ë
é
=
5.23
15.3
40
04
33
15
5
7
,
5.3
5.2
,
7
3
5
5
meanOverall
3
2
1
321
S
S
S
µµµµµµ
µ
Zero covariance to lead to data samples distributed horizontally.
Positive covariance to lead to data samples distributed along the y = x line.
Negative covariance to lead to data samples distributed along the y = -x line.

37. In Matlab J

38. It’s Working … J
-5 0 5 10 15 20
-5
0
5
10
15
20
X1
- the first feature
X2
-thesecondfeature
x1
x2
µ1
µ2
µ3
µ

39. Computing LDA Projection Vectors
( )( )
å
åå
å
Î
""
=
=
==
--=
ixi
i
x
ii
x
C
i
T
iiiB
x
N
and
N
N
x
N
where
NS
w
µ
µµ
µµµµ
1
11
1
( )( )
å
å
å
Î
Î
=
=
--=
=
i
i
xi
i
T
i
x
ii
C
i
iW
x
N
and
xxSwhere
SS
w
w
µ
µµ
1
1
Recall …
BW SS 1-

40. Let’s visualize the projection vectors W
-15 -10 -5 0 5 10 15 20 25
-10
-5
0
5
10
15
20
25
X1
- the first feature
X2
-thesecondfeature

41. Projection … y = WTx
Along first projection vector
-5 0 5 10 15 20 25
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
y
p(y|wi
)
Classes PDF : using the first projection vector with eigen value = 4508.2089

42. Projection … y = WTx
Along second projection vector
-10 -5 0 5 10 15 20
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
y
p(y|wi
)
Classes PDF : using the second projection vector with eigen value = 1878.8511

43. Which is Better?!!!
• Apparently, the projection vector that has the highest eigen
value provides higher discrimination power between classes
-10 -5 0 5 10 15 20
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
y
p(y|wi
)
Classes PDF : using the second projection vector with eigen value = 1878.8511
-5 0 5 10 15 20 25
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
y
p(y|wi
)
Classes PDF : using the first projection vector with eigen value = 4508.2089

44. PCA vs LDA

45. Limitations of LDA L
• LDA produces at most C-1 feature projections
– If the classification error estimates establish that more features are needed, some other method must be
employed to provide those additional features
• LDA is a parametric method since it assumes unimodal Gaussian likelihoods
– If the distributions are significantly non-Gaussian, the LDA projections will not be able to preserve any
complex structure of the data, which may be needed for classification.

46. Limitations of LDA L
• LDA will fail when the discriminatory information is not in the mean but
rather in the variance of the data

47. Thank You