Linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA) are statistical techniques used to classify observations into groups based on measurements of predictor variables. LDA assumes normal distributions and equal covariance matrices across groups, while QDA allows unequal covariance matrices. The document discusses these techniques, provides an example applying LDA and QDA to Fisher's iris data, and evaluates classification performance using an apparent error rate.

1. Linear Discriminant Analysis and Its
Variations
Abu Minhajuddin
CSE 8331
Department of Statistical Science
Southern Methodist University
April 27, 2002

2. Plan…
The Problem
Linear Discriminant Analysis
Quadratic Discriminant Analysis
Other Extensions
Evaluation of the Method
An Example
Summary

3. The Problem…
Classification Problem:
Data on p predictors
Unknown group membership
Training Situation:
Data on p predictors,
Membership of one of g groups

4. The Problem…
Sepal Width
PetalWidth
2.0 2.5 3.0 3.5 4.0
0.51.01.52.02.5
Fisher’s Iris Data: Identify the three species?

5. Linear Discriminant Analysis
Classify the item x at hand to one of J groups based on
measurements on p predictors.
Rule: Assign x to group j that has the closest mean
j = 1, 2, …, J
Distance Measure: Mahalanobis Distance….
Takes the spread of the data into Consideration

6. Linear Discriminant Analysis
Distance Measure:
For j = 1, 2, …, J, compute
( ) ( ) ( )xSxd jpl
T
jj
xxx −−=
−1
Assign x to the group for which dj is minimum
Spl is the pooled estimate of the covariance
matrix

7. Linear Discriminant AnalysisLinear Discriminant Analysis
…or equivalently, assign x to the group
for which
( ) xSxSxL jpl
T
jpl
T
jj
xx
11
2
1 −−
−=
is a maximum.
(Notice the linear form of the equation!)

8. Linear Discriminant Analysis
…optimal if….
• Multivariate normal distribution for the
observation in each of the groups
• Equal covariance matrix for all groups
• Equal prior probability for each group
• Equal costs for misclassification

9. Linear Discriminant Analysis
Relaxing the assumption of equal prior
probabilities…
( ) xSxSxpL jpl
T
jpl
T
j
j
xjx
11
2
1
ln −−
−= +
p j being the prior probability for the jth
group.

10. Linear Discriminant Analysis
Relaxing the assumption of equal covariance
matrices…
( )
( ) ( )jx
jj
x
xxSx
SpQ
j
T
j
j
−
−
−
−−
=
1
ln
2
1
ln
result?…Quadratic Discriminant Analysis

11. Quadratic Discriminant Analysis
Rule: assign to group j if is the
largest.
( )xQj
Optimal if the J groups of
measurements are multivariate normal

12. Other Extensions & Related Methods
Relaxing the assumption of normality…
Kernel density based LDA and QDA
Other extensions…..
Regularized discriminant analysis
Penalized discriminant analysis
Flexible discriminant analysis

13. Other Extensions & Related Methods
Related Methods:
Logistic regression for binary classification
Multinomial logistic regression
These methods models the probability of
being in a class as a linear function of the
predictor.

14. Evaluations of the Methods
Classification Table (confusion matrix)
Actual
group
Number of
observations
Predicted group
A B
A
B
nA
nB
n11
n21
n12
n22

15. Evaluations of the Methods
Apparent Error Rate (APER):
APER = # misclassified/Total # of cases
….underestimates the actual error rate.
Improved estimate of APER:
Holdout Method or cross validation

16. An Example: Fisher’s Iris Data
Actual
Group
Number of
Observations
Predicted Group
Setosa Versicolo
r
Virginica
Setosa
Versicolor
Virginica
50
50
50
50
0
0
0
48
1
0
2
49
Table 1: Linear Discriminant Analysis
(APER = 0.0200)

17. An Example: Fisher’s Iris DataAn Example: Fisher’s Iris Data
Actual
Group
Number of
Observations
Predicted Group
Setosa Versicolo
r
Virginica
Setosa
Versicolor
Virginica
50
50
50
50
0
0
0
47
1
0
3
49
Table 1: Quadratic Discriminant Analysis
(APER = 0.0267)

18. An Example: Fisher’s Iris Data
Sepal Width
PetalWidth
2.0 2.5 3.0 3.5 4.0
0.51.01.52.02.5
ss ss s
s
s
ss
s
ss
ss
s
ss
s ss
s
s
s
s
ss
s
ssss
s
s
ss s s
s
s s
ss
s
s
s
s
ss ss
v
v
v
v
v
v
v
vv
v
v
v
v
v
v
v
v
v
v
v
v
vv
v
v
vv v
v
v
v
v
v
v
v
v
v
vv
v
v
v
v
v
v
v
v
v
v
v
c
cc
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
cc
c
c
c
c
c
c
c
c
c
c
c cc
c
c
c
c
c
c
cc
c
c
ss ss s
s
s
ss
s
ss
ss
s
ss
s ss
s
s
s
s
ss
s
ssss
s
s
ss s s
s
s s
ss
s
s
s
s
ss ss
v
v
v
v
v
v
v
v
v
vvv
v
v
vv
v
v
v
vv
v
vvvv
v
vvv
v
vv
v
v
v
v
vv
v
vvv
v
vv
v
v
v
v
v
c
cc
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
cc
c
c
c
c
c
c
c
c
c
c cc
c
c
c
c
c
c
cc
c
c
c
ss ss s
s
s
ss
s
ss
ss
s
ss
s ss
s
s
s
s
ss
s
ssss
s
s
ss s s
s
s s
ss
s
s
s
s
ss ss
v
v
v
v
v
v
v
v
v
vvv
v
v
vv
v
v
v
vv
v
vvvv
v
vvv
v
vv
v
v
v
v
vv
v
vvv
v
vv
v
v
v
v
v
c
cc
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
cc
c
c
c
c
c
c
c
c
c
c cc
c
c
c
c
c
c
cc
c
c
c
ss ss s
s
s
ss
s
ss
ss
s
ss
s ss
s
s
s
s
ss
s
ssss
s
s
ss s s
s
s s
ss
s
s
s
s
ss ss
v
v
v
v
v
v
v
vv
v
v
v
v
v
v
v
v
v
v
v
v
vv
v
v
vv v
v
v
v
v
v
v
v
v
v
vv
v
v
v
v
v
v
v
v
v
v
v
c
cc
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
cc
c
c
c
c
c
c
c
c
c
c
c cc
c
c
c
c
c
c
cc
c
c
ss ss s
s
s
ss
s
ss
ss
s
ss
s ss
s
s
s
s
ss
s
ssss
s
s
ss s s
s
s s
ss
s
s
s
s
ss ss
v
v
v
v
v
v
v
v
v
vvv
v
v
vv
v
v
v
vv
v
vvvv
v
vvv
v
vv
v
v
v
v
vv
v
vvv
v
vv
v
v
v
v
v
c
cc
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
cc
c
c
c
c
c
c
c
c
c
c cc
c
c
c
c
c
c
cc
c
c
c
ss ss s
s
s
ss
s
ss
ss
s
ss
s ss
s
s
s
s
ss
s
ssss
s
s
ss s s
s
s s
ss
s
s
s
s
ss ss
v
v
v
v
v
v
v
vv
v
v
v
v
v
v
v
v
v
v
v
v
vv
v
v
vv v
v
v
v
v
v
v
v
v
v
vv
v
v
v
v
v
v
v
v
v
v
v
c
cc
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
cc
c
c
c
c
c
c
c
c
c
c
c cc
c
c
c
c
c
c
cc
c
c
ss ss s
s
s
ss
s
ss
ss
s
ss
s ss
s
s
s
s
ss
s
ssss
s
s
ss s s
s
s s
ss
s
s
s
s
ss ss
v
v
v
v
v
v
v
v
v
vvv
v
v
vv
v
v
v
vv
v
vvvv
v
vvv
v
vv
v
v
v
v
vv
v
vvv
v
vv
v
v
v
v
v
c
cc
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
cc
c
c
c
c
c
c
c
c
c
c cc
c
c
c
c
c
c
cc
c
c
c
ss ss s
s
s
ss
s
ss
ss
s
ss
s ss
s
s
s
s
ss
s
ssss
s
s
ss s s
s
s s
ss
s
s
s
s
ss ss
v
v
v
v
v
v
v
vv
v
v
v
v
v
v
v
v
v
v
v
v
vv
v
v
vv v
v
v
v
v
v
v
v
v
v
vv
v
v
v
v
v
v
v
v
v
v
v
c
cc
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
cc
c
c
c
c
c
c
c
c
c
c
c cc
c
c
c
c
c
c
cc
c
c

19. An Example: Fisher’s Iris Data
Sepal Width
PetalWidth
2.0 2.5 3.0 3.5 4.0
0.51.01.52.02.5
ss ss s
s
s
ss
s
ss
ss
s
ss
s ss
s
s
s
s
ss
s
ssss
s
s
ss s s
s
s s
ss
s
s
s
s
ss ss
v
v
v
v
v
v
v
vv
v
v
v
v
v
v
v
v
v
v
v
v
vv
v
v
vv v
v
v
v
v
v
v
v
v
v
vv
v
v
v
v
v
v
v
v
v
v
v
c
cc
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
cc
c
c
c
c
c
c
c
c
c
c
c cc
c
c
c
c
c
c
cc
c
c
Sepal Width
PetalWidth
2.0 2.5 3.0 3.5 4.0
0.51.01.52.02.5
++ ++ +
+
+
++
+
++
++
+
++
+ ++
+
+
+
+
++
+
++++
+
+
++ + +
+
+ +
++
+
+
+
+
++ ++
o
o
o
o
o
o
o
o
o
ooo
o
o
oo
o
o
o
oo
o
oooo
o
ooo
o
oo
o
o
o
o
oo
o
ooo
o
oo
o
o
o
o
o
x
xx
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
xx
x
x
x
x
x
x
x
x
x
x xx
x
x
x
x
x
x
xx
x
x
x

20. Summary
LDA is a powerful tool available for
classification.
Widely implemented through various
software
Theoretical properties well researched
SAS implementation available for
large data sets.