Discriminant Analysis-lecture 8

4,403 views

Published on

Published in: Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,403
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
161
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Discriminant Analysis-lecture 8

  1. 1. 4/30/2012Linear Discriminant Analysis Proposed by Fisher (1936) for classifying an observation into one of two possible groups based on many measurements x1,x2,…xp. Seek a linear transformation of the variables Y=w1x1+w2x2+..+wpxp + a constant 1
  2. 2. 4/30/2012 Linear Discriminant Analysis  Discriminant analysis – creates an equation which will minimize the possibility of misclassifying cases into their respective groups or categories.The purposes of discriminant analysis (DA)  Discriminant Function Analysis (DA) undertakes the same task as multiple linear regression by predicting an outcome.  However, multiple linear regression is limited to cases where the dependent is numerical  But many interesting variables are categorical, 2
  3. 3. 4/30/2012  The objective of DA is to perform dimensionality reduction while preserving as much of the class discriminatory information as possible  Assume we have a set of D-dimensional samples {x 1, x2, …, xN}, N1 of which belong to class ω1, and N2 to class ω2.  We seek to obtain a scalar y by projecting the samples x onto a line y = wTx•The top two distributions overlap too much and do notdiscriminate too well compared to the bottom set.•Misclassification will be minimal in the lower pair,•whereas many will be misclassified in the top pair. 3
  4. 4. 4/30/2012 Linear Discriminant Analysis Assume variance matrices equal Classify the item x at hand to one of J groups based on measurements on p predictors. Rule: Assign x to group j that has the closest mean j = 1, 2, …, J Distance Measure: Mahalanobis Distance. Linear Discriminant Analysis Distance Measure: For j = 1, 2, …, J, compute d j  x    x  x j Spl  x  x j T 1Assign x to the group for which dj is minimumS is the pooled estimate of the covariance pl matrix 4
  5. 5. 4/30/2012…or equivalently, assign x to thegroup for which L x   x S 1 1 1 x T T j j pl 2 x S x j pl j is a maximum. (Notice the linear form of the equation!)Linear Discriminant Analysis…optimal if….• Multivariate normal distribution for the observation in each of the groups• Equal covariance matrix for all groups• Equal prior probability for each group• Equal costs for misclassification 5
  6. 6. 4/30/2012Relaxing the assumption of equal priorprobabilities…L x   ln p j  x S T 1 1 1 x T j j pl 2 x S x j pl j pj being the prior probability for the jth group.Relaxing the assumption of equalcovariance matrices… 1 Q  x   ln p j  ln S j j 2  x  x  S x  x j  T 1 j jresult?…Quadratic DiscriminantAnalysis 6
  7. 7. 4/30/2012Quadratic Discriminant Analysis Rule: assign to group j if Q x  j is the largest.Optimal ifthe J groups of measurements aremultivariate normal Other Extensions & Related Methods Relaxing the assumption of normality… Kernel density based LDA and QDA Other extensions….. Regularized discriminant analysis Penalized discriminant analysis Flexible discriminant analysis 7
  8. 8. 4/30/2012Evaluations of the Methods Classification Table (confusion matrix) Predicted groupActual group Number of observations A BA nA n11 n12B nB n21 n22Evaluations of the MethodsApparent Error Rate (APER): # misclassified APER = Total # of cases….underestimates the actual error rate.Improved estimate of APER:Holdout Method or cross validation 8
  9. 9. 4/30/2012Fishers iris dataset •The data were collected by Anderson and used by Fisher to formulate the linear discriminant analysis (LDA or DA). •The dataset gives the measurements in centimeters of the following variables: 1- sepal length, 2- sepal width, 3- petal length, and 4- petal width, this for 50 fowers from each of the 3 species of iris considered. •The species considered are Iris setosa, versicolor, and virginica setosa versicolor virginica 9
  10. 10. 4/30/2012An Example: Fisher’s Iris DataActual Number of Predicted Group ObservationsGroup Setosa Versicolo Virginica rSetosa 50 50 0 0Versicolor 50 0 48 2Virginica 50 0 1 49Table 1: Linear Discriminant Analysis (APER = 0.0200) An Example: Fisher’s Iris DataActual Number of Predicted Group ObservationsGroup Setosa Versicolo Virginica rSetosa 50 50 0 0Versicolor 50 0 47 3Virginica 50 0 1 49Table 1: Quadratic Discriminant Analysis (APER = 0.0267) 10
  11. 11. 4/30/2012 An Example: Fisher’s Iris Data 2.5 v v v v v v v v v v v v v v v v v 2.0 v v v v v v v v v v v v v v c v v c v c v c c 1.5 c v c c v c c c c v c c c c c cPetal Width c c c c c c c c c c c c 1.0 c c c c c c s 0.5 s s s s s s s s s s s s s s s s s s s s s s s s s s s 2. 0 2. 5 3. 0 3. 5 4. 0 S Wi t epal dh An Example: Fisher’s Iris Data 2.5 o v o v o v o v o v o v o v o v o v o v o v o v o v o v o v o v o v 2.0 o v o v o v o v o v o v o v o v o v o v o v o v o v o v o c v o v x c o c o v x c x c 1.5 x o c v x c x c v x c x c x c x c o v x c x c x c x c x c x c Petal Width x c x c x c x c x c x c x c x c x c x c x c x c 1.0 x c x c x c x c x c x c + s 0.5 + s + s + s + s + s + s + s + s + s + s + s + s + s + s + s + s + s + s + s + s + s + s + s + s + s + s + s 2. 0 2. 5 3. 0 3. 5 4. 0 S Wi t epal dh 11
  12. 12. 4/30/2012SummaryLDA is a powerful tool available forclassification. Widely implemented through various software Theoretical properties well researched 12

×