IIT Roorkee

1. INDIAN INSTITUTE OF TECHNOLOGY ROORKEE
bala@cs.iitr.ac.in
https://faculty.iitr.ac.in/cs/bala/
CSN-382 (Lecture 10)
Dr. R. Balasubramanian
Professor
Department of Computer Science and Engineering
Mehta Family School of Data Science and Artificial Intelligence
Indian Institute of Technology Roorkee
Roorkee 247 667
Machine Learning

2. 2
● Signal – all valid values for a variable (shows
between max and min values for x axis and y
axis). Represents a valid data.
● Noise – The spread of data points across the
best fit line. For a given value of x, there are
multiple values of y (some on line and some
around the line). This spread is due to random
factors.
● Signal to Noise Ratio – Variance of signal /
variance in noise.
● Greater the SNR the better the model will be.
X min X max
Signal
Y max
Y min
+
+
+
+
+
+
+
+
+
+ +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ +
+
+
+ +
+
+
+
+
+
+
+
+
+
+
PCA (Signal to noise ratio)

3. 3
● PCA can also be used to reduce
dimensions.
● Arrange all eigenvectors along with
corresponding eigenvalues in
descending order of eigenvalues.
● Plot a cumulative eigenvalue graph.
● Eigenvectors with insignificant
contribution to total eigenvalues
can be removed from analysis.
PCA for dimensionality reduction

4. 4
Advantages Disadvantages
● Helps is reducing dimensions
● Correlated features are
removed
● Improves performance of an
algorithm
● Low noise sensitivity
● Assumes that feature set is correlated
● Sensitive to outliers
● High variance axis is treated as PC,
and low variance axes are treated as
noise
● Covariance matrix are difficult to be
evaluated in an accurate manner
Advantages and disadvantages

5. 5
● Dimensionality reduction
● Improving signal to noise ratio
● Helps in removing correlation between variables
● To speed up the convergence of Neural networks
● Computer vision (Face recognition)
Applications of PCA

6. 6
Feature Selection
► Instance based learning (kNN, last class)
 Not useful if the number of features is large.
► Feature Reduction
 Features contain information about the target.
► More features means better information or more information,
and better discriminative power or better classification power.
 But this may not be true always

7. 7
Curse of Dimensionality

8. 8
Curse of Dimensionality
► Irrelevant features
 In algorithm such as k nearest neighbor these irrelevant features
introduce noise and they fool the learning algorithm.
► Redundant features
 If you have a fixed number of training examples and redundant
features which do not contribute additional information they may
lead to degradation in performance of the learning algorithm.
► These irrelevant features and redundant features can confuse
learner, especially when you have limited training examples and
limited computational resources.
► Large number of features and limited training examples
 Overfitting

9. 9
To overcome Curse of Dimensionality
► Feature Selection
► Feature Extraction

10. 10
Feature Selection
► Given Set of initial features 𝐹 = {𝑥1, 𝑥2, 𝑥3, … , 𝑥𝑛}
► we can find 𝐹′ = {𝑥1′, 𝑥2′, 𝑥3′, … , 𝑥𝑚′}⊂ 𝐹
► We want to find a subset 𝐹′ of those features in 𝐹 so that it
optimizes certain criteria.
► How feature selection is differing from feature extraction.
► Feature selection in problems like hyperspectral imaging.
► From 𝑛 features set, we can have 2𝑛 possible feature subsets.
 Optimized algorithm in polynomial time
 Heuristic
 Greedy algorithm
 Randomized algorithm

11. 11
Feature Subset Evaluation
► Unsupervised (Filter method)
► Supervised (Wrapper method)

12. 12
Feature Selection Steps
► Feature Selection is an optimization problem:
► Step 1: Search the space of possible feature subsets.
► Step 2: Pick the subset that is optimal or near optimal w.r.t
some optimal function.

13. 13
Feature Selection Steps
► Search Strategies
 Optimum
 Heuristic
 Randomized
► Evaluation Methods
 Filter methods
 Wrapper methods

14. 14
Evaluating Feature Subset
► Supervised (Wrapper method)
 Train using selected subset
 Estimate error on validation dataset
► Unsupervised (Filter method)
 Look at input only
 Select the subset that has most input

15. 15
Evaluation Strategies

16. 16
Two different frameworks of feature
selection
► Find uncorrelated features in the reduced features
► Heuristic algorithms
 Forward Selection Algorithm
 Backward Selection Algorithm
► Forward Selection Algorithm
 Start with empty feature set and then you add features one by one
► Backward Selection Algorithm
 In backward search you start with the full feature set. Then you try
removing features from the features that you have.

17. 17
Feature Selection
► Univariate (looks at each feature independently of others)
 Pearson correlation coefficient
 F-Score
 Chi-Square
 Signal to noise ratio
► Rank features by importance
► Ranking cut-off determined by user
► Univariate methods measure some type of correlation between
two random variables.
► The label 𝑦𝑖 and a fixed feature, 𝑥𝑖𝑗 for fixed 𝑗

18. 18
Pearson correlation coefficient
► Please refer lecture 4 slides

19. 19
● Signal – all valid values for a variable (shows
between max and min values for x axis and y
axis). Represents a valid data.
● Noise – The spread of data points across the
best fit line. For a given value of x, there are
multiple values of y (some on line and some
around the line). This spread is due to random
factors.
● Signal to Noise Ratio – Variance of signal /
variance in noise.
● Greater the SNR the better the model will be.
X min X max
Signal
Y max
Y min
+
+
+
+
+
+
+
+
+
+ +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ +
+
+
+ +
+
+
+
+
+
+
+
+
+
+
Signal to noise ratio

20. 20
Multivariate Feature Selection
► Multivariate (consider all features simultaneously).
► Consider the vector 𝑤 for any linear classifier.
► Classification of point 𝑥 is given by 𝑤𝑇𝑥 + 𝑤0.
► Small entries of 𝑤 will have little effect on dot product and hence
those features are less relevant.
► For example if 𝑤 = (10, 0.01, −9) then features 0 and 2 are
contributing more to a dot product than feature 1.
 A ranking of features given by this 𝑤 are 0,2 and 1.
► The 𝑤 can be obtained any of linear classifiers.

21. 21
Multivariate Feature Selection
► A variant of this approach is called recursive feature elimination
 Compute 𝑤 on all features
 Remove features with smallest 𝑤𝑖
 Precompute 𝑤 on reduced data
 Goto step 2 if stopping criteria doesn’t meet.

22. 22
Linear Discriminant Analysis

23. 23
● Linear Discriminant Analysis is a supervised learning algorithm for
classification.
● Similar to PCA, it can be used for dimensionality reduction, by
projecting the input data to a linear subspace consisting of the
directions which maximize the separation between classes.
● It is a linear transformation technique.
● It can be used as a pre-processing stage for pattern-classification.
● The purpose of LDA is to lower the dimension space with a good
separability between the classes.
● It assumes that the features are normally distributed.
Linear Discriminant Analysis

24. 24
Objective of LDA

25. 25
● Fisher’s LDA aims to maximise
equation (1), maximize the distance
between means and minimize the
variance within classes
● Equation-1 can be rewritten with
two new terms:
○ Between class matrix (SB)
○ Within class matrix (SW)
Here, W is a unit vector
onto which the data points
are to be projected.
Objective of LDA

26. 26
● Upon differentiating the equation
(2) w.r.t W and equating with 0, we
get a generalized eigenvalue-
eigenvector problem
○ SBW = vSwW
○ Sw
-1SBW = vW
■ Where v = eigenvalue
■ W = eigenvector
Objective of LDA

27. 27
LDA
Matrix
● SB represents how precisely the data is
scattered across the classes
● Goal is to maximize SB. i.e. the distance
between the two classes should be
higher
Between
Class
Matrix(SB)
Step:2
● SW captures how precisely the data is
scattered within the class
● Goal is to minimize SW. i.e. the distance
between the elements of the class
should be minimum
Within
Class
Matrix(SW)
LDA Matrix

28. 28
Linear Discriminant Analysis - Procedure

29. 29
Thank You!