KNN

1. Program Studi Teknik Informatika
Fakultas Teknik – Universitas Surabaya
Dimensionality Reduction:
Principal Component Analysis
Week 12
1604C055 - Machine Learning

2. Dimensionality reduction
• Dimensionality reduction is a process to transform data from a high-
dimensional space into new data in a low-dimensional space such that
the new data still has some meaningful properties of the original data.
• A high-dimensional data in machine learning leads to:
– High computational demands
– Low generalization performance
– Poor error estimates
• Some techniques:
– Principal component analysis (PCA)
– Linear discriminant analysis (LDA)
– Deep Learning: Autoencoders

3. Principal component analysis (PCA)
• PCA is a statistical techniques used to reduce the dimensions of
data/variables/features without losing the intrinsic information
contained in the original data.
• PCA is categorized as unsupervised learning
• PCA works by transforming the original variables into new variables,
called principal components
• Principal components:
– Uncorrelated variables
– Ordered such that the first few principal components retain the most
variation in the original variables

4. Principal component analysis (PCA)
PC1
PC2

5. Principal component analysis (PCA)
• Transformation from 2D to 1D:
– Green: without PCA
– Blue: with PCA
• Transformation without PCA
causes the new data close to
each other.
• Transformation with PCA
increase the distance of each
data
PC

6. Reduce data from 2D to 1D
Andrew

7. Reduce data from 3D to 2D

8. PCA Algorithm
•

9. Covariance
•

10. Covariance

11. Covariance: example
No.
1 4 3
2 1 9
3 4 7
4 8 2
5 9 3
6 7 -2
7 5 4
8 3 4
9 3 2
10 9 -1

12. Covariance: example
No.
1 4 3
2 1 9
3 4 7
4 8 2
5 9 3
6 7 -2
7 5 4
8 3 4
9 3 2
10 9 -1

13. Eigenvalue and eigenvector
•

14. Eigenvalue and eigenvector: example
•

15. Eigenvalue and eigenvector: example
•

16. Eigenvalue and eigenvector: example
•

17. Eigenvalue and eigenvector of
covariance matrix
•

18. Eigenvalue and eigenvector of
covariance matrix
•

19. Eigenvalue and eigenvector of
covariance matrix: example
•

20. Transform to PC coordinate system
•

21. Transform to PC coordinate system:
example
No.
1 4 3 -0.72945009
2 1 9 -7.29463721
3 4 7 -3.86347132
4 8 2 2.53959559
5 9 3 2.37747538
6 7 -2 5.05223172
7 5 4 -0.8915703
8 3 4 -2.13434049
9 3 2 -0.56732988
10 9 -1 5.5114966

22. Transform to PC coordinate system:
example
•

23. Choosing the number of PCs
•

24. Scree plot
Find the "elbow" of the graph where
the eigenvalues seem to level off is
found .Components to the left of this
point should be retained as
significant
Elbow

25. PCA in Python with numpy

26. PCA in Python with numpy

27. PCA in Python with numpy

28. PCA in Python with
sklearn.decomposition.PCA

29. PCA in Python with
sklearn.decomposition.PCA

30. PCA in Python with
sklearn.decomposition.PCA
Elbow

31. PCA in Python with
sklearn.decomposition.PCA

32. Assignment
• Download dataset here:
https://drive.google.com/drive/folders/1fXfv0VECkys55fnlqxPEuiL3C
-3KyheV?usp=sharing
• This is digit mnist dataset which contain images of handwritten digits
(range from 0-4). The distribution of digit label:
– digit 0-3: 100 for each digit
– digit 4: 200
• Code in the next slide is provided to read dataset where the final
output is a matrix “original_data” (row is for the number of image
being read, 600 images, and column is for image features, which is
from image pixels = 784 pixels = 28 pixels × 28 pixels).

34. Assignment
• Perform PCA to reduce the dimension of dataset from 784 D to any
number of dimension that would give the optimal result. Save it to
matrix “reduced_data”.
• Choose the best classification algorithm that you think would give
the best result to predict the digit label.
• Perform classification for both “original_data” and “reduced_data”
using the same classification algorithm chosen before. Compare the
result for both of them.

35. Assignment
• You could perform any data pre-processing techniques to the
dataset before used to train the model such that the best model is
obtained.
• Before feeding to classifier, split the dataset into training and testing
data. Use StratifiedShuffleSplit from scikit-learn with n_splits=1 and
ratio of 70%:30% for training:testing data.
• Evaluate the model using accuracy and F1 Score (weighted).
• State your conclusion.