# 5. Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis

The seminar series will focus on the mathematical background needed for machine learning. The first set of the seminars will be on "Linear Algebra for Machine Learning". Here are the slides of the fifth part which is discussing singular value decomposition and principal component analysis. Here are the slides of the first part which was discussing linear systems: https://www.slideshare.net/CeniBabaogluPhDinMat/linear-algebra-for-machine-learning-linear-systems/1 Here are the slides of the second part which was discussing basis and dimension: https://www.slideshare.net/CeniBabaogluPhDinMat/2-linear-algebra-for-machine-learning-basis-and-dimension Here are the slides of the third part which is discussing factorization and linear transformations. https://www.slideshare.net/CeniBabaogluPhDinMat/3-linear-algebra-for-machine-learning-factorization-and-linear-transformations-130813437 Here are the slides of the fourth part which is discussing eigenvalues and eigenvectors. https://www.slideshare.net/CeniBabaogluPhDinMat/4-linear-algebra-for-machine-learning-eigenvalues-eigenvectors-and-diagonalization

1 of 24

## Recommended

Singular Value Decompostion (SVD)
Singular Value Decompostion (SVD)Isaac Yowetu

3. Linear Algebra for Machine Learning: Factorization and Linear Transformations
3. Linear Algebra for Machine Learning: Factorization and Linear TransformationsCeni Babaoglu, PhD

1. Linear Algebra for Machine Learning: Linear Systems
1. Linear Algebra for Machine Learning: Linear SystemsCeni Babaoglu, PhD

An overview of gradient descent optimization algorithms
An overview of gradient descent optimization algorithms Hakky St

## What's hot

Spectral clustering
Spectral clusteringSOYEON KIM

Introduction to Statistical Machine Learning
Introduction to Statistical Machine Learningmahutte

Kernels and Support Vector Machines
Kernels and Support Vector MachinesEdgar Marca

Neural Networks: Radial Bases Functions (RBF)
Neural Networks: Radial Bases Functions (RBF)Mostafa G. M. Mostafa

Minimum spanning tree
Minimum spanning treeAmit Rathi

Classification Based Machine Learning Algorithms
Classification Based Machine Learning AlgorithmsMd. Main Uddin Rony

Maths-->>Eigenvalues and eigenvectors
Maths-->>Eigenvalues and eigenvectorsJaydev Kishnani

Lecture 18: Gaussian Mixture Models and Expectation Maximization
Lecture 18: Gaussian Mixture Models and Expectation Maximizationbutest

Fuzzy set and its application
Fuzzy set and its applicationKalaivananRaja

Support Vector Machines
Support Vector Machinesnextlib

K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...Simplilearn

Eigenvectors & Eigenvalues: The Road to Diagonalisation
Eigenvectors & Eigenvalues: The Road to DiagonalisationChristopher Gratton

Unsupervised learning clustering

Partial differential equations
Partial differential equationsaman1894

Support vector machine
Support vector machineRishabh Gupta

### What's hot(20)

Spectral clustering
Spectral clustering

Introduction to Statistical Machine Learning
Introduction to Statistical Machine Learning

K Nearest Neighbor Algorithm
K Nearest Neighbor Algorithm

Kernels and Support Vector Machines
Kernels and Support Vector Machines

Neural Networks: Radial Bases Functions (RBF)
Neural Networks: Radial Bases Functions (RBF)

Minimum spanning tree
Minimum spanning tree

Classification Based Machine Learning Algorithms
Classification Based Machine Learning Algorithms

Maths-->>Eigenvalues and eigenvectors
Maths-->>Eigenvalues and eigenvectors

Lecture 18: Gaussian Mixture Models and Expectation Maximization
Lecture 18: Gaussian Mixture Models and Expectation Maximization

Fuzzy set and its application
Fuzzy set and its application

Support Vector Machines
Support Vector Machines

K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...

Vector space
Vector space

Eigenvectors & Eigenvalues: The Road to Diagonalisation
Eigenvectors & Eigenvalues: The Road to Diagonalisation

Unsupervised learning clustering
Unsupervised learning clustering

Partial differential equations
Partial differential equations

lattice
lattice

Maths
Maths

Support vector machine
Support vector machine

## Similar to 5. Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis

Tutorial on EM algorithm – Part 1
Tutorial on EM algorithm – Part 1Loc Nguyen

4. Linear Algebra for Machine Learning: Eigenvalues, Eigenvectors and Diagona...
4. Linear Algebra for Machine Learning: Eigenvalues, Eigenvectors and Diagona...Ceni Babaoglu, PhD

MODULE_05-Matrix Decomposition.pptx
MODULE_05-Matrix Decomposition.pptxAlokSingh205089

Numerical Solutions of Burgers' Equation Project Report
Numerical Solutions of Burgers' Equation Project ReportShikhar Agarwal

A Parallel Eigensolver Using Contour Integration For Generalized Eigenvalue P...
A Parallel Eigensolver Using Contour Integration For Generalized Eigenvalue P...James Heller

2016 - Nonlinear Eigenvalue Problems And Contour Integrals
2016 - Nonlinear Eigenvalue Problems And Contour IntegralsDereck Downing

A Differential Approach To Solve The Inverse Eigenvalue Problem Derived From ...
A Differential Approach To Solve The Inverse Eigenvalue Problem Derived From ...Scott Faria

Cs6402 design and analysis of algorithms may june 2016 answer key
Cs6402 design and analysis of algorithms may june 2016 answer keyappasami

2. Linear Algebra for Machine Learning: Basis and Dimension
2. Linear Algebra for Machine Learning: Basis and DimensionCeni Babaoglu, PhD

An Iterative Adaptive Finite Element Method For Elliptic Eigenvalue Problems
An Iterative Adaptive Finite Element Method For Elliptic Eigenvalue ProblemsKaren Benoit

Dmitrii Tihonkih - The Iterative Closest Points Algorithm and Affine Transfo...
Dmitrii Tihonkih - The Iterative Closest Points Algorithm and Affine Transfo...AIST

The Generalized Difference Operator of the 퐧 퐭퐡 Kind
The Generalized Difference Operator of the 퐧 퐭퐡 KindDr. Amarjeet Singh

Algebraic Structure 1. Intro.pdf
Algebraic Structure 1. Intro.pdfMinhNguynQuang44

Solution to linear equhgations
Solution to linear equhgationsRobin Singh

Generalized fractional-order hybrid of block-pulse functions and Bernoulli po...
Generalized fractional-order hybrid of block-pulse functions and Bernoulli po...OctavianPostavaru

Count-Distinct Problem
Count-Distinct ProblemKai Zhang

### Similar to 5. Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis(20)

Tutorial on EM algorithm – Part 1
Tutorial on EM algorithm – Part 1

4. Linear Algebra for Machine Learning: Eigenvalues, Eigenvectors and Diagona...
4. Linear Algebra for Machine Learning: Eigenvalues, Eigenvectors and Diagona...

MODULE_05-Matrix Decomposition.pptx
MODULE_05-Matrix Decomposition.pptx

Numerical Solutions of Burgers' Equation Project Report
Numerical Solutions of Burgers' Equation Project Report

A Parallel Eigensolver Using Contour Integration For Generalized Eigenvalue P...
A Parallel Eigensolver Using Contour Integration For Generalized Eigenvalue P...

2016 - Nonlinear Eigenvalue Problems And Contour Integrals
2016 - Nonlinear Eigenvalue Problems And Contour Integrals

A Differential Approach To Solve The Inverse Eigenvalue Problem Derived From ...
A Differential Approach To Solve The Inverse Eigenvalue Problem Derived From ...

Cs6402 design and analysis of algorithms may june 2016 answer key
Cs6402 design and analysis of algorithms may june 2016 answer key

2. Linear Algebra for Machine Learning: Basis and Dimension
2. Linear Algebra for Machine Learning: Basis and Dimension

Brute force
Brute force

document(1).pdf
document(1).pdf

An Iterative Adaptive Finite Element Method For Elliptic Eigenvalue Problems
An Iterative Adaptive Finite Element Method For Elliptic Eigenvalue Problems

Dmitrii Tihonkih - The Iterative Closest Points Algorithm and Affine Transfo...
Dmitrii Tihonkih - The Iterative Closest Points Algorithm and Affine Transfo...

The Generalized Difference Operator of the 퐧 퐭퐡 Kind
The Generalized Difference Operator of the 퐧 퐭퐡 Kind

D026017036
D026017036

Algebraic Structure 1. Intro.pdf
Algebraic Structure 1. Intro.pdf

Solution to linear equhgations
Solution to linear equhgations

Generalized fractional-order hybrid of block-pulse functions and Bernoulli po...
Generalized fractional-order hybrid of block-pulse functions and Bernoulli po...

Count-Distinct Problem
Count-Distinct Problem

Creative, Technical, and Academic WritingMYDA ANGELICA SUAN

A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdf
A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdfDr.M.Geethavani

Organic Synthesis and Estimation of Functional Groups
Organic Synthesis and Estimation of Functional GroupsDr.M.Geethavani

Odontogenesis and its related anomiles.pptx
Odontogenesis and its related anomiles.pptxMennat Allah Alkaram

Plant Genetic Resources, Germplasm, gene pool - Copy.pptx
Plant Genetic Resources, Germplasm, gene pool - Copy.pptxAKSHAYMAGAR17

BTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaFaheema Hasan

BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptx
BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptxProf. Kanchan Kumari

Data Modeling - Entity Relationship Diagrams-1.pdf
Data Modeling - Entity Relationship Diagrams-1.pdfChristalin Nelson

GIÁO ÁN TIẾNG ANH GLOBAL SUCCESS LỚP 11 (CẢ NĂM) THEO CÔNG VĂN 5512 (2 CỘT) N...
GIÁO ÁN TIẾNG ANH GLOBAL SUCCESS LỚP 11 (CẢ NĂM) THEO CÔNG VĂN 5512 (2 CỘT) N...Nguyen Thanh Tu Collection

Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...
Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...AKSHAYMAGAR17

VPEC BROUCHER FOR ALL COURSES UPDATED FEB 2024
VPEC BROUCHER FOR ALL COURSES UPDATED FEB 2024avesmalik2

Digital Footprints to Career Pathways - Building a Strong Professional Online...
Digital Footprints to Career Pathways - Building a Strong Professional Online...Sue Beckingham

BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...Nguyen Thanh Tu Collection

Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdfSUMIT TIWARI

Group_Presentation_Gun_Island_Amitav_Ghosh.pptx
Group_Presentation_Gun_Island_Amitav_Ghosh.pptxPooja Bhuva

spring_bee_bot_creations_erd primary.pdf
spring_bee_bot_creations_erd primary.pdfKonstantina Koutsodimou

DISCOURSE: TEXT AS CONNECTED DISCOURSE
DISCOURSE: TEXT AS CONNECTED DISCOURSEMYDA ANGELICA SUAN

Overview of Databases and Data Modelling-2.pdf
Overview of Databases and Data Modelling-2.pdfChristalin Nelson

2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptxMaryPotorti1

skeletal system complete details with joints and its types
skeletal system complete details with joints and its typesMinaxi patil. CATALLYST

A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdf
A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdf

Organic Synthesis and Estimation of Functional Groups
Organic Synthesis and Estimation of Functional Groups

Odontogenesis and its related anomiles.pptx
Odontogenesis and its related anomiles.pptx

Plant Genetic Resources, Germplasm, gene pool - Copy.pptx
Plant Genetic Resources, Germplasm, gene pool - Copy.pptx

BTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic Leukemia

BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptx
BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptx

Data Modeling - Entity Relationship Diagrams-1.pdf
Data Modeling - Entity Relationship Diagrams-1.pdf

GIÁO ÁN TIẾNG ANH GLOBAL SUCCESS LỚP 11 (CẢ NĂM) THEO CÔNG VĂN 5512 (2 CỘT) N...
GIÁO ÁN TIẾNG ANH GLOBAL SUCCESS LỚP 11 (CẢ NĂM) THEO CÔNG VĂN 5512 (2 CỘT) N...

Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...
Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...

VPEC BROUCHER FOR ALL COURSES UPDATED FEB 2024
VPEC BROUCHER FOR ALL COURSES UPDATED FEB 2024

Digital Footprints to Career Pathways - Building a Strong Professional Online...
Digital Footprints to Career Pathways - Building a Strong Professional Online...

BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...

Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf

Group_Presentation_Gun_Island_Amitav_Ghosh.pptx
Group_Presentation_Gun_Island_Amitav_Ghosh.pptx

spring_bee_bot_creations_erd primary.pdf
spring_bee_bot_creations_erd primary.pdf

DISCOURSE: TEXT AS CONNECTED DISCOURSE
DISCOURSE: TEXT AS CONNECTED DISCOURSE

Overview of Databases and Data Modelling-2.pdf
Overview of Databases and Data Modelling-2.pdf

2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx

skeletal system complete details with joints and its types
skeletal system complete details with joints and its types

### 5. Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis

• 1. Seminar Series on Linear Algebra for Machine Learning Part 5: Singular Value Decomposition and Principal Component Analysis Dr. Ceni Babaoglu Data Science Laboratory Ryerson University cenibabaoglu.com Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 2. Overview 1 Spectral Decomposition 2 Singular Value Decomposition 3 Principal Component Analysis 4 References Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 3. Spectral Decomposition An n × n symmetric matrix A can be expressed as the matrix product A = PDPT where D is a diagonal matrix and P is an orthogonal matrix. The diagonal entries of D are the eigenvalues of A, λ1, λ2, · · · , λn, and the columns of P are associated orthonormal eigenvectors x1, x2, · · · , xn. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 4. Spectral Decomposition The expression A = PDPT is called the spectral decomposition of A. We can write it as A = x1 x2 · · · xn         λ1 0 · · · · · · 0 0 λ2 0 · · · 0 ... 0 ... ... 0 ... ... ... ... 0 0 0 · · · 0 λn              xT 1 xT 2 ... xT n      Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 5. Spectral Decomposition The matrix product DPT gives DPT =         λ1 0 · · · · · · 0 0 λ2 0 · · · 0 ... 0 ... ... 0 ... ... ... ... 0 0 0 · · · 0 λn              xT 1 xT 2 ... xT n      =      λ1xT 1 λ2xT 2 ... λnxT n      A = x1 x2 · · · xn      λ1xT 1 λ2xT 2 ... λnxT n      Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 6. Spectral Decomposition We can express A as a linear combination of the matrices xj xT j , and the coeﬃcients are the eigenvalues of A, A = n j=1 λj xj xT j = λ1x1xT 1 + λ2x2xT 2 + · · · + λnxnxT n . Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 7. Singular Value Decomposition Singular Value Decomposition is based on a theorem from linear algebra which says the following: a rectangular matrix A can be broken down into the product of three matrices: an orthogonal matrix U, a diagonal matrix S, the transpose of an orthogonal matrix V . Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 8. Singular Value Decomposition Let A be an m × n real matrix. Then there exist orthogonal matrices U of size m × m and V of size n × n such that A = USV T where S is an m × n matrix with nondiagonal entries all zero and s11 ≥ s12 ≥ · · · ≥ spp ≥ 0, p = min{m, n}. the diagonal entries of S are called the singular values of A, the columns of U are called the left singular vectors of A, the columns of V are called the right singular vectors of A, the factorization USV T is called the singular value decomposition of A. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 9. Singular Value Decomposition A = USV T , UT U = I, V T V = I, the columns of U are orthonormal eigenvectors of AAT , the columns of V are orthonormal eigenvectors of AT A, S is a diagonal matrix containing the square roots of eigenvalues from U or V in descending order. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 10. Example Let’s ﬁnd the singular value decomposition of A= 3 1 1 −1 3 1 . In order to ﬁnd U, we start with AAT . AAT = 3 1 1 −1 3 1   3 −1 1 3 1 1   = 11 1 1 11 Eigenvalues: λ1 = 12 and λ2 = 10. Eigenvectors: u1 = 1 1 and u2 = 1 −1 . Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 11. Example Using Gram-Schmidt Process v1 = u1 w1 = v1 |v1| = 1/ √ 2 1/ √ 2 v2 = u2 − (u2, v1) (v1, v1) v1 v2 = 1 −1 − 0 1 1 = 1 −1 − [0, 0] = 1 −1 w2 = v2 |v2| = 1 √ 2 , −1 √ 2 U = 1/ √ 2 1/ √ 2 1/ √ 2 −1/ √ 2 Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 12. Example The calculation of V is similar. V is based on AT A, so we have AT A = 3 −1 1 3 1 1 3 1 1 −1 3 1 = 10 0 2 0 10 4 2 4 2 We ﬁnd the eigenvalues of AT A Eigenvalues: λ1 = 12, λ2 = 10 and λ3 = 0. Eigenvectors: u1 = 1 2 1 , u2 = 2 −1 0 and u3 = 1 2 −5 . Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 13. Example v1 = u1, w1 = v1 |v1| = 1 √ 6 , 2 √ 6 , 1 √ 6 v2 = u2 − (u2, v1) (v1, v1) v1 = [2, −1, 0] w2 = v2 |v2| = 2 √ 5 , −1 √ 5 , 0 Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 14. Example v3 = u3 − (u3, v1) (v1, v1) v1 − (u3, v2) (v2, v2) v2 = −2 3 , −4 3 , 10 3 w3 = v3 |v3| = 1 √ 30 , 2 √ 30 , −5 √ 30 V =    1√ 6 2√ 5 1√ 30 2√ 6 −1√ 5 2√ 30 1√ 6 0 −5√ 30    , V T =    1√ 6 2√ 6 1√ 6 2√ 5 −1√ 5 0 1√ 30 2√ 30 −5√ 30    Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 15. Example A = USV T A = 1√ 2 1√ 2 1√ 2 −1√ 2 √ 12 0 0 0 √ 10 0   1√ 6 2√ 6 1√ 6 2√ 5 −1√ 5 0 1√ 30 2√ 30 −5√ 30   = 3 1 1 −1 3 1 Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 16. Principal Component Analysis Let λ1, λ2, · · · , λn be the eigenvalues of A and x1, x2, · · · , xn be a set of associated orthonormal eigenvectors. Then the spectral decomposition of A is given by A = λ1x1xT 1 + λ2x2xT 2 + · · · + λnxnxT n . If A is a real n × n matrix with real eigenvalues λ1, λ2, · · · , λn, then an eigenvalue of largest magnitude is called a dominant eigenvalue of A. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 17. Principal Component Analysis Let X be the multivariate data matrix X =         x11 x12 · · · x1k · · · x1p x21 x22 · · · x2k · · · x2p ... ... ... ... xj1 xj2 · · · xjk · · · xjp ... ... ... ... xn1 xn2 · · · xnk · · · xnp         . The measure of association between the ith and kth variables in the multivariate data matrix is given by the sample covariance sik = 1 n n j=1 (xji − xi ) (xjk − xk) , i = 1, 2, . . . , p, k = 1, 2, . . . , p. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 18. Principal Component Analysis Let Sn be the p × p covariance matrix associated with the multivariate data matrix X. Sn =      s11 s12 · · · s1p s21 s22 · · · s2p ... ... ... ... sp1 sp2 · · · spp      Let the eigenvalues of Sn be λj , j = 1, 2, . . . , p, where λ1 ≥ λ2 ≥ · · · ≥ λp ≥ 0 and let the associated orthonormal eigenvectors be uj , j = 1, 2, . . . , p. The ith principal component yi is given by the linear combination of the columns of X, where the coeﬃcients are the entries of the eigenvector ui , yi = Xui Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 19. Principal Component Analysis The variance of yi is λi , The covariance of yi and yk, i = k, is zero. If some of the eigenvalues are repeated, then the choices of the associated eigenvectors are not unique; hence the principal components are not unique.     Proportion of the total variance due to the k th principal component     = λk λ1 + λ2 + · · · + λp , k = 1, 2, . . . , p Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 20. Example Let’s compute the ﬁrst and second principal components for the multivariate data matrix X given by X =         39 21 59 28 18 10 21 13 14 13 22 10         . We ﬁrst ﬁnd the sample means x1 ≈ 28.8 and x2 ≈ 15.8. Then we take the matrix of sample means as x = 28.8 15.8 Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 21. Example The variances are s11 ≈ 243.1 and s22 ≈ 43.1 while the covariances are s12 = s21 ≈ 97.8. We take the covariance matrix as Sn = 243.1 97.8 97.8 43.1 Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 22. Example Eigenvalues: λ1 = 282.9744 and λ2 = 3.2256, Eigenvectors: u1 = 0.9260 0.3775 and u2 = 0.3775 −0.9260 . We ﬁnd the ﬁrst principal component as y1 = 0.9260 col1(X) + 0.3775 col2(X). It follows that y1 accounts for the proportion λ1 λ1 + λ2 ( about 98.9%) of the total variance of X. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 23. Example We ﬁnd the second principal component as y2 = 0.3775 col1(X) − 0.9260 col2(X). It follows that y2 accounts for the proportion λ2 λ1 + λ2 ( about 0.011%) of the total variance of X. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
• 24. References Linear Algebra With Applications, 7th Edition by Steven J. Leon. Elementary Linear Algebra with Applications, 9th Edition by Bernard Kolman and David Hill. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
Current LanguageEnglish
Español
Portugues
Français
Deutsche