SlideShare a Scribd company logo

5. Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis

The seminar series will focus on the mathematical background needed for machine learning. The first set of the seminars will be on "Linear Algebra for Machine Learning". Here are the slides of the fifth part which is discussing singular value decomposition and principal component analysis. Here are the slides of the first part which was discussing linear systems: https://www.slideshare.net/CeniBabaogluPhDinMat/linear-algebra-for-machine-learning-linear-systems/1 Here are the slides of the second part which was discussing basis and dimension: https://www.slideshare.net/CeniBabaogluPhDinMat/2-linear-algebra-for-machine-learning-basis-and-dimension Here are the slides of the third part which is discussing factorization and linear transformations. https://www.slideshare.net/CeniBabaogluPhDinMat/3-linear-algebra-for-machine-learning-factorization-and-linear-transformations-130813437 Here are the slides of the fourth part which is discussing eigenvalues and eigenvectors. https://www.slideshare.net/CeniBabaogluPhDinMat/4-linear-algebra-for-machine-learning-eigenvalues-eigenvectors-and-diagonalization

1 of 24
Download to read offline
Seminar Series on
Linear Algebra for Machine Learning
Part 5: Singular Value Decomposition and
Principal Component Analysis
Dr. Ceni Babaoglu
Data Science Laboratory
Ryerson University
cenibabaoglu.com
Dr. Ceni Babaoglu cenibabaoglu.com
Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
Overview
1 Spectral Decomposition
2 Singular Value Decomposition
3 Principal Component Analysis
4 References
Dr. Ceni Babaoglu cenibabaoglu.com
Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
Spectral Decomposition
An n × n symmetric matrix A can be expressed as the matrix
product
A = PDPT
where D is a diagonal matrix and P is an orthogonal matrix.
The diagonal entries of D are the eigenvalues of A,
λ1, λ2, · · · , λn, and the columns of P are associated
orthonormal eigenvectors x1, x2, · · · , xn.
Dr. Ceni Babaoglu cenibabaoglu.com
Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
Spectral Decomposition
The expression
A = PDPT
is called the spectral decomposition of A. We can write it as
A = x1 x2 · · · xn








λ1 0 · · · · · · 0
0 λ2 0 · · · 0
... 0
...
... 0
...
...
...
... 0
0 0 · · · 0 λn













xT
1
xT
2
...
xT
n





Dr. Ceni Babaoglu cenibabaoglu.com
Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
Spectral Decomposition
The matrix product DPT gives
DPT
=








λ1 0 · · · · · · 0
0 λ2 0 · · · 0
... 0
...
... 0
...
...
...
... 0
0 0 · · · 0 λn













xT
1
xT
2
...
xT
n





=





λ1xT
1
λ2xT
2
...
λnxT
n





A = x1 x2 · · · xn





λ1xT
1
λ2xT
2
...
λnxT
n





Dr. Ceni Babaoglu cenibabaoglu.com
Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
Spectral Decomposition
We can express A as a linear combination of the matrices
xj xT
j , and the coefficients are the eigenvalues of A,
A =
n
j=1
λj xj xT
j = λ1x1xT
1 + λ2x2xT
2 + · · · + λnxnxT
n .
Dr. Ceni Babaoglu cenibabaoglu.com
Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
Ad

Recommended

Singular Value Decompostion (SVD)
Singular Value Decompostion (SVD)Singular Value Decompostion (SVD)
Singular Value Decompostion (SVD)Isaac Yowetu
 
3. Linear Algebra for Machine Learning: Factorization and Linear Transformations
3. Linear Algebra for Machine Learning: Factorization and Linear Transformations3. Linear Algebra for Machine Learning: Factorization and Linear Transformations
3. Linear Algebra for Machine Learning: Factorization and Linear TransformationsCeni Babaoglu, PhD
 
1. Linear Algebra for Machine Learning: Linear Systems
1. Linear Algebra for Machine Learning: Linear Systems1. Linear Algebra for Machine Learning: Linear Systems
1. Linear Algebra for Machine Learning: Linear SystemsCeni Babaoglu, PhD
 
An overview of gradient descent optimization algorithms
An overview of gradient descent optimization algorithms An overview of gradient descent optimization algorithms
An overview of gradient descent optimization algorithms Hakky St
 

More Related Content

What's hot

Spectral clustering
Spectral clusteringSpectral clustering
Spectral clusteringSOYEON KIM
 
Introduction to Statistical Machine Learning
Introduction to Statistical Machine LearningIntroduction to Statistical Machine Learning
Introduction to Statistical Machine Learningmahutte
 
Kernels and Support Vector Machines
Kernels and Support Vector  MachinesKernels and Support Vector  Machines
Kernels and Support Vector MachinesEdgar Marca
 
Neural Networks: Radial Bases Functions (RBF)
Neural Networks: Radial Bases Functions (RBF)Neural Networks: Radial Bases Functions (RBF)
Neural Networks: Radial Bases Functions (RBF)Mostafa G. M. Mostafa
 
Minimum spanning tree
Minimum spanning treeMinimum spanning tree
Minimum spanning treeAmit Rathi
 
Classification Based Machine Learning Algorithms
Classification Based Machine Learning AlgorithmsClassification Based Machine Learning Algorithms
Classification Based Machine Learning AlgorithmsMd. Main Uddin Rony
 
Maths-->>Eigenvalues and eigenvectors
Maths-->>Eigenvalues and eigenvectorsMaths-->>Eigenvalues and eigenvectors
Maths-->>Eigenvalues and eigenvectorsJaydev Kishnani
 
Lecture 18: Gaussian Mixture Models and Expectation Maximization
Lecture 18: Gaussian Mixture Models and Expectation MaximizationLecture 18: Gaussian Mixture Models and Expectation Maximization
Lecture 18: Gaussian Mixture Models and Expectation Maximizationbutest
 
Fuzzy set and its application
Fuzzy set and its applicationFuzzy set and its application
Fuzzy set and its applicationKalaivananRaja
 
Support Vector Machines
Support Vector MachinesSupport Vector Machines
Support Vector Machinesnextlib
 
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...Simplilearn
 
Eigenvectors & Eigenvalues: The Road to Diagonalisation
Eigenvectors & Eigenvalues: The Road to DiagonalisationEigenvectors & Eigenvalues: The Road to Diagonalisation
Eigenvectors & Eigenvalues: The Road to DiagonalisationChristopher Gratton
 
Unsupervised learning clustering
Unsupervised learning clusteringUnsupervised learning clustering
Unsupervised learning clusteringArshad Farhad
 
Gradient descent method
Gradient descent methodGradient descent method
Gradient descent methodSanghyuk Chun
 
Partial differential equations
Partial differential equationsPartial differential equations
Partial differential equationsaman1894
 
Support vector machine
Support vector machineSupport vector machine
Support vector machineRishabh Gupta
 

What's hot (20)

Spectral clustering
Spectral clusteringSpectral clustering
Spectral clustering
 
Introduction to Statistical Machine Learning
Introduction to Statistical Machine LearningIntroduction to Statistical Machine Learning
Introduction to Statistical Machine Learning
 
K Nearest Neighbor Algorithm
K Nearest Neighbor AlgorithmK Nearest Neighbor Algorithm
K Nearest Neighbor Algorithm
 
Kernels and Support Vector Machines
Kernels and Support Vector  MachinesKernels and Support Vector  Machines
Kernels and Support Vector Machines
 
Neural Networks: Radial Bases Functions (RBF)
Neural Networks: Radial Bases Functions (RBF)Neural Networks: Radial Bases Functions (RBF)
Neural Networks: Radial Bases Functions (RBF)
 
Minimum spanning tree
Minimum spanning treeMinimum spanning tree
Minimum spanning tree
 
Classification Based Machine Learning Algorithms
Classification Based Machine Learning AlgorithmsClassification Based Machine Learning Algorithms
Classification Based Machine Learning Algorithms
 
Maths-->>Eigenvalues and eigenvectors
Maths-->>Eigenvalues and eigenvectorsMaths-->>Eigenvalues and eigenvectors
Maths-->>Eigenvalues and eigenvectors
 
Lecture 18: Gaussian Mixture Models and Expectation Maximization
Lecture 18: Gaussian Mixture Models and Expectation MaximizationLecture 18: Gaussian Mixture Models and Expectation Maximization
Lecture 18: Gaussian Mixture Models and Expectation Maximization
 
Fuzzy set and its application
Fuzzy set and its applicationFuzzy set and its application
Fuzzy set and its application
 
Support Vector Machines
Support Vector MachinesSupport Vector Machines
Support Vector Machines
 
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...
K Means Clustering Algorithm | K Means Clustering Example | Machine Learning ...
 
Vector space
Vector spaceVector space
Vector space
 
Eigenvectors & Eigenvalues: The Road to Diagonalisation
Eigenvectors & Eigenvalues: The Road to DiagonalisationEigenvectors & Eigenvalues: The Road to Diagonalisation
Eigenvectors & Eigenvalues: The Road to Diagonalisation
 
Unsupervised learning clustering
Unsupervised learning clusteringUnsupervised learning clustering
Unsupervised learning clustering
 
Gradient descent method
Gradient descent methodGradient descent method
Gradient descent method
 
Partial differential equations
Partial differential equationsPartial differential equations
Partial differential equations
 
lattice
 lattice lattice
lattice
 
Maths
MathsMaths
Maths
 
Support vector machine
Support vector machineSupport vector machine
Support vector machine
 

Similar to 5. Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis

Tutorial on EM algorithm – Part 1
Tutorial on EM algorithm – Part 1Tutorial on EM algorithm – Part 1
Tutorial on EM algorithm – Part 1Loc Nguyen
 
4. Linear Algebra for Machine Learning: Eigenvalues, Eigenvectors and Diagona...
4. Linear Algebra for Machine Learning: Eigenvalues, Eigenvectors and Diagona...4. Linear Algebra for Machine Learning: Eigenvalues, Eigenvectors and Diagona...
4. Linear Algebra for Machine Learning: Eigenvalues, Eigenvectors and Diagona...Ceni Babaoglu, PhD
 
MODULE_05-Matrix Decomposition.pptx
MODULE_05-Matrix Decomposition.pptxMODULE_05-Matrix Decomposition.pptx
MODULE_05-Matrix Decomposition.pptxAlokSingh205089
 
Numerical Solutions of Burgers' Equation Project Report
Numerical Solutions of Burgers' Equation Project ReportNumerical Solutions of Burgers' Equation Project Report
Numerical Solutions of Burgers' Equation Project ReportShikhar Agarwal
 
A Parallel Eigensolver Using Contour Integration For Generalized Eigenvalue P...
A Parallel Eigensolver Using Contour Integration For Generalized Eigenvalue P...A Parallel Eigensolver Using Contour Integration For Generalized Eigenvalue P...
A Parallel Eigensolver Using Contour Integration For Generalized Eigenvalue P...James Heller
 
2016 - Nonlinear Eigenvalue Problems And Contour Integrals
2016 - Nonlinear Eigenvalue Problems And Contour Integrals2016 - Nonlinear Eigenvalue Problems And Contour Integrals
2016 - Nonlinear Eigenvalue Problems And Contour IntegralsDereck Downing
 
A Differential Approach To Solve The Inverse Eigenvalue Problem Derived From ...
A Differential Approach To Solve The Inverse Eigenvalue Problem Derived From ...A Differential Approach To Solve The Inverse Eigenvalue Problem Derived From ...
A Differential Approach To Solve The Inverse Eigenvalue Problem Derived From ...Scott Faria
 
Cs6402 design and analysis of algorithms may june 2016 answer key
Cs6402 design and analysis of algorithms may june 2016 answer keyCs6402 design and analysis of algorithms may june 2016 answer key
Cs6402 design and analysis of algorithms may june 2016 answer keyappasami
 
2. Linear Algebra for Machine Learning: Basis and Dimension
2. Linear Algebra for Machine Learning: Basis and Dimension2. Linear Algebra for Machine Learning: Basis and Dimension
2. Linear Algebra for Machine Learning: Basis and DimensionCeni Babaoglu, PhD
 
An Iterative Adaptive Finite Element Method For Elliptic Eigenvalue Problems
An Iterative Adaptive Finite Element Method For Elliptic Eigenvalue ProblemsAn Iterative Adaptive Finite Element Method For Elliptic Eigenvalue Problems
An Iterative Adaptive Finite Element Method For Elliptic Eigenvalue ProblemsKaren Benoit
 
Dmitrii Tihonkih - The Iterative Closest Points Algorithm and Affine Transfo...
Dmitrii Tihonkih - The Iterative Closest Points Algorithm and  Affine Transfo...Dmitrii Tihonkih - The Iterative Closest Points Algorithm and  Affine Transfo...
Dmitrii Tihonkih - The Iterative Closest Points Algorithm and Affine Transfo...AIST
 
The Generalized Difference Operator of the 퐧 퐭퐡 Kind
The Generalized Difference Operator of the 퐧 퐭퐡 KindThe Generalized Difference Operator of the 퐧 퐭퐡 Kind
The Generalized Difference Operator of the 퐧 퐭퐡 KindDr. Amarjeet Singh
 
Algebraic Structure 1. Intro.pdf
Algebraic Structure 1. Intro.pdfAlgebraic Structure 1. Intro.pdf
Algebraic Structure 1. Intro.pdfMinhNguynQuang44
 
Solution to linear equhgations
Solution to linear equhgationsSolution to linear equhgations
Solution to linear equhgationsRobin Singh
 
Generalized fractional-order hybrid of block-pulse functions and Bernoulli po...
Generalized fractional-order hybrid of block-pulse functions and Bernoulli po...Generalized fractional-order hybrid of block-pulse functions and Bernoulli po...
Generalized fractional-order hybrid of block-pulse functions and Bernoulli po...OctavianPostavaru
 
Count-Distinct Problem
Count-Distinct ProblemCount-Distinct Problem
Count-Distinct ProblemKai Zhang
 

Similar to 5. Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis (20)

Tutorial on EM algorithm – Part 1
Tutorial on EM algorithm – Part 1Tutorial on EM algorithm – Part 1
Tutorial on EM algorithm – Part 1
 
4. Linear Algebra for Machine Learning: Eigenvalues, Eigenvectors and Diagona...
4. Linear Algebra for Machine Learning: Eigenvalues, Eigenvectors and Diagona...4. Linear Algebra for Machine Learning: Eigenvalues, Eigenvectors and Diagona...
4. Linear Algebra for Machine Learning: Eigenvalues, Eigenvectors and Diagona...
 
MODULE_05-Matrix Decomposition.pptx
MODULE_05-Matrix Decomposition.pptxMODULE_05-Matrix Decomposition.pptx
MODULE_05-Matrix Decomposition.pptx
 
Numerical Solutions of Burgers' Equation Project Report
Numerical Solutions of Burgers' Equation Project ReportNumerical Solutions of Burgers' Equation Project Report
Numerical Solutions of Burgers' Equation Project Report
 
A Parallel Eigensolver Using Contour Integration For Generalized Eigenvalue P...
A Parallel Eigensolver Using Contour Integration For Generalized Eigenvalue P...A Parallel Eigensolver Using Contour Integration For Generalized Eigenvalue P...
A Parallel Eigensolver Using Contour Integration For Generalized Eigenvalue P...
 
2016 - Nonlinear Eigenvalue Problems And Contour Integrals
2016 - Nonlinear Eigenvalue Problems And Contour Integrals2016 - Nonlinear Eigenvalue Problems And Contour Integrals
2016 - Nonlinear Eigenvalue Problems And Contour Integrals
 
A Differential Approach To Solve The Inverse Eigenvalue Problem Derived From ...
A Differential Approach To Solve The Inverse Eigenvalue Problem Derived From ...A Differential Approach To Solve The Inverse Eigenvalue Problem Derived From ...
A Differential Approach To Solve The Inverse Eigenvalue Problem Derived From ...
 
Cs6402 design and analysis of algorithms may june 2016 answer key
Cs6402 design and analysis of algorithms may june 2016 answer keyCs6402 design and analysis of algorithms may june 2016 answer key
Cs6402 design and analysis of algorithms may june 2016 answer key
 
2. Linear Algebra for Machine Learning: Basis and Dimension
2. Linear Algebra for Machine Learning: Basis and Dimension2. Linear Algebra for Machine Learning: Basis and Dimension
2. Linear Algebra for Machine Learning: Basis and Dimension
 
Brute force
Brute forceBrute force
Brute force
 
document(1).pdf
document(1).pdfdocument(1).pdf
document(1).pdf
 
An Iterative Adaptive Finite Element Method For Elliptic Eigenvalue Problems
An Iterative Adaptive Finite Element Method For Elliptic Eigenvalue ProblemsAn Iterative Adaptive Finite Element Method For Elliptic Eigenvalue Problems
An Iterative Adaptive Finite Element Method For Elliptic Eigenvalue Problems
 
Dmitrii Tihonkih - The Iterative Closest Points Algorithm and Affine Transfo...
Dmitrii Tihonkih - The Iterative Closest Points Algorithm and  Affine Transfo...Dmitrii Tihonkih - The Iterative Closest Points Algorithm and  Affine Transfo...
Dmitrii Tihonkih - The Iterative Closest Points Algorithm and Affine Transfo...
 
The Generalized Difference Operator of the 퐧 퐭퐡 Kind
The Generalized Difference Operator of the 퐧 퐭퐡 KindThe Generalized Difference Operator of the 퐧 퐭퐡 Kind
The Generalized Difference Operator of the 퐧 퐭퐡 Kind
 
D026017036
D026017036D026017036
D026017036
 
Algebraic Structure 1. Intro.pdf
Algebraic Structure 1. Intro.pdfAlgebraic Structure 1. Intro.pdf
Algebraic Structure 1. Intro.pdf
 
Ada notes
Ada notesAda notes
Ada notes
 
Solution to linear equhgations
Solution to linear equhgationsSolution to linear equhgations
Solution to linear equhgations
 
Generalized fractional-order hybrid of block-pulse functions and Bernoulli po...
Generalized fractional-order hybrid of block-pulse functions and Bernoulli po...Generalized fractional-order hybrid of block-pulse functions and Bernoulli po...
Generalized fractional-order hybrid of block-pulse functions and Bernoulli po...
 
Count-Distinct Problem
Count-Distinct ProblemCount-Distinct Problem
Count-Distinct Problem
 

Recently uploaded

Creative, Technical, and Academic Writing
Creative, Technical, and Academic WritingCreative, Technical, and Academic Writing
Creative, Technical, and Academic WritingMYDA ANGELICA SUAN
 
A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdf
A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdfA LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdf
A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdfDr.M.Geethavani
 
Organic Synthesis and Estimation of Functional Groups
Organic Synthesis and Estimation of Functional GroupsOrganic Synthesis and Estimation of Functional Groups
Organic Synthesis and Estimation of Functional GroupsDr.M.Geethavani
 
Odontogenesis and its related anomiles.pptx
Odontogenesis and its related anomiles.pptxOdontogenesis and its related anomiles.pptx
Odontogenesis and its related anomiles.pptxMennat Allah Alkaram
 
Plant Genetic Resources, Germplasm, gene pool - Copy.pptx
Plant Genetic Resources, Germplasm, gene pool - Copy.pptxPlant Genetic Resources, Germplasm, gene pool - Copy.pptx
Plant Genetic Resources, Germplasm, gene pool - Copy.pptxAKSHAYMAGAR17
 
BTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaBTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaFaheema Hasan
 
BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptx
BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptxBBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptx
BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptxProf. Kanchan Kumari
 
Data Modeling - Entity Relationship Diagrams-1.pdf
Data Modeling - Entity Relationship Diagrams-1.pdfData Modeling - Entity Relationship Diagrams-1.pdf
Data Modeling - Entity Relationship Diagrams-1.pdfChristalin Nelson
 
GIÁO ÁN TIẾNG ANH GLOBAL SUCCESS LỚP 11 (CẢ NĂM) THEO CÔNG VĂN 5512 (2 CỘT) N...
GIÁO ÁN TIẾNG ANH GLOBAL SUCCESS LỚP 11 (CẢ NĂM) THEO CÔNG VĂN 5512 (2 CỘT) N...GIÁO ÁN TIẾNG ANH GLOBAL SUCCESS LỚP 11 (CẢ NĂM) THEO CÔNG VĂN 5512 (2 CỘT) N...
GIÁO ÁN TIẾNG ANH GLOBAL SUCCESS LỚP 11 (CẢ NĂM) THEO CÔNG VĂN 5512 (2 CỘT) N...Nguyen Thanh Tu Collection
 
Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...
Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...
Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...AKSHAYMAGAR17
 
VPEC BROUCHER FOR ALL COURSES UPDATED FEB 2024
VPEC BROUCHER FOR ALL COURSES UPDATED FEB 2024VPEC BROUCHER FOR ALL COURSES UPDATED FEB 2024
VPEC BROUCHER FOR ALL COURSES UPDATED FEB 2024avesmalik2
 
Digital Footprints to Career Pathways - Building a Strong Professional Online...
Digital Footprints to Career Pathways - Building a Strong Professional Online...Digital Footprints to Career Pathways - Building a Strong Professional Online...
Digital Footprints to Career Pathways - Building a Strong Professional Online...Sue Beckingham
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...Nguyen Thanh Tu Collection
 
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdfDiploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdfSUMIT TIWARI
 
Group_Presentation_Gun_Island_Amitav_Ghosh.pptx
Group_Presentation_Gun_Island_Amitav_Ghosh.pptxGroup_Presentation_Gun_Island_Amitav_Ghosh.pptx
Group_Presentation_Gun_Island_Amitav_Ghosh.pptxPooja Bhuva
 
spring_bee_bot_creations_erd primary.pdf
spring_bee_bot_creations_erd primary.pdfspring_bee_bot_creations_erd primary.pdf
spring_bee_bot_creations_erd primary.pdfKonstantina Koutsodimou
 
DISCOURSE: TEXT AS CONNECTED DISCOURSE
DISCOURSE:   TEXT AS CONNECTED DISCOURSEDISCOURSE:   TEXT AS CONNECTED DISCOURSE
DISCOURSE: TEXT AS CONNECTED DISCOURSEMYDA ANGELICA SUAN
 
Overview of Databases and Data Modelling-2.pdf
Overview of Databases and Data Modelling-2.pdfOverview of Databases and Data Modelling-2.pdf
Overview of Databases and Data Modelling-2.pdfChristalin Nelson
 
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptxMaryPotorti1
 
skeletal system complete details with joints and its types
skeletal system complete details with joints and its typesskeletal system complete details with joints and its types
skeletal system complete details with joints and its typesMinaxi patil. CATALLYST
 

Recently uploaded (20)

Creative, Technical, and Academic Writing
Creative, Technical, and Academic WritingCreative, Technical, and Academic Writing
Creative, Technical, and Academic Writing
 
A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdf
A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdfA LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdf
A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdf
 
Organic Synthesis and Estimation of Functional Groups
Organic Synthesis and Estimation of Functional GroupsOrganic Synthesis and Estimation of Functional Groups
Organic Synthesis and Estimation of Functional Groups
 
Odontogenesis and its related anomiles.pptx
Odontogenesis and its related anomiles.pptxOdontogenesis and its related anomiles.pptx
Odontogenesis and its related anomiles.pptx
 
Plant Genetic Resources, Germplasm, gene pool - Copy.pptx
Plant Genetic Resources, Germplasm, gene pool - Copy.pptxPlant Genetic Resources, Germplasm, gene pool - Copy.pptx
Plant Genetic Resources, Germplasm, gene pool - Copy.pptx
 
BTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaBTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic Leukemia
 
BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptx
BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptxBBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptx
BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptx
 
Data Modeling - Entity Relationship Diagrams-1.pdf
Data Modeling - Entity Relationship Diagrams-1.pdfData Modeling - Entity Relationship Diagrams-1.pdf
Data Modeling - Entity Relationship Diagrams-1.pdf
 
GIÁO ÁN TIẾNG ANH GLOBAL SUCCESS LỚP 11 (CẢ NĂM) THEO CÔNG VĂN 5512 (2 CỘT) N...
GIÁO ÁN TIẾNG ANH GLOBAL SUCCESS LỚP 11 (CẢ NĂM) THEO CÔNG VĂN 5512 (2 CỘT) N...GIÁO ÁN TIẾNG ANH GLOBAL SUCCESS LỚP 11 (CẢ NĂM) THEO CÔNG VĂN 5512 (2 CỘT) N...
GIÁO ÁN TIẾNG ANH GLOBAL SUCCESS LỚP 11 (CẢ NĂM) THEO CÔNG VĂN 5512 (2 CỘT) N...
 
Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...
Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...
Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...
 
VPEC BROUCHER FOR ALL COURSES UPDATED FEB 2024
VPEC BROUCHER FOR ALL COURSES UPDATED FEB 2024VPEC BROUCHER FOR ALL COURSES UPDATED FEB 2024
VPEC BROUCHER FOR ALL COURSES UPDATED FEB 2024
 
Digital Footprints to Career Pathways - Building a Strong Professional Online...
Digital Footprints to Career Pathways - Building a Strong Professional Online...Digital Footprints to Career Pathways - Building a Strong Professional Online...
Digital Footprints to Career Pathways - Building a Strong Professional Online...
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
 
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdfDiploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
 
Group_Presentation_Gun_Island_Amitav_Ghosh.pptx
Group_Presentation_Gun_Island_Amitav_Ghosh.pptxGroup_Presentation_Gun_Island_Amitav_Ghosh.pptx
Group_Presentation_Gun_Island_Amitav_Ghosh.pptx
 
spring_bee_bot_creations_erd primary.pdf
spring_bee_bot_creations_erd primary.pdfspring_bee_bot_creations_erd primary.pdf
spring_bee_bot_creations_erd primary.pdf
 
DISCOURSE: TEXT AS CONNECTED DISCOURSE
DISCOURSE:   TEXT AS CONNECTED DISCOURSEDISCOURSE:   TEXT AS CONNECTED DISCOURSE
DISCOURSE: TEXT AS CONNECTED DISCOURSE
 
Overview of Databases and Data Modelling-2.pdf
Overview of Databases and Data Modelling-2.pdfOverview of Databases and Data Modelling-2.pdf
Overview of Databases and Data Modelling-2.pdf
 
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
 
skeletal system complete details with joints and its types
skeletal system complete details with joints and its typesskeletal system complete details with joints and its types
skeletal system complete details with joints and its types
 

5. Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis

  • 1. Seminar Series on Linear Algebra for Machine Learning Part 5: Singular Value Decomposition and Principal Component Analysis Dr. Ceni Babaoglu Data Science Laboratory Ryerson University cenibabaoglu.com Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 2. Overview 1 Spectral Decomposition 2 Singular Value Decomposition 3 Principal Component Analysis 4 References Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 3. Spectral Decomposition An n × n symmetric matrix A can be expressed as the matrix product A = PDPT where D is a diagonal matrix and P is an orthogonal matrix. The diagonal entries of D are the eigenvalues of A, λ1, λ2, · · · , λn, and the columns of P are associated orthonormal eigenvectors x1, x2, · · · , xn. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 4. Spectral Decomposition The expression A = PDPT is called the spectral decomposition of A. We can write it as A = x1 x2 · · · xn         λ1 0 · · · · · · 0 0 λ2 0 · · · 0 ... 0 ... ... 0 ... ... ... ... 0 0 0 · · · 0 λn              xT 1 xT 2 ... xT n      Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 5. Spectral Decomposition The matrix product DPT gives DPT =         λ1 0 · · · · · · 0 0 λ2 0 · · · 0 ... 0 ... ... 0 ... ... ... ... 0 0 0 · · · 0 λn              xT 1 xT 2 ... xT n      =      λ1xT 1 λ2xT 2 ... λnxT n      A = x1 x2 · · · xn      λ1xT 1 λ2xT 2 ... λnxT n      Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 6. Spectral Decomposition We can express A as a linear combination of the matrices xj xT j , and the coefficients are the eigenvalues of A, A = n j=1 λj xj xT j = λ1x1xT 1 + λ2x2xT 2 + · · · + λnxnxT n . Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 7. Singular Value Decomposition Singular Value Decomposition is based on a theorem from linear algebra which says the following: a rectangular matrix A can be broken down into the product of three matrices: an orthogonal matrix U, a diagonal matrix S, the transpose of an orthogonal matrix V . Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 8. Singular Value Decomposition Let A be an m × n real matrix. Then there exist orthogonal matrices U of size m × m and V of size n × n such that A = USV T where S is an m × n matrix with nondiagonal entries all zero and s11 ≥ s12 ≥ · · · ≥ spp ≥ 0, p = min{m, n}. the diagonal entries of S are called the singular values of A, the columns of U are called the left singular vectors of A, the columns of V are called the right singular vectors of A, the factorization USV T is called the singular value decomposition of A. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 9. Singular Value Decomposition A = USV T , UT U = I, V T V = I, the columns of U are orthonormal eigenvectors of AAT , the columns of V are orthonormal eigenvectors of AT A, S is a diagonal matrix containing the square roots of eigenvalues from U or V in descending order. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 10. Example Let’s find the singular value decomposition of A= 3 1 1 −1 3 1 . In order to find U, we start with AAT . AAT = 3 1 1 −1 3 1   3 −1 1 3 1 1   = 11 1 1 11 Eigenvalues: λ1 = 12 and λ2 = 10. Eigenvectors: u1 = 1 1 and u2 = 1 −1 . Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 11. Example Using Gram-Schmidt Process v1 = u1 w1 = v1 |v1| = 1/ √ 2 1/ √ 2 v2 = u2 − (u2, v1) (v1, v1) v1 v2 = 1 −1 − 0 1 1 = 1 −1 − [0, 0] = 1 −1 w2 = v2 |v2| = 1 √ 2 , −1 √ 2 U = 1/ √ 2 1/ √ 2 1/ √ 2 −1/ √ 2 Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 12. Example The calculation of V is similar. V is based on AT A, so we have AT A = 3 −1 1 3 1 1 3 1 1 −1 3 1 = 10 0 2 0 10 4 2 4 2 We find the eigenvalues of AT A Eigenvalues: λ1 = 12, λ2 = 10 and λ3 = 0. Eigenvectors: u1 = 1 2 1 , u2 = 2 −1 0 and u3 = 1 2 −5 . Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 13. Example v1 = u1, w1 = v1 |v1| = 1 √ 6 , 2 √ 6 , 1 √ 6 v2 = u2 − (u2, v1) (v1, v1) v1 = [2, −1, 0] w2 = v2 |v2| = 2 √ 5 , −1 √ 5 , 0 Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 14. Example v3 = u3 − (u3, v1) (v1, v1) v1 − (u3, v2) (v2, v2) v2 = −2 3 , −4 3 , 10 3 w3 = v3 |v3| = 1 √ 30 , 2 √ 30 , −5 √ 30 V =    1√ 6 2√ 5 1√ 30 2√ 6 −1√ 5 2√ 30 1√ 6 0 −5√ 30    , V T =    1√ 6 2√ 6 1√ 6 2√ 5 −1√ 5 0 1√ 30 2√ 30 −5√ 30    Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 15. Example A = USV T A = 1√ 2 1√ 2 1√ 2 −1√ 2 √ 12 0 0 0 √ 10 0   1√ 6 2√ 6 1√ 6 2√ 5 −1√ 5 0 1√ 30 2√ 30 −5√ 30   = 3 1 1 −1 3 1 Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 16. Principal Component Analysis Let λ1, λ2, · · · , λn be the eigenvalues of A and x1, x2, · · · , xn be a set of associated orthonormal eigenvectors. Then the spectral decomposition of A is given by A = λ1x1xT 1 + λ2x2xT 2 + · · · + λnxnxT n . If A is a real n × n matrix with real eigenvalues λ1, λ2, · · · , λn, then an eigenvalue of largest magnitude is called a dominant eigenvalue of A. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 17. Principal Component Analysis Let X be the multivariate data matrix X =         x11 x12 · · · x1k · · · x1p x21 x22 · · · x2k · · · x2p ... ... ... ... xj1 xj2 · · · xjk · · · xjp ... ... ... ... xn1 xn2 · · · xnk · · · xnp         . The measure of association between the ith and kth variables in the multivariate data matrix is given by the sample covariance sik = 1 n n j=1 (xji − xi ) (xjk − xk) , i = 1, 2, . . . , p, k = 1, 2, . . . , p. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 18. Principal Component Analysis Let Sn be the p × p covariance matrix associated with the multivariate data matrix X. Sn =      s11 s12 · · · s1p s21 s22 · · · s2p ... ... ... ... sp1 sp2 · · · spp      Let the eigenvalues of Sn be λj , j = 1, 2, . . . , p, where λ1 ≥ λ2 ≥ · · · ≥ λp ≥ 0 and let the associated orthonormal eigenvectors be uj , j = 1, 2, . . . , p. The ith principal component yi is given by the linear combination of the columns of X, where the coefficients are the entries of the eigenvector ui , yi = Xui Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 19. Principal Component Analysis The variance of yi is λi , The covariance of yi and yk, i = k, is zero. If some of the eigenvalues are repeated, then the choices of the associated eigenvectors are not unique; hence the principal components are not unique.     Proportion of the total variance due to the k th principal component     = λk λ1 + λ2 + · · · + λp , k = 1, 2, . . . , p Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 20. Example Let’s compute the first and second principal components for the multivariate data matrix X given by X =         39 21 59 28 18 10 21 13 14 13 22 10         . We first find the sample means x1 ≈ 28.8 and x2 ≈ 15.8. Then we take the matrix of sample means as x = 28.8 15.8 Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 21. Example The variances are s11 ≈ 243.1 and s22 ≈ 43.1 while the covariances are s12 = s21 ≈ 97.8. We take the covariance matrix as Sn = 243.1 97.8 97.8 43.1 Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 22. Example Eigenvalues: λ1 = 282.9744 and λ2 = 3.2256, Eigenvectors: u1 = 0.9260 0.3775 and u2 = 0.3775 −0.9260 . We find the first principal component as y1 = 0.9260 col1(X) + 0.3775 col2(X). It follows that y1 accounts for the proportion λ1 λ1 + λ2 ( about 98.9%) of the total variance of X. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 23. Example We find the second principal component as y2 = 0.3775 col1(X) − 0.9260 col2(X). It follows that y2 accounts for the proportion λ2 λ1 + λ2 ( about 0.011%) of the total variance of X. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis
  • 24. References Linear Algebra With Applications, 7th Edition by Steven J. Leon. Elementary Linear Algebra with Applications, 9th Edition by Bernard Kolman and David Hill. Dr. Ceni Babaoglu cenibabaoglu.com Linear Algebra for Machine Learning: Singular Value Decomposition and Principal Component Analysis