Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Loading in …5
×

of

Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

2

Share

# Singular value decomposition (SVD)

Slides for this video
https://www.youtube.com/watch?v=DG7YTlGnCEo

See all

See all

### Singular value decomposition (SVD)

1. 1. Singular Value Decomposition Luis Serrano
2. 2. Announcements
3. 3. github.com/luisguiserrano/singular_value_decomposition
4. 4. https://www.manning.com/books/grokking-machine-learning Discount code: serranoyt Grokking Machine Learning By Luis G. Serrano
5. 5. Transformations
6. 6. Transformations Stretch (or compress) horizontally
7. 7. Transformations Stretch (or compress) horizontally
8. 8. Transformations Stretch (or compress) horizontally
9. 9. Transformations Stretch (or compress) horizontally
10. 10. Transformations Stretch (or compress) vertically
11. 11. Transformations Stretch (or compress) vertically
12. 12. Transformations Stretch (or compress) vertically
13. 13. Transformations Stretch (or compress) vertically
14. 14. Transformations Rotate
15. 15. Puzzle (easy)
16. 16. Puzzle (easy)
17. 17. Puzzle (easy)
18. 18. Puzzle (easy)
19. 19. Puzzle (easy)
20. 20. Puzzle (hard)
21. 21. Puzzle (hard)
22. 22. Puzzle (hard)
23. 23. Puzzle (hard)
24. 24. Puzzle (hard)
25. 25. Solution
26. 26. Solution
27. 27. Solution
28. 28. Solution
29. 29. Solution
30. 30. Solution
31. 31. Solution
32. 32. Linear transformations
33. 33. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 What does this have to do with matrices? (p,q) (3p+0q, 4p+5q) (1,0) (3, 4) (0,1) (0, 5) (-1,0) (0,-1) (-3, -4) (0, -5) 3 0 4 5[ ]A =
34. 34. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation matrices cos(θ) −sin(θ) sin(θ) cos(θ)[ ] θ
35. 35. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Stretching matrices σ1 0 0 σ2 [ ] σ1 σ2
36. 36. Stretching matrices σ1 0 0 σ2 [ ] -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 σ1 σ2
37. 37. Stretching matrices σ1 0 0 σ2 [ ] -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 σ1 σ2
38. 38. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 What does this have to do with matrices? 3 0 4 5[ ]A = cos(θ) sin(θ) −sin(θ) cos(θ)[ ] cos(ϕ) sin(ϕ) −sin(ϕ) cos(ϕ)[ ] σ1 0 0 σ2 [ ]
39. 39. Singular value decomposition 3 0 4 5[ ] cos(θ) sin(θ) −sin(θ) cos(θ)[ ] cos(ϕ) sin(ϕ) −sin(ϕ) cos(ϕ)[ ] σ1 0 0 σ2 [ ]= A = UΣV†
40. 40. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 SVD A = UΣV† 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] 1/ 2 1/ 2 −1/ 2 1/ 2[ ] Rotation of θ = − π 4 = − 45o 3 0 4 5[ ] =
41. 41. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Rotation of θ = − π 4 = − 45o 3 0 4 5[ ] =
42. 42. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Horizontal scaling by 3 5 3 0 4 5[ ] =
43. 43. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Scaling 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Horizontal scaling by 3 5 Vertical scaling by 5 3 0 4 5[ ] =
44. 44. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Vertical scaling by 5 3 0 4 5[ ] =
45. 45. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Rotation of θ = arctan(3) = 71.72o 3 0 4 5[ ] =
46. 46. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Rotation of θ = arctan(3) = 71.72o 3 0 4 5[ ] =
47. 47. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 A = UΣV† A Σ V† U
48. 48. Dimensionality reduction
49. 49. 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Difference between these two matrices?
50. 50. 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Difference between these two matrices? 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ?
51. 51. = 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Rank 1 matrices 16 numbers 8 numbers
52. 52. = 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ? Higher rank matrices 16 numbers
53. 53. Rank of a matrix = Rank 1 = Rank 2 = Rank 3 = Rank 4
54. 54. ∼ 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ? Approximation by a rank one matrix
55. 55. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4σ4 = U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 0 1 2 3 4 0 1 2 3 4 A
56. 56. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
57. 57. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
58. 58. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = + 0 1 2 3 4 0 1 2 3 4 A
59. 59. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4V1 V2 V3 V4σ1 σ2 σ3 σ4+ = + + Rank 1 Rank 1 Rank 1 Rank 1 TinySmallMediumLarge 0 1 2 3 4 0 1 2 3 4 A
60. 60. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2V1 V2σ1 σ2+ = Rank 1 Rank 1 MediumLarge 0 1 2 3 4 0 1 2 3 4 A
61. 61. 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 0.15+ Tiny 0.79 -0.29 -0.54 -0.04 -0.89 -0.23 0.15 0.36 0 1 2 3 4 0 1 2 3 4 21.2 6.4 4.9 0.15 = -0.55 -0.52 -0.49 -0.43 0.26 -0.4 0.65 -0.59 0.07 0.7 -0.22 -0.68 0.79 -0.29 -0.54 -0.04 -0.21 0.37 -0.13 -0.89 -0.52 -0.7 0.43 -0.23 -0.48 -0.21 -0.84 0.15 -0.67 0.57 0.31 0.36 4.9+ 0.07 0.7 -0.22 -0.68 -0.13 0.43 -0.84 0.31 Small 6.4+ 0.26 -0.4 0.65 -0.59 0.37 -0.7 -0.21 0.57 Medium 21.2 -0.55 -0.52 -0.49 -0.43 -0.21 -0.52 -0.48 -0.67 Large 2.51 2.37 2.22 1.97 6.07 5,72 5.37 4.77 5.63 5,31 4.99 4.43 7.88 7.43 6.98 6.19 3.15 1.41 3.79 0.56 4.87 7.53 2.44 7.43 5.28 5.85 4.12 5.22 8.85 5.96 9.36 4.03 3.1 0.96 3.93 0.99 5.03 8.99 1.98 6 4.98 3.01 5.01 8 8.96 7.02 9.03 3 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3
62. 62. Rank of a matrix = Rank 1 = Rank 2 = Rank 3 = Rank 4
63. 63. Dimensionality reduction
64. 64. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] A -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 A = UΣV† 0.316 −0.949 0.949 0.316[ ]U 6.71 0 0 0.44[ ]Σ 0.7071 0.7071 −0.7071 0.7071[ ] V† 6.71 0.44
65. 65. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 0 0 0.44[ ] 1.8 1.2 4.4 4.6[ ] A = UΣV† A Σ -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 0.44 0 0 0.7071 0.7071 −0.7071 0.7071[ ] V† 0.316 −0.949 0.949 0.316[ ]U
66. 66. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] A = UΣV† A Σ -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 [ 6.71 0 0 0.44 ] 0.44 0 0 0.316 −0.949 0.949 0.316[ ]U 0.7071 0.7071 −0.7071 0.7071[ ] V†
67. 67. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] A = UΣV† A Σ -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 [ 6.71 0 0 0.44 ] 0.44 0 0 1.5 1.5 4.5 4.5[ ] 0.316 −0.949 0.949 0.316[ ]U 0.7071 0.7071 −0.7071 0.7071[ ] V†
68. 68. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.5 1.5 4.5 4.5[ ] Rank 1
69. 69. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] Rank 2 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.5 1.5 4.5 4.5[ ] Rank 1
70. 70. 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Rank 1 matrices
71. 71. = 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Rank 1 matrices 16 numbers 8 numbers
72. 72. = 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ? Higher rank matrices 16 numbers
73. 73. Approximation by rank one matrices = + + … 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3
74. 74. U 0 1 2 3 4 0 1 2 3 4 V† Σ= U1 U2 U3 U4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 0 1 2 3 4 0 1 2 3 4 A
75. 75. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4σ4 = U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 0 1 2 3 4 0 1 2 3 4 A
76. 76. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
77. 77. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
78. 78. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = + 0 1 2 3 4 0 1 2 3 4 A
79. 79. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4V1 V2 V3 V4σ1 σ2 σ3 σ4+ = + + Rank 1 Rank 1 Rank 1 Rank 1 TinySmallMediumLarge 0 1 2 3 4 0 1 2 3 4 A
80. 80. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2V1 V2σ1 σ2+ = Rank 1 Rank 1 MediumLarge 0 1 2 3 4 0 1 2 3 4 A
81. 81. Rectangular matrices
82. 82. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4= 0 1 2 3 4 0 1 2 3 4 5 6 A V1 V2 V3 V4 V5 V6 σ1 σ2 σ3 σ4 0 0 0 0 0 0 0 0 No square matrix? No problem!
83. 83. Image compression
84. 84. 0 1 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 0 0 0 1 1 1 0 0 0 0 0 1 0 0 0 Rank 4 www.github.com/luisguiserrano/singular_value_decomposition
85. 85. Thank you!
86. 86. Similar videos on dimensionality reduction Matrix Factorization Principal Component Analysis
87. 87. https://www.manning.com/books/grokking-machine-learning Discount code: serranoyt Grokking Machine Learning By Luis G. Serrano
88. 88. Thank you! @luis_likes_math Subscribe, like, share, comment! youtube.com/c/LuisSerrano http://serrano.academy
• #### donnabeers2

Jul. 28, 2021
• #### ShalinShah52

Dec. 18, 2020

Slides for this video https://www.youtube.com/watch?v=DG7YTlGnCEo

Total views

132

On Slideshare

0

From embeds

0

Number of embeds

4

Downloads

25

Shares

0

Comments

0

Likes

2