Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

2

Share

Download to read offline

Singular value decomposition (SVD)

Download to read offline

Slides for this video
https://www.youtube.com/watch?v=DG7YTlGnCEo

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Singular value decomposition (SVD)

  1. 1. Singular Value Decomposition Luis Serrano
  2. 2. Announcements
  3. 3. github.com/luisguiserrano/singular_value_decomposition
  4. 4. https://www.manning.com/books/grokking-machine-learning Discount code: serranoyt Grokking Machine Learning By Luis G. Serrano
  5. 5. Transformations
  6. 6. Transformations Stretch (or compress) horizontally
  7. 7. Transformations Stretch (or compress) horizontally
  8. 8. Transformations Stretch (or compress) horizontally
  9. 9. Transformations Stretch (or compress) horizontally
  10. 10. Transformations Stretch (or compress) vertically
  11. 11. Transformations Stretch (or compress) vertically
  12. 12. Transformations Stretch (or compress) vertically
  13. 13. Transformations Stretch (or compress) vertically
  14. 14. Transformations Rotate
  15. 15. Puzzle (easy)
  16. 16. Puzzle (easy)
  17. 17. Puzzle (easy)
  18. 18. Puzzle (easy)
  19. 19. Puzzle (easy)
  20. 20. Puzzle (hard)
  21. 21. Puzzle (hard)
  22. 22. Puzzle (hard)
  23. 23. Puzzle (hard)
  24. 24. Puzzle (hard)
  25. 25. Solution
  26. 26. Solution
  27. 27. Solution
  28. 28. Solution
  29. 29. Solution
  30. 30. Solution
  31. 31. Solution
  32. 32. Linear transformations
  33. 33. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 What does this have to do with matrices? (p,q) (3p+0q, 4p+5q) (1,0) (3, 4) (0,1) (0, 5) (-1,0) (0,-1) (-3, -4) (0, -5) 3 0 4 5[ ]A =
  34. 34. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation matrices cos(θ) −sin(θ) sin(θ) cos(θ)[ ] θ
  35. 35. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Stretching matrices σ1 0 0 σ2 [ ] σ1 σ2
  36. 36. Stretching matrices σ1 0 0 σ2 [ ] -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 σ1 σ2
  37. 37. Stretching matrices σ1 0 0 σ2 [ ] -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 σ1 σ2
  38. 38. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 What does this have to do with matrices? 3 0 4 5[ ]A = cos(θ) sin(θ) −sin(θ) cos(θ)[ ] cos(ϕ) sin(ϕ) −sin(ϕ) cos(ϕ)[ ] σ1 0 0 σ2 [ ]
  39. 39. Singular value decomposition 3 0 4 5[ ] cos(θ) sin(θ) −sin(θ) cos(θ)[ ] cos(ϕ) sin(ϕ) −sin(ϕ) cos(ϕ)[ ] σ1 0 0 σ2 [ ]= A = UΣV†
  40. 40. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 SVD A = UΣV† 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] 1/ 2 1/ 2 −1/ 2 1/ 2[ ] Rotation of θ = − π 4 = − 45o 3 0 4 5[ ] =
  41. 41. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Rotation of θ = − π 4 = − 45o 3 0 4 5[ ] =
  42. 42. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Horizontal scaling by 3 5 3 0 4 5[ ] =
  43. 43. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Scaling 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Horizontal scaling by 3 5 Vertical scaling by 5 3 0 4 5[ ] =
  44. 44. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Vertical scaling by 5 3 0 4 5[ ] =
  45. 45. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Rotation of θ = arctan(3) = 71.72o 3 0 4 5[ ] =
  46. 46. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Rotation of θ = arctan(3) = 71.72o 3 0 4 5[ ] =
  47. 47. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 A = UΣV† A Σ V† U
  48. 48. Dimensionality reduction
  49. 49. 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Difference between these two matrices?
  50. 50. 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Difference between these two matrices? 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ?
  51. 51. = 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Rank 1 matrices 16 numbers 8 numbers
  52. 52. = 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ? Higher rank matrices 16 numbers
  53. 53. Rank of a matrix = Rank 1 = Rank 2 = Rank 3 = Rank 4
  54. 54. ∼ 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ? Approximation by a rank one matrix
  55. 55. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4σ4 = U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 0 1 2 3 4 0 1 2 3 4 A
  56. 56. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
  57. 57. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
  58. 58. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = + 0 1 2 3 4 0 1 2 3 4 A
  59. 59. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4V1 V2 V3 V4σ1 σ2 σ3 σ4+ = + + Rank 1 Rank 1 Rank 1 Rank 1 TinySmallMediumLarge 0 1 2 3 4 0 1 2 3 4 A
  60. 60. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2V1 V2σ1 σ2+ = Rank 1 Rank 1 MediumLarge 0 1 2 3 4 0 1 2 3 4 A
  61. 61. 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 0.15+ Tiny 0.79 -0.29 -0.54 -0.04 -0.89 -0.23 0.15 0.36 0 1 2 3 4 0 1 2 3 4 21.2 6.4 4.9 0.15 = -0.55 -0.52 -0.49 -0.43 0.26 -0.4 0.65 -0.59 0.07 0.7 -0.22 -0.68 0.79 -0.29 -0.54 -0.04 -0.21 0.37 -0.13 -0.89 -0.52 -0.7 0.43 -0.23 -0.48 -0.21 -0.84 0.15 -0.67 0.57 0.31 0.36 4.9+ 0.07 0.7 -0.22 -0.68 -0.13 0.43 -0.84 0.31 Small 6.4+ 0.26 -0.4 0.65 -0.59 0.37 -0.7 -0.21 0.57 Medium 21.2 -0.55 -0.52 -0.49 -0.43 -0.21 -0.52 -0.48 -0.67 Large 2.51 2.37 2.22 1.97 6.07 5,72 5.37 4.77 5.63 5,31 4.99 4.43 7.88 7.43 6.98 6.19 3.15 1.41 3.79 0.56 4.87 7.53 2.44 7.43 5.28 5.85 4.12 5.22 8.85 5.96 9.36 4.03 3.1 0.96 3.93 0.99 5.03 8.99 1.98 6 4.98 3.01 5.01 8 8.96 7.02 9.03 3 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3
  62. 62. Rank of a matrix = Rank 1 = Rank 2 = Rank 3 = Rank 4
  63. 63. Dimensionality reduction
  64. 64. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] A -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 A = UΣV† 0.316 −0.949 0.949 0.316[ ]U 6.71 0 0 0.44[ ]Σ 0.7071 0.7071 −0.7071 0.7071[ ] V† 6.71 0.44
  65. 65. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 0 0 0.44[ ] 1.8 1.2 4.4 4.6[ ] A = UΣV† A Σ -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 0.44 0 0 0.7071 0.7071 −0.7071 0.7071[ ] V† 0.316 −0.949 0.949 0.316[ ]U
  66. 66. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] A = UΣV† A Σ -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 [ 6.71 0 0 0.44 ] 0.44 0 0 0.316 −0.949 0.949 0.316[ ]U 0.7071 0.7071 −0.7071 0.7071[ ] V†
  67. 67. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] A = UΣV† A Σ -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 [ 6.71 0 0 0.44 ] 0.44 0 0 1.5 1.5 4.5 4.5[ ] 0.316 −0.949 0.949 0.316[ ]U 0.7071 0.7071 −0.7071 0.7071[ ] V†
  68. 68. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.5 1.5 4.5 4.5[ ] Rank 1
  69. 69. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] Rank 2 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.5 1.5 4.5 4.5[ ] Rank 1
  70. 70. 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Rank 1 matrices
  71. 71. = 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Rank 1 matrices 16 numbers 8 numbers
  72. 72. = 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ? Higher rank matrices 16 numbers
  73. 73. Approximation by rank one matrices = + + … 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3
  74. 74. U 0 1 2 3 4 0 1 2 3 4 V† Σ= U1 U2 U3 U4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 0 1 2 3 4 0 1 2 3 4 A
  75. 75. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4σ4 = U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 0 1 2 3 4 0 1 2 3 4 A
  76. 76. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
  77. 77. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
  78. 78. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = + 0 1 2 3 4 0 1 2 3 4 A
  79. 79. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4V1 V2 V3 V4σ1 σ2 σ3 σ4+ = + + Rank 1 Rank 1 Rank 1 Rank 1 TinySmallMediumLarge 0 1 2 3 4 0 1 2 3 4 A
  80. 80. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2V1 V2σ1 σ2+ = Rank 1 Rank 1 MediumLarge 0 1 2 3 4 0 1 2 3 4 A
  81. 81. Rectangular matrices
  82. 82. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4= 0 1 2 3 4 0 1 2 3 4 5 6 A V1 V2 V3 V4 V5 V6 σ1 σ2 σ3 σ4 0 0 0 0 0 0 0 0 No square matrix? No problem!
  83. 83. Image compression
  84. 84. 0 1 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 0 0 0 1 1 1 0 0 0 0 0 1 0 0 0 Rank 4 www.github.com/luisguiserrano/singular_value_decomposition
  85. 85. Thank you!
  86. 86. Similar videos on dimensionality reduction Matrix Factorization Principal Component Analysis
  87. 87. https://www.manning.com/books/grokking-machine-learning Discount code: serranoyt Grokking Machine Learning By Luis G. Serrano
  88. 88. Thank you! @luis_likes_math Subscribe, like, share, comment! youtube.com/c/LuisSerrano http://serrano.academy
  • donnabeers2

    Jul. 28, 2021
  • ShalinShah52

    Dec. 18, 2020

Slides for this video https://www.youtube.com/watch?v=DG7YTlGnCEo

Views

Total views

132

On Slideshare

0

From embeds

0

Number of embeds

4

Actions

Downloads

25

Shares

0

Comments

0

Likes

2

×