Successfully reported this slideshow.
Your SlideShare is downloading. ×

Singular value decomposition (SVD)

Ad

Singular Value Decomposition
Luis Serrano

Ad

Announcements

Ad

github.com/luisguiserrano/singular_value_decomposition

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Upcoming SlideShare
Algebra
Algebra
Loading in …3
×

Check these out next

1 of 89 Ad
1 of 89 Ad

More Related Content

Singular value decomposition (SVD)

  1. 1. Singular Value Decomposition Luis Serrano
  2. 2. Announcements
  3. 3. github.com/luisguiserrano/singular_value_decomposition
  4. 4. https://www.manning.com/books/grokking-machine-learning Discount code: serranoyt Grokking Machine Learning By Luis G. Serrano
  5. 5. Transformations
  6. 6. Transformations Stretch (or compress) horizontally
  7. 7. Transformations Stretch (or compress) horizontally
  8. 8. Transformations Stretch (or compress) horizontally
  9. 9. Transformations Stretch (or compress) horizontally
  10. 10. Transformations Stretch (or compress) vertically
  11. 11. Transformations Stretch (or compress) vertically
  12. 12. Transformations Stretch (or compress) vertically
  13. 13. Transformations Stretch (or compress) vertically
  14. 14. Transformations Rotate
  15. 15. Puzzle (easy)
  16. 16. Puzzle (easy)
  17. 17. Puzzle (easy)
  18. 18. Puzzle (easy)
  19. 19. Puzzle (easy)
  20. 20. Puzzle (hard)
  21. 21. Puzzle (hard)
  22. 22. Puzzle (hard)
  23. 23. Puzzle (hard)
  24. 24. Puzzle (hard)
  25. 25. Solution
  26. 26. Solution
  27. 27. Solution
  28. 28. Solution
  29. 29. Solution
  30. 30. Solution
  31. 31. Solution
  32. 32. Linear transformations
  33. 33. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 What does this have to do with matrices? (p,q) (3p+0q, 4p+5q) (1,0) (3, 4) (0,1) (0, 5) (-1,0) (0,-1) (-3, -4) (0, -5) 3 0 4 5[ ]A =
  34. 34. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation matrices cos(θ) −sin(θ) sin(θ) cos(θ)[ ] θ
  35. 35. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Stretching matrices σ1 0 0 σ2 [ ] σ1 σ2
  36. 36. Stretching matrices σ1 0 0 σ2 [ ] -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 σ1 σ2
  37. 37. Stretching matrices σ1 0 0 σ2 [ ] -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 σ1 σ2
  38. 38. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 What does this have to do with matrices? 3 0 4 5[ ]A = cos(θ) sin(θ) −sin(θ) cos(θ)[ ] cos(ϕ) sin(ϕ) −sin(ϕ) cos(ϕ)[ ] σ1 0 0 σ2 [ ]
  39. 39. Singular value decomposition 3 0 4 5[ ] cos(θ) sin(θ) −sin(θ) cos(θ)[ ] cos(ϕ) sin(ϕ) −sin(ϕ) cos(ϕ)[ ] σ1 0 0 σ2 [ ]= A = UΣV†
  40. 40. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 SVD A = UΣV† 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] 1/ 2 1/ 2 −1/ 2 1/ 2[ ] Rotation of θ = − π 4 = − 45o 3 0 4 5[ ] =
  41. 41. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Rotation of θ = − π 4 = − 45o 3 0 4 5[ ] =
  42. 42. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Horizontal scaling by 3 5 3 0 4 5[ ] =
  43. 43. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Scaling 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Horizontal scaling by 3 5 Vertical scaling by 5 3 0 4 5[ ] =
  44. 44. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Vertical scaling by 5 3 0 4 5[ ] =
  45. 45. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Rotation of θ = arctan(3) = 71.72o 3 0 4 5[ ] =
  46. 46. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Rotation of θ = arctan(3) = 71.72o 3 0 4 5[ ] =
  47. 47. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 A = UΣV† A Σ V† U
  48. 48. Dimensionality reduction
  49. 49. 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Difference between these two matrices?
  50. 50. 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Difference between these two matrices? 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ?
  51. 51. = 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Rank 1 matrices 16 numbers 8 numbers
  52. 52. = 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ? Higher rank matrices 16 numbers
  53. 53. Rank of a matrix = Rank 1 = Rank 2 = Rank 3 = Rank 4
  54. 54. ∼ 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ? Approximation by a rank one matrix
  55. 55. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4σ4 = U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 0 1 2 3 4 0 1 2 3 4 A
  56. 56. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
  57. 57. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
  58. 58. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = + 0 1 2 3 4 0 1 2 3 4 A
  59. 59. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4V1 V2 V3 V4σ1 σ2 σ3 σ4+ = + + Rank 1 Rank 1 Rank 1 Rank 1 TinySmallMediumLarge 0 1 2 3 4 0 1 2 3 4 A
  60. 60. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2V1 V2σ1 σ2+ = Rank 1 Rank 1 MediumLarge 0 1 2 3 4 0 1 2 3 4 A
  61. 61. 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 0.15+ Tiny 0.79 -0.29 -0.54 -0.04 -0.89 -0.23 0.15 0.36 0 1 2 3 4 0 1 2 3 4 21.2 6.4 4.9 0.15 = -0.55 -0.52 -0.49 -0.43 0.26 -0.4 0.65 -0.59 0.07 0.7 -0.22 -0.68 0.79 -0.29 -0.54 -0.04 -0.21 0.37 -0.13 -0.89 -0.52 -0.7 0.43 -0.23 -0.48 -0.21 -0.84 0.15 -0.67 0.57 0.31 0.36 4.9+ 0.07 0.7 -0.22 -0.68 -0.13 0.43 -0.84 0.31 Small 6.4+ 0.26 -0.4 0.65 -0.59 0.37 -0.7 -0.21 0.57 Medium 21.2 -0.55 -0.52 -0.49 -0.43 -0.21 -0.52 -0.48 -0.67 Large 2.51 2.37 2.22 1.97 6.07 5,72 5.37 4.77 5.63 5,31 4.99 4.43 7.88 7.43 6.98 6.19 3.15 1.41 3.79 0.56 4.87 7.53 2.44 7.43 5.28 5.85 4.12 5.22 8.85 5.96 9.36 4.03 3.1 0.96 3.93 0.99 5.03 8.99 1.98 6 4.98 3.01 5.01 8 8.96 7.02 9.03 3 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3
  62. 62. Rank of a matrix = Rank 1 = Rank 2 = Rank 3 = Rank 4
  63. 63. Dimensionality reduction
  64. 64. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] A -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 A = UΣV† 0.316 −0.949 0.949 0.316[ ]U 6.71 0 0 0.44[ ]Σ 0.7071 0.7071 −0.7071 0.7071[ ] V† 6.71 0.44
  65. 65. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 0 0 0.44[ ] 1.8 1.2 4.4 4.6[ ] A = UΣV† A Σ -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 0.44 0 0 0.7071 0.7071 −0.7071 0.7071[ ] V† 0.316 −0.949 0.949 0.316[ ]U
  66. 66. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] A = UΣV† A Σ -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 [ 6.71 0 0 0.44 ] 0.44 0 0 0.316 −0.949 0.949 0.316[ ]U 0.7071 0.7071 −0.7071 0.7071[ ] V†
  67. 67. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] A = UΣV† A Σ -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 [ 6.71 0 0 0.44 ] 0.44 0 0 1.5 1.5 4.5 4.5[ ] 0.316 −0.949 0.949 0.316[ ]U 0.7071 0.7071 −0.7071 0.7071[ ] V†
  68. 68. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.5 1.5 4.5 4.5[ ] Rank 1
  69. 69. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] Rank 2 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.5 1.5 4.5 4.5[ ] Rank 1
  70. 70. 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Rank 1 matrices
  71. 71. = 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Rank 1 matrices 16 numbers 8 numbers
  72. 72. = 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ? Higher rank matrices 16 numbers
  73. 73. Approximation by rank one matrices = + + … 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3
  74. 74. U 0 1 2 3 4 0 1 2 3 4 V† Σ= U1 U2 U3 U4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 0 1 2 3 4 0 1 2 3 4 A
  75. 75. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4σ4 = U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 0 1 2 3 4 0 1 2 3 4 A
  76. 76. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
  77. 77. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
  78. 78. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = + 0 1 2 3 4 0 1 2 3 4 A
  79. 79. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4V1 V2 V3 V4σ1 σ2 σ3 σ4+ = + + Rank 1 Rank 1 Rank 1 Rank 1 TinySmallMediumLarge 0 1 2 3 4 0 1 2 3 4 A
  80. 80. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2V1 V2σ1 σ2+ = Rank 1 Rank 1 MediumLarge 0 1 2 3 4 0 1 2 3 4 A
  81. 81. Rectangular matrices
  82. 82. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4= 0 1 2 3 4 0 1 2 3 4 5 6 A V1 V2 V3 V4 V5 V6 σ1 σ2 σ3 σ4 0 0 0 0 0 0 0 0 No square matrix? No problem!
  83. 83. Image compression
  84. 84. 0 1 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 0 0 0 1 1 1 0 0 0 0 0 1 0 0 0 Rank 4 www.github.com/luisguiserrano/singular_value_decomposition
  85. 85. Thank you!
  86. 86. Similar videos on dimensionality reduction Matrix Factorization Principal Component Analysis
  87. 87. https://www.manning.com/books/grokking-machine-learning Discount code: serranoyt Grokking Machine Learning By Luis G. Serrano
  88. 88. Thank you! @luis_likes_math Subscribe, like, share, comment! youtube.com/c/LuisSerrano http://serrano.academy

×