The document discusses singular value decomposition (SVD), which is a way to decompose a matrix A into three matrices: A = UΣV^T. U and V are orthogonal matrices, and Σ is a diagonal matrix containing the singular values of A. SVD can be used to perform dimensionality reduction by approximating A using only the top k singular values/vectors in Σ, U, and V^T. This reduces the number of parameters needed to represent A while retaining most of its information.

1. Singular Value Decomposition
Luis Serrano

3. Announcements

4. github.com/luisguiserrano/singular_value_decomposition

5. https://www.manning.com/books/grokking-machine-learning
Discount code: serranoyt
Grokking Machine
Learning
By Luis G. Serrano

6. Transformations

7. Transformations
Stretch (or compress) horizontally

8. Transformations
Stretch (or compress) horizontally

9. Transformations
Stretch (or compress) horizontally

10. Transformations
Stretch (or compress) horizontally

11. Transformations
Stretch (or compress) vertically

12. Transformations
Stretch (or compress) vertically

13. Transformations
Stretch (or compress) vertically

14. Transformations
Stretch (or compress) vertically

15. Transformations
Rotate

16. Puzzle (easy)

17. Puzzle (easy)

18. Puzzle (easy)

19. Puzzle (easy)

20. Puzzle (easy)

21. Puzzle (hard)

22. Puzzle (hard)

23. Puzzle (hard)

24. Puzzle (hard)

25. Puzzle (hard)

26. Solution

27. Solution

28. Solution

29. Solution

30. Solution

31. Solution

32. Solution

33. Linear transformations

34. -7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
What does this have to do with matrices?
(p,q) (3p+0q, 4p+5q)
(1,0) (3, 4)
(0,1) (0, 5)
(-1,0)
(0,-1)
(-3, -4)
(0, -5)
3 0
4 5[ ]A =

35. -7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Rotation matrices
cos(θ) −sin(θ)
sin(θ) cos(θ)[ ]
θ

36. -7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Stretching matrices
σ1 0
0 σ2
[ ]
σ1
σ2

37. Stretching matrices
σ1 0
0 σ2
[ ]
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
σ1
σ2

38. Stretching matrices
σ1 0
0 σ2
[ ]
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
σ1
σ2

39. -7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
What does this have to do with matrices?
3 0
4 5[ ]A =
cos(θ) sin(θ)
−sin(θ) cos(θ)[ ]
cos(ϕ) sin(ϕ)
−sin(ϕ) cos(ϕ)[ ]
σ1 0
0 σ2
[ ]

40. Singular value decomposition
3 0
4 5[ ]
cos(θ) sin(θ)
−sin(θ) cos(θ)[ ]
cos(ϕ) sin(ϕ)
−sin(ϕ) cos(ϕ)[ ]
σ1 0
0 σ2
[ ]=
A = UΣV†

41. -7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
SVD A = UΣV†
3 5 0
0 5[ ]
1/ 10 −3/ 10
3/ 10 1/ 10[ ]
0.7071 0.7071
−0.7071 0.7071[ ]
6.708 0
0 2.236[ ]
0.316 −0.949
0.949 0.316[ ]
1/ 2 1/ 2
−1/ 2 1/ 2[ ]
Rotation of θ = −
π
4
= − 45o
3 0
4 5[ ] =

42. -7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Rotation
1/ 2 1/ 2
−1/ 2 1/ 2[ ]
3 5 0
0 5[ ]
1/ 10 −3/ 10
3/ 10 1/ 10[ ]
0.7071 0.7071
−0.7071 0.7071[ ]
6.708 0
0 2.236[ ]
0.316 −0.949
0.949 0.316[ ]
A = UΣV†
Rotation of θ = −
π
4
= − 45o
3 0
4 5[ ] =

43. -7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Rotation
1/ 2 1/ 2
−1/ 2 1/ 2[ ]
3 5 0
0 5[ ]
1/ 10 −3/ 10
3/ 10 1/ 10[ ]
0.7071 0.7071
−0.7071 0.7071[ ]
6.708 0
0 2.236[ ]
0.316 −0.949
0.949 0.316[ ]
A = UΣV†
Horizontal scaling by 3 5
3 0
4 5[ ] =

44. -7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Scaling
1/ 2 1/ 2
−1/ 2 1/ 2[ ]
3 5 0
0 5[ ]
1/ 10 −3/ 10
3/ 10 1/ 10[ ]
0.7071 0.7071
−0.7071 0.7071[ ]
6.708 0
0 2.236[ ]
0.316 −0.949
0.949 0.316[ ]
A = UΣV†
Horizontal scaling by 3 5
Vertical scaling by 5
3 0
4 5[ ] =

45. -7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Rotation
1/ 2 1/ 2
−1/ 2 1/ 2[ ]
3 5 0
0 5[ ]
1/ 10 −3/ 10
3/ 10 1/ 10[ ]
0.7071 0.7071
−0.7071 0.7071[ ]
6.708 0
0 2.236[ ]
0.316 −0.949
0.949 0.316[ ]
A = UΣV†
Vertical scaling by 5
3 0
4 5[ ] =

46. -7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Rotation
1/ 2 1/ 2
−1/ 2 1/ 2[ ]
3 5 0
0 5[ ]
1/ 10 −3/ 10
3/ 10 1/ 10[ ]
0.7071 0.7071
−0.7071 0.7071[ ]
6.708 0
0 2.236[ ]
0.316 −0.949
0.949 0.316[ ]
A = UΣV†
Rotation of θ = arctan(3) = 71.72o
3 0
4 5[ ] =

47. -7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Rotation
1/ 2 1/ 2
−1/ 2 1/ 2[ ]
3 5 0
0 5[ ]
1/ 10 −3/ 10
3/ 10 1/ 10[ ]
0.7071 0.7071
−0.7071 0.7071[ ]
6.708 0
0 2.236[ ]
0.316 −0.949
0.949 0.316[ ]
A = UΣV†
Rotation of θ = arctan(3) = 71.72o
3 0
4 5[ ] =

48. -7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
-7
-5
-3
-1
1
3
5
7
-7
-5
-3
-1
1
3
5
7
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
A = UΣV†
A
Σ
V† U

49. Dimensionality reduction

50. 1 2 3 4
-1 -2 -3 -4
2 4 6 8
10 20 30 40
1 2 3 4
1
-1
2
10
Difference between these two matrices?

51. 1 2 3 4
-1 -2 -3 -4
2 4 6 8
10 20 30 40
1 2 3 4
1
-1
2
10
Difference between these two matrices?
3 1 4 1
5 9 2 6
5 3 5 8
9 7 9 3
? ? ? ?
?
?
?
?

52. =
1 2 3 4
-1 -2 -3 -4
2 4 6 8
10 20 30 40
1 2 3 4
1
-1
2
10
Rank 1 matrices
16 numbers 8 numbers

53. =
3 1 4 1
5 9 2 6
5 3 5 8
9 7 9 3
? ? ? ?
?
?
?
?
Higher rank matrices
16 numbers

54. Rank of a matrix
=
Rank 1
=
Rank 2
=
Rank 3
=
Rank 4

55. ∼
3 1 4 1
5 9 2 6
5 3 5 8
9 7 9 3
? ? ? ?
?
?
?
?
Approximation by a rank one matrix

56. 0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4σ4
= U1 U2 U3 U4
V1
V2
V3
V4
σ1
σ2
σ3
σ4
0
1
2
3
4
0 1 2 3 4
A

57. 0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1
U2 U3 U4
V1
V2
V3
V4
σ1
σ2
σ3
σ4
+
=
0
1
2
3
4
0 1 2 3 4
A

58. 0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2
U3 U4
V1 V2
V3
V4
σ1 σ2
σ3
σ4
+
=
0
1
2
3
4
0 1 2 3 4
A

59. 0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2 U3
U4
V1 V2 V3
V4
σ1 σ2 σ3
σ4
+
=
+
0
1
2
3
4
0 1 2 3 4
A

60. 0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2 U3 U4V1 V2 V3 V4σ1 σ2 σ3 σ4+
=
+ +
Rank 1 Rank 1 Rank 1 Rank 1
TinySmallMediumLarge
0
1
2
3
4
0 1 2 3 4
A

61. 0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2V1 V2σ1 σ2+
=
Rank 1 Rank 1
MediumLarge
0
1
2
3
4
0 1 2 3 4
A

62. 3 1 4 1
5 9 2 6
5 3 5 8
9 7 9 3
0.15+
Tiny
0.79 -0.29 -0.54 -0.04
-0.89
-0.23
0.15
0.36
0
1
2
3
4
0 1 2 3 4
21.2
6.4
4.9
0.15
=
-0.55 -0.52 -0.49 -0.43
0.26 -0.4 0.65 -0.59
0.07 0.7 -0.22 -0.68
0.79 -0.29 -0.54 -0.04
-0.21 0.37 -0.13 -0.89
-0.52 -0.7 0.43 -0.23
-0.48 -0.21 -0.84 0.15
-0.67 0.57 0.31 0.36
4.9+ 0.07 0.7 -0.22 -0.68
-0.13
0.43
-0.84
0.31
Small
6.4+ 0.26 -0.4 0.65 -0.59
0.37
-0.7
-0.21
0.57
Medium
21.2 -0.55 -0.52 -0.49 -0.43
-0.21
-0.52
-0.48
-0.67
Large
2.51 2.37 2.22 1.97
6.07 5,72 5.37 4.77
5.63 5,31 4.99 4.43
7.88 7.43 6.98 6.19
3.15 1.41 3.79 0.56
4.87 7.53 2.44 7.43
5.28 5.85 4.12 5.22
8.85 5.96 9.36 4.03
3.1 0.96 3.93 0.99
5.03 8.99 1.98 6
4.98 3.01 5.01 8
8.96 7.02 9.03 3
3 1 4 1
5 9 2 6
5 3 5 8
9 7 9 3

63. Rank of a matrix
=
Rank 1
=
Rank 2
=
Rank 3
=
Rank 4

64. Dimensionality reduction

65. -7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
1.8 1.2
4.4 4.6[ ]
A
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
-7
-5
-3
-1
1
3
5
7
-7
-5
-3
-1
1
3
5
7
A = UΣV†
0.316 −0.949
0.949 0.316[ ]U
6.71 0
0 0.44[ ]Σ
0.7071 0.7071
−0.7071 0.7071[ ] V†
6.71
0.44

66. -7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
-7
-5
-3
-1
1
3
5
7
-7
-5
-3
-1
1
3
5
7
6.71 0
0 0.44[ ]
1.8 1.2
4.4 4.6[ ]
A = UΣV†
A
Σ
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
6.71
0.44
0
0
0.7071 0.7071
−0.7071 0.7071[ ] V†
0.316 −0.949
0.949 0.316[ ]U

67. -7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
-7
-5
-3
-1
1
3
5
7
-7
-5
-3
-1
1
3
5
7
1.8 1.2
4.4 4.6[ ]
A = UΣV†
A
Σ
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
6.71
[
6.71 0
0 0.44 ]
0.44
0
0
0.316 −0.949
0.949 0.316[ ]U
0.7071 0.7071
−0.7071 0.7071[ ] V†

68. -7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
-7
-5
-3
-1
1
3
5
7
-7
-5
-3
-1
1
3
5
7
1.8 1.2
4.4 4.6[ ]
A = UΣV†
A
Σ
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
6.71
[
6.71 0
0 0.44 ]
0.44
0
0
1.5 1.5
4.5 4.5[ ]
0.316 −0.949
0.949 0.316[ ]U
0.7071 0.7071
−0.7071 0.7071[ ] V†

69. -7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
1.5 1.5
4.5 4.5[ ]
Rank 1

70. -7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
1.8 1.2
4.4 4.6[ ]
Rank 2
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
1.5 1.5
4.5 4.5[ ]
Rank 1

71. 1 2 3 4
-1 -2 -3 -4
2 4 6 8
10 20 30 40
1 2 3 4
1
-1
2
10
Rank 1 matrices

72. =
1 2 3 4
-1 -2 -3 -4
2 4 6 8
10 20 30 40
1 2 3 4
1
-1
2
10
Rank 1 matrices
16 numbers 8 numbers

73. =
3 1 4 1
5 9 2 6
5 3 5 8
9 7 9 3
? ? ? ?
?
?
?
?
Higher rank matrices
16 numbers

74. Approximation by rank one matrices
= + + …
3 1 4 1
5 9 2 6
5 3 5 8
9 7 9 3

75. U
0
1
2
3
4
0 1 2 3 4
V†
Σ= U1 U2 U3 U4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
0
1
2
3
4
0 1 2 3 4
A

76. 0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4σ4
= U1 U2 U3 U4
V1
V2
V3
V4
σ1
σ2
σ3
σ4
0
1
2
3
4
0 1 2 3 4
A

77. 0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1
U2 U3 U4
V1
V2
V3
V4
σ1
σ2
σ3
σ4
+
=
0
1
2
3
4
0 1 2 3 4
A

78. 0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2
U3 U4
V1 V2
V3
V4
σ1 σ2
σ3
σ4
+
=
0
1
2
3
4
0 1 2 3 4
A

79. 0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2 U3
U4
V1 V2 V3
V4
σ1 σ2 σ3
σ4
+
=
+
0
1
2
3
4
0 1 2 3 4
A

80. 0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2 U3 U4V1 V2 V3 V4σ1 σ2 σ3 σ4+
=
+ +
Rank 1 Rank 1 Rank 1 Rank 1
TinySmallMediumLarge
0
1
2
3
4
0 1 2 3 4
A

81. 0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2V1 V2σ1 σ2+
=
Rank 1 Rank 1
MediumLarge
0
1
2
3
4
0 1 2 3 4
A

82. Rectangular matrices

83. 0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4=
0
1
2
3
4
0 1 2 3 4 5 6
A
V1
V2
V3
V4
V5
V6
σ1
σ2
σ3
σ4
0 0
0 0
0 0
0 0
No square matrix? No problem!

84. Image compression

85. 0 1 1 0 1 1 0
1 1 1 1 1 1 1
1 1 1 1 1 1 1
0 1 1 1 1 1 0
0 0 1 1 1 0 0
0 0 0 1 0 0 0
Rank 4
www.github.com/luisguiserrano/singular_value_decomposition

86. Thank you!

87. Similar videos on dimensionality reduction
Matrix Factorization Principal Component Analysis

88. https://www.manning.com/books/grokking-machine-learning
Discount code: serranoyt
Grokking Machine
Learning
By Luis G. Serrano

89. Thank you!
@luis_likes_math
Subscribe, like,
share, comment!
youtube.com/c/LuisSerrano
http://serrano.academy