### SlideShare for iOS

by Linkedin Corporation

FREE - On the App Store

- Total Views
- 1,018
- Views on SlideShare
- 1,018
- Embed Views

- Likes
- 4
- Downloads
- 87
- Comments
- 0

No embeds

Uploaded via SlideShare as Adobe PDF

© All Rights Reserved

- 1. Linear Algebra Review CS221: Introduction to Articial Intelligence Carlos Fernandez-Granda 10/11/2011CS221 Linear Algebra Review 10/11/2011 1 / 37
- 2. Index1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 2 / 37
- 3. Vectors1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 3 / 37
- 4. VectorsWhat is a vector ? CS221 Linear Algebra Review 10/11/2011 4 / 37
- 5. VectorsWhat is a vector ? An ordered tuple of numbers : u 1 u 2 u= . . . un CS221 Linear Algebra Review 10/11/2011 4 / 37
- 6. VectorsWhat is a vector ? An ordered tuple of numbers : u 1 u 2 u= . . . un A quantity that has a magnitude and a direction. CS221 Linear Algebra Review 10/11/2011 4 / 37
- 7. VectorsVector spaces More formally a vector is an element of a vector space. CS221 Linear Algebra Review 10/11/2011 5 / 37
- 8. VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : CS221 Linear Algebra Review 10/11/2011 5 / 37
- 9. VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : If vectors u and v ∈ V , then u + v ∈ V . CS221 Linear Algebra Review 10/11/2011 5 / 37
- 10. VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : If vectors u and v ∈ V , then u + v ∈ V . If vector u ∈ V , then α u ∈ V for any scalar α. CS221 Linear Algebra Review 10/11/2011 5 / 37
- 11. VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : If vectors u and v ∈ V , then u + v ∈ V . If vector u ∈ V , then α u ∈ V for any scalar α. There exists 0 ∈ V such that u + 0 = u for any u ∈ V . CS221 Linear Algebra Review 10/11/2011 5 / 37
- 12. VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : If vectors u and v ∈ V , then u + v ∈ V . If vector u ∈ V , then α u ∈ V for any scalar α. There exists 0 ∈ V such that u + 0 = u for any u ∈ V . A subspace is a subset of a vector space that is also a vector space. CS221 Linear Algebra Review 10/11/2011 5 / 37
- 13. VectorsExamples Euclidean space Rn . CS221 Linear Algebra Review 10/11/2011 6 / 37
- 14. VectorsExamples Euclidean space Rn . The span of any set of vectors {u1 , u2 , . . . , un }, dened as : span (u1 , u2 , . . . , un ) = {α1 u1 + α2 u2 + · · · + αn un | αi ∈ R} CS221 Linear Algebra Review 10/11/2011 6 / 37
- 15. VectorsSubspaces A line through the origin in Rn (span of a vector). CS221 Linear Algebra Review 10/11/2011 7 / 37
- 16. VectorsSubspaces A line through the origin in Rn (span of a vector). A plane in Rn (span of two vectors). CS221 Linear Algebra Review 10/11/2011 7 / 37
- 17. VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. CS221 Linear Algebra Review 10/11/2011 8 / 37
- 18. VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. CS221 Linear Algebra Review 10/11/2011 8 / 37
- 19. VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . CS221 Linear Algebra Review 10/11/2011 8 / 37
- 20. VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . If a vector space has a basis with d vectors, its dimension is d. CS221 Linear Algebra Review 10/11/2011 8 / 37
- 21. VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . If a vector space has a basis with d vectors, its dimension is d. Any other basis of the same space will consist of exactly d vectors. CS221 Linear Algebra Review 10/11/2011 8 / 37
- 22. VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . If a vector space has a basis with d vectors, its dimension is d. Any other basis of the same space will consist of exactly d vectors. The dimension of a vector space can be innite (function spaces). CS221 Linear Algebra Review 10/11/2011 8 / 37
- 23. VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. CS221 Linear Algebra Review 10/11/2011 9 / 37
- 24. VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : CS221 Linear Algebra Review 10/11/2011 9 / 37
- 25. VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : Homogeneity ||α u|| = |α| ||u|| CS221 Linear Algebra Review 10/11/2011 9 / 37
- 26. VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : Homogeneity ||α u|| = |α| ||u|| Triangle inequality ||u + v|| ≤ ||u|| + ||v|| CS221 Linear Algebra Review 10/11/2011 9 / 37
- 27. VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : Homogeneity ||α u|| = |α| ||u|| Triangle inequality ||u + v|| ≤ ||u|| + ||v|| Point separation ||u|| = 0 if and only if u = 0. CS221 Linear Algebra Review 10/11/2011 9 / 37
- 28. VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : Homogeneity ||α u|| = |α| ||u|| Triangle inequality ||u + v|| ≤ ||u|| + ||v|| Point separation ||u|| = 0 if and only if u = 0. Examples : Manhattan or 1 norm : ||u||1 = i |ui |. Euclidean or 2 norm : ||u||2 = i ui . 2 CS221 Linear Algebra Review 10/11/2011 9 / 37
- 29. VectorsInner Product Inner product between u and v : u, v = n uv. i =1 i i CS221 Linear Algebra Review 10/11/2011 10 / 37
- 30. VectorsInner Product Inner product between u and v : u, v = n=1 ui vi . i It is the projection of one vector onto the other. CS221 Linear Algebra Review 10/11/2011 10 / 37
- 31. VectorsInner Product Inner product between u and v : u, v = n=1 ui vi . i It is the projection of one vector onto the other. Related to the Euclidean norm : u, u = ||u||2 . 2 CS221 Linear Algebra Review 10/11/2011 10 / 37
- 32. VectorsInner ProductThe inner product is a measure of correlation between two vectors, scaledby the norms of the vectors : CS221 Linear Algebra Review 10/11/2011 11 / 37
- 33. VectorsInner ProductThe inner product is a measure of correlation between two vectors, scaledby the norms of the vectors : If u, v > 0, u and v are aligned. CS221 Linear Algebra Review 10/11/2011 11 / 37
- 34. VectorsInner ProductThe inner product is a measure of correlation between two vectors, scaledby the norms of the vectors : If u, v > 0, u and v are aligned. If u, v < 0, u and v are opposed. CS221 Linear Algebra Review 10/11/2011 11 / 37
- 35. VectorsInner ProductThe inner product is a measure of correlation between two vectors, scaledby the norms of the vectors : If u, v > 0, u and v are aligned. If u, v < 0, u and v are opposed. If u, v = 0, u and v are orthogonal. CS221 Linear Algebra Review 10/11/2011 11 / 37
- 36. VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. CS221 Linear Algebra Review 10/11/2011 12 / 37
- 37. VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. CS221 Linear Algebra Review 10/11/2011 12 / 37
- 38. VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. Example : x = α1 b1 + α2 b2 . CS221 Linear Algebra Review 10/11/2011 12 / 37
- 39. VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. Example : x = α1 b1 + α2 b2 . We compute x, b1 : x, b1 = α1 b1 + α2 b2 , b1 = α1 b1 , b1 + α2 b2 , b1 By linearity. = α1 + 0 By orthonormality. CS221 Linear Algebra Review 10/11/2011 12 / 37
- 40. VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. Example : x = α1 b1 + α2 b2 . We compute x, b1 : x, b1 = α1 b1 + α2 b2 , b1 = α1 b1 , b1 + α2 b2 , b1 By linearity. = α1 + 0 By orthonormality. Likewise, α2 = x, b2 . CS221 Linear Algebra Review 10/11/2011 12 / 37
- 41. Matrices1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 13 / 37
- 42. MatricesLinear Operator A linear operator L : U → V is a map from a vector space U to another vector space V that satises : CS221 Linear Algebra Review 10/11/2011 14 / 37
- 43. MatricesLinear Operator A linear operator L : U → V is a map from a vector space U to another vector space V that satises : L (u1 + u2 ) = L (u1 ) + L (u2 ) . CS221 Linear Algebra Review 10/11/2011 14 / 37
- 44. MatricesLinear Operator A linear operator L : U → V is a map from a vector space U to another vector space V that satises : . L (u1 + u2 ) = L (u1 ) + L (u2 ) L (α u) = α L (u) for any scalar α. CS221 Linear Algebra Review 10/11/2011 14 / 37
- 45. MatricesLinear Operator A linear operator L : U → V is a map from a vector space U to another vector space V that satises : . L (u1 + u2 ) = L (u1 ) + L (u2 ) L (α u) = α L (u) for any scalar α. If the dimension n of U and m of V are nite, L can be represented by an m × n matrix A. A A An 11 12 ... 1 A 21 A 22 ... A n 2 A= ... Am 1 Am 2 ... Amn CS221 Linear Algebra Review 10/11/2011 14 / 37
- 46. MatricesMatrix Vector Multiplication A A An u v 11 12 ... 1 1 1 A 21 A 22 ... A nu 2 2 v 2 Au = = ... . . . . . . Am 1 Am 2 ... Amn un vm CS221 Linear Algebra Review 10/11/2011 15 / 37
- 47. MatricesMatrix Vector Multiplication A A An u v 11 12 ... 1 1 1 A 21 A 22 ... A nu 2 2 v 2 Au = = ... . . . . . . Am 1 Am 2 ... Amn un vm CS221 Linear Algebra Review 10/11/2011 15 / 37
- 48. MatricesMatrix Vector Multiplication A A An u v 11 12 ... 1 1 1 A 21 A 22 ... A nu 2 2 v 2 Au = = ... . . . . . . Am 1 Am 2 ... Amn un vm CS221 Linear Algebra Review 10/11/2011 15 / 37
- 49. MatricesMatrix Vector Multiplication A A An u v 11 12 ... 1 1 1 A 21 A 22 ... A nu 2 2 v 2 Au = = ... . . . . . . Am 1 Am 2 ... Amn un vm CS221 Linear Algebra Review 10/11/2011 15 / 37
- 50. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp 11 12 ... 1 11 12 ... 1 A 21 A 22 ... B A n 2 21 B22 ... B p 2 AB = ... ... Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p 11 12 1 AB AB 21 22 . . . AB p 2 = ... ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
- 51. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp 11 12 ... 1 11 12 ... 1 A 21 A 22 ... B A n 2 21 B22 ... B p 2 AB = ... ... Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p 11 12 1 AB AB 21 22 . . . AB p 2 = ... ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
- 52. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp 11 12 ... 1 11 12 ... 1 A 21 A 22 ... B A n 2 21 B22 ... B p 2 AB = ... ... Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p 11 12 1 AB AB 21 22 . . . AB p 2 = ... ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
- 53. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp 11 12 ... 1 11 12 ... 1 A 21 A 22 ... B A n 2 21 B22 ... B p 2 AB = ... ... Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p 11 12 1 AB AB 21 22 . . . AB p 2 = ... ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
- 54. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp 11 12 ... 1 11 12 ... 1 A 21 A 22 ... B A n 2 21 B22 ... B p 2 AB = ... ... Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p 11 12 1 AB AB 21 22 . . . AB p 2 = ... ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
- 55. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp 11 12 ... 1 11 12 ... 1 A 21 A 22 ... B A n 2 21 B22 ... B p 2 AB = ... ... Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p 11 12 1 AB AB 21 22 . . . AB p 2 = ... ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
- 56. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp 11 12 ... 1 11 12 ... 1 A 21 A 22 ... B A n 2 21 B22 ... B p 2 AB = ... ... Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p 11 12 1 AB AB 21 22 . . . AB p 2 = ... ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
- 57. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp 11 12 ... 1 11 12 ... 1 A 21 A 22 ... B A n 2 21 B22 ... B p 2 AB = ... ... Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p 11 12 1 AB AB 21 22 . . . AB p 2 = ... ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
- 58. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp 11 12 ... 1 11 12 ... 1 A 21 A 22 ... B A n 2 21 B22 ... B p 2 AB = ... ... Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p 11 12 1 AB AB 21 22 . . . AB p 2 = ... ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
- 59. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp 11 12 ... 1 11 12 ... 1 A 21 A 22 ... B A n 2 21 B22 ... B p 2 AB = ... ... Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p 11 12 1 AB AB 21 22 . . . AB p 2 = ... ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
- 60. MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1 A12 A22 . . . An2 AT = ... A1m A2m . . . Anm CS221 Linear Algebra Review 10/11/2011 17 / 37
- 61. MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1 A12 A22 . . . An2 AT = ... A1m A2m . . . Anm Some simple properties : CS221 Linear Algebra Review 10/11/2011 17 / 37
- 62. MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1 A12 A22 . . . An2 AT = ... A1m A2m . . . Anm Some simple properties : T AT =A CS221 Linear Algebra Review 10/11/2011 17 / 37
- 63. MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1 A12 A22 . . . An2 AT = ... A1m A2m . . . Anm Some simple properties : T AT = A T (A + B) = AT + BT CS221 Linear Algebra Review 10/11/2011 17 / 37
- 64. MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1 A12 A22 . . . An2 AT = ... A1m A2m . . . Anm Some simple properties : T AT = A T (A + B) = AT + BT T (AB) = BT AT CS221 Linear Algebra Review 10/11/2011 17 / 37
- 65. MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1 A12 A22 . . . An2 AT = ... A1m A2m . . . Anm Some simple properties : T AT = A T (A + B) = AT + BT T (AB) = BT AT The inner product for vectors can be represented as u, v = uT v. CS221 Linear Algebra Review 10/11/2011 17 / 37
- 66. MatricesIdentity Operator 1 0 ... 0 0 1 . . . 0 I= ... 0 0 ... 1 CS221 Linear Algebra Review 10/11/2011 18 / 37
- 67. MatricesIdentity Operator 1 0 ... 0 0 1 . . . 0 I= ... 0 0 ... 1 For any matrix A, AI = A. CS221 Linear Algebra Review 10/11/2011 18 / 37
- 68. MatricesIdentity Operator 1 0 ... 0 0 1 . . . 0 I= ... 0 0 ... 1 For any matrix A, AI = A. I is the identity operator for the matrix product. CS221 Linear Algebra Review 10/11/2011 18 / 37
- 69. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. CS221 Linear Algebra Review 10/11/2011 19 / 37
- 70. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : CS221 Linear Algebra Review 10/11/2011 19 / 37
- 71. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. CS221 Linear Algebra Review 10/11/2011 19 / 37
- 72. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . CS221 Linear Algebra Review 10/11/2011 19 / 37
- 73. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : CS221 Linear Algebra Review 10/11/2011 19 / 37
- 74. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : Span of the rows of A. CS221 Linear Algebra Review 10/11/2011 19 / 37
- 75. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : Span of the rows of A. Linear subspace of Rn . CS221 Linear Algebra Review 10/11/2011 19 / 37
- 76. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : Span of the rows of A. Linear subspace of Rn . Important fact : The column and row spaces of any matrix have the same dimension. CS221 Linear Algebra Review 10/11/2011 19 / 37
- 77. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : Span of the rows of A. Linear subspace of Rn . Important fact : The column and row spaces of any matrix have the same dimension. This dimension is the rank of the matrix. CS221 Linear Algebra Review 10/11/2011 19 / 37
- 78. MatricesRange and Null space Let A be an m × n matrix. CS221 Linear Algebra Review 10/11/2011 20 / 37
- 79. MatricesRange and Null space Let A be an m × n matrix. Range : CS221 Linear Algebra Review 10/11/2011 20 / 37
- 80. MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . CS221 Linear Algebra Review 10/11/2011 20 / 37
- 81. MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . CS221 Linear Algebra Review 10/11/2011 20 / 37
- 82. MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . Null space : CS221 Linear Algebra Review 10/11/2011 20 / 37
- 83. MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . Null space : u belongs to the null space if Au = 0. CS221 Linear Algebra Review 10/11/2011 20 / 37
- 84. MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . Null space : u belongs to the null space if Au = 0. Subspace of Rn . CS221 Linear Algebra Review 10/11/2011 20 / 37
- 85. MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . Null space : u belongs to the null space if Au = 0. Subspace of Rn . Every vector in the null space is orthogonal to the rows of A. The null space and row space of a matrix are orthogonal. CS221 Linear Algebra Review 10/11/2011 20 / 37
- 86. MatricesRange and Column SpaceAnother interpretation of the matrix vector product (Ai is column i of A) : u 1 u 2 Au = A1 A2 ... An . . . un = u1 A1 + u2 A2 + · · · + un An CS221 Linear Algebra Review 10/11/2011 21 / 37
- 87. MatricesRange and Column SpaceAnother interpretation of the matrix vector product (Ai is column i of A) : u 1 u 2 Au = A1 A2 ... An . . . un = u1 A1 + u2 A2 + · · · + un An The result is a linear combination of the columns of A. CS221 Linear Algebra Review 10/11/2011 21 / 37
- 88. MatricesRange and Column SpaceAnother interpretation of the matrix vector product (Ai is column i of A) : u 1 u 2 Au = A1 A2 ... An . . . un = u1 A1 + u2 A2 + · · · + un An The result is a linear combination of the columns of A. For any matrix, the range is equal to the column space. CS221 Linear Algebra Review 10/11/2011 21 / 37
- 89. MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. CS221 Linear Algebra Review 10/11/2011 22 / 37
- 90. MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. CS221 Linear Algebra Review 10/11/2011 22 / 37
- 91. MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. In this case, the action of the matrix is invertible. CS221 Linear Algebra Review 10/11/2011 22 / 37
- 92. MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. In this case, the action of the matrix is invertible. The inversion is also linear and consequently can be represented by another matrix A−1 . CS221 Linear Algebra Review 10/11/2011 22 / 37
- 93. MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. In this case, the action of the matrix is invertible. The inversion is also linear and consequently can be represented by another matrix A−1 . A−1 is the only matrix such that A−1 A = AA−1 = I. CS221 Linear Algebra Review 10/11/2011 22 / 37
- 94. MatricesOrthogonal Matrices An orthogonal matrix U satises UT U = I. CS221 Linear Algebra Review 10/11/2011 23 / 37
- 95. MatricesOrthogonal Matrices An orthogonal matrix U satises UT U = I. Equivalently, U has orthonormal columns. CS221 Linear Algebra Review 10/11/2011 23 / 37
- 96. MatricesOrthogonal Matrices An orthogonal matrix U satises UT U = I. Equivalently, U has orthonormal columns. Applying an orthogonal matrix to two vectors does not change their inner product : Uu, Uv = (Uu) Uv T = uT UT Uv = uT v = u, v CS221 Linear Algebra Review 10/11/2011 23 / 37
- 97. MatricesOrthogonal Matrices An orthogonal matrix U satises UT U = I. Equivalently, U has orthonormal columns. Applying an orthogonal matrix to two vectors does not change their inner product : Uu, Uv = (Uu) Uv T = uT UT Uv = uT v = u, v Example : Matrices that represent rotations are orthogonal. CS221 Linear Algebra Review 10/11/2011 23 / 37
- 98. MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. CS221 Linear Algebra Review 10/11/2011 24 / 37
- 99. MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. The determinant of a square matrix A is denoted by |A|. CS221 Linear Algebra Review 10/11/2011 24 / 37
- 100. MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. The determinant of a square matrix A is denoted by |A|. ab For a 2 × 2 matrix A = c d : CS221 Linear Algebra Review 10/11/2011 24 / 37
- 101. MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. The determinant of a square matrix A is denoted by |A|. ab For a 2 × 2 matrix A = c d : |A| = ad − bc CS221 Linear Algebra Review 10/11/2011 24 / 37
- 102. MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. The determinant of a square matrix A is denoted by |A|. ab For a 2 × 2 matrix A = c d : |A| = ad − bc The absolute value of |A| is the area of the parallelogram given by the rows of A. CS221 Linear Algebra Review 10/11/2011 24 / 37
- 103. MatricesProperties of the Determinant The denition can be generalized to larger matrices. CS221 Linear Algebra Review 10/11/2011 25 / 37
- 104. MatricesProperties of the Determinant The denition can be generalized to larger matrices. |A| = AT . CS221 Linear Algebra Review 10/11/2011 25 / 37
- 105. MatricesProperties of the Determinant The denition can be generalized to larger matrices. |A| = AT . |AB| = |A| |B| CS221 Linear Algebra Review 10/11/2011 25 / 37
- 106. MatricesProperties of the Determinant The denition can be generalized to larger matrices. |A| = AT . |AB| = |A| |B| |A| = 0 if and only if A is not invertible. CS221 Linear Algebra Review 10/11/2011 25 / 37
- 107. MatricesProperties of the Determinant The denition can be generalized to larger matrices. |A| = AT . |AB| = |A| |B| |A| = 0 if and only if A is not invertible. If A is invertible, then A−1 = |A | . 1 CS221 Linear Algebra Review 10/11/2011 25 / 37
- 108. Matrix Decompositions1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 26 / 37
- 109. Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. CS221 Linear Algebra Review 10/11/2011 27 / 37
- 110. Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ > 1) or contracts (λ < 1) eigenvectors but does not rotate them. CS221 Linear Algebra Review 10/11/2011 27 / 37
- 111. Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ > 1) or contracts (λ < 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A − λI, which is consequently not invertible. CS221 Linear Algebra Review 10/11/2011 27 / 37
- 112. Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ > 1) or contracts (λ < 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A − λI, which is consequently not invertible. The eigenvalues of A are the roots of the equation |A − λI| = 0. CS221 Linear Algebra Review 10/11/2011 27 / 37
- 113. Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ > 1) or contracts (λ < 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A − λI, which is consequently not invertible. The eigenvalues of A are the roots of the equation |A − λI| = 0. Not the way eigenvalues are calculated in practice. CS221 Linear Algebra Review 10/11/2011 27 / 37
- 114. Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ > 1) or contracts (λ < 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A − λI, which is consequently not invertible. The eigenvalues of A are the roots of the equation |A − λI| = 0. Not the way eigenvalues are calculated in practice. Eigenvalues and eigenvectors can be complex valued, even if all the entries of A are real. CS221 Linear Algebra Review 10/11/2011 27 / 37
- 115. Matrix DecompositionsEigendecomposition of a Matrix Let A be an n × n square matrix with n linearly independent eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn . CS221 Linear Algebra Review 10/11/2011 28 / 37
- 116. Matrix DecompositionsEigendecomposition of a Matrix Let A be an n × n square matrix with n linearly independent eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn . We dene the matrices : P= p1 . . . pn λ1 0 . . . 0 0 λ2 . . . 0 Λ= ... 0 0 ... λn CS221 Linear Algebra Review 10/11/2011 28 / 37
- 117. Matrix DecompositionsEigendecomposition of a Matrix Let A be an n × n square matrix with n linearly independent eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn . We dene the matrices : P= p1 . . . pn λ1 0 . . . 0 0 λ2 . . . 0 Λ= ... 0 0 ... λn A satises AP = PΛ. CS221 Linear Algebra Review 10/11/2011 28 / 37
- 118. Matrix DecompositionsEigendecomposition of a Matrix Let A be an n × n square matrix with n linearly independent eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn . We dene the matrices : P= p1 . . . pn λ1 0 . . . 0 0 λ2 . . . 0 Λ= ... 0 0 ... λn A satises AP = PΛ. P is full rank. Inverting it yields the eigendecomposition of A : A = PΛ P− 1 CS221 Linear Algebra Review 10/11/2011 28 / 37
- 119. Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 CS221 Linear Algebra Review 10/11/2011 29 / 37
- 120. Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i CS221 Linear Algebra Review 10/11/2011 29 / 37
- 121. Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i |A| = |Λ| = n=1 λi . i CS221 Linear Algebra Review 10/11/2011 29 / 37
- 122. Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i |A| = |Λ| = n=1 λi . i The rank of A is equal to the number of nonzero eigenvalues. CS221 Linear Algebra Review 10/11/2011 29 / 37
- 123. Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i |A| = |Λ| = n=1 λi . i The rank of A is equal to the number of nonzero eigenvalues. If λ is a nonzero eigenvalue of A, λ is an eigenvalue of A−1 with the 1 same eigenvector. CS221 Linear Algebra Review 10/11/2011 29 / 37
- 124. Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i |A| = |Λ| = n=1 λi . i The rank of A is equal to the number of nonzero eigenvalues. If λ is a nonzero eigenvalue of A, λ is an eigenvalue of A−1 with the 1 same eigenvector. The eigendecomposition makes it possible to compute matrix powers eciently : m Am = PΛP−1 = PΛP−1 PΛP−1 . . . PΛP−1 = PΛm P−1 . CS221 Linear Algebra Review 10/11/2011 29 / 37
- 125. Matrix DecompositionsMatlab Example CS221 Linear Algebra Review 10/11/2011 30 / 37
- 126. Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. CS221 Linear Algebra Review 10/11/2011 31 / 37
- 127. Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. CS221 Linear Algebra Review 10/11/2011 31 / 37
- 128. Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. The eigenvectors of symmetric matrices are orthonormal. CS221 Linear Algebra Review 10/11/2011 31 / 37
- 129. Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. The eigenvectors of symmetric matrices are orthonormal. Consequently, the eigendecomposition becomes : A = UΛUT for Λ real and U orthogonal. CS221 Linear Algebra Review 10/11/2011 31 / 37
- 130. Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. The eigenvectors of symmetric matrices are orthonormal. Consequently, the eigendecomposition becomes : A = UΛUT for Λ real and U orthogonal. The eigenvectors of A are an orthonormal basis for the column space (and the row space, since they are equal). CS221 Linear Algebra Review 10/11/2011 31 / 37
- 131. Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : CS221 Linear Algebra Review 10/11/2011 32 / 37
- 132. Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). CS221 Linear Algebra Review 10/11/2011 32 / 37
- 133. Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). 2 Scaling of each coecient Ui , u by the corresponding eigenvalue (multiplication by Λ). CS221 Linear Algebra Review 10/11/2011 32 / 37
- 134. Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). 2 Scaling of each coecient Ui , u by the corresponding eigenvalue (multiplication by Λ). 3 Linear combination of the eigenvectors scaled by the resulting coecient (multiplication by U). CS221 Linear Algebra Review 10/11/2011 32 / 37
- 135. Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). 2 Scaling of each coecient Ui , u by the corresponding eigenvalue (multiplication by Λ). 3 Linear combination of the eigenvectors scaled by the resulting coecient (multiplication by U). Summarizing : n Au = λi Ui , u Ui i =1 CS221 Linear Algebra Review 10/11/2011 32 / 37
- 136. Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). 2 Scaling of each coecient Ui , u by the corresponding eigenvalue (multiplication by Λ). 3 Linear combination of the eigenvectors scaled by the resulting coecient (multiplication by U). Summarizing : n Au = λi Ui , u Ui i =1 It would be great to generalize this to all matrices... CS221 Linear Algebra Review 10/11/2011 32 / 37
- 137. Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT CS221 Linear Algebra Review 10/11/2011 33 / 37
- 138. Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT The columns of U are an orthonormal basis for the column space of A. CS221 Linear Algebra Review 10/11/2011 33 / 37
- 139. Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT The columns of U are an orthonormal basis for the column space of A. The columns of V are an orthonormal basis for the row space of A. CS221 Linear Algebra Review 10/11/2011 33 / 37
- 140. Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT The columns of U are an orthonormal basis for the column space of A. The columns of V are an orthonormal basis for the row space of A. Σ is diagonal. The entries on its diagonal σi = Σii are positive real numbers, called the singular values of A. CS221 Linear Algebra Review 10/11/2011 33 / 37
- 141. Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT The columns of U are an orthonormal basis for the column space of A. The columns of V are an orthonormal basis for the row space of A. Σ is diagonal. The entries on its diagonal σi = Σii are positive real numbers, called the singular values of A. The action of A on a vector u can be decomposed into : n Au = σi Vi , u Ui i =1 CS221 Linear Algebra Review 10/11/2011 33 / 37
- 142. Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT CS221 Linear Algebra Review 10/11/2011 34 / 37
- 143. Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT The rank of a matrix is equal to the number of nonzero singular values. CS221 Linear Algebra Review 10/11/2011 34 / 37
- 144. Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT The rank of a matrix is equal to the number of nonzero singular values. The largest singular value σ1 is the solution to the optimization problem : ||Ax||2 σ1 = max x=0 ||x||2 CS221 Linear Algebra Review 10/11/2011 34 / 37
- 145. Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT The rank of a matrix is equal to the number of nonzero singular values. The largest singular value σ1 is the solution to the optimization problem : ||Ax||2 σ1 = max x=0 ||x||2 It can be veried that the largest singular value satises the properties of a norm, it is called the spectral norm of the matrix. CS221 Linear Algebra Review 10/11/2011 34 / 37
- 146. Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT The rank of a matrix is equal to the number of nonzero singular values. The largest singular value σ1 is the solution to the optimization problem : ||Ax||2 σ1 = max x=0 ||x||2 It can be veried that the largest singular value satises the properties of a norm, it is called the spectral norm of the matrix. In statistics analyzing data with the singular value decomposition is called Principal Component Analysis. CS221 Linear Algebra Review 10/11/2011 34 / 37
- 147. Application : Image Compression1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 35 / 37
- 148. Application : Image CompressionCompression using the Singular Value Decomposition The singular value decomposition can be viewed as a sum of rank 1 matrices : CS221 Linear Algebra Review 10/11/2011 36 / 37
- 149. Application : Image CompressionCompression using the Singular Value Decomposition The singular value decomposition can be viewed as a sum of rank 1 matrices : If some of the singular values are very small, we can discard them. CS221 Linear Algebra Review 10/11/2011 36 / 37
- 150. Application : Image CompressionCompression using the Singular Value Decomposition The singular value decomposition can be viewed as a sum of rank 1 matrices : If some of the singular values are very small, we can discard them. This is a form of lossy compression. CS221 Linear Algebra Review 10/11/2011 36 / 37
- 151. Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. CS221 Linear Algebra Review 10/11/2011 37 / 37
- 152. Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : CS221 Linear Algebra Review 10/11/2011 37 / 37
- 153. Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : Full representation : 512 × 512 =262 144 CS221 Linear Algebra Review 10/11/2011 37 / 37
- 154. Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : Full representation : 512 × 512 =262 144 Rank 10 approximation : 512 × 10 + 10 + 512 × 10 =10 250 CS221 Linear Algebra Review 10/11/2011 37 / 37
- 155. Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : Full representation : 512 × 512 =262 144 Rank 10 approximation : 512 × 10 + 10 + 512 × 10 =10 250 Rank 40 approximation : 512 × 40 + 40 + 512 × 40 =41 000 CS221 Linear Algebra Review 10/11/2011 37 / 37
- 156. Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : Full representation : 512 × 512 =262 144 Rank 10 approximation : 512 × 10 + 10 + 512 × 10 =10 250 Rank 40 approximation : 512 × 40 + 40 + 512 × 40 =41 000 Rank 80 approximation : 512 × 80 + 80 + 512 × 80 =82 000 CS221 Linear Algebra Review 10/11/2011 37 / 37

Full NameComment goes here.Noah Fallatah, Field Engineer at Saudi Aramco 14 hours agorafaelvf1 month agoSaffrony Institute Of Technologyat Saffrony Institute Of Technology 2 months agoHan Xu, Researcher at UNSW (The University of New South Wales) 1 year ago