Cs221 linear algebra

2,119 views

Published on

Published in: Technology, Education

Cs221 linear algebra

  1. 1. Linear Algebra Review CS221: Introduction to Articial Intelligence Carlos Fernandez-Granda 10/11/2011CS221 Linear Algebra Review 10/11/2011 1 / 37
  2. 2. Index1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 2 / 37
  3. 3. Vectors1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 3 / 37
  4. 4. VectorsWhat is a vector ? CS221 Linear Algebra Review 10/11/2011 4 / 37
  5. 5. VectorsWhat is a vector ? An ordered tuple of numbers : u   1 u 2  u=  . . .  un CS221 Linear Algebra Review 10/11/2011 4 / 37
  6. 6. VectorsWhat is a vector ? An ordered tuple of numbers : u   1 u 2  u=  . . .  un A quantity that has a magnitude and a direction. CS221 Linear Algebra Review 10/11/2011 4 / 37
  7. 7. VectorsVector spaces More formally a vector is an element of a vector space. CS221 Linear Algebra Review 10/11/2011 5 / 37
  8. 8. VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : CS221 Linear Algebra Review 10/11/2011 5 / 37
  9. 9. VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : If vectors u and v ∈ V , then u + v ∈ V . CS221 Linear Algebra Review 10/11/2011 5 / 37
  10. 10. VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : If vectors u and v ∈ V , then u + v ∈ V . If vector u ∈ V , then α u ∈ V for any scalar α. CS221 Linear Algebra Review 10/11/2011 5 / 37
  11. 11. VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : If vectors u and v ∈ V , then u + v ∈ V . If vector u ∈ V , then α u ∈ V for any scalar α. There exists 0 ∈ V such that u + 0 = u for any u ∈ V . CS221 Linear Algebra Review 10/11/2011 5 / 37
  12. 12. VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : If vectors u and v ∈ V , then u + v ∈ V . If vector u ∈ V , then α u ∈ V for any scalar α. There exists 0 ∈ V such that u + 0 = u for any u ∈ V . A subspace is a subset of a vector space that is also a vector space. CS221 Linear Algebra Review 10/11/2011 5 / 37
  13. 13. VectorsExamples Euclidean space Rn . CS221 Linear Algebra Review 10/11/2011 6 / 37
  14. 14. VectorsExamples Euclidean space Rn . The span of any set of vectors {u1 , u2 , . . . , un }, dened as : span (u1 , u2 , . . . , un ) = {α1 u1 + α2 u2 + · · · + αn un | αi ∈ R} CS221 Linear Algebra Review 10/11/2011 6 / 37
  15. 15. VectorsSubspaces A line through the origin in Rn (span of a vector). CS221 Linear Algebra Review 10/11/2011 7 / 37
  16. 16. VectorsSubspaces A line through the origin in Rn (span of a vector). A plane in Rn (span of two vectors). CS221 Linear Algebra Review 10/11/2011 7 / 37
  17. 17. VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. CS221 Linear Algebra Review 10/11/2011 8 / 37
  18. 18. VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. CS221 Linear Algebra Review 10/11/2011 8 / 37
  19. 19. VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . CS221 Linear Algebra Review 10/11/2011 8 / 37
  20. 20. VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . If a vector space has a basis with d vectors, its dimension is d. CS221 Linear Algebra Review 10/11/2011 8 / 37
  21. 21. VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . If a vector space has a basis with d vectors, its dimension is d. Any other basis of the same space will consist of exactly d vectors. CS221 Linear Algebra Review 10/11/2011 8 / 37
  22. 22. VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . If a vector space has a basis with d vectors, its dimension is d. Any other basis of the same space will consist of exactly d vectors. The dimension of a vector space can be innite (function spaces). CS221 Linear Algebra Review 10/11/2011 8 / 37
  23. 23. VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. CS221 Linear Algebra Review 10/11/2011 9 / 37
  24. 24. VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : CS221 Linear Algebra Review 10/11/2011 9 / 37
  25. 25. VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : Homogeneity ||α u|| = |α| ||u|| CS221 Linear Algebra Review 10/11/2011 9 / 37
  26. 26. VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : Homogeneity ||α u|| = |α| ||u|| Triangle inequality ||u + v|| ≤ ||u|| + ||v|| CS221 Linear Algebra Review 10/11/2011 9 / 37
  27. 27. VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : Homogeneity ||α u|| = |α| ||u|| Triangle inequality ||u + v|| ≤ ||u|| + ||v|| Point separation ||u|| = 0 if and only if u = 0. CS221 Linear Algebra Review 10/11/2011 9 / 37
  28. 28. VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : Homogeneity ||α u|| = |α| ||u|| Triangle inequality ||u + v|| ≤ ||u|| + ||v|| Point separation ||u|| = 0 if and only if u = 0. Examples : Manhattan or 1 norm : ||u||1 = i |ui |. Euclidean or 2 norm : ||u||2 = i ui . 2 CS221 Linear Algebra Review 10/11/2011 9 / 37
  29. 29. VectorsInner Product Inner product between u and v : u, v = n uv. i =1 i i CS221 Linear Algebra Review 10/11/2011 10 / 37
  30. 30. VectorsInner Product Inner product between u and v : u, v = n=1 ui vi . i It is the projection of one vector onto the other. CS221 Linear Algebra Review 10/11/2011 10 / 37
  31. 31. VectorsInner Product Inner product between u and v : u, v = n=1 ui vi . i It is the projection of one vector onto the other. Related to the Euclidean norm : u, u = ||u||2 . 2 CS221 Linear Algebra Review 10/11/2011 10 / 37
  32. 32. VectorsInner ProductThe inner product is a measure of correlation between two vectors, scaledby the norms of the vectors : CS221 Linear Algebra Review 10/11/2011 11 / 37
  33. 33. VectorsInner ProductThe inner product is a measure of correlation between two vectors, scaledby the norms of the vectors : If u, v 0, u and v are aligned. CS221 Linear Algebra Review 10/11/2011 11 / 37
  34. 34. VectorsInner ProductThe inner product is a measure of correlation between two vectors, scaledby the norms of the vectors : If u, v 0, u and v are aligned. If u, v 0, u and v are opposed. CS221 Linear Algebra Review 10/11/2011 11 / 37
  35. 35. VectorsInner ProductThe inner product is a measure of correlation between two vectors, scaledby the norms of the vectors : If u, v 0, u and v are aligned. If u, v 0, u and v are opposed. If u, v = 0, u and v are orthogonal. CS221 Linear Algebra Review 10/11/2011 11 / 37
  36. 36. VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. CS221 Linear Algebra Review 10/11/2011 12 / 37
  37. 37. VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. CS221 Linear Algebra Review 10/11/2011 12 / 37
  38. 38. VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. Example : x = α1 b1 + α2 b2 . CS221 Linear Algebra Review 10/11/2011 12 / 37
  39. 39. VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. Example : x = α1 b1 + α2 b2 . We compute x, b1 : x, b1 = α1 b1 + α2 b2 , b1 = α1 b1 , b1 + α2 b2 , b1 By linearity. = α1 + 0 By orthonormality. CS221 Linear Algebra Review 10/11/2011 12 / 37
  40. 40. VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. Example : x = α1 b1 + α2 b2 . We compute x, b1 : x, b1 = α1 b1 + α2 b2 , b1 = α1 b1 , b1 + α2 b2 , b1 By linearity. = α1 + 0 By orthonormality. Likewise, α2 = x, b2 . CS221 Linear Algebra Review 10/11/2011 12 / 37
  41. 41. Matrices1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 13 / 37
  42. 42. MatricesLinear Operator A linear operator L : U → V is a map from a vector space U to another vector space V that satises : CS221 Linear Algebra Review 10/11/2011 14 / 37
  43. 43. MatricesLinear Operator A linear operator L : U → V is a map from a vector space U to another vector space V that satises : L (u1 + u2 ) = L (u1 ) + L (u2 ) . CS221 Linear Algebra Review 10/11/2011 14 / 37
  44. 44. MatricesLinear Operator A linear operator L : U → V is a map from a vector space U to another vector space V that satises : . L (u1 + u2 ) = L (u1 ) + L (u2 ) L (α u) = α L (u) for any scalar α. CS221 Linear Algebra Review 10/11/2011 14 / 37
  45. 45. MatricesLinear Operator A linear operator L : U → V is a map from a vector space U to another vector space V that satises : . L (u1 + u2 ) = L (u1 ) + L (u2 ) L (α u) = α L (u) for any scalar α. If the dimension n of U and m of V are nite, L can be represented by an m × n matrix A. A A An   11 12 ... 1 A 21 A 22 ... A n 2 A=   ...  Am 1 Am 2 ... Amn CS221 Linear Algebra Review 10/11/2011 14 / 37
  46. 46. MatricesMatrix Vector Multiplication A A An u v      11 12 ... 1 1 1 A 21 A 22 ... A nu 2 2  v 2   Au =   =  ...  . . .  . . .  Am 1 Am 2 ... Amn un vm CS221 Linear Algebra Review 10/11/2011 15 / 37
  47. 47. MatricesMatrix Vector Multiplication A A An u v      11 12 ... 1 1 1 A 21 A 22 ... A nu 2 2  v 2   Au =   =  ...  . . .  . . .  Am 1 Am 2 ... Amn un vm CS221 Linear Algebra Review 10/11/2011 15 / 37
  48. 48. MatricesMatrix Vector Multiplication A A An u v      11 12 ... 1 1 1 A 21 A 22 ... A nu 2 2  v 2   Au =   =  ...  . . .  . . .  Am 1 Am 2 ... Amn un vm CS221 Linear Algebra Review 10/11/2011 15 / 37
  49. 49. MatricesMatrix Vector Multiplication A A An u v      11 12 ... 1 1 1 A 21 A 22 ... A nu 2 2  v 2   Au =   =  ...  . . .  . . .  Am 1 Am 2 ... Amn un vm CS221 Linear Algebra Review 10/11/2011 15 / 37
  50. 50. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
  51. 51. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
  52. 52. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
  53. 53. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
  54. 54. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
  55. 55. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
  56. 56. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
  57. 57. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
  58. 58. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
  59. 59. MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
  60. 60. MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1    A12 A22 . . . An2  AT =    ...  A1m A2m . . . Anm CS221 Linear Algebra Review 10/11/2011 17 / 37
  61. 61. MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1    A12 A22 . . . An2  AT =    ...  A1m A2m . . . Anm Some simple properties : CS221 Linear Algebra Review 10/11/2011 17 / 37
  62. 62. MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1    A12 A22 . . . An2  AT =    ...  A1m A2m . . . Anm Some simple properties : T AT =A CS221 Linear Algebra Review 10/11/2011 17 / 37
  63. 63. MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1    A12 A22 . . . An2  AT =    ...  A1m A2m . . . Anm Some simple properties : T AT = A T (A + B) = AT + BT CS221 Linear Algebra Review 10/11/2011 17 / 37
  64. 64. MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1    A12 A22 . . . An2  AT =    ...  A1m A2m . . . Anm Some simple properties : T AT = A T (A + B) = AT + BT T (AB) = BT AT CS221 Linear Algebra Review 10/11/2011 17 / 37
  65. 65. MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1    A12 A22 . . . An2  AT =    ...  A1m A2m . . . Anm Some simple properties : T AT = A T (A + B) = AT + BT T (AB) = BT AT The inner product for vectors can be represented as u, v = uT v. CS221 Linear Algebra Review 10/11/2011 17 / 37
  66. 66. MatricesIdentity Operator 1 0 ... 0   0 1 . . . 0 I=   ...  0 0 ... 1 CS221 Linear Algebra Review 10/11/2011 18 / 37
  67. 67. MatricesIdentity Operator 1 0 ... 0   0 1 . . . 0 I=   ...  0 0 ... 1 For any matrix A, AI = A. CS221 Linear Algebra Review 10/11/2011 18 / 37
  68. 68. MatricesIdentity Operator 1 0 ... 0   0 1 . . . 0 I=   ...  0 0 ... 1 For any matrix A, AI = A. I is the identity operator for the matrix product. CS221 Linear Algebra Review 10/11/2011 18 / 37
  69. 69. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. CS221 Linear Algebra Review 10/11/2011 19 / 37
  70. 70. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : CS221 Linear Algebra Review 10/11/2011 19 / 37
  71. 71. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. CS221 Linear Algebra Review 10/11/2011 19 / 37
  72. 72. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . CS221 Linear Algebra Review 10/11/2011 19 / 37
  73. 73. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : CS221 Linear Algebra Review 10/11/2011 19 / 37
  74. 74. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : Span of the rows of A. CS221 Linear Algebra Review 10/11/2011 19 / 37
  75. 75. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : Span of the rows of A. Linear subspace of Rn . CS221 Linear Algebra Review 10/11/2011 19 / 37
  76. 76. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : Span of the rows of A. Linear subspace of Rn . Important fact : The column and row spaces of any matrix have the same dimension. CS221 Linear Algebra Review 10/11/2011 19 / 37
  77. 77. MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : Span of the rows of A. Linear subspace of Rn . Important fact : The column and row spaces of any matrix have the same dimension. This dimension is the rank of the matrix. CS221 Linear Algebra Review 10/11/2011 19 / 37
  78. 78. MatricesRange and Null space Let A be an m × n matrix. CS221 Linear Algebra Review 10/11/2011 20 / 37
  79. 79. MatricesRange and Null space Let A be an m × n matrix. Range : CS221 Linear Algebra Review 10/11/2011 20 / 37
  80. 80. MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . CS221 Linear Algebra Review 10/11/2011 20 / 37
  81. 81. MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . CS221 Linear Algebra Review 10/11/2011 20 / 37
  82. 82. MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . Null space : CS221 Linear Algebra Review 10/11/2011 20 / 37
  83. 83. MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . Null space : u belongs to the null space if Au = 0. CS221 Linear Algebra Review 10/11/2011 20 / 37
  84. 84. MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . Null space : u belongs to the null space if Au = 0. Subspace of Rn . CS221 Linear Algebra Review 10/11/2011 20 / 37
  85. 85. MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . Null space : u belongs to the null space if Au = 0. Subspace of Rn . Every vector in the null space is orthogonal to the rows of A. The null space and row space of a matrix are orthogonal. CS221 Linear Algebra Review 10/11/2011 20 / 37
  86. 86. MatricesRange and Column SpaceAnother interpretation of the matrix vector product (Ai is column i of A) : u   1 u 2   Au = A1 A2 ... An  . . .  un = u1 A1 + u2 A2 + · · · + un An CS221 Linear Algebra Review 10/11/2011 21 / 37
  87. 87. MatricesRange and Column SpaceAnother interpretation of the matrix vector product (Ai is column i of A) : u   1 u 2   Au = A1 A2 ... An  . . .  un = u1 A1 + u2 A2 + · · · + un An The result is a linear combination of the columns of A. CS221 Linear Algebra Review 10/11/2011 21 / 37
  88. 88. MatricesRange and Column SpaceAnother interpretation of the matrix vector product (Ai is column i of A) : u   1 u 2   Au = A1 A2 ... An  . . .  un = u1 A1 + u2 A2 + · · · + un An The result is a linear combination of the columns of A. For any matrix, the range is equal to the column space. CS221 Linear Algebra Review 10/11/2011 21 / 37
  89. 89. MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. CS221 Linear Algebra Review 10/11/2011 22 / 37
  90. 90. MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. CS221 Linear Algebra Review 10/11/2011 22 / 37
  91. 91. MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. In this case, the action of the matrix is invertible. CS221 Linear Algebra Review 10/11/2011 22 / 37
  92. 92. MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. In this case, the action of the matrix is invertible. The inversion is also linear and consequently can be represented by another matrix A−1 . CS221 Linear Algebra Review 10/11/2011 22 / 37
  93. 93. MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. In this case, the action of the matrix is invertible. The inversion is also linear and consequently can be represented by another matrix A−1 . A−1 is the only matrix such that A−1 A = AA−1 = I. CS221 Linear Algebra Review 10/11/2011 22 / 37
  94. 94. MatricesOrthogonal Matrices An orthogonal matrix U satises UT U = I. CS221 Linear Algebra Review 10/11/2011 23 / 37
  95. 95. MatricesOrthogonal Matrices An orthogonal matrix U satises UT U = I. Equivalently, U has orthonormal columns. CS221 Linear Algebra Review 10/11/2011 23 / 37
  96. 96. MatricesOrthogonal Matrices An orthogonal matrix U satises UT U = I. Equivalently, U has orthonormal columns. Applying an orthogonal matrix to two vectors does not change their inner product : Uu, Uv = (Uu) Uv T = uT UT Uv = uT v = u, v CS221 Linear Algebra Review 10/11/2011 23 / 37
  97. 97. MatricesOrthogonal Matrices An orthogonal matrix U satises UT U = I. Equivalently, U has orthonormal columns. Applying an orthogonal matrix to two vectors does not change their inner product : Uu, Uv = (Uu) Uv T = uT UT Uv = uT v = u, v Example : Matrices that represent rotations are orthogonal. CS221 Linear Algebra Review 10/11/2011 23 / 37
  98. 98. MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. CS221 Linear Algebra Review 10/11/2011 24 / 37
  99. 99. MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. The determinant of a square matrix A is denoted by |A|. CS221 Linear Algebra Review 10/11/2011 24 / 37
  100. 100. MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. The determinant of a square matrix A is denoted by |A|. ab For a 2 × 2 matrix A = c d : CS221 Linear Algebra Review 10/11/2011 24 / 37
  101. 101. MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. The determinant of a square matrix A is denoted by |A|. ab For a 2 × 2 matrix A = c d : |A| = ad − bc CS221 Linear Algebra Review 10/11/2011 24 / 37
  102. 102. MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. The determinant of a square matrix A is denoted by |A|. ab For a 2 × 2 matrix A = c d : |A| = ad − bc The absolute value of |A| is the area of the parallelogram given by the rows of A. CS221 Linear Algebra Review 10/11/2011 24 / 37
  103. 103. MatricesProperties of the Determinant The denition can be generalized to larger matrices. CS221 Linear Algebra Review 10/11/2011 25 / 37
  104. 104. MatricesProperties of the Determinant The denition can be generalized to larger matrices. |A| = AT . CS221 Linear Algebra Review 10/11/2011 25 / 37
  105. 105. MatricesProperties of the Determinant The denition can be generalized to larger matrices. |A| = AT . |AB| = |A| |B| CS221 Linear Algebra Review 10/11/2011 25 / 37
  106. 106. MatricesProperties of the Determinant The denition can be generalized to larger matrices. |A| = AT . |AB| = |A| |B| |A| = 0 if and only if A is not invertible. CS221 Linear Algebra Review 10/11/2011 25 / 37
  107. 107. MatricesProperties of the Determinant The denition can be generalized to larger matrices. |A| = AT . |AB| = |A| |B| |A| = 0 if and only if A is not invertible. If A is invertible, then A−1 = |A | . 1 CS221 Linear Algebra Review 10/11/2011 25 / 37
  108. 108. Matrix Decompositions1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 26 / 37
  109. 109. Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. CS221 Linear Algebra Review 10/11/2011 27 / 37
  110. 110. Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ 1) or contracts (λ 1) eigenvectors but does not rotate them. CS221 Linear Algebra Review 10/11/2011 27 / 37
  111. 111. Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ 1) or contracts (λ 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A − λI, which is consequently not invertible. CS221 Linear Algebra Review 10/11/2011 27 / 37
  112. 112. Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ 1) or contracts (λ 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A − λI, which is consequently not invertible. The eigenvalues of A are the roots of the equation |A − λI| = 0. CS221 Linear Algebra Review 10/11/2011 27 / 37
  113. 113. Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ 1) or contracts (λ 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A − λI, which is consequently not invertible. The eigenvalues of A are the roots of the equation |A − λI| = 0. Not the way eigenvalues are calculated in practice. CS221 Linear Algebra Review 10/11/2011 27 / 37
  114. 114. Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ 1) or contracts (λ 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A − λI, which is consequently not invertible. The eigenvalues of A are the roots of the equation |A − λI| = 0. Not the way eigenvalues are calculated in practice. Eigenvalues and eigenvectors can be complex valued, even if all the entries of A are real. CS221 Linear Algebra Review 10/11/2011 27 / 37
  115. 115. Matrix DecompositionsEigendecomposition of a Matrix Let A be an n × n square matrix with n linearly independent eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn . CS221 Linear Algebra Review 10/11/2011 28 / 37
  116. 116. Matrix DecompositionsEigendecomposition of a Matrix Let A be an n × n square matrix with n linearly independent eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn . We dene the matrices : P= p1 . . . pn λ1 0 . . . 0    0 λ2 . . . 0 Λ=   ...  0 0 ... λn CS221 Linear Algebra Review 10/11/2011 28 / 37
  117. 117. Matrix DecompositionsEigendecomposition of a Matrix Let A be an n × n square matrix with n linearly independent eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn . We dene the matrices : P= p1 . . . pn λ1 0 . . . 0    0 λ2 . . . 0 Λ=   ...  0 0 ... λn A satises AP = PΛ. CS221 Linear Algebra Review 10/11/2011 28 / 37
  118. 118. Matrix DecompositionsEigendecomposition of a Matrix Let A be an n × n square matrix with n linearly independent eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn . We dene the matrices : P= p1 . . . pn λ1 0 . . . 0    0 λ2 . . . 0 Λ=   ...  0 0 ... λn A satises AP = PΛ. P is full rank. Inverting it yields the eigendecomposition of A : A = PΛ P− 1 CS221 Linear Algebra Review 10/11/2011 28 / 37
  119. 119. Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 CS221 Linear Algebra Review 10/11/2011 29 / 37
  120. 120. Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i CS221 Linear Algebra Review 10/11/2011 29 / 37
  121. 121. Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i |A| = |Λ| = n=1 λi . i CS221 Linear Algebra Review 10/11/2011 29 / 37
  122. 122. Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i |A| = |Λ| = n=1 λi . i The rank of A is equal to the number of nonzero eigenvalues. CS221 Linear Algebra Review 10/11/2011 29 / 37
  123. 123. Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i |A| = |Λ| = n=1 λi . i The rank of A is equal to the number of nonzero eigenvalues. If λ is a nonzero eigenvalue of A, λ is an eigenvalue of A−1 with the 1 same eigenvector. CS221 Linear Algebra Review 10/11/2011 29 / 37
  124. 124. Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i |A| = |Λ| = n=1 λi . i The rank of A is equal to the number of nonzero eigenvalues. If λ is a nonzero eigenvalue of A, λ is an eigenvalue of A−1 with the 1 same eigenvector. The eigendecomposition makes it possible to compute matrix powers eciently : m Am = PΛP−1 = PΛP−1 PΛP−1 . . . PΛP−1 = PΛm P−1 . CS221 Linear Algebra Review 10/11/2011 29 / 37
  125. 125. Matrix DecompositionsMatlab Example CS221 Linear Algebra Review 10/11/2011 30 / 37
  126. 126. Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. CS221 Linear Algebra Review 10/11/2011 31 / 37
  127. 127. Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. CS221 Linear Algebra Review 10/11/2011 31 / 37
  128. 128. Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. The eigenvectors of symmetric matrices are orthonormal. CS221 Linear Algebra Review 10/11/2011 31 / 37
  129. 129. Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. The eigenvectors of symmetric matrices are orthonormal. Consequently, the eigendecomposition becomes : A = UΛUT for Λ real and U orthogonal. CS221 Linear Algebra Review 10/11/2011 31 / 37
  130. 130. Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. The eigenvectors of symmetric matrices are orthonormal. Consequently, the eigendecomposition becomes : A = UΛUT for Λ real and U orthogonal. The eigenvectors of A are an orthonormal basis for the column space (and the row space, since they are equal). CS221 Linear Algebra Review 10/11/2011 31 / 37
  131. 131. Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : CS221 Linear Algebra Review 10/11/2011 32 / 37
  132. 132. Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). CS221 Linear Algebra Review 10/11/2011 32 / 37
  133. 133. Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). 2 Scaling of each coecient Ui , u by the corresponding eigenvalue (multiplication by Λ). CS221 Linear Algebra Review 10/11/2011 32 / 37
  134. 134. Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). 2 Scaling of each coecient Ui , u by the corresponding eigenvalue (multiplication by Λ). 3 Linear combination of the eigenvectors scaled by the resulting coecient (multiplication by U). CS221 Linear Algebra Review 10/11/2011 32 / 37
  135. 135. Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). 2 Scaling of each coecient Ui , u by the corresponding eigenvalue (multiplication by Λ). 3 Linear combination of the eigenvectors scaled by the resulting coecient (multiplication by U). Summarizing : n Au = λi Ui , u Ui i =1 CS221 Linear Algebra Review 10/11/2011 32 / 37
  136. 136. Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). 2 Scaling of each coecient Ui , u by the corresponding eigenvalue (multiplication by Λ). 3 Linear combination of the eigenvectors scaled by the resulting coecient (multiplication by U). Summarizing : n Au = λi Ui , u Ui i =1 It would be great to generalize this to all matrices... CS221 Linear Algebra Review 10/11/2011 32 / 37
  137. 137. Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT CS221 Linear Algebra Review 10/11/2011 33 / 37
  138. 138. Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT The columns of U are an orthonormal basis for the column space of A. CS221 Linear Algebra Review 10/11/2011 33 / 37
  139. 139. Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT The columns of U are an orthonormal basis for the column space of A. The columns of V are an orthonormal basis for the row space of A. CS221 Linear Algebra Review 10/11/2011 33 / 37
  140. 140. Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT The columns of U are an orthonormal basis for the column space of A. The columns of V are an orthonormal basis for the row space of A. Σ is diagonal. The entries on its diagonal σi = Σii are positive real numbers, called the singular values of A. CS221 Linear Algebra Review 10/11/2011 33 / 37
  141. 141. Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT The columns of U are an orthonormal basis for the column space of A. The columns of V are an orthonormal basis for the row space of A. Σ is diagonal. The entries on its diagonal σi = Σii are positive real numbers, called the singular values of A. The action of A on a vector u can be decomposed into : n Au = σi Vi , u Ui i =1 CS221 Linear Algebra Review 10/11/2011 33 / 37
  142. 142. Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT CS221 Linear Algebra Review 10/11/2011 34 / 37
  143. 143. Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT The rank of a matrix is equal to the number of nonzero singular values. CS221 Linear Algebra Review 10/11/2011 34 / 37
  144. 144. Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT The rank of a matrix is equal to the number of nonzero singular values. The largest singular value σ1 is the solution to the optimization problem : ||Ax||2 σ1 = max x=0 ||x||2 CS221 Linear Algebra Review 10/11/2011 34 / 37
  145. 145. Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT The rank of a matrix is equal to the number of nonzero singular values. The largest singular value σ1 is the solution to the optimization problem : ||Ax||2 σ1 = max x=0 ||x||2 It can be veried that the largest singular value satises the properties of a norm, it is called the spectral norm of the matrix. CS221 Linear Algebra Review 10/11/2011 34 / 37
  146. 146. Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT The rank of a matrix is equal to the number of nonzero singular values. The largest singular value σ1 is the solution to the optimization problem : ||Ax||2 σ1 = max x=0 ||x||2 It can be veried that the largest singular value satises the properties of a norm, it is called the spectral norm of the matrix. In statistics analyzing data with the singular value decomposition is called Principal Component Analysis. CS221 Linear Algebra Review 10/11/2011 34 / 37
  147. 147. Application : Image Compression1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 35 / 37
  148. 148. Application : Image CompressionCompression using the Singular Value Decomposition The singular value decomposition can be viewed as a sum of rank 1 matrices : CS221 Linear Algebra Review 10/11/2011 36 / 37
  149. 149. Application : Image CompressionCompression using the Singular Value Decomposition The singular value decomposition can be viewed as a sum of rank 1 matrices : If some of the singular values are very small, we can discard them. CS221 Linear Algebra Review 10/11/2011 36 / 37
  150. 150. Application : Image CompressionCompression using the Singular Value Decomposition The singular value decomposition can be viewed as a sum of rank 1 matrices : If some of the singular values are very small, we can discard them. This is a form of lossy compression. CS221 Linear Algebra Review 10/11/2011 36 / 37
  151. 151. Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. CS221 Linear Algebra Review 10/11/2011 37 / 37
  152. 152. Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : CS221 Linear Algebra Review 10/11/2011 37 / 37
  153. 153. Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : Full representation : 512 × 512 =262 144 CS221 Linear Algebra Review 10/11/2011 37 / 37
  154. 154. Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : Full representation : 512 × 512 =262 144 Rank 10 approximation : 512 × 10 + 10 + 512 × 10 =10 250 CS221 Linear Algebra Review 10/11/2011 37 / 37
  155. 155. Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : Full representation : 512 × 512 =262 144 Rank 10 approximation : 512 × 10 + 10 + 512 × 10 =10 250 Rank 40 approximation : 512 × 40 + 40 + 512 × 40 =41 000 CS221 Linear Algebra Review 10/11/2011 37 / 37
  156. 156. Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : Full representation : 512 × 512 =262 144 Rank 10 approximation : 512 × 10 + 10 + 512 × 10 =10 250 Rank 40 approximation : 512 × 40 + 40 + 512 × 40 =41 000 Rank 80 approximation : 512 × 80 + 80 + 512 × 80 =82 000 CS221 Linear Algebra Review 10/11/2011 37 / 37

×