• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Cs221 linear algebra
 

Cs221 linear algebra

on

  • 832 views

 

Statistics

Views

Total Views
832
Views on SlideShare
832
Embed Views
0

Actions

Likes
1
Downloads
70
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Cs221 linear algebra Cs221 linear algebra Presentation Transcript

    • Linear Algebra Review CS221: Introduction to Articial Intelligence Carlos Fernandez-Granda 10/11/2011CS221 Linear Algebra Review 10/11/2011 1 / 37
    • Index1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 2 / 37
    • Vectors1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 3 / 37
    • VectorsWhat is a vector ? CS221 Linear Algebra Review 10/11/2011 4 / 37
    • VectorsWhat is a vector ? An ordered tuple of numbers : u   1 u 2  u=  . . .  un CS221 Linear Algebra Review 10/11/2011 4 / 37
    • VectorsWhat is a vector ? An ordered tuple of numbers : u   1 u 2  u=  . . .  un A quantity that has a magnitude and a direction. CS221 Linear Algebra Review 10/11/2011 4 / 37
    • VectorsVector spaces More formally a vector is an element of a vector space. CS221 Linear Algebra Review 10/11/2011 5 / 37
    • VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : CS221 Linear Algebra Review 10/11/2011 5 / 37
    • VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : If vectors u and v ∈ V , then u + v ∈ V . CS221 Linear Algebra Review 10/11/2011 5 / 37
    • VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : If vectors u and v ∈ V , then u + v ∈ V . If vector u ∈ V , then α u ∈ V for any scalar α. CS221 Linear Algebra Review 10/11/2011 5 / 37
    • VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : If vectors u and v ∈ V , then u + v ∈ V . If vector u ∈ V , then α u ∈ V for any scalar α. There exists 0 ∈ V such that u + 0 = u for any u ∈ V . CS221 Linear Algebra Review 10/11/2011 5 / 37
    • VectorsVector spaces More formally a vector is an element of a vector space. A vector space V is a set that contains all linear combinations of its elements : If vectors u and v ∈ V , then u + v ∈ V . If vector u ∈ V , then α u ∈ V for any scalar α. There exists 0 ∈ V such that u + 0 = u for any u ∈ V . A subspace is a subset of a vector space that is also a vector space. CS221 Linear Algebra Review 10/11/2011 5 / 37
    • VectorsExamples Euclidean space Rn . CS221 Linear Algebra Review 10/11/2011 6 / 37
    • VectorsExamples Euclidean space Rn . The span of any set of vectors {u1 , u2 , . . . , un }, dened as : span (u1 , u2 , . . . , un ) = {α1 u1 + α2 u2 + · · · + αn un | αi ∈ R} CS221 Linear Algebra Review 10/11/2011 6 / 37
    • VectorsSubspaces A line through the origin in Rn (span of a vector). CS221 Linear Algebra Review 10/11/2011 7 / 37
    • VectorsSubspaces A line through the origin in Rn (span of a vector). A plane in Rn (span of two vectors). CS221 Linear Algebra Review 10/11/2011 7 / 37
    • VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. CS221 Linear Algebra Review 10/11/2011 8 / 37
    • VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. CS221 Linear Algebra Review 10/11/2011 8 / 37
    • VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . CS221 Linear Algebra Review 10/11/2011 8 / 37
    • VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . If a vector space has a basis with d vectors, its dimension is d. CS221 Linear Algebra Review 10/11/2011 8 / 37
    • VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . If a vector space has a basis with d vectors, its dimension is d. Any other basis of the same space will consist of exactly d vectors. CS221 Linear Algebra Review 10/11/2011 8 / 37
    • VectorsLinear Independence and Basis of Vector Spaces A vector u is linearly independent of a set of vectors {u1 , u2 , . . . , un } if it does not lie in their span. A set of vectors is linearly independent if every vector is linearly independent of the rest. A basis of a vector space V is a linearly independent set of vectors whose span is equal to V . If a vector space has a basis with d vectors, its dimension is d. Any other basis of the same space will consist of exactly d vectors. The dimension of a vector space can be innite (function spaces). CS221 Linear Algebra Review 10/11/2011 8 / 37
    • VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. CS221 Linear Algebra Review 10/11/2011 9 / 37
    • VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : CS221 Linear Algebra Review 10/11/2011 9 / 37
    • VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : Homogeneity ||α u|| = |α| ||u|| CS221 Linear Algebra Review 10/11/2011 9 / 37
    • VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : Homogeneity ||α u|| = |α| ||u|| Triangle inequality ||u + v|| ≤ ||u|| + ||v|| CS221 Linear Algebra Review 10/11/2011 9 / 37
    • VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : Homogeneity ||α u|| = |α| ||u|| Triangle inequality ||u + v|| ≤ ||u|| + ||v|| Point separation ||u|| = 0 if and only if u = 0. CS221 Linear Algebra Review 10/11/2011 9 / 37
    • VectorsNorm of a Vector A norm ||u|| measures the magnitude of a vector. Mathematically, any function from the vector space to R that satises : Homogeneity ||α u|| = |α| ||u|| Triangle inequality ||u + v|| ≤ ||u|| + ||v|| Point separation ||u|| = 0 if and only if u = 0. Examples : Manhattan or 1 norm : ||u||1 = i |ui |. Euclidean or 2 norm : ||u||2 = i ui . 2 CS221 Linear Algebra Review 10/11/2011 9 / 37
    • VectorsInner Product Inner product between u and v : u, v = n uv. i =1 i i CS221 Linear Algebra Review 10/11/2011 10 / 37
    • VectorsInner Product Inner product between u and v : u, v = n=1 ui vi . i It is the projection of one vector onto the other. CS221 Linear Algebra Review 10/11/2011 10 / 37
    • VectorsInner Product Inner product between u and v : u, v = n=1 ui vi . i It is the projection of one vector onto the other. Related to the Euclidean norm : u, u = ||u||2 . 2 CS221 Linear Algebra Review 10/11/2011 10 / 37
    • VectorsInner ProductThe inner product is a measure of correlation between two vectors, scaledby the norms of the vectors : CS221 Linear Algebra Review 10/11/2011 11 / 37
    • VectorsInner ProductThe inner product is a measure of correlation between two vectors, scaledby the norms of the vectors : If u, v > 0, u and v are aligned. CS221 Linear Algebra Review 10/11/2011 11 / 37
    • VectorsInner ProductThe inner product is a measure of correlation between two vectors, scaledby the norms of the vectors : If u, v > 0, u and v are aligned. If u, v < 0, u and v are opposed. CS221 Linear Algebra Review 10/11/2011 11 / 37
    • VectorsInner ProductThe inner product is a measure of correlation between two vectors, scaledby the norms of the vectors : If u, v > 0, u and v are aligned. If u, v < 0, u and v are opposed. If u, v = 0, u and v are orthogonal. CS221 Linear Algebra Review 10/11/2011 11 / 37
    • VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. CS221 Linear Algebra Review 10/11/2011 12 / 37
    • VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. CS221 Linear Algebra Review 10/11/2011 12 / 37
    • VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. Example : x = α1 b1 + α2 b2 . CS221 Linear Algebra Review 10/11/2011 12 / 37
    • VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. Example : x = α1 b1 + α2 b2 . We compute x, b1 : x, b1 = α1 b1 + α2 b2 , b1 = α1 b1 , b1 + α2 b2 , b1 By linearity. = α1 + 0 By orthonormality. CS221 Linear Algebra Review 10/11/2011 12 / 37
    • VectorsOrthonormal Basis The vectors in an orthonormal basis have unit Euclidean norm and are orthogonal to each other. To express a vector x in an orthonormal basis, we can just take inner products. Example : x = α1 b1 + α2 b2 . We compute x, b1 : x, b1 = α1 b1 + α2 b2 , b1 = α1 b1 , b1 + α2 b2 , b1 By linearity. = α1 + 0 By orthonormality. Likewise, α2 = x, b2 . CS221 Linear Algebra Review 10/11/2011 12 / 37
    • Matrices1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 13 / 37
    • MatricesLinear Operator A linear operator L : U → V is a map from a vector space U to another vector space V that satises : CS221 Linear Algebra Review 10/11/2011 14 / 37
    • MatricesLinear Operator A linear operator L : U → V is a map from a vector space U to another vector space V that satises : L (u1 + u2 ) = L (u1 ) + L (u2 ) . CS221 Linear Algebra Review 10/11/2011 14 / 37
    • MatricesLinear Operator A linear operator L : U → V is a map from a vector space U to another vector space V that satises : . L (u1 + u2 ) = L (u1 ) + L (u2 ) L (α u) = α L (u) for any scalar α. CS221 Linear Algebra Review 10/11/2011 14 / 37
    • MatricesLinear Operator A linear operator L : U → V is a map from a vector space U to another vector space V that satises : . L (u1 + u2 ) = L (u1 ) + L (u2 ) L (α u) = α L (u) for any scalar α. If the dimension n of U and m of V are nite, L can be represented by an m × n matrix A. A A An   11 12 ... 1 A 21 A 22 ... A n 2 A=   ...  Am 1 Am 2 ... Amn CS221 Linear Algebra Review 10/11/2011 14 / 37
    • MatricesMatrix Vector Multiplication A A An u v      11 12 ... 1 1 1 A 21 A 22 ... A nu 2 2  v 2   Au =   =  ...  . . .  . . .  Am 1 Am 2 ... Amn un vm CS221 Linear Algebra Review 10/11/2011 15 / 37
    • MatricesMatrix Vector Multiplication A A An u v      11 12 ... 1 1 1 A 21 A 22 ... A nu 2 2  v 2   Au =   =  ...  . . .  . . .  Am 1 Am 2 ... Amn un vm CS221 Linear Algebra Review 10/11/2011 15 / 37
    • MatricesMatrix Vector Multiplication A A An u v      11 12 ... 1 1 1 A 21 A 22 ... A nu 2 2  v 2   Au =   =  ...  . . .  . . .  Am 1 Am 2 ... Amn un vm CS221 Linear Algebra Review 10/11/2011 15 / 37
    • MatricesMatrix Vector Multiplication A A An u v      11 12 ... 1 1 1 A 21 A 22 ... A nu 2 2  v 2   Au =   =  ...  . . .  . . .  Am 1 Am 2 ... Amn un vm CS221 Linear Algebra Review 10/11/2011 15 / 37
    • MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
    • MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
    • MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
    • MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
    • MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
    • MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
    • MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
    • MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
    • MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
    • MatricesMatrix ProductApplying two linear operators A and B denes another linear operator,which can be represented by another matrix AB. A A An B B Bp    11 12 ... 1 11 12 ... 1 A 21 A 22 ...  B A n 2 21 B22 ... B p 2 AB =    ...  ...  Am Am 1 2 ... Amn Bn 1 Bn 2 ... Bnp AB AB . . . AB p   11 12 1  AB AB 21 22 . . . AB p  2 =   ...  ABm 1 ABm 2 ... ABmpNote that if A is m × n and B is n × p, then AB is m × p. CS221 Linear Algebra Review 10/11/2011 16 / 37
    • MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1    A12 A22 . . . An2  AT =    ...  A1m A2m . . . Anm CS221 Linear Algebra Review 10/11/2011 17 / 37
    • MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1    A12 A22 . . . An2  AT =    ...  A1m A2m . . . Anm Some simple properties : CS221 Linear Algebra Review 10/11/2011 17 / 37
    • MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1    A12 A22 . . . An2  AT =    ...  A1m A2m . . . Anm Some simple properties : T AT =A CS221 Linear Algebra Review 10/11/2011 17 / 37
    • MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1    A12 A22 . . . An2  AT =    ...  A1m A2m . . . Anm Some simple properties : T AT = A T (A + B) = AT + BT CS221 Linear Algebra Review 10/11/2011 17 / 37
    • MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1    A12 A22 . . . An2  AT =    ...  A1m A2m . . . Anm Some simple properties : T AT = A T (A + B) = AT + BT T (AB) = BT AT CS221 Linear Algebra Review 10/11/2011 17 / 37
    • MatricesTranspose of a Matrix The transpose of a matrix is obtained by ipping the rows and columns. A11 A21 . . . An1    A12 A22 . . . An2  AT =    ...  A1m A2m . . . Anm Some simple properties : T AT = A T (A + B) = AT + BT T (AB) = BT AT The inner product for vectors can be represented as u, v = uT v. CS221 Linear Algebra Review 10/11/2011 17 / 37
    • MatricesIdentity Operator 1 0 ... 0   0 1 . . . 0 I=   ...  0 0 ... 1 CS221 Linear Algebra Review 10/11/2011 18 / 37
    • MatricesIdentity Operator 1 0 ... 0   0 1 . . . 0 I=   ...  0 0 ... 1 For any matrix A, AI = A. CS221 Linear Algebra Review 10/11/2011 18 / 37
    • MatricesIdentity Operator 1 0 ... 0   0 1 . . . 0 I=   ...  0 0 ... 1 For any matrix A, AI = A. I is the identity operator for the matrix product. CS221 Linear Algebra Review 10/11/2011 18 / 37
    • MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. CS221 Linear Algebra Review 10/11/2011 19 / 37
    • MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : CS221 Linear Algebra Review 10/11/2011 19 / 37
    • MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. CS221 Linear Algebra Review 10/11/2011 19 / 37
    • MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . CS221 Linear Algebra Review 10/11/2011 19 / 37
    • MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : CS221 Linear Algebra Review 10/11/2011 19 / 37
    • MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : Span of the rows of A. CS221 Linear Algebra Review 10/11/2011 19 / 37
    • MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : Span of the rows of A. Linear subspace of Rn . CS221 Linear Algebra Review 10/11/2011 19 / 37
    • MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : Span of the rows of A. Linear subspace of Rn . Important fact : The column and row spaces of any matrix have the same dimension. CS221 Linear Algebra Review 10/11/2011 19 / 37
    • MatricesColumn Space, Row Space and Rank Let A be an m × n matrix. Column space : Span of the columns of A. Linear subspace of Rm . Row space : Span of the rows of A. Linear subspace of Rn . Important fact : The column and row spaces of any matrix have the same dimension. This dimension is the rank of the matrix. CS221 Linear Algebra Review 10/11/2011 19 / 37
    • MatricesRange and Null space Let A be an m × n matrix. CS221 Linear Algebra Review 10/11/2011 20 / 37
    • MatricesRange and Null space Let A be an m × n matrix. Range : CS221 Linear Algebra Review 10/11/2011 20 / 37
    • MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . CS221 Linear Algebra Review 10/11/2011 20 / 37
    • MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . CS221 Linear Algebra Review 10/11/2011 20 / 37
    • MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . Null space : CS221 Linear Algebra Review 10/11/2011 20 / 37
    • MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . Null space : u belongs to the null space if Au = 0. CS221 Linear Algebra Review 10/11/2011 20 / 37
    • MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . Null space : u belongs to the null space if Au = 0. Subspace of Rn . CS221 Linear Algebra Review 10/11/2011 20 / 37
    • MatricesRange and Null space Let A be an m × n matrix. Range : Set of vectors equal to Au for some u ∈ Rn . Linear subspace of Rm . Null space : u belongs to the null space if Au = 0. Subspace of Rn . Every vector in the null space is orthogonal to the rows of A. The null space and row space of a matrix are orthogonal. CS221 Linear Algebra Review 10/11/2011 20 / 37
    • MatricesRange and Column SpaceAnother interpretation of the matrix vector product (Ai is column i of A) : u   1 u 2   Au = A1 A2 ... An  . . .  un = u1 A1 + u2 A2 + · · · + un An CS221 Linear Algebra Review 10/11/2011 21 / 37
    • MatricesRange and Column SpaceAnother interpretation of the matrix vector product (Ai is column i of A) : u   1 u 2   Au = A1 A2 ... An  . . .  un = u1 A1 + u2 A2 + · · · + un An The result is a linear combination of the columns of A. CS221 Linear Algebra Review 10/11/2011 21 / 37
    • MatricesRange and Column SpaceAnother interpretation of the matrix vector product (Ai is column i of A) : u   1 u 2   Au = A1 A2 ... An  . . .  un = u1 A1 + u2 A2 + · · · + un An The result is a linear combination of the columns of A. For any matrix, the range is equal to the column space. CS221 Linear Algebra Review 10/11/2011 21 / 37
    • MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. CS221 Linear Algebra Review 10/11/2011 22 / 37
    • MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. CS221 Linear Algebra Review 10/11/2011 22 / 37
    • MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. In this case, the action of the matrix is invertible. CS221 Linear Algebra Review 10/11/2011 22 / 37
    • MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. In this case, the action of the matrix is invertible. The inversion is also linear and consequently can be represented by another matrix A−1 . CS221 Linear Algebra Review 10/11/2011 22 / 37
    • MatricesMatrix Inverse For an n × n matrix A : rank + dim(null space) = n. If dim(null space)=0 then A is full rank. In this case, the action of the matrix is invertible. The inversion is also linear and consequently can be represented by another matrix A−1 . A−1 is the only matrix such that A−1 A = AA−1 = I. CS221 Linear Algebra Review 10/11/2011 22 / 37
    • MatricesOrthogonal Matrices An orthogonal matrix U satises UT U = I. CS221 Linear Algebra Review 10/11/2011 23 / 37
    • MatricesOrthogonal Matrices An orthogonal matrix U satises UT U = I. Equivalently, U has orthonormal columns. CS221 Linear Algebra Review 10/11/2011 23 / 37
    • MatricesOrthogonal Matrices An orthogonal matrix U satises UT U = I. Equivalently, U has orthonormal columns. Applying an orthogonal matrix to two vectors does not change their inner product : Uu, Uv = (Uu) Uv T = uT UT Uv = uT v = u, v CS221 Linear Algebra Review 10/11/2011 23 / 37
    • MatricesOrthogonal Matrices An orthogonal matrix U satises UT U = I. Equivalently, U has orthonormal columns. Applying an orthogonal matrix to two vectors does not change their inner product : Uu, Uv = (Uu) Uv T = uT UT Uv = uT v = u, v Example : Matrices that represent rotations are orthogonal. CS221 Linear Algebra Review 10/11/2011 23 / 37
    • MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. CS221 Linear Algebra Review 10/11/2011 24 / 37
    • MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. The determinant of a square matrix A is denoted by |A|. CS221 Linear Algebra Review 10/11/2011 24 / 37
    • MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. The determinant of a square matrix A is denoted by |A|. ab For a 2 × 2 matrix A = c d : CS221 Linear Algebra Review 10/11/2011 24 / 37
    • MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. The determinant of a square matrix A is denoted by |A|. ab For a 2 × 2 matrix A = c d : |A| = ad − bc CS221 Linear Algebra Review 10/11/2011 24 / 37
    • MatricesTrace and Determinant The trace is the sum of the diagonal elements of a square matrix. The determinant of a square matrix A is denoted by |A|. ab For a 2 × 2 matrix A = c d : |A| = ad − bc The absolute value of |A| is the area of the parallelogram given by the rows of A. CS221 Linear Algebra Review 10/11/2011 24 / 37
    • MatricesProperties of the Determinant The denition can be generalized to larger matrices. CS221 Linear Algebra Review 10/11/2011 25 / 37
    • MatricesProperties of the Determinant The denition can be generalized to larger matrices. |A| = AT . CS221 Linear Algebra Review 10/11/2011 25 / 37
    • MatricesProperties of the Determinant The denition can be generalized to larger matrices. |A| = AT . |AB| = |A| |B| CS221 Linear Algebra Review 10/11/2011 25 / 37
    • MatricesProperties of the Determinant The denition can be generalized to larger matrices. |A| = AT . |AB| = |A| |B| |A| = 0 if and only if A is not invertible. CS221 Linear Algebra Review 10/11/2011 25 / 37
    • MatricesProperties of the Determinant The denition can be generalized to larger matrices. |A| = AT . |AB| = |A| |B| |A| = 0 if and only if A is not invertible. If A is invertible, then A−1 = |A | . 1 CS221 Linear Algebra Review 10/11/2011 25 / 37
    • Matrix Decompositions1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 26 / 37
    • Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. CS221 Linear Algebra Review 10/11/2011 27 / 37
    • Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ > 1) or contracts (λ < 1) eigenvectors but does not rotate them. CS221 Linear Algebra Review 10/11/2011 27 / 37
    • Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ > 1) or contracts (λ < 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A − λI, which is consequently not invertible. CS221 Linear Algebra Review 10/11/2011 27 / 37
    • Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ > 1) or contracts (λ < 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A − λI, which is consequently not invertible. The eigenvalues of A are the roots of the equation |A − λI| = 0. CS221 Linear Algebra Review 10/11/2011 27 / 37
    • Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ > 1) or contracts (λ < 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A − λI, which is consequently not invertible. The eigenvalues of A are the roots of the equation |A − λI| = 0. Not the way eigenvalues are calculated in practice. CS221 Linear Algebra Review 10/11/2011 27 / 37
    • Matrix DecompositionsEigenvalues and Eigenvectors An eigenvalue λ of a square matrix A satises : Au = λu for some vector u, which we call an eigenvector. Geometrically, the operator A expands (λ > 1) or contracts (λ < 1) eigenvectors but does not rotate them. If u is an eigenvector of A, it is in the null space of A − λI, which is consequently not invertible. The eigenvalues of A are the roots of the equation |A − λI| = 0. Not the way eigenvalues are calculated in practice. Eigenvalues and eigenvectors can be complex valued, even if all the entries of A are real. CS221 Linear Algebra Review 10/11/2011 27 / 37
    • Matrix DecompositionsEigendecomposition of a Matrix Let A be an n × n square matrix with n linearly independent eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn . CS221 Linear Algebra Review 10/11/2011 28 / 37
    • Matrix DecompositionsEigendecomposition of a Matrix Let A be an n × n square matrix with n linearly independent eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn . We dene the matrices : P= p1 . . . pn λ1 0 . . . 0    0 λ2 . . . 0 Λ=   ...  0 0 ... λn CS221 Linear Algebra Review 10/11/2011 28 / 37
    • Matrix DecompositionsEigendecomposition of a Matrix Let A be an n × n square matrix with n linearly independent eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn . We dene the matrices : P= p1 . . . pn λ1 0 . . . 0    0 λ2 . . . 0 Λ=   ...  0 0 ... λn A satises AP = PΛ. CS221 Linear Algebra Review 10/11/2011 28 / 37
    • Matrix DecompositionsEigendecomposition of a Matrix Let A be an n × n square matrix with n linearly independent eigenvectors p1 . . . pn and eigenvalues λ1 . . . λn . We dene the matrices : P= p1 . . . pn λ1 0 . . . 0    0 λ2 . . . 0 Λ=   ...  0 0 ... λn A satises AP = PΛ. P is full rank. Inverting it yields the eigendecomposition of A : A = PΛ P− 1 CS221 Linear Algebra Review 10/11/2011 28 / 37
    • Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 CS221 Linear Algebra Review 10/11/2011 29 / 37
    • Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i CS221 Linear Algebra Review 10/11/2011 29 / 37
    • Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i |A| = |Λ| = n=1 λi . i CS221 Linear Algebra Review 10/11/2011 29 / 37
    • Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i |A| = |Λ| = n=1 λi . i The rank of A is equal to the number of nonzero eigenvalues. CS221 Linear Algebra Review 10/11/2011 29 / 37
    • Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i |A| = |Λ| = n=1 λi . i The rank of A is equal to the number of nonzero eigenvalues. If λ is a nonzero eigenvalue of A, λ is an eigenvalue of A−1 with the 1 same eigenvector. CS221 Linear Algebra Review 10/11/2011 29 / 37
    • Matrix DecompositionsProperties of the Eigendecomposition Not all matrices are diagonalizable (have an eigendecomposition). Example : ( 1 1 ). 0 1 Trace(A) = Trace(Λ) = n=1 λi . i |A| = |Λ| = n=1 λi . i The rank of A is equal to the number of nonzero eigenvalues. If λ is a nonzero eigenvalue of A, λ is an eigenvalue of A−1 with the 1 same eigenvector. The eigendecomposition makes it possible to compute matrix powers eciently : m Am = PΛP−1 = PΛP−1 PΛP−1 . . . PΛP−1 = PΛm P−1 . CS221 Linear Algebra Review 10/11/2011 29 / 37
    • Matrix DecompositionsMatlab Example CS221 Linear Algebra Review 10/11/2011 30 / 37
    • Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. CS221 Linear Algebra Review 10/11/2011 31 / 37
    • Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. CS221 Linear Algebra Review 10/11/2011 31 / 37
    • Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. The eigenvectors of symmetric matrices are orthonormal. CS221 Linear Algebra Review 10/11/2011 31 / 37
    • Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. The eigenvectors of symmetric matrices are orthonormal. Consequently, the eigendecomposition becomes : A = UΛUT for Λ real and U orthogonal. CS221 Linear Algebra Review 10/11/2011 31 / 37
    • Matrix DecompositionsEigendecomposition of a Symmetric Matrix If A = AT then A is symmetric. The eigenvalues of symmetric matrices are real. The eigenvectors of symmetric matrices are orthonormal. Consequently, the eigendecomposition becomes : A = UΛUT for Λ real and U orthogonal. The eigenvectors of A are an orthonormal basis for the column space (and the row space, since they are equal). CS221 Linear Algebra Review 10/11/2011 31 / 37
    • Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : CS221 Linear Algebra Review 10/11/2011 32 / 37
    • Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). CS221 Linear Algebra Review 10/11/2011 32 / 37
    • Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). 2 Scaling of each coecient Ui , u by the corresponding eigenvalue (multiplication by Λ). CS221 Linear Algebra Review 10/11/2011 32 / 37
    • Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). 2 Scaling of each coecient Ui , u by the corresponding eigenvalue (multiplication by Λ). 3 Linear combination of the eigenvectors scaled by the resulting coecient (multiplication by U). CS221 Linear Algebra Review 10/11/2011 32 / 37
    • Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). 2 Scaling of each coecient Ui , u by the corresponding eigenvalue (multiplication by Λ). 3 Linear combination of the eigenvectors scaled by the resulting coecient (multiplication by U). Summarizing : n Au = λi Ui , u Ui i =1 CS221 Linear Algebra Review 10/11/2011 32 / 37
    • Matrix DecompositionsEigendecomposition of a Symmetric Matrix The action of a symmetric matrix on a vector Au = UΛUT u can be decomposed into : 1 Projection of u onto the column space of A (multiplication by UT ). 2 Scaling of each coecient Ui , u by the corresponding eigenvalue (multiplication by Λ). 3 Linear combination of the eigenvectors scaled by the resulting coecient (multiplication by U). Summarizing : n Au = λi Ui , u Ui i =1 It would be great to generalize this to all matrices... CS221 Linear Algebra Review 10/11/2011 32 / 37
    • Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT CS221 Linear Algebra Review 10/11/2011 33 / 37
    • Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT The columns of U are an orthonormal basis for the column space of A. CS221 Linear Algebra Review 10/11/2011 33 / 37
    • Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT The columns of U are an orthonormal basis for the column space of A. The columns of V are an orthonormal basis for the row space of A. CS221 Linear Algebra Review 10/11/2011 33 / 37
    • Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT The columns of U are an orthonormal basis for the column space of A. The columns of V are an orthonormal basis for the row space of A. Σ is diagonal. The entries on its diagonal σi = Σii are positive real numbers, called the singular values of A. CS221 Linear Algebra Review 10/11/2011 33 / 37
    • Matrix DecompositionsSingular Value Decomposition Every matrix has a singular value decomposition : A = UΣVT The columns of U are an orthonormal basis for the column space of A. The columns of V are an orthonormal basis for the row space of A. Σ is diagonal. The entries on its diagonal σi = Σii are positive real numbers, called the singular values of A. The action of A on a vector u can be decomposed into : n Au = σi Vi , u Ui i =1 CS221 Linear Algebra Review 10/11/2011 33 / 37
    • Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT CS221 Linear Algebra Review 10/11/2011 34 / 37
    • Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT The rank of a matrix is equal to the number of nonzero singular values. CS221 Linear Algebra Review 10/11/2011 34 / 37
    • Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT The rank of a matrix is equal to the number of nonzero singular values. The largest singular value σ1 is the solution to the optimization problem : ||Ax||2 σ1 = max x=0 ||x||2 CS221 Linear Algebra Review 10/11/2011 34 / 37
    • Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT The rank of a matrix is equal to the number of nonzero singular values. The largest singular value σ1 is the solution to the optimization problem : ||Ax||2 σ1 = max x=0 ||x||2 It can be veried that the largest singular value satises the properties of a norm, it is called the spectral norm of the matrix. CS221 Linear Algebra Review 10/11/2011 34 / 37
    • Matrix DecompositionsProperties of the Singular Value Decomposition The eigenvalues of the symmetric matrix AT A are equal to the square of the singular values of A : AT A = VΣUT UΣVT = VΣ2 VT The rank of a matrix is equal to the number of nonzero singular values. The largest singular value σ1 is the solution to the optimization problem : ||Ax||2 σ1 = max x=0 ||x||2 It can be veried that the largest singular value satises the properties of a norm, it is called the spectral norm of the matrix. In statistics analyzing data with the singular value decomposition is called Principal Component Analysis. CS221 Linear Algebra Review 10/11/2011 34 / 37
    • Application : Image Compression1 Vectors2 Matrices3 Matrix Decompositions4 Application : Image Compression CS221 Linear Algebra Review 10/11/2011 35 / 37
    • Application : Image CompressionCompression using the Singular Value Decomposition The singular value decomposition can be viewed as a sum of rank 1 matrices : CS221 Linear Algebra Review 10/11/2011 36 / 37
    • Application : Image CompressionCompression using the Singular Value Decomposition The singular value decomposition can be viewed as a sum of rank 1 matrices : If some of the singular values are very small, we can discard them. CS221 Linear Algebra Review 10/11/2011 36 / 37
    • Application : Image CompressionCompression using the Singular Value Decomposition The singular value decomposition can be viewed as a sum of rank 1 matrices : If some of the singular values are very small, we can discard them. This is a form of lossy compression. CS221 Linear Algebra Review 10/11/2011 36 / 37
    • Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. CS221 Linear Algebra Review 10/11/2011 37 / 37
    • Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : CS221 Linear Algebra Review 10/11/2011 37 / 37
    • Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : Full representation : 512 × 512 =262 144 CS221 Linear Algebra Review 10/11/2011 37 / 37
    • Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : Full representation : 512 × 512 =262 144 Rank 10 approximation : 512 × 10 + 10 + 512 × 10 =10 250 CS221 Linear Algebra Review 10/11/2011 37 / 37
    • Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : Full representation : 512 × 512 =262 144 Rank 10 approximation : 512 × 10 + 10 + 512 × 10 =10 250 Rank 40 approximation : 512 × 40 + 40 + 512 × 40 =41 000 CS221 Linear Algebra Review 10/11/2011 37 / 37
    • Application : Image CompressionCompression using the Singular Value Decomposition Truncating the singular value decomposition allows us to represent the matrix with less parameters. For a 512 × 512 matrix : Full representation : 512 × 512 =262 144 Rank 10 approximation : 512 × 10 + 10 + 512 × 10 =10 250 Rank 40 approximation : 512 × 40 + 40 + 512 × 40 =41 000 Rank 80 approximation : 512 × 80 + 80 + 512 × 80 =82 000 CS221 Linear Algebra Review 10/11/2011 37 / 37