Upcoming SlideShare
Loading in …5
×

# SVD review

2,396 views

Published on

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
Your message goes here
• Be the first to comment

No Downloads
Views
Total views
2,396
On SlideShare
0
From Embeds
0
Number of Embeds
44
Actions
Shares
0
Downloads
67
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

### SVD review

1. 1. CS 395/495-26: Spring 2004 <ul><li>IBMR: Singular Value Decomposition (SVD Review) </li></ul><ul><li>Jack Tumblin </li></ul><ul><li>[email_address] </li></ul>
2. 2. Matrix Multiply: A Change-of-Axes <ul><li>Matrix Multiply: Ax = b </li></ul><ul><ul><li>x and b are column vectors </li></ul></ul><ul><ul><li>A has m rows, n columns </li></ul></ul>x 1 … x n a 11 … a 1n … … a m1 … a mn = b 1 … b m x 1 x 2 b 3 b 2 b 1 Ax=b x b Input (N dim) (2D) Output (M-dim) (3D)
3. 3. Matrix Multiply: A Change-of-Axes <ul><li>Matrix Multiply: Ax = b </li></ul><ul><ul><li>x and b are column vectors </li></ul></ul><ul><ul><li>A has m rows, n columns </li></ul></ul><ul><li>Matrix multiply is just a set of dot-products; it ‘changes coordinates’ for vector x, makes b. </li></ul><ul><ul><li>Rows of A = A 1 ,A 2 ,A 3 ,... = new coordinate axes </li></ul></ul><ul><ul><li>Ax = a dot product for each new axis </li></ul></ul>x 1 … x n a 11 … a 1n … … a m1 … a mn = b 1 … b m x 1 x 2 A 2 A 1 A 3 b 3 b 2 b 1 Ax=b x b Input (N dim) (2D) Output (M-dim) (3D)
4. 4. <ul><li>Sphere of all unit-length x  ellipsoid of b </li></ul><ul><li>Output ellipsoid’s axes are always perpendicular (true for any ellipsoid) </li></ul><ul><li>**THUS** we can make ┴ , unit-length axes… </li></ul>How Does Matrix ‘stretch space’? x 1 x 2 b 3 b 2 b 1 Ax=b 1 Input (N dim.) Output (M dim.) u 1 u 2 x 1 ellipsoid (disk) b
5. 5. <ul><li>Sphere of all unit-length x  ellipsoid of b </li></ul><ul><li>Output ellipsoid’s axes form orthonormal basis vectors U i : </li></ul><ul><li>Basis vectors U i are columns of U matrix </li></ul><ul><li>SVD(A) = U SV T columns of U = OUTPUT basis vectors </li></ul>SVD finds: ‘Output’ Vectors U , and… x 1 x 2 b 3 b 2 b 1 Ax=b 1 Input (N dim.) Output (M dim.) u 1 u 2 x 1 ellipsoid (disk) b
6. 6. SVD finds: …‘Input’ Vectors V , and… <ul><li>For each U i , make a matching V i that: </li></ul><ul><ul><li>Transforms to U i (with scaling s i ) : s i (A V i ) = U i </li></ul></ul><ul><ul><li>Forms an orthonormal basis of input space </li></ul></ul><ul><li>SVD(A) = US V T columns of V = INPUT basis vectors </li></ul>x 1 x 2 Ax=b x Input (N dim.) b 3 b 2 b 1 Output (M dim.) u 1 u 2 v 2 v 1 b
7. 7. SVD finds: …‘Scale’ factors S . <ul><li>We have Unit-length Output U i , vectors, </li></ul><ul><li>Each from a Unit-length Input V i vector, so </li></ul><ul><li>So we need ‘scale factor’ (singular values) s i to link input to output: s i (A V i ) = U i </li></ul><ul><li>SVD(A) = U S V T </li></ul>x 1 x 2 Ax=b x Input (N dim.) b 3 b 2 b 1 Output (M dim.) u 1 u 2 v 2 v 1 s 2 s 1 b
8. 8. SVD Review: What is it? <ul><li>Finish: SVD(A) = USV T </li></ul><ul><ul><li>add ‘ missing ’ U i or V i , define these s i =0 . </li></ul></ul><ul><ul><li>Singular matrix S : diagonals are s i : ‘v-to-u scale factors’ </li></ul></ul><ul><ul><li>Matrix U , Matrix V have columns U i and V i </li></ul></ul>x 1 x 2 Ax=b x Input (N dim.) b 3 b 2 b 1 Output (M dim.) u 1 u 3 u 2 v 2 v 1 A = USV T ‘Find Input & Output Axes, Linked by Scale’ b
9. 9. ‘ Let SVDs Explain it All for You’ <ul><ul><li>cool! U and V are Orthonormal! U -1 = U T , V -1 = V T </li></ul></ul><ul><ul><li>Rank(A)? == # of non-zero singular values s i </li></ul></ul><ul><ul><li>‘ ill conditioned’? Some s i are nearly zero </li></ul></ul><ul><ul><li>‘ Invert’ a non-square matrix A ? Amazing! pseudo-inverse: A=USV T ; A VS -1 U T = I; so A -1 = V S -1 U T </li></ul></ul>x 1 x 2 Ax=b x Input (N dim.) b 3 b 2 b 1 Output (M dim.) u 1 u 3 u 2 v 2 v 1 A = USV T ‘Find Input & Output Axes, Linked by Scale’
10. 10. ‘ Solve for the Null Space’ A x =0 <ul><ul><li>Easy! x is the V i axis that doesn’t affect the output ( s i = 0 ) </li></ul></ul><ul><ul><li>are all s i nonzero? then the only answer is x=0. </li></ul></ul><ul><ul><li>Null ‘space’ because V i is a DIMENSION--- x = Vi * a </li></ul></ul><ul><ul><li>More than 1 zero-valued s i ? null space >1 dimension ... x = a V i1 + b V i2 + … </li></ul></ul>x 1 x 2 Ax=b x Input (N dim.) b 3 b 2 b 1 Output (M dim.) u 1 u 3 u 2 v 2 v 1 A = USV T ‘Find Input & Output Axes, Linked by Scale’
11. 11. More on SVDs: Handout <ul><li>Zisserman Appendix 3, pg 556 (cryptic) </li></ul><ul><li>Linear Algebra books </li></ul><ul><li>Good online introduction: (your handout) Leach, S. “Singular Value Decomposition – A primer,” Unpublished Manuscript, Department of Computer Science, Brown University. http://www.cs.brown.edu/research/ai/dynamics/tutorial/Postscript/ http://nextlib.lifegoo.com/user/jack/article/1791 </li></ul><ul><li>Google on Singular Value Decomposition... </li></ul>
12. 12. END