The document discusses eigenvalues and eigenvectors of symmetric matrices. It provides an overview of linear transformations and how they can be represented by matrices. It then discusses how eigenvalues and eigenvectors are defined as vectors that do not change direction under a linear transformation except for their sign. The document outlines methods for computing eigenvalues and eigenvectors, including using tridiagonal matrices, householder transformations, and sturm sequences to optimize the computation. Faster algorithms are needed as the current methods are slow.

1. EIGENVALUES
AND
EIGENVECTORS
OF SYMMETRIC
MATRICES
Ivan Mateev
19/03/2013

2. LINEAR TRANSFORMATIONS:
RECAP
A linear transformation
Matrix
19/03/2013

3. LINEAR TRANSFORMATIONS:
“UNDER THE HOOD”
Shapes can be viewed as collections of points
Lots of points*
*Infinitely many
19/03/2013

4. LINEAR TRANSFORMATIONS:
VECTOR SPACES
Each two points define a vector
19/03/2013

5. LINEAR TRANSFORMATIONS:
VECTOR SPACES
Each two points define a vector
Instead of a collection of points*, a shape can be seen as a collections of
vectors*
*Infinitely many
19/03/2013

6. LINEAR TRANSFORMATIONS:
VECTOR SPACES
Each two points define a vector
Instead of a collection of points*, shapes can be seen as collections of
vectors*
“Vector spaces”!
19/03/2013
*Infinitely many

7. LINEAR TRANSFORMATIONS:
VECTOR SPACES
A linear transformation a function, which transform one vector space
into another.
19/03/2013

8. LINEAR TRANSFORMATIONS
Scaling
Shearing
19/03/2013

9. SCALING AS A VECTOR
TRANSFORMATION
Scaling
19/03/2013

10. SCALING AS A VECTOR
TRANSFORMATION
19/03/2013

11. SCALING AS A VECTOR
TRANSFORMATION
19/03/2013

12. SCALING AS A VECTOR
TRANSFORMATION
The red and blue vectors didn’t change (much)!
19/03/2013

13. SCALING AS A VECTOR
TRANSFORMATION
The red and blue vectors didn’t change (much)!
They are actually parallel to the vectors (1,0) and (0,1) 19/03/2013

14. SHEARING AS A VECTOR
TRANSFORMATION
Shearing
The red vectors are preserved
Actually all vectors, parallel to (1,0) are preserved 19/03/2013

15. EIGENVECTORS & EIGENVALUES
 A non-zero vector, which doesn’t change
direction in the transformation (except for
it’s sign) is called an Eigenvector.
 Each eigenvector has a number associated
with it, which is called an Eigenvalue.
 Literally means “own vector” and “own
value” from German.
 Historically eigenvalues were first used in
differential equations and later found many
applications in matrix theory.
 But how do we find them?
19/03/2013

16. EIGENVECTORS & EIGENVALUES:
A HOW-TO GUIDE
19/03/2013

17. EIGENVECTORS & EIGENVALUES:
A HOW-TO GUIDE
19/03/2013

18. EIGENVECTORS & EIGENVALUES:
CLARIFICATION
19/03/2013

19. EIGENVECTORS & EIGENVALUES:
CLARIFICATION
The equation:
19/03/2013

20. EIGENVECTORS & EIGENVALUES:
(SOME SCIENCY) APPLICATIONS
 Differential equations
 Quantum Physics
 Basic QP is all about matrices and rotations
 Other areas of mathematics
 A bit more on this later 
19/03/2013

21. EIGENVECTORS & EIGENVALUES:
(SOME) APPLICATIONS
Computer Graphics/Physics:
 Signal processing (filtering)
 Rigid body dynamics
 LOD Optimizations
 Continuum and Fracture Mechanics (realistic
destruction/tearing)
 Fractal Object Generation (Fractional Brownian Motion)
 Pathinding/Spacial partitioning:
 Spectral partitioning of graphs
 Graph theory (matrix representation of graphs)
 Computer Vision/AI:
19/03/2013
 Geometric Data Analysis, Face recognition.

22. FACE RECOGNITION:
IMFRE
Ivan’s Magical Face Recognition
Extravaganza
19/03/2013

23. IMFRE:
HIGH-LEVEL IDEA
IMFRE is an idea for a two step Pseudo-Algorithm
 Step 1: Learning Phase
Machine takes 100 photos of a person’s face;
Resolves a method for storing the person’s face;
Step 2: Recognition Phase
Takes a picture and tests it against the method;
19/03/2013

24. IMFRE:
LEARNING PHASE
19/03/2013

25. IMFRE:
LEARNING PHASE
19/03/2013

26. IMFRE:
LEARNING PHASE
 Assume that one picture is a linear transformation of the other
By analogy:
19/03/2013

27. IMFRE:
LEARNING PHASE
 Keep taking pictures!
 After each picture taken, repeat the step from previous slide with
every new picture against every old picture!
19/03/2013

28. IMFRE:
MAGIC
 Afterall the pictures have been taken, we should have about
100000 transformation matrices.
The “magic” is that we already have functionality to mix all
100000 transformations. And the result of that mix should be a
linear transformation that can map 1 image to every one of the
others.
Magic
19/03/2013

29. IMFRE:
MAGIC
Magic
After all of that T is our new mixed transformation
Fly, matrix!
*Fly,Fly!
matrix! Fly!
19/03/2013

30. IMFRE:
EIGENFACES
 This is what the
eigenvectors of T look like.
 It’s called an eigenface
 In the Recognition Phase,
IMFRE will take a picture,
transform it using T and
compare how much it looks
like one of he stock photos.
19/03/2013

31. FACE RECOGNITION:
BUT SERIOUSLY
 The actual eigenface algorithm was developed
in 1987
 Two steps like IMFRE.
 But is actually quite inaccurate for faces and
couldn’t handle lighting very well.
 On the other hand it’s ok for general object
recognition: boxes, triangles, etc…
19/03/2013

32. PART 2
COMPUTING
19/03/2013

33. EIGENVALUES & EIGENVECTORS:
RECAP
19/03/2013

34. EIGENVALUES & EIGENVECTORS:
COMPUTING (2D)
 Lets say you have a 2D transformation that you need to find the
eigenvalue of:
The solution in code would be quite simple:
19/03/2013

35. 19/03/2013

36. “But I wanna do
the cool stuff!!!”
19/03/2013

37. BRUTE FORCE EIGENVALUES:
CONSIDERATIONS
We’ve seen earlier that the characteristic
equation is a polynomial of degree n. Let
that polynomial be C(x).
 We can use an iteration method to find
the roots of C(x), but we need to know:
 The range, in which the roots are
 What is a usefull iteration method
or our problem?
19/03/2013

38. BRUTE FORCE EIGENVALUES:
GERSHGORIN CIRCLE THEOREM
19/03/2013

39. BRUTE FORCE EIGENVALUES:
BISECTION METHOD
 Board time!!!
 Problem! Your program might get stuck…
 Furthermore, you have to compute C(x) on every step…
 So it turns out that brute force is not good idea… (fo’ real)
 But we can still take Bisection and GCT and look for a diferent
solution!
19/03/2013

40. EIGENVALUE COMPUTING:
ENTER TRIDIAGONAL MATRICES
19/03/2013

41. TRIDIAGONAL MATRICES:
STURM SEQUENCES
19/03/2013

42. TRIDIAGONAL MATRICES:
HOUSEHOLDER TRANSORMATION
19/03/2013

43. TRIDIAGONAL MATRICES:
STURM SEQUENCES + BISECTION
19/03/2013

44. EIGENVALUES + EIGENVECTORS:
PUTTING IT ALL TOGETHER
19/03/2013

45. OPTIMIZATIONS
 As it is, this stuff is slow!!!
19/03/2013

46. OPTIMIZATIONS
 As it is, this stuff is slow!!!
 Mathematical optimizations
Bisection over each GCs
Newton’s method or Runge-Khutta, instead of Bisection
Use Sturm’s sequence to find a closed form characteristics
equation (if you dare) + Newton’s method
19/03/2013

47.  As it is, this stuff is slow!!!
OPTIMIZATIONS
 Mathematical optimizations
Bisection over each GCs
Newton’s method or Runge-Khutta, instead of Bisection
Use Sturm’s sequence to find a closed form characteristics
equation (if you dare) + Newton’s method
 Hardware+Mathematical optimization:
 Calculating GCs can be done in parallel
 Bisection over each GCs + threading
 Householder Reduction is the biggest bottleneck
Crunch inner body of the loop into one expression and..
Look for a large-scale solution: OpenCL, CUDA 19/03/2013

48. LINKS/REFERENCES/EXTRA
READING
*Note: Articles, other than [1] and [2] were used as sources of other p.o.vs
 [1] NUMERICAL RECIPES IN FORTRAN 77: THE ART OF SCIENTIFIC
COMPUTING, “Reduction of a Symmetric Matrix to Tridiagonal Form: Givens and
Householder Reductions” pages 462 – 469 available @ http://www.mpi-
hd.mpg.de/astrophysik/HEA/internal/Numerical_Recipes/f11-2.pdf
 [2] Technische Universiteit Eindhoven CASA – “Iterative Techniques For Solving
Eigenvalue Problems” @
http://www.win.tue.nl/casa/meetings/seminar/previous/_abstract051109_files/presentatio
n_full.pdf
 Numerische Mathematik 9 (1967), W. BARTH, R. S. MARTIN and J. H.
WILKINSON, “Calculation of the Eigenvalues of a Symmetric Tridiagonal Matrix by
the Method of Bisection” Pages 386 - 393 @
http://www.maths.ed.ac.uk/~aar/papers/bamawi.pdf
 Johns Hopkins University Department of Physics & Astronomy - "Lecture 10:
Eigenvectors and eigenvalues (Numerical Recipes, Chapter 11)" @
http://www.pha.jhu.edu/~neufeld/numerical/lecturenotes10.pdf
 UC Berkley College of Engineering, "A New O(n^2) Algorithm for the Symmetric
Tridiagonal Eigenvalue/Eigenvector Problem“ (1997), Inderjit Singh Dhillon @
http://www.eecs.berkeley.edu/Pubs/TechRpts/1997/5888.html 19/03/2013