Robert CollinsCSE486, Penn State                             Lecture 20:                     The Eight-Point Algorithm    ...
Robert CollinsCSE486, Penn State                                      Reminder:            -0.00310695 -0.0025646       2....
Robert CollinsCSE486, Penn State                     Essential/Fundamental Matrix      The essential and fundamental matri...
Robert CollinsCSE486, Penn State                       E/F Matrix Summary    Longuet-Higgins equation     Epipolar lines: ...
Robert Collins               Computing F from Point MatchesCSE486, Penn State         • Assume that you have m corresponde...
Robert CollinsCSE486, Penn State                     Computing F
Robert CollinsCSE486, Penn State                     Computing F
Robert CollinsCSE486, Penn State                      Computing F    Given m point correspondences…     Think: how many po...
Robert CollinsCSE486, Penn State                     How Many Points?   Unlike a homography, where each point corresponden...
Robert CollinsCSE486, Penn State                     Solving Homogeneous Systems    Self-study         Assume that we need...
Robert CollinsCSE486, Penn State                     Least Square solution    Self-study    We want Ax as close to 0 as po...
Robert CollinsCSE486, Penn State                     Optimization with constraints    Self-study        Define the followi...
Robert CollinsCSE486, Penn State                     Optimization with constraints    Self-study    Taking derivatives wrt...
Robert CollinsCSE486, Penn State                     Optimization with constraints    Self-study      •x is an eigenvector...
Robert CollinsCSE486, Penn State    Self-study           We can find the eigenvectors and            eigenvalues of ATA by...
Robert CollinsCSE486, Penn State   Singular Value Decomposition    Self-study                  (SVD)    Any m x n matrix A...
Robert CollinsCSE486, Penn State                       SVD Properties    Self-study  • The columns of U are the eigenvecto...
Robert Collins                Computing F: The 8 pt AlgorithmCSE486, Penn State            8   8    8   8    8   8   8   8...
Robert CollinsCSE486, Penn State                     Algorithm EIGHT_POINT    The input is formed by m point correspondenc...
Robert CollinsCSE486, Penn State                     Algorithm EIGHT_POINT    F must be singular (remember, it is rank 2, ...
Robert CollinsCSE486, Penn State                        Numerical Details    Self-study    • The coordinates of correspond...
Robert CollinsCSE486, Penn State                        A Practical Issue         How to “rectify” the images so that any ...
Robert Collins               Stereo RectificationCSE486, Penn State• Image Reprojection     – reproject image planes onto ...
Robert CollinsCSE486, Penn State                     General Idea                       Apply homography representing a   ...
Robert CollinsCSE486, Penn State                     General Idea, continued
Robert CollinsCSE486, Penn State                       Image Rectification                       Method from T&V 7.3.7    ...
Robert CollinsCSE486, Penn State                        Image Rectification         • Rectification involves two rotations...
Robert CollinsCSE486, Penn State                     Image Rectification    Build the rotation:      with:      where T is...
Robert CollinsCSE486, Penn State                      Image Rectification    Build the rotation:      with:         Verify...
Robert CollinsCSE486, Penn State                      Algorithm Rectification         • Build the matrix Rrect         • S...
Robert CollinsCSE486, Penn State                     Example
Robert CollinsCSE486, Penn State                      Example                     Rectified Pair
Robert CollinsCSE486, Penn State                         A Better Approach              The traditional method does not wo...
Robert CollinsCSE486, Penn State                                  Polar rectification                                  (Po...
Robert CollinsCSE486, Penn State                     polar rectification: example
Robert CollinsCSE486, Penn State                     polar rectification: example
Robert CollinsCSE486, Penn State                     Example                                    epipoles                  ...
Robert CollinsCSE486, Penn State                      Example (cont)   Rectified images
Robert CollinsCSE486, Penn State                     Example (cont)                                      Depth Maps       ...
Robert CollinsCSE486, Penn State                     Example (cont)    Depth map in    pixel coords                       ...
Upcoming SlideShare
Loading in …5
×

Lecture20

660 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
660
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
25
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Lecture20

  1. 1. Robert CollinsCSE486, Penn State Lecture 20: The Eight-Point Algorithm Readings T&V 7.3 and 7.4
  2. 2. Robert CollinsCSE486, Penn State Reminder: -0.00310695 -0.0025646 2.96584 F= -0.028094 13.1905 -0.00771621 -29.2007 56.3813 -9999.79
  3. 3. Robert CollinsCSE486, Penn State Essential/Fundamental Matrix The essential and fundamental matrices are 3x3 matrices that “encode” the epipolar geometry of two views. Motivation: Given a point in one image, multiplying by the essential/fundamental matrix will tell us which epipolar line to search along in the second view.
  4. 4. Robert CollinsCSE486, Penn State E/F Matrix Summary Longuet-Higgins equation Epipolar lines: Epipoles: E vs F: E works in film coords (calibrated cameras) F works in pixel coords (uncalibrated cameras)
  5. 5. Robert Collins Computing F from Point MatchesCSE486, Penn State • Assume that you have m correspondences • Each correspondence satisfies: • F is a 3x3 matrix (9 entries) • Set up a HOMOGENEOUS linear system with 9 unknowns
  6. 6. Robert CollinsCSE486, Penn State Computing F
  7. 7. Robert CollinsCSE486, Penn State Computing F
  8. 8. Robert CollinsCSE486, Penn State Computing F Given m point correspondences… Think: how many points do we need?
  9. 9. Robert CollinsCSE486, Penn State How Many Points? Unlike a homography, where each point correspondence contributes two constraints (rows in the linear system of equations), for estimating the essential/fundamental matrix, each point only contributes one constraint (row). [because the Longuet-Higgins / Epipolar constraint is a scalar eqn.] Thus need at least 8 points. Hence: The Eight Point algorithm!
  10. 10. Robert CollinsCSE486, Penn State Solving Homogeneous Systems Self-study Assume that we need the non trivial solution of: with m equations and n unknowns, m >= n – 1 and rank(A) = n-1 Since the norm of x is arbitrary, we will look for a solution with norm ||x|| = 1
  11. 11. Robert CollinsCSE486, Penn State Least Square solution Self-study We want Ax as close to 0 as possible and ||x|| =1:
  12. 12. Robert CollinsCSE486, Penn State Optimization with constraints Self-study Define the following cost: This cost is called the LAGRANGIAN cost and λ is called the LAGRANGIAN multiplier The Lagrangian incorporates the constraints into the cost function by introducing extra variables.
  13. 13. Robert CollinsCSE486, Penn State Optimization with constraints Self-study Taking derivatives wrt to x and λ: •The first equation is an eigenvector problem • The second equation is the original constraint
  14. 14. Robert CollinsCSE486, Penn State Optimization with constraints Self-study •x is an eigenvector of ATA with eigenvalue λ: eλ •We want the eigenvector with smallest eigenvalue
  15. 15. Robert CollinsCSE486, Penn State Self-study We can find the eigenvectors and eigenvalues of ATA by finding the Singular Value Decomposition of A
  16. 16. Robert CollinsCSE486, Penn State Singular Value Decomposition Self-study (SVD) Any m x n matrix A can be written as the product of 3 matrices: Where: • U is m x m and its columns are orthonormal vectors • V is n x n and its columns are orthonormal vectors • D is m x n diagonal and its diagonal elements are called the singular values of A, and are such that: σ1 ¸ σ2 ¸ … σn ¸ 0
  17. 17. Robert CollinsCSE486, Penn State SVD Properties Self-study • The columns of U are the eigenvectors of AAT • The columns of V are the eigenvectors of ATA • The squares of the diagonal elements of D are the eigenvalues of AAT and ATA
  18. 18. Robert Collins Computing F: The 8 pt AlgorithmCSE486, Penn State 8 8 8 8 8 8 8 8 8 8 8 8 A has rank 8 •Find eigenvector of ATA with smallest eigenvalue!
  19. 19. Robert CollinsCSE486, Penn State Algorithm EIGHT_POINT The input is formed by m point correspondences, m >= 8 • Construct the m x 9 matrix A • Find the SVD of A: A = UDVT • The entries of F are the components of the column of V corresponding to the least s.v.
  20. 20. Robert CollinsCSE486, Penn State Algorithm EIGHT_POINT F must be singular (remember, it is rank 2, since it is important for it to have a left and right nullspace, i.e. the epipoles). To enforce rank 2 constraint: • Find the SVD of F: F = UfDfVfT • Set smallest s.v. of F to 0 to create D’f • Recompute F: F = UfD’fVfT
  21. 21. Robert CollinsCSE486, Penn State Numerical Details Self-study • The coordinates of corresponding points can have a wide range leading to numerical instabilities. • It is better to first normalize them so they have average 0 and stddev 1 and denormalize F at the end: F = (T’)-1 Fn T Hartley preconditioning algorithm: this was an “optional” topic in Lecture 16. Go back and look if you want to know more.
  22. 22. Robert CollinsCSE486, Penn State A Practical Issue How to “rectify” the images so that any scan-line stereo alorithm that works for simple stereo can be used to find dense matches (i.e. compute a disparity image for every pixel). rectify general epipolar lines parallel epipolar lines
  23. 23. Robert Collins Stereo RectificationCSE486, Penn State• Image Reprojection – reproject image planes onto common plane parallel to line between optical centers• Notice, only focal point of camera really mattersSeitz, UW
  24. 24. Robert CollinsCSE486, Penn State General Idea Apply homography representing a virtual rotation to bring the image planes parallel with the baseline (epipoles go to infinity).
  25. 25. Robert CollinsCSE486, Penn State General Idea, continued
  26. 26. Robert CollinsCSE486, Penn State Image Rectification Method from T&V 7.3.7 Assuming extrinsic parameters R & T are known, compute a 3D rotation that makes conjugate epipolar lines collinear and parallel to the horizontal image axis Remember: a rotation around focal point of camera is just a 2D homography in the image! Note: this method from the book assumes calibrated cameras (we can recover R,T from the E matrix). In a moment, we will see a more general approch that uses F matrix.
  27. 27. Robert CollinsCSE486, Penn State Image Rectification • Rectification involves two rotations: – First rotation sends epipoles to infinity – Second rotation makes epipolar lines parallel • Rotate the left and right cameras with first R1 (constructed from translation T) • Rotate the right camera with the R matrix • Adjust scales in both camera references
  28. 28. Robert CollinsCSE486, Penn State Image Rectification Build the rotation: with: where T is just a unit vector representing the epipole in the left image. We know how to compute this from E, from last class.
  29. 29. Robert CollinsCSE486, Penn State Image Rectification Build the rotation: with: Verify that this homography maps e1 to [1 0 0]’
  30. 30. Robert CollinsCSE486, Penn State Algorithm Rectification • Build the matrix Rrect • Set Rl = Rrect and Rr = R.Rrect • For each left point pl = (x,y,f)T – compute Rl pl = (x’, y’, z’)T – Compute p’l = f/z’ (x’, y’ z’)T • Repeat above for the right camera with Rr
  31. 31. Robert CollinsCSE486, Penn State Example
  32. 32. Robert CollinsCSE486, Penn State Example Rectified Pair
  33. 33. Robert CollinsCSE486, Penn State A Better Approach The traditional method does not work when epipoles are in the image (for example, camera translating forward) Good paper: “Simple and efficient rectification methods for general motion”, Marc Pollefeys, R.Koch, L.VanGool, ICCV99. General idea: polar rectification around the epipoles.
  34. 34. Robert CollinsCSE486, Penn State Polar rectification (Pollefeys et al. ICCV’99) Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose Δθ so that no pixels are compressed original image rectified image Works for all relative motions Guarantees minimal image size
  35. 35. Robert CollinsCSE486, Penn State polar rectification: example
  36. 36. Robert CollinsCSE486, Penn State polar rectification: example
  37. 37. Robert CollinsCSE486, Penn State Example epipoles Input images
  38. 38. Robert CollinsCSE486, Penn State Example (cont) Rectified images
  39. 39. Robert CollinsCSE486, Penn State Example (cont) Depth Maps sparse interpolated
  40. 40. Robert CollinsCSE486, Penn State Example (cont) Depth map in pixel coords Views of texture mapped depth surface

×