Robert CollinsCSE486, Penn State                         Lecture 31:                Object Recognition: SIFT Keys
Robert CollinsCSE486, Penn State                        Motivation        • Want to recognize a known objects from        ...
Robert Collins                 Local Feature based ApproachesCSE486, Penn State         • Represent appearance of object b...
Robert CollinsCSE486, Penn State                           Simple Example         • Represent object by set of 11x11 inten...
Robert CollinsCSE486, Penn State                     Simple Example         • Match patches to new image using NCC.
Robert CollinsCSE486, Penn State                      Simple Example         • Find matches consistent with affine        ...
Robert CollinsCSE486, Penn State                       Simple Example         • Inlier matches let you solve for location ...
Robert CollinsCSE486, Penn State                     Problem with Simple Example         Using NCC to match intensity patc...
Robert CollinsCSE486, Penn State                       More General : SIFT KeysDavid G. Lowe, "Distinctive image features ...
Robert CollinsCSE486, Penn State                     SIFT Keys: General Idea        • Reliably extract same image points r...
Robert CollinsCSE486, Penn State                           SIFT Keys: General Idea         Want to detect/match same featu...
Robert CollinsCSE486, Penn State                        Recall: Scale Space   Basic idea: different scales are appropriate...
Robert Collins                     Scale SelectionCSE486, Penn State                               “Laplacian” operator.
Robert CollinsCSE486, Penn State                      Local Scale Space Maxima         Lindeberg proposes that the natural...
Robert CollinsCSE486, Penn State                       Recall: LoG Blob Finding      LoG filter extrema locates “blobs”   ...
Robert CollinsCSE486, Penn State                     Extrema in Space and Scale                                           ...
Robert CollinsCSE486, Penn State                           SIFT Keys: General Idea         Want to detect/match same featu...
Robert CollinsCSE486, Penn State                     Sift Key Steps
Robert CollinsCSE486, Penn State                     Example                      •Keypoint location = extrema location   ...
Robert CollinsCSE486, Penn State                                Example (continued)                                       ...
Robert CollinsCSE486, Penn State                     Example (continued)                     .*                     =     ...
Robert CollinsCSE486, Penn State                        Example (continued)    weighted gradient    magnitude             ...
Robert CollinsCSE486, Penn State                        Example (continued)    weighted gradient    magnitude             ...
Robert CollinsCSE486, Penn State                           Example (continued)            There may be multiple orientatio...
Robert CollinsCSE486, Penn State                     Example of KeyPoint Detection            Each keypoint has a center p...
Robert CollinsCSE486, Penn State                        SIFT Vector     to make things more insensitive to changes in     ...
Robert CollinsCSE486, Penn State                     SIFT Vector
Robert CollinsCSE486, Penn State                     Sift Key Matching
Robert CollinsCSE486, Penn State                     Model Verification
Robert Collins                 Application: Object RecognitionCSE486, Penn State                                    Comput...
Robert Collins                 Application: Object RecognitionCSE486, Penn State                                find them ...
Robert Collins                 Application: Object RecognitionCSE486, Penn State         For sets of 3 SIFT key matches, c...
Robert Collins                 Application: Object RecognitionCSE486, Penn State     Note: since these are local, parts-ba...
Robert Collins               Application: Landmark RecognitionCSE486, Penn State
Robert CollinsCSE486, Penn State                         Application: Generating PanoramasBrown and Lowe, ICCV03
Robert Collins               Application: Generating PanoramasCSE486, Penn StateBrown and Lowe, ICCV03
Robert CollinsCSE486, Penn State                           State of the Art             Fully affine invariant local featu...
Robert CollinsCSE486, Penn State                       State of the Art     Affine invariant descriptors can handle larger...
Robert CollinsCSE486, Penn State                     State of the Art
Robert CollinsCSE486, Penn State                            For More Information       SIFT keys          David G. Lowe, "...
Upcoming SlideShare
Loading in...5
×

Lecture31

802

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
802
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
52
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Lecture31

  1. 1. Robert CollinsCSE486, Penn State Lecture 31: Object Recognition: SIFT Keys
  2. 2. Robert CollinsCSE486, Penn State Motivation • Want to recognize a known objects from unknown viewpoints. find them in an image database of models
  3. 3. Robert Collins Local Feature based ApproachesCSE486, Penn State • Represent appearance of object by little intensity/feature patches. • Try to match patches from object to image • Geometrically consistent matches tell you the location and pose of the object
  4. 4. Robert CollinsCSE486, Penn State Simple Example • Represent object by set of 11x11 intensity templates extracted around Harris corners. harris corners our object “model”
  5. 5. Robert CollinsCSE486, Penn State Simple Example • Match patches to new image using NCC.
  6. 6. Robert CollinsCSE486, Penn State Simple Example • Find matches consistent with affine transformation using RANSAC
  7. 7. Robert CollinsCSE486, Penn State Simple Example • Inlier matches let you solve for location and pose of object in the image.
  8. 8. Robert CollinsCSE486, Penn State Problem with Simple Example Using NCC to match intensity patches puts restrictions on the amount of overall rotation and scaling allowed between the model and the image appearance. model template ncc ncc ncc matches well no match no match
  9. 9. Robert CollinsCSE486, Penn State More General : SIFT KeysDavid G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal ofComputer Vision, 60, 2 (2004), pp. 91-110.
  10. 10. Robert CollinsCSE486, Penn State SIFT Keys: General Idea • Reliably extract same image points regardless of new magnification and rotation of the image. • Normalize image patches, extract feature vector • Match feature vectors using correlation
  11. 11. Robert CollinsCSE486, Penn State SIFT Keys: General Idea Want to detect/match same features regardless of Translation : easy, almost every feature extraction and correlation matching algorithm in vision is translation invariant Rotation : harder. Guess a canonical orientation for each patch from local gradients Scaling : hardest of all. Create a multi-scale representation of the image and appeal to scale space theory to determine correct scale at each point.
  12. 12. Robert CollinsCSE486, Penn State Recall: Scale Space Basic idea: different scales are appropriate for describing different objects in the image, and we may not know the correct scale/size ahead of time.
  13. 13. Robert Collins Scale SelectionCSE486, Penn State “Laplacian” operator.
  14. 14. Robert CollinsCSE486, Penn State Local Scale Space Maxima Lindeberg proposes that the natural scale for describing a feature is the scale at which a normalized derivative for detecting that feature achieves a local maximum both spatially and in scale. DnormL is the DoG operator, in this case. Scale Example for blob detection
  15. 15. Robert CollinsCSE486, Penn State Recall: LoG Blob Finding LoG filter extrema locates “blobs” maxima = dark blobs on light background minima = light blobs on dark background Scale of blob (size ; radius in pixels) is determined by the sigma parameter of the LoG filter. LoG sigma = 2 LoG sigma = 10
  16. 16. Robert CollinsCSE486, Penn State Extrema in Space and Scale DOG level L-1 Scale (sigma) DOG level L DOG level L+1 Space Hint: when finding maxima or minima at level L, use DownSample or UpSample as necessary to make DOG images at level L-1 and L+1 the same size as L.
  17. 17. Robert CollinsCSE486, Penn State SIFT Keys: General Idea Want to detect/match same features regardless of Translation : easy, almost every feature extraction and correlation matching algorithm in vision is translation invariant Rotation : harder. Guess a canonical orientation for each patch from local gradients Scaling : hardest of all. Create a multi-scale representation of the image and appeal to scale space theory to determine correct scale at each point.
  18. 18. Robert CollinsCSE486, Penn State Sift Key Steps
  19. 19. Robert CollinsCSE486, Penn State Example •Keypoint location = extrema location •Keypoint scale is scale of the DOG image
  20. 20. Robert CollinsCSE486, Penn State Example (continued) gradient magnitude gaussian image (at closest scale, from pyramid) gradient orientation
  21. 21. Robert CollinsCSE486, Penn State Example (continued) .* = gradient weighted by 2D weighted gradient magnitude gaussian kernel magnitude
  22. 22. Robert CollinsCSE486, Penn State Example (continued) weighted gradient magnitude weighted orientation histogram. Each bucket contains sum of weighted gradient magnitudes corresponding to angles that fall within that bucket. gradient orientation 36 buckets 10 degree range of angles in each bucket, i.e. 0 <=ang<10 : bucket 1 10<=ang<20 : bucket 2 20<=ang<30 : bucket 3 …
  23. 23. Robert CollinsCSE486, Penn State Example (continued) weighted gradient magnitude weighted orientation histogram. peak 80% of peak value gradient orientation 20-30 degrees Orientation of keypoint is approximately 25 degrees
  24. 24. Robert CollinsCSE486, Penn State Example (continued) There may be multiple orientations. peak Second peak 80% of peak value In this case, generate duplicate keypoints, one with orientation at 25 degrees, one at 155 degrees. Design decision: you may want to limit number of possible multiple peaks to two.
  25. 25. Robert CollinsCSE486, Penn State Example of KeyPoint Detection Each keypoint has a center point (location), an orientation (rotation) and a radius (scale). At this point, we could try to correlate patches (after first normalizing to a canonical orientation and scale).
  26. 26. Robert CollinsCSE486, Penn State SIFT Vector to make things more insensitive to changes in lighting or small changes in geometry, Lowe constructs feature vector from image gradients.
  27. 27. Robert CollinsCSE486, Penn State SIFT Vector
  28. 28. Robert CollinsCSE486, Penn State Sift Key Matching
  29. 29. Robert CollinsCSE486, Penn State Model Verification
  30. 30. Robert Collins Application: Object RecognitionCSE486, Penn State Compute SIFT keys of models and store in a database
  31. 31. Robert Collins Application: Object RecognitionCSE486, Penn State find them in an image database of models
  32. 32. Robert Collins Application: Object RecognitionCSE486, Penn State For sets of 3 SIFT key matches, compute affine transformation and perform model verification.
  33. 33. Robert Collins Application: Object RecognitionCSE486, Penn State Note: since these are local, parts-based descriptors, they perform well even when some parts are missing (i.e. under occlusion).
  34. 34. Robert Collins Application: Landmark RecognitionCSE486, Penn State
  35. 35. Robert CollinsCSE486, Penn State Application: Generating PanoramasBrown and Lowe, ICCV03
  36. 36. Robert Collins Application: Generating PanoramasCSE486, Penn StateBrown and Lowe, ICCV03
  37. 37. Robert CollinsCSE486, Penn State State of the Art Fully affine invariant local feature descriptors.
  38. 38. Robert CollinsCSE486, Penn State State of the Art Affine invariant descriptors can handle larger changes in viewpoint.
  39. 39. Robert CollinsCSE486, Penn State State of the Art
  40. 40. Robert CollinsCSE486, Penn State For More Information SIFT keys David G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, 60, 2 (2004), pp. 91-110. Affine local-feature methods
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×