Machine Vision

1. Course: Machine Vision
Image Matching
Session 12
D5627 – I Gede Putra Kusuma Negara, B.Eng., PhD

2. Outline
• Definition
• Invariant features
• Scale Invariant Feature Transform
• Feature matching

3. Definition

4. Definition
• Image matching can be defined as “the process of bringing two
images geometrically into agreement so that corresponding pixels in
the two images correspond to the same physical region of the scene
being imaged
• Image matching problem is accomplished by transforming (e.g.,
translating, rotating, scaling) one of the images in such a way that the
similarity with the other image is maximized in some sense
• The 3D nature of real-world scenarios makes this solution complex to
achieve, specially because images can be taken from arbitrary
viewpoints and in different illumination conditions.

5. Image Matching
• Image matching is a fundamental aspect of many problems in
computer vision, including object and scene recognition, content-
based image retrieval, stereo correspondence, motion tracking,
texture classification and video data mining.
• It is a complex problem, that remains challenging due to partial
occlusions, image deformations, and viewpoint or lighting changes
that may occur across different images

6. Image Matching (example)
by Diva Sian
by swashford

7. Feature Matching
• What stuff in the left image matches with stuff on the right?
• This information is important for for automatic panorama stitching

8. Harder Case
by Diva Sian by scgbt

9. Even Harder Case
NASA Mars Rover images with SIFT feature matches
Figure by Noah Snavely

10. Image Matching Procedure
1. Detection of distinguished regions
2. Local invariant description of these regions
3. Definition of the correspondence space
4. searching of a globally consistent subset of correspondences.

11. Invariant Features

12. Invariant Local Features
1. At an interesting
point, let’s define a
coordinate system
(x,y axis)
2. Use the coordinate
system to pull out a
patch at that point

13. Invariant Local Features
• Algorithm for finding points and representing their patches should produce similar
results even when conditions vary
• Buzzword is “invariance”
– Geometric invariance: translation, rotation, scale
– Photometric invariance: brightness, exposure, …etc.
Feature Descriptors

14. What makes good feature?
• We would like to align by matching local features of these 3 images
• What would be the good local features (ones easy to match)?

15. Local Measure of Uniqueness
Suppose we only consider a small window of pixels
• What defines whether a feature is a good or bad candidate?
• How does the window change when we shift it?
• Shifting the window in any direction causes a big change
“flat” region:
no change in all
directions
“edge”:
no change along
the edge direction
“corner”:
significant change
in all directions

16. Harris Corner Detector
(example)

17. Invariance
Suppose we are comparing two images I1 and I2
– I2 may be a transformed version of I1
– What kinds of transformations are we likely to encounter in
practice?
We’d like to find the same features regardless of the
transformation
– This is called transformational invariance
– Most feature methods are designed to be invariant to
• Translation, 2D rotation, scale
– They can usually also handle
• Limited 3D rotations (SIFT works up to about 60 degrees)
• Limited affine transformations (2D rotation, scale, shear)
• Limited illumination/contrast changes

18. How to achieve invariance?
1. Make sure our detector is invariant
– Harris is invariant to translation and rotation
– Scale is trickier
• common approach is to detect features at many scales using a Gaussian pyramid
(e.g., MOPS)
• More sophisticated methods find “the best scale” to represent each feature (e.g.,
SIFT)
2. Design an invariant feature descriptor
– A descriptor captures the information in a region around the
detected feature point
– The simplest descriptor: a square window of pixels
• What’s this invariant to?
– Let’s look at some better approaches…

19. Scale invariant detection
• Suppose we are looking for
corners
• Key idea: find scale that
gives local maximum of f
– f is a local maximum in both
position and scale
– Common definition of f:
Laplacian
(or difference between two
Gaussian filtered images with
different sigmas)

20. Slide from Tinne Tuytelaars
Lindeberg et al, 1996
Slide from Tinne Tuytelaars
Lindeberg et al., 1996

22. Rotation invariance for feature
descriptors
Find dominant orientation of the image patch
• This is given by x+, the eigenvector of H corresponding to +
– + is the larger eigenvalue
• Rotate the patch according to this angle
Figure by Matthew Brown

23. Multiscale Oriented PatcheS
descriptor
Take 40x40 square window
around detected feature
• Scale to 1/5 size (using
prefiltering)
• Rotate to horizontal
• Sample 8x8 square
window centered at
feature
• Intensity normalize the
window by subtracting
the mean, dividing by the
standard deviation in the
window (both window I
and aI+b will match)
8 pixels

24. Detections at Multiple Scales

25. Scale Invariant Feature Transform

26. Scale Invariant Feature
Transform
Basic idea:
• Take 16x16 square window
around detected feature
• Compute edge orientation (angle
of the gradient - 90 ) for each
pixel
• Throw out weak edges
(threshold gradient magnitude)
• Create histogram of surviving
edge orientations
0 2

27. SIFT Descriptor
Full version
• Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown
below)
• Compute an orientation histogram for each cell
• 16 cells * 8 orientations = 128 dimensional descriptor

28. SIFT Properties
Extraordinarily robust matching technique
• Can handle changes in viewpoint
• Up to about 60 degree out of plane rotation
• Can handle significant changes in illumination, sometimes even day
vs. night (next example)
• Fast and efficient—can run in real time
• Lots of code available
http://people.csail.mit.edu/albert/ladypack/wiki/index.php?title=Kno
wn_implementations_of_SIFT

29. SIFT in Action

30. Feature Matching

31. Feature Matching
Given a feature in I1, how to find
the best match in I2?
1. Define distance function that
compares two descriptors
2. Test all the features in I2, find the one
with min distance
How to define the difference
between two features f1, f2?
• Simple approach is SSD(f1, f2)
– Sum of square differences between
entries of the two descriptors
– Can give good scores to very
ambiguous (bad) matches
I1
f1
f2
I2

32. Feature Distance
How to define the difference
between two features f1, f2?
• Better approach: ratio
distance = SSD(f1, f2) /
SSD(f1, f2’)
– f2 is best SSD match to f1 in I2
– f2’ is 2nd best SSD match to f1 in
I2
– gives small values for ambiguous
matches
f1
f2
f2’
I1
I2

33. Evaluating the Results
• How can we measure the performance of a feature matcher?
50
75
200
feature distance

34. True/false Positive
The distance threshold affects performance
• True positives = # of detected matches that are correct
– Suppose we want to maximize these—how to choose threshold?
• False positives = # of detected matches that are incorrect
– Suppose we want to minimize these—how to choose threshold?
50
75
200
feature distance
false match
true match

35. Applications
Features are used for:
• Image alignment (e.g., mosaics)
• 3D reconstruction
• Motion tracking
• Object recognition
• Indexing and database retrieval
• Robot navigation
• … etc.

36. Acknowledgment
Some of slides in this PowerPoint presentation are adaptation from
various slides, many thanks to:
1. Deva Ramanan, Department of Computer Science, University of
California at Irvine (http://www.ics.uci.edu/~dramanan/)

37. Thank You