AAAI08 tutorial: visual object recognition

Visual Object Recognition
Visual Object Recognition Tutorial Computing
Perceptual and Sensory Augmented

Bastian Leibe & Kristen Grauman
Computer Vision Laboratory Department of Computer Sciences
ETH Zurich University of Texas in Austin

Chicago, 14.07.2008


K. Grauman, B. Leibe
?
???
Identification vs. Categorization

2

Object Categorization
• How to recognize ANY car

• How to recognize ANY cow

3

What could be done with recognition algorithms?
There is a wide range of applications, including…

Autonomous robots Navigation, driver safety Situated search

Content-based retrieval and analysis for
Medical image
images and videos
analysis

Object Categorization
• Task Description
“Given a small number of training images of a category,

recognize a-priori unknown instances of that category and assign
the correct category label.”

• Which categories are feasible visually?

Extensively studied in Cognitive Psychology,
e.g. [Brown’58]

“Fido” German dog animal living
shepherd being

5

Visual Object Categories

• Basic Level Categories in human categorization
[Rosch 76, Lakoff 87]

The highest level at which category members have similar
perceived shape
The highest level at which a single mental image reflects the

entire category
The level at which human subjects are usually fastest at
identifying category members
The first level named and understood by children
The highest level at which a person uses similar motor actions
for interaction with category members

6

Visual Object Categories
• Basic-level categories in humans seem to be defined
predominantly visually.

• There is evidence that humans (usually)
…
start with basic-level categorization
before doing identification.

animal
⇒ Basic-level categorization is easier
Abstract
and faster for humans than object levels
… …
identification! quadruped
⇒ Most promising starting point
…
for visual classification
Basic level dog cat cow

German
Doberman
shepherd

Individual … “Fido” …
level 7

Other Types of Categories
• Functional Categories
e.g. chairs = “something you can sit on”

8

Other Types of Categories
• Ad-hoc categories
e.g. “something you can find in an office environment”

9

Levels of Object Categorization

“cow”

“car”

“motorbike”

• Different levels of recognition
Which object class is in the image? ⇒ Obj/Img classification
Where is it in the image? ⇒ Detection/Localization
Where exactly ― which pixels? ⇒ Figure/Ground
segmentation

10

Challenges: robustness

Illumination Object pose Clutter

Occlusions Intra-class Viewpoint
appearance

Challenges: robustness

• Detection in Crowded Scenes
Learn object variability
– Changes in appearance, scale, and articulation
Compensate for clutter, overlap, and occlusion

12


Challenges: context and human experience

Challenges: context and human experience

Context cues Dynamics

Image credit: D. Hoeim Video credit: J. Davis

Challenges: scale, efficiency
• Thousands to millions of pixels in an image
• Estimated 30 Gigapixels of image/video content
generated per second

• About half of the cerebral cortex in primates is devoted
to processing visual information [Felleman and van
Essen 1991]

• 3,000-30,000 human recognizable object categories
• 30+ degrees of freedom in the pose of articulated
objects (humans)
• Billions of images indexed by Google Image Search
• 18 billion+ prints produced from digital camera images
in 2004
• 295.5 million camera phones sold in 2005



Less

More
Challenges: learning with minimal supervision

Rough evolution of focus in recognition research

1980s 1990s to early 2000s Currently


This tutorial
• Intended for broad AAAI audience
Assuming basic familiarity with machine learning, linear algebra,

probability
Not assuming significant vision background

• Our goals
Describe main approaches to recognition
Highlight past successes and future challenges
Provide the pointers (to literature and tools) that would allow
you to take advantage of existing techniques in your research

• Questions welcome

18

Outline

1. Detection with Global Appearance & Sliding Windows

2. Local Invariant Features: Detection & Description
3. Specific Object Recognition with Local Features

― Coffee Break ―
4. Visual Words: Indexing, Bags of Words Categorization
5. Matching Local Features
6. Part-Based Models for Categorization
7. Current Challenges and Research Directions

19

Outline




2

Detection via classification: Main idea

Basic component: a binary classifier

Car/non-car
Classifier

No, notcar.
Yes, a car.



If object may be in a cluttered scene, slide a window
around looking for it.

Car/non-car
Classifier


Fleshing out this
pipeline a bit more,
we need to:

1. Obtain training data
2. Define features
3. Define classifier

Training examples

Car/non-car
Classifier
Feature
extraction


• Consider all subwindows in an image
Sample at multiple scales and positions

• Make a decision per window:
“Does this contain object category X or not?”

• In this section, we’ll focus specifically on methods
using a global representation (i.e., not part-based,
not local features).

6

Feature extraction:
global appearance
Feature
extraction

Simple holistic descriptions of image content
grayscale / color histogram
vector of pixel intensities


Eigenfaces: global appearance description
An early appearance-based approach to face recognition

Generate low-

dimensional
representation
Mean of appearance
with a linear

Eigenvectors computed
Training images
from covariance matrix subspace.

... Project new
images to “face
≈ space”.
+ + ++
Mean Recognition via
nearest neighbors
in face space
Turk & Pentland, 1991

Feature extraction: global appearance
• Pixel-based representations sensitive to small shifts

• Color or grayscale-based appearance description can be
sensitive to illumination and intra-class appearance
variation
Cartoon example:
an albino koala


Gradient-based representations
• Consider edges, contours, and (oriented) intensity
gradients


Gradient-based representations:
Matching edge templates
• Example: Chamfer matching

Input Edges Distance Template Best
image detected transform shape match

At each window position,
compute average min
distance between points on
template (T) and input (I).

Gavrila & Philomin ICCV 1999

Matching edge templates
• Chamfer matching

Hierarchy of templates

Gavrila & Philomin ICCV 1999

Gradient-based representations
• Consider edges, contours, and (oriented) intensity
gradients

• Summarize local distribution of gradients with histogram
Locally orderless: offers invariance to small shifts and rotations
Contrast-normalization: try to correct for variable illumination


Histograms of oriented gradients (HoG)

Map each grid cell in the input
window to a histogram counting
the gradients per orientation.

Code available:
http://pascal.inrialpes.fr/soft/olt/
Dalal & Triggs, CVPR 2005

SIFT descriptor

Local patch descriptor
(more on this later)

Code: http://vision.ucla.edu/~vedaldi/code/sift/sift.html
Binary: http://www.cs.ubc.ca/~lowe/keypoints/
Lowe, ICCV 1999

Biologically inspired features

Convolve with Gabor filters at
multiple orientations
Pool nearby units (max)
Intermediate layers compare input
to prototype patches

Serre, Wolf, Poggio, CVPR 2005
Mutch & Lowe, CVPR 2006 K. Grauman, B. Leibe

Rectangular features

Compute differences between sums of pixels in rectangles
Captures contrast in adjacent spatial regions
Similar to Haar wavelets, efficient to compute

Viola & Jones, CVPR 2001

Shape context descriptor

Count the number of points
inside each bin, e.g.:

Count = 4

...
Count = 10

Log-polar binning: more
precision for nearby points,
more flexibility for farther
points.

Local descriptor
Belongie, Malik & Puzicha, ICCV 2001 (more on this later)

Classifier construction

• How to compute a decision for each

subwindow?

Image feature


Discriminative vs. generative models

Pr(image, car ) Pr(image, ¬car ) Generative: separately

0.1

0.05
model class-conditional
and prior densities
0
0 10 20 30 40 50 60 70
image feature

Pr(car | image) Pr(¬car | image) Discriminative: directly
1
x = data
model posterior
0.5

0
0 10 20 30 40 50 60 70
image feature

Plots from Antonio Torralba 2007 K. Grauman, B. Leibe

Discriminative vs. generative models
• Generative:
+ possibly interpretable

+ can draw samples
- models variability unimportant to classification task
- often hard to build good model with few parameters

• Discriminative:
+ appealing when infeasible to model data itself
+ excel in practice
- often can’t provide uncertainty in predictions
- non-interpretable

21

Discriminative methods
Nearest neighbor Neural networks

106 examples

Shakhnarovich, Viola, Darrell 2003 LeCun, Bottou, Bengio, Haffner 1998
Berg, Berg, Malik 2005... Rowley, Baluja, Kanade 1998

…

Support Vector Machines Boosting Conditional Random Fields

Guyon, Vapnik Viola, Jones 2001, McCallum, Freitag, Pereira
Heisele, Serre, Poggio, Torralba et al. 2004, 2000; Kumar, Hebert 2003
2001,… Opelt et al. 2006,… …

K. Grauman, B. Leibe Slide adapted from Antonio Torralba

Boosting
• Build a strong classifier by combining number of “weak
classifiers”, which need only be better than chance

• Sequential learning process: at each iteration, add a
weak classifier
• Flexible to choice of weak learner

including fast simple classifiers that alone may be inaccurate

• We’ll look at Freund & Schapire’s AdaBoost algorithm
Easy to implement
Base learning algorithm for Viola-Jones face detector

23

AdaBoost: Intuition

Consider a 2-d feature
space with positive and

negative examples.

Each weak classifier splits

the training examples with
at least 50% accuracy.

Examples misclassified by
a previous weak learner
are given more emphasis
at future rounds.

Figure adapted from Freund and Schapire
24


AdaBoost: Intuition

25

AdaBoost: Intuition

Final classifier is
combination of the
weak classifiers

26

AdaBoost Algorithm
Start with
uniform weights
on training
examples
{x1,…xn}

Evaluate

weighted error
for each feature,
pick best.

Incorrectly classified -> more weight
Correctly classified -> less weight

Final classifier is combination of the
weak ones, weighted according to
error they had.
Freund & Schapire 1995

Cascading classifiers for detection
For efficiency, apply less
accurate but faster classifiers
first to immediately discard

windows that clearly appear to
be negative; e.g.,

Filter for promising regions with an
initial inexpensive classifier
Build a chain of classifiers, choosing
cheap ones with low false negative
rates early in the chain

Fleuret & Geman, IJCV 2001
Rowley et al., PAMI 1998
Viola & Jones, CVPR 2001 28
K. Grauman, B. Leibe Figure from Viola & Jones CVPR 2001

Example: Face detection
• Frontal faces are a good example of a class where
global appearance models + a sliding window

detection approach fit well:
Regular 2D structure
Center of face almost shaped like a “patch”/window

• Now we’ll take AdaBoost and see how the Viola-
Jones face detector works
29

Feature extraction
“Rectangular” filters
Feature output is difference
between adjacent regions

Value at (x,y) is
sum of pixels
Efficiently computable above and to the
with integral image: any left of (x,y)
sum can be computed
in constant time
Avoid scaling images
scale features directly Integral image
for same cost

Viola & Jones, CVPR 2001 30

Large library of filters
Considering all
possible filter
parameters:

position, scale,
and type:

180,000+
possible features
associated with
each 24 x 24
window

Use AdaBoost both to select the informative
features and to form the classifier


AdaBoost for feature+classifier selection
• Want to select the single rectangle feature and threshold
that best separates positive (faces) and negative (non-
faces) training examples, in terms of weighted error.

Resulting weak classifier:

For next round, reweight the
…

examples according to errors,
Outputs of a possible choose another filter/threshold
rectangle feature on
combo.
faces and non-faces.


Viola-Jones Face Detector: Summary

Train cascade of
classifiers with

AdaBoost

ow h
ind eac
Faces

bw o
New image

su ply t

Ap
Selected features,
Non-faces thresholds, and weights

• Train with 5K positives, 350M negatives
• Real-time detector using 38 layer cascade
• 6061 features in final layer
• [Implementation available in OpenCV:
http://www.intel.com/technology/computing/opencv/]
33

Viola-Jones Face Detector: Results

First two features
selected

34

Profile Features
Detecting profile faces requires training separate
detector with profile examples.


Paul Viola, ICCV tutorial

Example application

Frontal faces

detected and
then tracked,
character

names inferred
with alignment
of script and
subtitles.

Everingham, M., Sivic, J. and Zisserman, A.
"Hello! My name is... Buffy" - Automatic naming of characters in TV video,
BMVC 2006.
http://www.robots.ox.ac.uk/~vgg/research/nface/index.html
40

Pedestrian detection
• Detecting upright, walking humans also possible using sliding
window’s appearance/texture; e.g.,

SVM with Haar wavelets Space-time rectangle SVM with HoGs [Dalal &
[Papageorgiou & Poggio, IJCV features [Viola, Jones & Triggs, CVPR 2005]
2000] Snow, ICCV 2003]


Highlights
• Sliding window detection and global appearance
descriptors:

Simple detection protocol to implement
Good feature choices critical
Past successes for certain classes

42

Limitations
• High computational complexity
For example: 250,000 locations x 30 orientations x 4 scales =
30,000,000 evaluations!

If training binary detectors independently, means cost increases
linearly with number of classes
• With so many windows, false positive rate better be low

43

Limitations (continued)
• Not all objects are “box” shaped

44

• Non-rigid, deformable objects not captured well with
representations assuming a fixed 2d structure; or must
assume fixed viewpoint

• Objects with less-regular textures not captured well
with holistic appearance-based descriptions

45

• If considering windows in isolation, context is lost

Sliding window Detector’s view

46
Figure credit: Derek Hoiem K. Grauman, B. Leibe

• In practice, often entails large, cropped training set
(expensive)

• Requiring good match to a global appearance description
can lead to sensitivity to partial occlusions

47
Image credit: Adam, Rivlin, & Shimshoni K. Grauman, B. Leibe

Outline




48

Outline




2

Motivation
• Global representations have major limitations
• Instead, describe and match only local regions

• Increased robustness to
Occlusions

Articulation
d dq
φ
φ
θq
θ

Intra-category variations

3

Approach
1. Find a set of
distinctive key-
points

A1
2. Define a region
around each
A2 A3 keypoint

3. Extract and
normalize the
region content
fA Similarity fB
measure
4. Compute a local
N pixels

descriptor from the
e.g. color e.g. color
normalized region
N pixels d ( f A, fB ) < T
5. Match local
descriptors
4

Requirements
• Region extraction needs to be repeatable and precise
Translation, rotation, scale changes

(Limited out-of-plane (≈affine) transformations)
≈
Lighting variations

• We need a sufficient number of regions to cover the
object

• The regions should contain “interesting” structure

5

Many Existing Detectors Available
• Hessian & Harris [Beaudet ‘78], [Harris ‘88]
• Laplacian, DoG [Lindeberg ‘98], [Lowe 1999]

• Harris-/Hessian-Laplace [Mikolajczyk & Schmid ‘01]
• Harris-/Hessian-Affine [Mikolajczyk & Schmid ‘04]
• EBR and IBR [Tuytelaars & Van Gool ‘04]

• MSER [Matas ‘02]
• Salient Regions [Kadir & Brady ‘01]
• Others…

6

Keypoint Localization

• Goals:
Repeatable detection
Precise localization
Interesting content
⇒ Look for two-dimensional signal changes
7

Hessian Detector [Beaudet78]
• Hessian determinant
Ixx

 I xx I xy 
Hessian ( I ) = 
 I xy I yy 


Iyy
Ixy

Intuition: Search for strong
derivatives in two
orthogonal directions
8

Hessian Detector [Beaudet78]
• Hessian determinant
Ixx

 I xx I xy 
Hessian ( I ) = 
 I xy I yy 


Iyy
Ixy

2
det( Hessian( I )) = I xx I yy − I xy
In Matlab:
I xx . ∗ I yy − ( I xy )^ 2
9

Hessian Detector – Responses [Beaudet78]

Effect: Responses mainly
on corners and strongly
textured areas.

10


Hessian Detector – Responses [Beaudet78]

11

Harris Detector [Harris88]
• Second moment matrix
(autocorrelation matrix)

 I x2 (σ D ) I x I y (σ D )
µ (σ I , σ D ) = g (σ I ) ∗  2 

 I x I y (σ D ) I y (σ D ) 


Intuition: Search for local
neighborhoods where the
image content has two main
directions (eigenvectors).

12


 I x2 (σ D ) I x I y (σ D )
µ (σ I , σ D ) = g (σ I ) ∗  2 

 I x I y (σ D ) I y (σ D ) 


Ix Iy

1. Image derivatives
gx(σD), gy(σD),

13


 I x2 (σ D ) I x I y (σ D )
µ (σ I , σ D ) = g (σ I ) ∗  2 

 I x I y (σ D ) I y (σ D ) 


Ix Iy

1. Image derivatives
gx(σD), gy(σD),

Ix2 Iy2 IxIy
2. Square of
derivatives
14


 I x2 (σ D ) I x I y (σ D )
µ (σ I , σ D ) = g (σ I ) ∗  2 

 I x I y (σ D ) I y (σ D ) 

Ix Iy
1. Image

derivatives
Iy
2. Square of Ix2 Iy2 IxIy
1. Image derivatives derivatives
gx(σD), gy(σD),

2.3. Square of
Gaussian
filter g(σI)
derivatives
g(Ix2) g(Iy2) g(IxIy)
15

 I x2 (σ D ) I x I y (σ D )

µ (σ I , σ D ) = g (σ I ) ∗  2  Ix Iy
 I x I y (σ D ) I y (σ D )  1. Image
 
derivatives

Ix2 Iy2 IxIy

2. Square of
derivatives Iy

3. Gaussian
filter g(σI)
g(Ix2) g(Iy2) g(IxIy)
4. Cornerness function – both eigenvalues are strong
har = det[µ (σ I ,σ D)] − α [trace(µ (σ I ,σ D))] =
g ( I x2 ) g ( I y ) − [ g ( I x I y )]2 − α [ g ( I x2 ) + g ( I y )]2
2 2

g(IxIy)
5. Non-maxima suppression har 16

Harris Detector – Responses [Harris88]

Effect: A very precise
corner detector.

17


Harris Detector – Responses [Harris88]

18

Automatic Scale Selection

f ( I i1Kim ( x, σ )) = f ( I i1Kim ( x′, σ ′))

Same operator responses if the patch contains the same image up
to scale factor
How to find corresponding patch sizes?

19

• Function responses for increasing scale (scale signature)

f ( I i1Kim ( x, σ )) f ( I i1Kim ( x′, σ ))
20


21


22


23


24


f ( I i1Kim ( x, σ )) f ( I i1Kim ( x′, σ ′))
25

What Is A Useful Signature Function?
• Laplacian-of-Gaussian = “blob” detector

26

Laplacian-of-Gaussian (LoG)
• Local maxima in scale
σ5
space of Laplacian-of-

Gaussian
σ4

Lxx (σ ) + Lyy (σ ) σ3

σ2

⇒ List of
σ (x, y, s)

27


Results: Laplacian-of-Gaussian

28

Difference-of-Gaussian (DoG)
• Difference of Gaussians as approximation of the
Laplacian-of-Gaussian

- =

29

DoG – Efficient Computation
• Computation in Gaussian scale pyramid

Sampling with
step σ4 =2

σ

σ

1 σ
Original image σ =2 4
σ

30


Results: Lowe’s DoG

31

Harris-Laplace [Mikolajczyk ‘01]
1. Initialization: Multiscale Harris corner detection

σ4

σ3

σ2

σ

Computing Harris function Detecting local maxima 32

Harris-Laplace [Mikolajczyk ‘01]
1. Initialization: Multiscale Harris corner detection
2. Scale selection based on Laplacian

(same procedure with Hessian ⇒ Hessian-Laplace)

Harris points

Harris-Laplace points
33

Maximally Stable Extremal Regions [Matas ‘02]
• Based on Watershed segmentation algorithm
• Select regions that stay stable over a large parameter

range

34


Example Results: MSER

35

You Can Try It At Home…
• For most local feature detectors, executables are
available online:

• http://robots.ox.ac.uk/~vgg/research/affine
• http://www.cs.ubc.ca/~lowe/keypoints/
• http://www.vision.ee.ethz.ch/~surf

36

Orientation Normalization
• Compute orientation histogram [Lowe, SIFT, 1999]
• Select dominant orientation

• Normalize: rotate to fixed orientation

0 2π
37
T. Tuytelaars, B. Leibe

Local Descriptors
• The ideal descriptor should be
Repeatable

Distinctive
Compact
Efficient

• Most available descriptors focus on edge/gradient
information
Capture texture information
Color still relatively seldomly used
(more suitable for homogenous regions)

38

Local Descriptors: SIFT Descriptor

Histogram of oriented gradients
• Captures important texture
information
• Robust to small translations /
affine deformations
[Lowe, ICCV 1999]

Local Descriptors: SURF
• Fast approximation of SIFT idea
Efficient computation by 2D box
filters & integral images

⇒ 6 times faster than SIFT
Equivalent quality for object
identification

• GPU implementation available
Feature extraction @ 100Hz
(detector + descriptor, 640×480 img)
http://www.vision.ee.ethz.ch/~surf

[Bay, ECCV’06], [Cornelis, CVGPU’08] 40

Local Descriptors: Shape Context

Count the number of points
inside each bin, e.g.:

Count = 4

...

Count = 10

Log-polar binning: more
precision for nearby points,
more flexibility for farther
points.

Belongie & Malik, ICCV 2001

Local Descriptors: Geometric Blur

Compute edges

at four
orientations
Extract a patch

in each channel

~
Apply spatially varying
blur and sub-sample
Example descriptor
(Idealized signal)

Berg & Malik, CVPR 2001

So, What Local Features Should I Use?
• There have been extensive evaluations/comparisons
[Mikolajczyk et al., IJCV’05, PAMI’05]

All detectors/descriptors shown here work well

• Best choice often application dependent

MSER works well for buildings and printed things
Harris-/Hessian-Laplace/DoG work well for many natural
categories

• More features are better
Combining several detectors often helps

43

Outline




44

Recognition with Local Features
• Image content is transformed into local features that
are invariant to translation, rotation, and scale
• Goal: Verify if they belong to a consistent configuration

Local Features,
e.g. SIFT
3
K. Grauman, B. Leibe Slide credit: David Lowe

Finding Consistent Configurations
• Global spatial models
Generalized Hough Transform [Lowe99]

RANSAC [Obdrzalek02, Chum05, Nister06]
Basic assumption: object is planar

• Assumption is often justified in practice
Valid for many structures on
buildings
Sufficient for small viewpoint
variations on 3D objects

4

Hough Transform
• Origin: Detection of straight lines in clutter
Basic idea: each candidate point votes
for all lines that it is consistent with.

Votes are accumulated in quantized array
Local maxima correspond to candidate lines

• Representation of a line
Usual form y = a x + b has a singularity around 90º.
Better parameterization: x cos(θ) + y sin(θ) = ρ

y
ρ
y ρ
θ
x
x θ
5

Hough Transform: Noisy Line

ρ

θ
Tokens Votes

• Problem: Finding the true maximum

7

Hough Transform: Noisy Input

ρ

θ
Tokens Votes

• Problem: Lots of spurious maxima

8

Generalized Hough Transform [Ballard81]
• Generalization for an arbitrary contour or shape
Choose reference point for the contour (e.g. center)

For each point on the contour remember where it is located
w.r.t. to the reference point
Remember radius r and angle φ
relative to the contour tangent

Recognition: whenever you find
a contour point, calculate the
tangent angle and ‘vote’ for all
possible reference points

Instead of reference point, can also vote for transformation
⇒ The same idea can be used with local features!

9
K. Grauman, B. Leibe Slide credit: Bernt Schiele

Gen. Hough Transform with Local Features
• For every feature, store possible “occurrences”

• For new image, let the matched features vote for
– Object identity
possible object positions – Pose
– Relative position

10

3D Object Recognition
• Gen. HT for Recognition [Lowe99]

Typically only 3 feature matches

needed for recognition
Extra matches provide robustness
Affine model can be used for
planar objects

12

View Interpolation
• Training
Training views from similar

viewpoints are clustered
based on feature matches.
Matching features between
adjacent views are linked.

• Recognition
Feature matches may be
spread over several
training viewpoints.
⇒ Use the known links to “transfer votes” to other viewpoints.

[Lowe01]
13

Recognition Using View Interpolation

[Lowe01]
14

Location Recognition

Training

[Lowe04]
15

Applications
• Sony Aibo
(Evolution Robotics)

• SIFT usage
Recognize

docking station
Communicate
with visual cards

• Other uses
Place recognition
Loop closure in SLAM

16

RANSAC (RANdom SAmple Consensus) [Fischler81]
• Randomly choose a minimal subset of data points
necessary to fit a model (a sample)
• Points within some distance threshold t of model are a

consensus set. Size of consensus set is model’s support.
• Repeat for N samples; model with biggest support is
most robust fit

Points within distance t of best model are inliers
Fit final model to all inliers

17

RANSAC: How many samples?
• How many samples are needed?
Suppose w is fraction of inliers (points from line).
n points needed to define hypothesis (2 for lines)

k samples chosen.

• Prob. that a single sample of n points is correct: w n

• Prob. that all samples fail is: (1 − wn ) k

⇒ Choose k high enough to keep this below desired failure
rate.

19

After RANSAC
• RANSAC divides data into inliers and outliers and yields
estimate computed from minimal set of inliers

• Improve this initial estimate with estimation over all
inliers (e.g. with standard least-squares minimization)
• But this may change inliers, so alternate fitting with re-

classification as inlier/outlier

21

Example: Finding Feature Matches
• Find best stereo match within a square search window
(here 300 pixels2)

• Global transformation model: epipolar geometry

from Hartley & Zisserman

22

Example: Finding Feature Matches
• Find best stereo match within a square search window
(here 300 pixels2)

• Global transformation model: epipolar geometry
before RANSAC after RANSAC

from Hartley & Zisserman

23

Comparison
Gen. Hough Transform RANSAC
• Advantages • Advantages
Very effective for recognizing General method suited to large

arbitrary shapes or objects range of problems
Can handle high percentage of Easy to implement
outliers (>95%) Independent of number of
Extracts groupings from clutter in dimensions

linear time

• Disadvantages • Disadvantages
Quantization issues Only handles moderate number of
Only practical for small number of outliers (<50%)
dimensions (up to 4)

• Improvements available • Many variants available, e.g.
Probabilistic Extensions PROSAC: Progressive RANSAC
[Leibe08] [Chum05]
Continuous Voting Space
Preemptive RANSAC [Nister05]

24

Example Applications

Mobile tourist guide
• Self-localization
• Object/building recognition
• Photo/video augmentation

B. Leibe
[Quack, Leibe, Van Gool, CIVR’08] 25

Web Demo: Movie Poster Recognition

50’000 movie
posters indexed

Query-by-image
from mobile phone
available in Switzer-
land
http://www.kooaba.com/en/products_engine.html#
26

Application: Large-Scale Retrieval

Query Results from 5k Flickr images (demo available for 100k set)
K. Grauman, B. Leibe [Philbin CVPR’07] 27

Application: Image Auto-Annotation
Moulin Rouge Old Town Square (Prague)

Tour Montparnasse Colosseum

Viktualienmarkt
Maypole

Left: Wikipedia image
Right: closest match from Flickr

28
[Quack CIVR’08] K. Grauman, B. Leibe

Outline




29

Outline



5. Matching Local Feature Sets

2

Global representations: limitations
• Success may rely on alignment
-> sensitive to viewpoint

• All parts of the image or window impact the description
-> sensitive to occlusion, clutter

3

Local representations
• Describe component regions or patches separately.
• Many options for detection & description…

Maximally Stable
Extremal Regions
Shape context Superpixels [Matas 02]
SIFT [Lowe 99] [Belongie 02] [Ren et al.]

Salient regions Harris-Affine Spin images Geometric Blur
[Kadir 01] [Mikolajczyk 04] [Johnson 99] [Berg 05]
4

Recall: Invariant local features
Subset of local feature types
designed to be invariant to y1

Scale y2
Translation …
Rotation yd

Affine transformations
Illumination
x1
x2
1) Detect interest points …
2) Extract descriptors xd

[Mikolajczyk01, Matas02, Tuytelaars04, Lowe99, Kadir01,… ]


Recognition with local feature sets
• Previously, we saw how to use
local invariant features + a global

spatial model to recognize
specific objects, using a planar
object assumption.

• Now, we’ll use local features for
Indexing-based recognition
Bags of words representations
Correspondence / matching kernels

6

Basic flow

…
…
Index each one into pool
of descriptors from

…

previously seen images

Detect or sample Describe
features features

List of positions, Associated list of
scales, d-dimensional
orientations descriptors

7

Indexing local features
• Each patch / region has a descriptor, which is a point in
some high-dimensional feature space (e.g., SIFT)


• When we see close points in feature space, we have
similar descriptors, which indicates similar local

content.

Figure credit: A. Zisserman K. Grauman, B. Leibe

• We saw in the previous section how to use voting and
pose clustering to identify objects using local features

Figure credit: David Lowe

10

• With potentially thousands of features per image, and
hundreds to millions of images to search, how to

efficiently find those that are relevant to a new
image?

Low-dimensional descriptors : can use standard efficient
data structures for nearest neighbor search

High-dimensional descriptors: approximate nearest
neighbor search methods more practical

Inverted file indexing schemes

11

Indexing local features: approximate
nearest neighbor search

Best-Bin First (BBF), a variant of k-d

trees that uses priority queue to
examine most promising branches
first [Beis & Lowe, CVPR 1997]

Locality-Sensitive Hashing (LSH), a
randomized hashing technique using
hash functions that map similar
points to the same bin, with high
probability [Indyk & Motwani, 1998]

12

Indexing local features: inverted file index
• For text documents,
an efficient way to

find all pages on
which a word occurs
is to use an index…

• We want to find all
images in which a
feature occurs.
• To use this idea,
we’ll need to map
our features to
“visual words”.
13

Visual words: main idea
• Extract some local features from a number of images …

e.g., SIFT descriptor space: each
point is 128-dimensional

14
Slide credit: D. Nister K. Grauman, B. Leibe


Slide credit: D. Nister

15



16



17


18


19

Map high-dimensional descriptors to tokens/words by
quantizing the feature space
• Quantize via

clustering, let
cluster centers be

the prototype
“words”

Descriptor space

20

Map high-dimensional descriptors to tokens/words by
quantizing the feature space
• Determine which

word to assign to
each new image

region by finding
the closest cluster
center.
Descriptor space

21

Visual words
• Example: each
group of patches
belongs to the

same visual word

Figure from Sivic & Zisserman, ICCV 2003
22

Visual words

• First explored for texture
and material

representations
• Texton = cluster center of
filter responses over

collection of images
• Describe textures and
materials based on
distribution of prototypical
texture elements.

Leung & Malik 1999; Varma &
Zisserman, 2002; Lazebnik,
Schmid & Ponce, 2003;

Visual words

• More recently used for
describing scenes and

objects for the sake of
indexing or classification.

Sivic & Zisserman 2003;
Csurka, Bray, Dance, & Fan
2004; many others.
24

Inverted file index for images
comprised of visual words
Word List of image
number numbers

Image credit: A. Zisserman K. Grauman, B. Leibe

Bags of visual words
• Summarize entire image
based on its distribution

(histogram) of word
occurrences.
• Analogous to bag of words

representation commonly
used for documents.

26
Image credit: Fei-Fei Li K. Grauman, B. Leibe

Video Google System
Query
region
1. Collect all words within
query region

2. Inverted file index to find
relevant frames
3. Compare word counts

4. Spatial verification

Retrieved frames
Retrieved frames
Sivic & Zisserman, ICCV 2003

• Demo online at :
http://www.robots.ox.ac.uk/~vgg/
research/vgoogle/index.html

27

Basic flow

…
…
of descriptors from

…


or

…
Detect or sample Describe Quantize to form
features features bag of words vector
for the image

28

Visual vocabulary formation
Issues:
• Sampling strategy

• Clustering / quantization algorithm
• Unsupervised vs. supervised
• What corpus provides features (universal vocabulary?)

• Vocabulary size, number of words

29

Sampling strategies

Sparse, at

Dense, uniformly Randomly
interest points
• To find specific, textured objects, sparse
sampling from interest points often more
reliable.
• Multiple complementary interest operators
offer more image coverage.
• For object categorization, dense sampling
offers better coverage.
Multiple interest
operators [See Nowak, Jurie & Triggs, ECCV 2006]
30
Image credits: F-F. Li, E. Nowak, J. Sivic K. Grauman, B. Leibe

Clustering / quantization methods
• k-means (typical choice), agglomerative clustering,
mean-shift,…

• Hierarchical clustering: allows faster insertion / word
assignment while still allowing large vocabularies

Vocabulary tree [Nister & Stewenius, CVPR 2006]

31

Example: Recognition with Vocabulary Tree
• Tree construction:

[Nister & Stewenius, CVPR’06]
32
K. Grauman, B. Leibe Slide credit: David Nister

Vocabulary Tree
• Training: Filling the tree

33

Vocabulary Tree

34

Vocabulary Tree

35

Vocabulary Tree

36

Vocabulary Tree

37

Vocabulary Tree
• Recognition

RANSAC
verification

38

Vocabulary Tree: Performance
• Evaluated on large databases
Indexing with up to 1M images

• Online recognition for database
of 50,000 CD covers

Retrieval in ~1s

• Find experimentally that large
vocabularies can be beneficial for
recognition


39

Vocabulary formation
• Ensembles of trees provide additional robustness

Moosmann, Jurie, & Triggs 2006; Yeh, Lee, & Darrell 2007;
Bosch, Zisserman, & Munoz 2007; …

Figure credit: F. Jurie K. Grauman, B. Leibe

Supervised vocabulary formation
• Recent work considers how to leverage labeled images
when constructing the vocabulary

Perronnin, Dance, Csurka, & Bressan, Adapted Vocabularies for
Generic Visual Categorization, ECCV 2006.
41

• Merge words that don’t aid in discriminability

Winn, Criminisi, & Minka, Object Categorization by Learned
Universal Visual Dictionary, ICCV 2005

• Consider vocabulary and classifier construction jointly.

Yang, Jin, Sukthankar, & Jurie, Discriminative Visual Codebook Generation
with Classifier Training for Object Category Recognition, CVPR 2008.
43

Learning and recognition with bag of
words histograms
• Bag of words representation makes it possible to
describe the unordered point set with a single vector

(of fixed dimension across image examples)

• Provides easy way to use distribution of feature types
with various learning algorithms requiring vector input.

44

words histograms
• …including unsupervised topic models designed for
documents.

• Hierarchical Bayesian text models (pLSA and LDA)
– Hoffman 2001, Blei, Ng & Jordan, 2004

– For object and scene categorization: Sivic et al. 2005,
Sudderth et al. 2005, Quelhas et al. 2005, Fei-Fei et al.
2005

45
Figure credit: Fei-Fei Li K. Grauman, B. Leibe

words histograms
• …including unsupervised topic models designed for
documents.
Probabilistic Latent

Semantic Analysis
d z w (pLSA)
N

D

“face”

Sivic et al. ICCV 2005
[pLSA code available at: http://www.robots.ox.ac.uk/~vgg/software/]
46
Figure credit: Fei-Fei Li K. Grauman, B. Leibe

Bags of words: pros and cons
+ flexible to geometry / deformations / viewpoint
+ compact summary of image content

+ provides vector representation for sets
+ has yielded good recognition results in practice

- basic model ignores geometry – must verify afterwards,
or encode via features
- background and foreground mixed when bag covers
whole image
- interest points or sampling: no guarantee to capture
object-level parts
- optimal vocabulary formation remains unclear
47

Outline




48

Basic flow

…
…
of descriptors from

…


or

…
Detect or sample Describe Quantize to form
features features or bag of words vector
for the image

Compute match
with another image
3

Local feature correspondences
• The matching between sets of local features helps to
establish overall similarity between objects or shapes.

• Assigned matches also useful for localization

Shape context Low-distortion matching [Berg & Malik 2005] Match kernel
[Belongie & [Wallraven,
Malik 2001] Caputo & Graf
2003]

4

Local feature correspondences
• Least cost match: minimize total cost between matched
points

min
π : X →Y
∑ x − π (x )
xi∈X
i i

• Least cost partial match: match all of smaller set to
some portion of larger set.

Pyramid match kernel (PMK)
• Optimal matching expensive relative to number of
features per image (m).

• PMK is approximate partial match for efficient
discriminative learning from sets of local features.

Optimal match: O(m3)
Greedy match: O(m2 log m)
Pyramid match: O(m)

[Grauman & Darrell, ICCV 2005]

6

Pyramid match kernel: pyramid extraction

,

Histogram
pyramid:

level i has bins
of size

7
K. Grauman


Histogram
intersection

K. Grauman
Pyramid match kernel: counting matches

8

Pyramid match kernel: counting new matches
Histogram
intersection

matches at this level matches at previous level

Difference in histogram intersections across
levels counts number of new pairs matched

9
K. Grauman

Pyramid match kernel

histogram pyramids

number of newly matched pairs at level i

measure of difficulty of
a match at level i
• For similarity, weights inversely proportional to bin size (or may
be learned discriminatively)
• Normalize kernel values to avoid favoring large sets
10
K. Grauman


K. Grauman
Example pyramid match

11


K. Grauman

12


K. Grauman

13

pyramid match

optimal match

K. Grauman

• Forms a Mercer kernel -> allows classification with SVMs,
use of other kernel methods

• Bounded error relative to optimal partial match
• Linear time -> efficient learning with large feature sets




ETH-80 data set
ETH
Accuracy

Time (s)
Mean number of features Mean number of features
Match [Wallraven et al.]
O(m2)
Pyramid match O(m)


• Use data-dependent pyramid partitions for high-d

feature spaces

Uniform pyramid bins Vocabulary-guided
pyramid bins

Code for PMK: http://people.csail.mit.edu/jjl/libpmk/

Matching smoothness & local geometry
• Solving for linear assignment means (non-overlapping)
features can be matched independently, ignoring

relative geometry.
• One alternative: simply expand feature vectors to
include spatial information before matching.

[ f1,…,f128, xa, ya ]
ya

xa
18

Spatial pyramid match kernel
• First quantize descriptors into words, then do one
pyramid match per word in image coordinate space.

Lazebnik, Schmid & Ponce, CVPR 2006

• Use correspondence to estimate parameterized
transformation, regularize to enforce smoothness

Shape context matching [Belongie, Malik, & Puzicha 2001]

Code: http://www.eecs.berkeley.edu/Research/Projects/CS/vision/shape/sc_digits.html

• Let matching cost include term to penalize distortion
between pairs of matched features.

Template Query
j j'

Rij
Si'j'

i i'
i i'

Approximate for efficient solutions: Berg & Malik, CVPR 2005;
Leordeanu & Hebert, ICCV 2005

Figure credit: Alex Berg K. Grauman, B. Leibe

• Compare “semi-local” features: consider configurations
or neighborhoods and co-occurrence relationships

Correlograms of Proximity
visual words distribution kernel
[Savarese, Winn, & [Ling & Soatto, ICCV Hyperfeatures: Agarwal &
Criminisi, CVPR 2006] 2007] Triggs, ECCV 2006]

Feature neighborhoods [Sivic Tiled neighborhood [Quack, Ferrari,
& Zisserman, CVPR 2004] Leibe, van Gool ICCV 2007]

• Learn or provide explicit object-specific shape model
[Next in the tutorial : part-based models]

x1
x6 x2

x5 x3
x4

Summary
• Local features are a useful, flexible representation
Invariance properties - typically built into the descriptor

Distinctive, especially helpful for identifying specific textured
objects
Breaking image into regions/parts gives tolerance to occlusions
and clutter

Mapping to visual words forms discrete tokens from image
regions

• Efficient methods available for
Indexing patches or regions
Comparing distributions of visual words
Matching features

24

Outline




25

Recognition of Object Categories
• We no longer have exact correspondences…

• On a local level, we
can still detect
similar parts.

• Represent objects
by their parts
⇒ Bag-of-features

• How can we
improve on this?
Encode structure

3
T. Tuytelaars, B. Leibe Slide credit: Rob Fergus

Part-Based Models

• Fischler & Elschlager 1973

• Model has two components
parts

(2D image fragments)
structure
(configuration of parts)

4

Different Connectivity Structures

O(N6) O(N2) O(N3) O(N2)

Fergus et al. ’03 Leibe et al. ’04, ‘08 Crandall et al. ‘05 Felzenszwalb &

Fei-Fei et al. ‘03 Crandall et al. ‘05 Huttenlocher ‘05
Fergus et al. ’05

Csurka ’04 Bouchard & Triggs ‘05 Carneiro & Lowe ‘06
Vasconcelos ‘00

5
K. Grauman, B. Leibe from [Carneiro & Lowe, ECCV’06]

Spatial Models Considered Here

Fully connected shape “Star” shape model

model
x1 x1
x6 x2 x6 x2

x5 x3 x5 x3
x4 x4

e.g. Constellation Model e.g. ISM
Parts fully connected Parts mutually independent
Recognition complexity: O(NP) Recognition complexity: O(NP)
Method: Exhaustive search Method: Gen. Hough Transform

6
K. Grauman, B. Leibe Slide credit: Rob Fergus

Constellation Model
• Joint model for appearance and shape

Gaussian shape pdf Gaussian part appearance pdf Gaussian
relative scale pdf

Log(scale)

Prob. of detection

0.8 0.75 0.9
7

Constellation Model
Gaussian shape pdf Gaussian part appearance pdf Gaussian
relative scale pdf

Log(scale)

Prob. of detection

0.8 0.75 0.9
Clutter model Uniform
Uniform shape pdf Gaussian appearance pdf relative scale pdf

Log(scale)

Poission pdf on # detections

8

Constellation Model: Learning Procedure
• Goal: Find regions & their location, scale & appearance
• Initialize model parameters

• Use EM and iterate to convergence
E-step: Compute assignments for which regions are
foreground/background

M-step: Update model parameters
• Trying to maximize likelihood – consistency in shape & appearance

9


Example: Motorbikes

10


Example: Motorbikes (2)

11


Example: Spotted Cats

12

Discussion: Constellation Model
• Advantages
Works well for many different object categories
Can adapt well to categories where

– Shape is more important
– Appearance is more important
Everything is learned from training data

Weakly-supervised training possible

• Disadvantages
Model contains many parameters that need to be estimated
Cost increases exponentially with increasing number of
parameters
⇒ Fully connected model restricted to small number of parts.

13

Implicit Shape Model (ISM)
• Basic ideas x1
Learn an appearance codebook x6 x2

Learn a star-topology structural model x5 x3
– Features are considered independent given obj. center x4

• Algorithm: probabilistic Gen. Hough Transform
Exact correspondences → Prob. match to object part
NN matching → Soft matching
Feature location on obj. → Part location distribution
Uniform votes → Probabilistic vote weighting
Quantized Hough array → Continuous Hough space

14

Codebook Representation
• Extraction of local object features
Interest Points (e.g. Harris detector)

Sparse representation of the object appearance

• Collect features from whole training set
• Example:

15

Gen. Hough Transform with Local Features
• For every feature, store possible “occurrences”

• For new image, let the matched features vote for
– Object identity
possible object positions – Pose
– Relative position

18

Implicit Shape Model - Representation
…
…
…
…

Training images
(+reference segmentation)
…

Appearance codebook

• Learn appearance codebook y y

Extract local features at interest points
Agglomerative clustering ⇒ codebook
s s
x x
• Learn spatial distributions y y

Match codebook to training images
Record matching positions on object
s s
x x
Spatial occurrence distributions
+ local figure-ground labels 19
B. Leibe

Implicit Shape Model - Recognition
Interest Points Matched Codebook Probabilistic
Entries Voting

y

Image Feature Interpretation Object
(Codebook match) Position
s
x
3D Voting Space
(continuous)
f Ci o,x

p(Ci f ) p(on , x Ci , l)

p(on , x f , l) = ∑ p (Ci f ) p (on , x Ci , l)
i

[Leibe04, Leibe08] 21

Implicit Shape Model - Recognition
Entries Voting

y

s
x
3D Voting Space
(continuous)

Backprojected Backprojection
Hypotheses of Maxima


Example: Results on Cows

Original image
24



Interest points
Original image
25


Matchedpoints
Interest patches
Original image
26


Interest points
Original image
Matched patches
Prob. Votes
27



1st hypothesis
28



2nd hypothesis
29



3rd hypothesis
30

Scale Invariant Voting
• Scale-invariant feature selection
Scale-invariant interest points

Rescale extracted patches
Match to constant-size codebook

• Generate scale votes
Scale as 3rd dimension in voting space

s Search
window

y
Search for maxima in 3D voting space
x

31

Scale Voting: Efficient Computation

s s s s

y y y y
x x

Scale votes Binned Candidate Refinement
accum. array maxima (MSME)

• Mean-Shift formulation for refinement
Scale-adaptive balloon density estimator

33

Detection Results
• Qualitative Performance
Recognizes different kinds of objects

Robust to clutter, occlusion, noise, low contrast

35

Figure-Ground Segregation
• Problem extensively studied in
Psychophysics

• Experiments with ambiguous
figure-ground stimuli
• Results:

Evidence that object recognition can
and does operate before figure-ground
organization
Interpreted as Gestalt cue familiarity.

M.A. Peterson, “Object Recognition Processes Can and Do Operate Before Figure-
Ground Organization”, Cur. Dir. in Psych. Sc., 3:105-111, 1994.

36

ISM – Top-Down Segmentation
Entries Voting

y

s
x
3D Voting Space
Segmentation (continuous)

p(figure) Backprojected Backprojection
Probabilities Hypotheses of Maxima

Segmentation: Probabilistic Formulation

• Influence of patch on object hypothesis (vote weight)

p( f , l o , x ) =
∑ p(o , x | C ) p(C
i n i i | f ) p( f,l )
n
p(on , x )

• Backprojection to features f and pixels p:
p(p = figure | on , x ) = ∑ p(p = figure | f , l, o , x ) p( f , l | o , x )
n n
p∈( f ,l )

Segmentation Influence on
information object hypothesis
38
K. Grauman, B. Leibe [Leibe04, Leibe08]

Segmentation

p(figure)

Original image Segmentation
p(figure)
p(ground)

p(ground)

• Interpretation of p(figure) map
per-pixel confidence in object hypothesis
Use for hypothesis verification
46


Example Results: Motorbikes

47

Example Results: Cows
• Training
112 hand-segmented images

• Results on novel sequences:

Single-frame recognition - No temporal continuity used!
48


Office chairs

B. Leibe
Example Results: Chairs
Dining room chairs

49


Training

Test
Output
Inferring Other Information: Part Labels

[Thomas07]
50


[Thomas07]
Inferring Other Information: Part Labels (2)

51

Inferring Other Information: Depth Maps

“Depth from a single image”

52
[Thomas07]

Application for Pedestrian Detection
• Estimating Articulation

[Leibe, Seemann, Schiele, CVPR’05]

• Rotation-Invariant Detection

d dq
φ
φ
θq
θ
[Mikolajczyk, Leibe, Schiele, CVPR’06]
53
B. Leibe

Outline




54

Outline



Highlight of some research topics not covered in the main tutorial
2


Benchmark Data
• What degree of difficulty do current datasets have?

Example: Caltech-101

A dataset that has

been about
mastered…

Images from the Caltech-101:
101-way multi-class classification problem

Example: Caltech256

Images from the Caltech-256:
256 multi-class recognition problem

Example: Pascal Visual Object Classes Challenge

Pascal VOC 2007:
Binary detection problems

http://pascallin.ecs.soton.ac.uk/challenges/VOC/



Example: LabelMe

http://labelme.csail.mit.edu/

Current challenges & ongoing research
• Multi-cue integration
• Finer level categorization

• View invariant recognition
• Unsupervised category discovery
• Learning from noisily labeled images

• Integration of segmentation and recognition
• Learning with text and images/video
• Use of video
• Context and scene layout

Multi-cue integration
• Single cues often not sufficient.
• Integrate multiple local and global cues.

9

Multi-Category Discrimination
• Distinguish similar categories.
• Need to look at specific details!

10

Multi-Aspect Recognition
• Detectors for different viewpoints ⇒ How can this be
improved?

11


[Hoiem, Rother, Winn, CVPR’07] [Thomas et al., CVPR’06]

12


[Rothganger et al., CVPR’03]

[Savarese & Fei-Fei, ICCV’07]

13

Unsupervised, semi-supervised category discovery
Topic models for images
Probabilistic Latent

Semantic Analysis
(pLSA)

“face”

Latent Dirichlet
“beach” Allocation
(LDA)

c π z w
D N
Sivic et al. ICCV 2005, Fei-Fei et al. ICCV 2005 Figure credit: Fei-Fei Li

Unsupervised, semi-supervised category discovery
Clustering cluttered images
Learning from noisy keyword-based image search results

Grauman & Darrell, CVPR 2006

Fergus et al. ECCV 2004, ICCV 2005

Li & Fei-Fei, CVPR 2007

Learning with text and images/video

Barnard et al. JMLR 2003

Berg, Berg, Edwards,
& Forsyth, NIPS 2006
Gupta et al. ECML 2008

Integrating segmentation + recognition

Borenstein & Ullman, ECCV 2002
Kumar et al. CVPR 2005

Kannan, Winn, & Rother, NIPS 2006
Tu, Chen, Yuille, Zhu, ICCV 2003


Antonio Torralba, IJCV 2003
Role of context, understanding scene layout

Role of context, understanding scene layout

Image World

Hoiem, Efros, & Hebert, CVPR 2006

Integration with Scene Geometry
• Goal: Find the ground plane
Restrict object location

Assume Gaussian size prior
⇒ Significantly reduced search space

Structure-from-Motion
y Search
corridor

s
x
Hough Volume

Dense stereo 20
B. Leibe

Extensions
• Combination with 3D Geometry

[Leibe, Cornelis, Cornelis, Van Gool, CVPR’07]

• Mobile Pedestrian Detection

[Ess, Leibe, Van Gool, ICCV’07]
21

Detections Using Ground Plane Constraints

left camera
1175 frames

22
B. Leibe [Leibe et al. CVPR’07]

Extensions: Tracking-by-Detection

• Spacetime trajectory analysis
Link up detections to form physically plausible ST trajectories
Select set of ST trajectories that best explain the data

23
[Leibe et al. CVPR’07]


B. Leibe
Dynamic Scene Analysis Results

[Leibe et al. CVPR’07]
24

Extensions (2)
• Combination 3D Reconstruction

[Cornelis, Leibe, Cornelis, Van Gool, 3DPVT’06]

25

Textured 3D Model

Original 3D Reconstruction
• Run-times
SfM + Bundle adjustment: 27-30 fps on CPU
Dense reconstruction: 36 fps on GPU
[Cornelis, Cornelis, Van Gool, CVPR’06]
26
B. Leibe

Improved 3D City Model
Enhancing your driving experience…

Original 3D Reconstruction

[Cornelis, Leibe, Cornelis, Van Gool, 3DPVT’06] 27


B. Leibe
Putting It All Together…
y
s
x

Q

VT
S
V
πd

t
π

I

z
oi

H1
D
di
1..n

H2

x
H i , ti

28


Mobile Pedestrian Tracking

[Ess, Leibe, Schindler, Van Gool, CVPR’08]
29


Mobile Tracking Through Crowds

[Ess, Leibe, Schindler, Van Gool, CVPR’08]
30

Extension: Recovering Articulations
1...N

• Idea: Only perform articulated tracking where it’s easy!
• Multi-person tracking
Solves hard data association problem
• Articulated tracking
Only on individual “tracklets” between occlusions

[Gammeter, Ess, Jaeggli, Schindler, Leibe, Van Gool, ECCV’08]
31
B. Leibe

Articulated Multi-Person Tracking

• Multi-Person tracking
Recovers trajectories and solves data association
Estimates 3D walking direction and speed
Detects occlusion events

32
B. Leibe

Articulated Tracking under Egomotion

33
B. Leibe


34

Summary
• Visual recognition is a challenging and very active
research area.

• We’ve covered some basic models and representations
that have been shown to be effective, and highlighted
some ongoing issues.

• See tutorial website for slides, links, references.
http://www.vision.ee.ethz.ch/~bleibe/teaching/tutorial-aaai08/

Thank you!


AAAI08 tutorial: visual object recognition

More Related Content

What's hot

Viewers also liked

Similar to AAAI08 tutorial: visual object recognition

More from zukun

Recently uploaded

AAAI08 tutorial: visual object recognition