Image parsing researchers face challenges in adapting techniques developed for "closed universe" datasets to the "open universe" setting of evolving datasets with millions of images and classes. They also need to develop methods to parse extremely high-resolution images and to model dynamic image interpretation as an autonomous sequential process.

1. Challenges to image parsing
researchers
Lana Lazebnik
UNC Chapel Hill
sky
mountain
building
person
car car
sidewalk
road

2. The past: “closed universe”
datasets
Tens of classes, hundreds of images, offline learning
Figure from Shotton et al. (2009)
He et al. (2004), Hoiem et al. (2005), Shotton et al. (2006, 2008, 2009), Verbeek and Triggs
(2007), Rabinovich et al. (2007), Galleguillos et al. (2008), Gould et al. (2009), etc.

3. The future: “open universe”
datasets
Evolving images, annotations
http://labelme.csail.mit.edu/

4. The future: “open universe”
datasets
Non-uniform class frequencies
12
Millions of Pixels
10
8
6
4
2
0

5. Which “closed universe”
techniques can survive in the
“open universe” setting?
 Combination of local cues?
 Multiple segmentations/grouping hypotheses?
 Context?
 Graphical models (MRFs, CRFs, etc.)?
 Offline learning and inference?

6. Learning from all of LabelMe
50K images, 232 labels
sky
window building
building car car
door road
sidewalk
road
sky
sky
tree car
building
road mountain
sun ceiling
sky
wall
sea
floor
Tighe & Lazebnik, work in progres

7. Learning from all of LabelMe
50K images, 232 labels
Per-class classification rates
SiftFlow Barcelona LM + Sun
100%
75%
50%
25%
0%
100%
75%
50%
25%
0%
Tighe & Lazebnik, work in progres

8. Challenge: Parsing high-res
images

9. Challenge: Dynamic image
interpretation
 Image parsing algorithms should become
autonomous decision-making agents
Visual “detective
task”: Where
was this photo
taken?

10. Challenge: Dynamic image
interpretation
 Image parsing algorithms should become
autonomous decision-making agents
Input

11. Summary
 Challenges to image parsing researchers:
 Learn to parse images from “open universe”
evolving datasets
 Try parsing gigapixel images!
 Develop active, sequential image interpretation
strategies