1. Humans & Machines
collaborating on vision
Pietro Perona
California Institute of Technology
NSF Workshop - Frontiers in Vision
Cambridge, 23 Aug 2011
Friday, August 26, 2011

2. “Collaborative vision’’ ?
Pietro Perona
California Institute of Technology
NSF Workshop - Frontiers in Vision
Cambridge, 23 Aug 2011
Friday, August 26, 2011

3. Objectives
• Sketch new area of research
• Sampler of initial work
• Drawing lessons
• Brainstorm: potential, way forward
Friday, August 26, 2011

4. Plan
• Define area (10’)
• Presentations (50’): Perona, Geman,
Grauman, Berg, Belongie
• Discussion (15’)
Friday, August 26, 2011

5. Definition
Friday, August 26, 2011

6. 6
Friday, August 26, 2011

7. ?
6
Friday, August 26, 2011

8. 7
Friday, August 26, 2011

9. Friday, August 26, 2011

10. Friday, August 26, 2011

11. Friday, August 26, 2011

12. 9
Friday, August 26, 2011

13. Lessons:
• Visual queries
• Easy for humans
• Difficult for machines
• Much information is available on line
• Pictures are digital dark matter
• Experts not providing visual knowledge
10
Friday, August 26, 2011

14. Unsupervised learning?
11
[Fergus et al., CVPR03]
Friday, August 26, 2011

15. Unsupervised learning?
11
[Fergus et al., CVPR03]
Friday, August 26, 2011

16. 12
Friday, August 26, 2011

17. Friday, August 26, 2011

18. Throat
Friday, August 26, 2011

19. Throat
Friday, August 26, 2011

20. Visual knowledge
Categorical (experts) 14 Task-oriented (practitioners)
Friday, August 26, 2011

21. World
Ob
n
ser
io
va
at
tio
rv
n
se
Ob Science,
Shared Education Users
expertise knowledge
Experts
ers
sw
es
An
eri
Models Image
Qu
annotations
Machine vision
Annotators Automata scientists
15
Friday, August 26, 2011

22. World
Ob
n
ser
io
va
at
tio
rv
n
se
Ob Science,
Shared Education Users
expertise knowledge
Experts
ers
sw
es
An
eri
Models Image
Qu
annotations
Machine vision
Annotators Automata scientists
15
Friday, August 26, 2011

23. World
Ob
n
ser
io
va
at
tio
rv
n
se
Ob Science,
Shared Education Users
expertise knowledge
Experts
ers
sw
es
An
eri
Models Image
Qu
annotations
Machine vision
Annotators Automata scientists
15
Friday, August 26, 2011

24. World
Ob
n
ser
io
va
at
tio
rv
n
se
Ob Science,
Shared Education Users
expertise knowledge
Experts
ers
sw
es
An
eri
Models Image
Qu
annotations
Machine vision
Annotators Automata scientists
15
Friday, August 26, 2011

25. Some progress...
Friday, August 26, 2011

26. DUCKS Waterbirds
Mallard American Black Duck
Canada Goose Red Necked Grebe Clutter
Friday, August 26, 2011

27. Multidimensional signals and annotators
p(xi | zi = 1)
2
xi
1 2
xi = (xi , xi ) 1
xi
p(xi | zi = 0)
Friday, August 26, 2011

28. Multidimensional signals and annotators
p(xi | zi = 1)
2
xi
1 2
xi = (xi , xi ) 1
xi
p(xi | zi = 0)
Friday, August 26, 2011

29. Multidimensional signals and annotators
p(xi | zi = 1)
2
xi
1 2
xi = (xi , xi ) 1
xi
τj 1 2
wj = (wj , wj )
p(xi | zi = 0)
Friday, August 26, 2011

30. α γ
Full model annotators
σj wj τj
β θz
M
zi xi yij lij
Ji ij
images N labels |Lij |
[Welinder et al., NIPS2010]
Friday, August 26, 2011

31. Is there a duck in the image?
2
xi
1
xi
Friday, August 26, 2011

32. Is there a duck in the image?
2
xi
1
xi
Friday, August 26, 2011

33. Is there a duck in the image?
2
xi
1
xi
Friday, August 26, 2011

34. Is there a duck in the image?
2
xi
1
xi
Friday, August 26, 2011

35. Is there a duck in the image?
2
xi
1
xi
Friday, August 26, 2011

36. Is there a duck in the image?
2
xi
1
xi
Friday, August 26, 2011

37. Is there a duck in the image?
2
xi
1
xi
Friday, August 26, 2011

38. Concluding...
Friday, August 26, 2011

39. Collaborative vision
100%
Automation
0% Performance 100%
Friday, August 26, 2011

40. Collaborative vision
100%
Automation
0% Performance 100%
Friday, August 26, 2011

41. Collaborative vision
100%
Automation
0% Performance 100%
Friday, August 26, 2011

42. Collaborative vision
100%
Automation
0% Performance 100%
Friday, August 26, 2011

43. Collaborative vision
100%
+
Automation
Applications
Training data
-
Complexity
Cost
0% Performance 100%
Friday, August 26, 2011

44. World
Ob
n
ser
io
va
at
tio
rv
n
se
Ob Science,
Shared Education Users
expertise knowledge
Experts
ers
sw
es
An
eri
Models Image
Qu
annotations
Machine vision
Annotators Automata scientists
24
Friday, August 26, 2011

45. New research directions
• Incremental learning
• Models of human vision, decision, attention
• Systems composed of machines and humans
• Performance bounds (humans, machines)
• Representations (human-machine-friendly)
• Extracting visual knowledge from experts
Friday, August 26, 2011