Pietro Perona gave a presentation at the NSF Workshop on Frontiers in Vision about applying computer vision techniques to scientific domains like studying animal behavior. He discussed how measuring behavior can provide insights into genes, brains and the nature of behavior itself. He presented examples of using vision to track and analyze fly behavior, detecting phenotypes. Perona emphasized that successful applications require working as a scientist and publishing in scientific venues to have impact, which also benefits machine vision by providing new datasets and challenges. Basic research is still needed in areas like tracking, hierarchical modeling and unsupervised learning.

1. Applications to Science
Pietro Perona
California Institute of Technology
NSF Workshop - Frontiers in Vision
Cambridge, 23 Aug 2011
Friday, August 26, 2011

2. Goals
• A few examples
• Implications for machine vision
• Lessons learned
• NSF’s role
Friday, August 26, 2011

3. Plan
• Intro (5’)
• Sketch of a few success stories (50’)
• Discussion (10’)
Friday, August 26, 2011

4. ‘Lunging’ (view from top)
Friday, August 26, 2011

5. Why measure behavior
Friday, August 26, 2011

6. Why measure behavior
• Genes <<>> Brains <<>> Behavior
Friday, August 26, 2011

7. Why measure behavior
• Genes <<>> Brains <<>> Behavior
• Ethology
Friday, August 26, 2011

8. Why measure behavior
• Genes <<>> Brains <<>> Behavior
• Ethology
• What is behavior?
Friday, August 26, 2011

9. Fly behavior
(as we understand it today)
Adapted from:
Kravitz et al.
PNAS April 16, 2002
vol. 99 no. 8
5664–5668
Friday, August 26, 2011

10. Friday, August 26, 2011

11. Friday, August 26, 2011

12. Detection performance
[Dankert et al., Nature Methods, April 2009]
Friday, August 26, 2011

13. Phenotyping
[Dankert et al., Nature Methods, 2009]
Friday, August 26, 2011

14. Ethograms
[Dankert et al., Nature Methods, April 2009]
Friday, August 26, 2011

15. Perception
PSYCHOLOGY
interaction, cooperation,
competition
plans, goals, behavior,
relationships ...
pose, movemes, actions,
activities, objects, scenes
SENSORY
images, trajectories
World
Friday, August 26, 2011

16. Action Perception
PSYCHOLOGY
interaction, cooperation,
PLANNING
group-level goals and plans
competition
SOCIAL NETWORK THEORY OF SOCIOLOGY
INDIVIDUAL
plans, goals, behavior,
individual goals and plans
relationships ...
PREFRONTAL CORTEX THEORY OF PSYCHOLOGY
pose, movemes, actions,
MOTOR
motor programs
activities, objects, scenes
SENSORY
MOTOR CORTEX RECOGNITION
sensor-based control images, trajectories
SPINAL CORD IMAGING,TRACKING
World
Friday, August 26, 2011

17. Lessons learned
• Image deluge in science
• Doing better than the scientists
• Payoffs in science, not in MV (short term)
‣ Must work as scientist
‣ Students must be interested in science too
‣ Publish in unfamiliar venues
‣ CV publications are suspicious
• Benefit to MV: new challenges and datasets
• Benefit to PI: fun, learning
Friday, August 26, 2011

18. Basic research needed
• Tracking, detection and identification
• Parts and pose
• Hierarchical models (for time series)
• Unsupervised discovery of categories
• Weakly supervised learning
Friday, August 26, 2011