This document provides an overview of advanced computer vision. It begins by stating the goal of computer vision is to extract meaning from pixels. It then discusses what types of information can be extracted from images, such as 3D and semantic information. The document outlines several challenges in computer vision like viewpoint and illumination variations. It also discusses successes in areas like optical character recognition, biometrics, and mobile visual search. Finally, it provides an introduction to topics covered in computer vision like early vision, multi-view geometry, and recognition.

1. Advanced Computer Vision

2. Today
• Introduction to computer vision
• Course overview
• Course requirements

3. The goal of computer vision
• To extract “meaning” from pixels
What we see What a computer sees
Source: S. Narasimhan

4. The goal of computer vision
• To extract “meaning” from pixels
Source: “80 million tiny images” by Torralba et al.
Humans are remarkably good at this…

5. What kind of information can be extracted
from an image?
• Metric 3D information
• Semantic information

6. Vision as measurement device
Real-time stereo Structure from motion
NASA Mars Rover
Pollefeys et al.
Reconstruction from
Internet photo collections
Goesele et al.

7. Vision as a source of semantic information
slide credit: Fei-Fei, Fergus & Torralba

8. Object categorization
sky
building
flag
wall
banner
bus
cars
bus
face
street lamp
slide credit: Fei-Fei, Fergus & Torralba

9. Scene and context categorization
• outdoor
• city
• traffic
• …
slide credit: Fei-Fei, Fergus & Torralba

10. Qualitative spatial information
slanted
rigid moving
object
horizontal
vertical
slide credit: Fei-Fei, Fergus & Torralba
rigid moving
object
non-rigid moving
object

11. Why study computer vision?
Personal photo albums
Surveillance and security
Movies, news, sports
Medical and scientific images
• Vision is useful: Images and video are everywhere!

12. Why study computer vision?
• Vision is useful
• Vision is interesting
• Vision is difficult
– Half of primate cerebral cortex is devoted to visual processing
– Achieving human-level visual perception is probably “AI-complete”

13. Why is computer vision difficult?

14. Challenges: viewpoint variation
Michelangelo 1475-1564 slide credit: Fei-Fei, Fergus & Torralba

15. Challenges: illumination
image credit: J. Koenderink

16. Challenges: scale
slide credit: Fei-Fei, Fergus & Torralba

17. Challenges: deformation
Xu, Beihong 1943
slide credit: Fei-Fei, Fergus & Torralba

18. Challenges: occlusion
Magritte, 1957 slide credit: Fei-Fei, Fergus & Torralba

19. Challenges: background clutter

20. Challenges: Motion

21. Challenges: object intra-class
variation
slide credit: Fei-Fei, Fergus & Torralba

22. Challenges: local ambiguity
slide credit: Fei-Fei, Fergus & Torralba

23. Challenges: local ambiguity
Source: Rob Fergus and Antonio Torralba

24. Challenges: local ambiguity
Source: Rob Fergus and Antonio Torralba

25. Challenges or opportunities?
• Images are confusing, but they also reveal the structure of
the world through numerous cues
• Our job is to interpret the cues!
Image source: J. Koenderink

26. Depth cues: Linear perspective

27. Depth cues: Aerial perspective

28. Depth ordering cues: Occlusion
Source: J. Koenderink

29. Shape cues: Texture gradient

30. Shape and lighting cues: Shading

31. Position and lighting cues: Cast shadows
Source: J. Koenderink

32. Grouping cues: Similarity (color, texture,
proximity)

33. Grouping cues: “Common fate”
Image credit: Arthus-Bertrand (via F. Durand)

34. Inherent ambiguity of the problem
• Many different 3D scenes could have given rise to a
particular 2D picture

35. Inherent ambiguity of the problem
• Many different 3D scenes could have given rise to a
particular 2D picture
• Possible solutions
– Bring in more constraints (more images)
– Use prior knowledge about the structure of the world
• Need a combination of geometric and statistical methods

36. Connections to other disciplines
Computer Vision
Image Processing
Machine Learning
Artificial Intelligence
Robotics
Cognitive science
Neuroscience
Computer Graphics

37. Origins of computer vision
L. G. Roberts, Machine Perception
of Three Dimensional Solids,
Ph.D. thesis, MIT Department of
Electrical Engineering, 1963.

38. Successes of computer vision to date

39. Optical character recognition (OCR)
Source: S. Seitz, N. Snavely
Digit recognition
yann.lecun.com
License plate readers
http://en.wikipedia.org/wiki/Automatic_number_plate_recognition
Sudoku grabber
http://sudokugrab.blogspot.com/
Automatic check processing

40. Biometrics
Fingerprint scanners on
many new laptops,
other devices
Face recognition systems now
beginning to appear more widely
http://www.sensiblevision.com/
Source: S. Seitz

41. Biometrics
How the Afghan Girl was Identified by Her Iris Patterns
Source: S. Seitz

42. Mobile visual search: Google Goggles

43. Face detection
Many new digital cameras now detect faces
• Canon, Sony, Fuji, …
Source: S. Seitz

44. Smile detection
Sony Cyber-shot® T70 Digital Still Camera Source: S. Seitz

45. Face recognition: Apple iPhoto software
http://www.apple.com/ilife/iphoto/

46. Automotive safety
Mobileye: Vision systems in high-end BMW, GM, Volvo models
• Pedestrian collision warning
• Forward collision warning
• Lane departure warning
• Headway monitoring and warning Source: A. Shashua, S. Seitz

47. Vision-based interaction: Xbox Kinect
http://www.xbox.com/en-US/Live/EngineeringBlog/122910-
HowYouBecometheController
http://electronics.howstuffworks.com/microsoft-kinect.htm
http://blogs.howstuffworks.com/2010/11/05/how-microsoft-
kinect-works-an-amazing-use-of-infrared-light/
http://www.ismashphone.com/2010/12/kinect-hacks-more-
interesting-than-the-devices-original-intention.html

48. Special effects: shape and motion capture
Source: S. Seitz

49. 3D visualization: Microsoft Photosynth
http://photosynth.net Source: S. Seitz

50. Vision for robotics, space exploration
Vision systems (JPL) used for several tasks
• Panorama stitching
• 3D terrain modeling
• Obstacle detection, position tracking
• For more, read “Computer Vision on Mars” by Matthies et al.
NASA'S Mars Exploration Rover Spirit captured this westward view from atop
a low plateau where Spirit spent the closing months of 2007.
Source: S. Seitz

51. The computer vision industry
• A list of companies here:
http://www.cs.ubc.ca/spider/lowe/vision.html

52. Basic Info
• Textbooks (suggested):
Forsyth & Ponce, Computer Vision: A Modern Approach
Richard Szeliski, Computer Vision: Algorithms and
Applications

53. Early vision
Cameras and sensors
Light and color
Linear filtering
Edge detection
* =
Feature extraction: corner and blob detection
• Basic image formation and processing

54. “Mid-level vision”
Fitting: Least squares
Hough transform
RANSAC
Alignment
• Fitting and grouping

55. Multi-view geometry
Projective structure from motion
Stereo
Affine structure from motion
Tomasi & Kanade (1993)
Epipolar geometry

56. Recognition
Patch description and matching Clustering and visual vocabularies
Bag-of-features models Classification
Sources: D. Lowe, L. Fei-Fei

57. Advanced Topics
Segmentation
Articulated models
Face detection
Motion and tracking