Glass allows for hands-free, point-of-view photography that is always available and instantly triggerable. This new type of photography enables capturing people met, interesting things seen, and information to remember. Computational photography techniques may also allow for low-light photography, seeing through objects, magnification, motion effects, and face recognition through Glass in the future. New business models and artistic styles could emerge from these new camera capabilities.

1. What Google Glass means for the
future of photography
Marc Levoy
Computer Science Department
Stanford University
TTI Vanguard [next]
December 4, 2014
mostly retired
from Stanford
to work at...

2. (Google)
https://www.youtube.com/watch?v=5R1snVxGNVs

4. © 2010 Marc Levoy
Outline
✦ a bit of history
✦ technical specs
✦ What’s different about taking a picture using Glass?
✦ writing apps for cameras
✦ superhero vision
4

5. © Marc Levoy
Head-mounted displays are not new
✦ conferences:
• International Symposium on Wearable Computers (ISWC)
• Pervasive and Ubiquitous Computing (Ubicomp)
5
Ivan Sutherland’s
“ultimate display” (1965)

6. © Marc Levoy
Head-mounted displays are not new
6
(Google image search results for “head mounted display”)

7. © Marc Levoy
Neither are head-mounted cameras
7
Steve Mann’s
Eyetap (1981)

8. © Marc Levoy
Neither are head-mounted cameras
8
(Google image search results for “head mounted camera”)

9. (gopro.com)
https://www.youtube.com/watch?v=A3PDXmYoF5U
(GoPro trailer)

10. (gopro.com)
https://www.youtube.com/watch?v=A3PDXmYoF5U
(GoPro trailer)

11. © Marc Levoy
Technical specs
✦ camera
• cell phone equivalent, 5Mpix still or 720p video, no flash
11

12. Nokia N95

13. Nokia N95

14. iPhone 5

15. Glass

16. © Marc Levoy
Technical specs
✦ camera
• cell phone equivalent, 5Mpix still or 720p video, no flash
✦ display
• heads-up, color, 640×360 pixels, focused at a distance
✦ sensors
• accelerometer, gyro, compass, GPS
✦ radios
• bluetooth tethering through your phone or direct Wifi
• no cellular modem (3G/4G)
✦ computing
• cell phone equivalent processor (OMAP4), memory, and OS
• SDKs are becoming available in stages16

17. © Marc Levoy
What’s different about
taking a picture using Glass?
✦ the capture experience
• hands-free
17

18. (picture by Sebastian Thrun)

19. © Marc Levoy
What’s different about
taking a picture using Glass?
✦ the capture experience
• hands-free
• point-of-view
19

21. © Marc Levoy
What’s different about
taking a picture using Glass?
✦ the capture experience
• hands-free
• point-of-view
• always available
21

23. © Marc Levoy
What’s different about
taking a picture using Glass?
✦ the capture experience
• hands-free
• point-of-view
• always available
• instantly triggerable
23

25. © Marc Levoy
What’s different about
taking a picture using Glass?
✦ the capture experience
• hands-free
• point-of-view
• always available
• instantly triggerable
• your eyes are unobstructed
25

28. (Google)
https://www.youtube.com/watch?v=cCmfpHYAAz0

29. © Marc Levoy
What’s different about
taking a picture using Glass?
✦ the capture experience
• hands-free
• point-of-view
• always available
• instantly triggerable
• your eyes are unobstructed
✦ new reasons to take pictures
• people I’ve met
29

32. © Marc Levoy
What’s different about
taking a picture using Glass?
✦ the capture experience
• hands-free
• point-of-view
• always available
• instantly triggerable
• your eyes are unobstructed
✦ new reasons to take pictures
• people I’ve met
• something cool I saw
32

34. © Marc Levoy
What’s different about
taking a picture using Glass?
✦ the capture experience
• hands-free
• point-of-view
• always available
• instantly triggerable
• your eyes are unobstructed
✦ new reasons to take pictures
• people I’ve met
• something cool I saw
• information I’d like to remember
34

37. © Marc Levoy
What’s different about
taking a picture using Glass?
✦ the capture experience
• hands-free
• point-of-view
• always available
• instantly triggerable
• your eyes are unobstructed
✦ new reasons to take pictures
• people I’ve met
• something cool I saw
• information I need to remember
• things I gotta do
37

39. © Marc Levoy
What’s different about
taking a picture using Glass?
✦ the capture experience
• hands-free
• point-of-view
• always available
• instantly triggerable
• your eyes are unobstructed
✦ new reasons to take pictures
• people I’ve met
• something cool I saw
• information I need to remember
• things I gotta do
• life logging
39

41. © Marc Levoy
What’s different about
taking a picture using Glass?
✦ the capture experience
• hands-free
• point-of-view
• always available
• instantly triggerable
• your eyes are unobstructed
✦ new reasons to take pictures
• people I’ve met
• something cool I saw
• information I need to remember
• things I gotta do
• life logging
41
Glass also takes
nice photographs

45. © Marc Levoy
Video and other formats
✦ straight video
45

46. (video not available)

47. © Marc Levoy
Video and other formats
✦ straight video
✦ slo-mo and timelapse
47

48. (video not available)

49. © Marc Levoy
Video and other formats
✦ straight video
✦ slo-mo and timelapse
✦ video hangouts and eye swapping
49

50. (Google)
https://www.youtube.com/watch?v=cCmfpHYAAz0

51. © Marc Levoy
Video and other formats
✦ straight video
✦ slo-mo and timelapse
✦ video hangouts and eye swapping
✦ every photograph is a cliplet
51

52. © Marc Levoy
Cliplets
52
Nikon V1 Motion Snapshot
(60fps for 1 second, slowed 2.5×, cross-dissolve to still, played with music)

53. (http://vimeo.com/36897783)

54. © Marc Levoy
Photomontage from bursts
54

55. © Marc Levoy
Photomontage from bursts
55

56. © Marc Levoy
Multiplicity
56 (Evan Sharboneau)

57. © Marc Levoy
Video and other formats
✦ straight video
✦ slo-mo and timelapse
✦ video hangouts and eye swapping
✦ every photograph is a cliplet
✦ cinemagraphs
57

58. © Marc Levoy
Cinemagraphs:
between photography and video
58
(cinemagraphs.com)

59. © Marc Levoy
Video and other formats
✦ straight video
✦ slo-mo and timelapse
✦ video hangouts and eye swapping
✦ every photograph is a cliplet
✦ cinemagraphs
✦ other games with video...
59

60. © Marc Levoy
Video synopsis
[Pritch et al.]
60
http://www.cs.huji.ac.il/~yaelpri/demos.php

61. © Marc Levoy
Dynamosaicing
[Rav-Acha, Pritch, et al. 2007]
61 http://www.cs.huji.ac.il/~yaelpri/demos.php

62. © Marc Levoy
What’s hard about computational video?
✦ finding effects that are fun, useful, and not creepy
✦ making these effects easy for consumers to author
✦ making them robust enough to work most of the time
✦ making them efficient enough to compute on the device
62

63. © Marc Levoy
The future of digital cameras
✦ the megapixel wars are over (long overdue)
✦ computational photography is the next battleground
in the camera industry (it’s already starting)
63
nope, still raging in the
cell phone marketplace
actually, it’s the main
battleground right now

64. © Marc Levoy
The future of digital cameras
✦ the megapixel wars are over (long overdue)
✦ computational photography is the next battleground
in the camera industry (it’s already starting)
64
• burst mode low-light
• auto-HDR
• all-focus
• background defocus
Casio EX ZR1000
• captures a burst of 40
shots at 30fps and
auto-selects the best 5
Nikon 1 V2

65. © Marc Levoy
The future of digital cameras
✦ the megapixel wars are over (long overdue)
✦ computational photography is the next battleground
in the camera industry (it’s already starting)
✦ how will these features appear to consumers?
• standard and invisible
• standard and visible (and disable-able)
• aftermarket plugins and apps for your camera
65

66. © Marc Levoy
Example computational photography app:
SynthCam for the iPhone
66
Free

68. while recording synthetic aperture photograph

69. single frame synthetic aperture photograph

70. single frame synthetic aperture photograph

71. strong defocus subtle defocus

72. tilt-shift “miniature-model” effect

73. © Marc Levoy
Superhero vision
73

74. (Hector Garcia-Molina)

75. © Marc Levoy
Superhero vision
✦ seeing in the dark
75

76. © Marc Levoy
✦ required a tripod
✦ can’t currently do this using a cell phone, but it’s not impossible
• dark current (if one shot) or read noise (if a burst) must be very low76
Digital photography
can easily exceed human vision
(Jesse Levinson Canon 10D, 28mm f/4, 3 min, ISO 100, 4 image pano)

77. single frame
(iPhone 4)
Low-light imaging using
burst-mode computational
photography

78. average of
~30 frames
(SynthCam)
SNR increases as
sqrt(# of frames)
Low-light imaging using
burst-mode computational
photography

79. single frame

80. average of
~30 frames

81. single frame

82. average of
~30 frames

83. © Marc Levoy
Superhero vision
✦ seeing in the dark
✦ seeing through objects
83

84. © Marc Levoy
Removing foreground objects
by translating the camera
✦ align the shots
✦ match histograms
✦ apply median filter
84

85. © Marc Levoy
Superhero vision
✦ seeing in the dark
✦ seeing through objects
✦ magnifying glass, telescopic vision
85

86. © Marc Levoy
Head-mounted magnifiers
86

87. © Marc Levoy
Camera-based magnifiers
✦ optical zoom
• requires a long optical path
✦ digital zoom (cropping)
• requires a high pixel count,
hence a thick camera
✦ super-resolution
• results typically look oversharpened
87
Nokia 808

88. © Marc Levoy
Beyond SLRs: Superhero vision
✦ seeing in the dark
✦ seeing through objects
✦ magnifying glass, telescopic vision
✦ slowing down motion
88

89. (Dogs in Slow Motion, Devin Graham)
https://www.youtube.com/watch?v=7ZIjkvdz4Ko

90. © Marc Levoy
Superhero vision
✦ seeing in the dark
✦ seeing through objects
✦ magnifying glass, telescopic vision
✦ slowing down motion
✦ motion magnification, change magnification
90

91. © Marc Levoy
Motion magnification
[Liu, SIGGRAPH 2005]
✦ can this be done using a (shaky) handheld camera?
✦ can it be computed on a (slow) mobile device?
91

92. © Marc Levoy
Change magnification
[Wu, SIGGRAPH 2012]
92
✦ how much SNR is needed to detect this signal?
✦ is it socially acceptable to run this on Glass?

93. © Marc Levoy
Superhero vision
✦ seeing in the dark
✦ seeing through objects
✦ magnifying glass, telescopic vision
✦ slowing down motion
✦ motion magnification, change magnification
✦ language translation
93

94. © Marc Levoy
Word Lens
(app for iOS and Android)
✦ mediocre translation, but clever user interface
✦ recently bought by Google, runs on Glass
94

95. © Marc Levoy
Superhero vision
✦ seeing in the dark
✦ seeing through objects
✦ magnifying glass, telescopic vision
✦ slowing down motion
✦ motion magnification, change magnification
✦ language translation
✦ face recognition
95

96. © Marc Levoy
If you met this man at a party...
96
•name: Jack Sparrow
•address: Black Pearl
•profession: pirate
•net worth: zero
•spouse: many
•criminal record: long

97. © Marc Levoy
Face recognition
✦ recognition from uncontrolled photos is still sci-fi
✦ Google has pro-actively prohibited it on Glass
✦ it could eventually work
✦ if it does, someone will build a device to do it
✦ giving up anonymity ≠ giving up privacy
97

98. © Marc Levoy
Parting thought: new business models
✦ remote medical diagnosis
• or agricultural, or mechanical
✦ pay or be paid for every picture you take
• posting pictures of empty parking spaces
✦ instantly available cameras improve safety and security
• “ok glass, report that unsafe driver!”
• personal safety in bad neighborhoods
98

99. © Marc Levoy
Parting thought: new aesthetics
✦ new technologies begin by mimicing old technologies,
but eventually develop their own idioms
99
Georges Méliès, 20,000 Leagues Under the Sea (1907)

100. Canon 5DII, 1/400 sec

101. Canon 5DII, average of 15 shots of 1/6 sec

102. (Marc Levoy)