This document provides an introduction to light fields and their applications. It discusses the plenoptic function, which describes the set of all light rays that can be observed. Light fields can be parameterized in different ways, including using position and angle or using two planes. Applications of light fields include digital image refocusing, 3D displays, camera arrays, and controlling camera or object motion. Modern implementations include the Lytro light field camera, which uses a microlens array in front of the image sensor. Other applications discussed include using light fields to reduce lens glare and developing thin, depth-sensing LCD displays.

1. Introduction to Light Fields

2. The 5D Plenoptic Function
Q: What is the set of all things that one can ever see?
A: The Plenoptic Function [Adelson and Bergen 1991]
(from plenus, complete or full, and optic)
P(q, f, l, t)

3. The 5D Plenoptic Function
Q: What is the set of all things that one can ever see?
A: The Plenoptic Function [Adelson and Bergen 1991]
(from plenus, complete or full, and optic)
P(q, f, l, t)

4. Position-Angle Parameterization
2D position
2D direction
s
q

5. Two-Plane Parameterization
2D position
2D position
s
u

6. Two-Plane Parameterization – camera array
2D position
2D position
s
u

7. Two-Plane Parameterization – SLR camera
2D position
2D position
s
u

8. Two-Plane Parameterization – cell phone
s
u

9. Two-Plane Parameterization – cell phone (pelican)
s
u

10. Application: Digital Image Refocusing
[Ng 2005]
Kodak 16-megapixel sensor
125μ square-sided microlenses

11. Application: 3D Displays
Parallax Panoramagram
[Kanolt 1918]
3DTV with Integral Imaging
[Okano et al. 1999]
MERL 3DTV
[Matusik and Pfister 2004]

12. Multiple Sensors

13. Static Camera Arrays
Stanford Multi-Camera Array
125 cameras using custom hardware
[Wilburn et al. 2002, Wilburn et al. 2005]
Distributed Light Field Camera
64 cameras with distributed rendering
[Yang et al. 2002]

14. Refocus

15. Temporal Multiplexing

16. Controlled Camera or Object Motion
Stanford Spherical Gantry
[Levoy and Hanrahan 1996]
Relighting with 4D Incident Light Fields
[Masselus et al. 2003]

17. Uncontrolled Camera or Object Motion
Unstructured Lumigraph Rendering
[Gortler et al. 1996; Buehler et al. 2001]

18. Spatial Multiplexing

19. Parallax Barriers (Pinhole Arrays)
[Ives 1903]
sensor
barrier
Spatially-multiplexed light field capture using masks (i.e., barriers):
• Cause severe attenuation  long exposures or lower SNR
• Impose fixed trade-off between spatial and angular resolution
(unless implemented with programmable masks, e.g. LCDs)

20. Integral Imaging (“Fly’s Eye” Lenslets)
[Lippmann 1908]
sensor
lenslet
f
Spatially-multiplexed light field capture using lenslets:
• Impose fixed trade-off between spatial and angular resolution

21. Light Field Photograph (Sensor)

22. Light Field Photograph (Decoded)
[The (New) Stanford Light Field Archive]
lookingup
looking to the right
Sample Image

23. DEMO
lenstoys: light field camera

24. Modern, Digital Implementations
Digital Light Field Photography
• Hand-held plenoptic camera [Ng et al. 2005]
• Heterodyne light field camera [Veeraraghavan et al. 2007]

25.   Marc Levoy

Light Field = Array of (Virtual) Cameras
Slide by Marc Levoy

26.   Marc Levoy

Sub-aperture
Virtual Camera =
Sub-aperture View
Light Field = Array of (Virtual) Cameras
Slide by Marc Levoy

27.   Marc Levoy

Sub-aperture
Virtual Camera =
Sub-aperture View
Light Field = Array of (Virtual) Cameras
Slide by Marc Levoy

28.   Marc Levoy

29. DEMO
http://lightfield.stanford.edu/aperture.swf?lightfield=data/chess_lf/preview.zip&zoom=1

30. The Lytro Light Field Camera

31. Stanford camera array

32. Microlens camera array
Digital
sensor
Microlens
array (MLA)

33. Lytro: microlens array in camera
Main lens
Microlens
array (MLA)
Digital
sensor
Lens is not
really thin,
but can be
treated as so.
330 × 330 hex
array, 13.9
micron pitch
Occluders are
not required.
14 Mpixel,
square cropped
to 11 Mpixels
Choose pixel in
same location
behind each
microlens

34. Microlens
array (MLA)
Main lens
Digital
sensor
# of sub-apertures
=
# of pixels behind
each microlens
(10 × 10)
# of pixels per
sub-aperture image
=
# of microlenses
(~ 120,000)
All rays pass through a
“sub-aperture”
Sub-aperture captures on camera view
Choose pixel in
same location
behind each
microlens
So why put the
microlens array
inside the camera?

35. Lytro camera has unusual shape
8x f/2 lens
Light field
sensor
Battery
43-343 mm
equivalent

36. Other Applications of Light Fields

37. Lens Glare Reduction
[Raskar, Agrawal, Wilson, Veeraraghavan SIGGRAPH 2008]
Glare/Flare due to camera lenses reduces contrast

38. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Reducing Glare
Glare Reduced Image
After removing outliersConventional Photo

39. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Enhancing Glare
Glare Enhanced ImageConventional Photo

40. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan

41. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan

42. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal,
Wilson & Veeraraghavan
Glare due to Lens Inter-Reflections
a
Sensor
b

43. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Effects of Glare on Image
• Hard to model, Low Frequency in 2D
• But reflection glare is outlier in 4D ray-space
Angular Variation
at pixel a
Lens Inter-reflections
a
Sensor
b

44. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Key Idea
• Lens Glare manifests as low frequency in 2D Image
• But Glare is highly view dependent
– manifests as outliers in 4D ray-space
• Reducing Glare == Remove outliers among rays
a
Sensor
b

45. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Reducing Glare
using
a Light Field Camera

46. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Single Shot Light Field Cameras
Using Mask, this paper
Using Lenslets, Ng et al. 2005
Mask
Adelson and Wang, 1992, Ng et al. 2005
Kanolt 1933, Veeraraghavan et al. 2007

47. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Captured Photo: LED off

48. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Captured Photo: LED On

49. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Each Disk: Angular Samples at that Spatial Location
No Glare

50. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
With Glare

51. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
x
y
u
v

52. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Sequence of Sub-Aperture Views
Average of all the
Light Field views
One of the
Light Field views
Low Res Traditional Camera
Photo
Glare Reduced Image

53. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Key Idea
• Reducing Glare == Remove outlier among angular samples
a
Sensor
b

54. Light-sensitive Displays Depth Cameras
Multi-touch Interaction
Multi-touch Interaction with Thin Displays

55. BiDi Screen: Thin, Depth-sensing LCDs
• Seamless transition from
multi-touch to gesture
• Thin form factor (LCD)

56. LCD Modifications

57. Diffuser
Camera
LCD
Lights
Prototype