The document discusses light field and coded aperture cameras. It describes the Stanford plenoptic camera which uses a microlens array to sample individual rays of light, capturing 14 pixels per lens. An alternative approach is a mask-based light field camera that uses a narrowband cosine mask to sample a coded combination of rays. This heterodyne approach captures half the brightness but avoids wasting pixels and issues with lens array alignment. The document outlines how such cameras can digitally refocus images and increase depth of field. It also discusses using the Fourier transform to compute a 4D light field from 2D photos captured with a mask.

1. Coded Photography

2. Light Field Inside a Camera
Lenslet-based Light Field camera
[Adelson and Wang, 1992, Ng et al. 2005 ]

3. Stanford Plenoptic Camera
[Ng et al 2005]
Contax medium format camera
Kodak 16-megapixel sensor
Adaptive Optics microlens array
125μ square-sided microlenses
4000
4000 pixels
292
292 lenses = 14
14 pixels per lens

4. Digital Refocusing
[Ng et al 2005]
Can we achieve this with a Mask alone?

5. Mask based Light Field Camera
Mask
Sensor
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

6. Mask based Light Field Camera
Mask
Sensor

7. Cosine Mask Used
Mask Tile
1/f0

8. Captured 2D Photo
Encoding due to
Mask
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

9. 2D
FFT
Traditional Camera Photo
Magnitude of 2D FFT
2D
FFT
Heterodyne Camera Photo
Magnitude of 2D FFT

10. 2D Photo
LED
In Focus Photo

11. Out of Focus Photo: Open Aperture

12. Out of Focus Photo: Coded Aperture

13. Captured Blurred
Photo
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

14. Refocused on
Person
Increase DoF +
large aperture

15. Engineering the PSF
when you cannot
capture Lightfield
Out of Focus Photo: Coded Aperture

16. Digital Refocusing
Captured Blurred Photo

17. Digital Refocusing
Refocused Image on Person

18. Mask?
Mask
Digital
Refocusing
Sensor

19. Mask?
Mask
Sensor
Sensor
Mask
Digital
Refocusing
Heterodyne
Light Field
Camera

20. 4D
Light Field
2D Photo

21. Computing 4D Light Field
2D Sensor Photo, 1800*1800
2D Fourier Transform
2D
FFT
9*9=81 spectral copies
Rearrange 2D tiles into 4D planes
4D IFFT
4D Light Field
200*200*9*9
200*200*9*9

22. Mask = more information?
Mask?
Sensor
Sensor
Mask
Digital Refocusing
Sensor
Mask
Heterodyne Light
Field Camera
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan], Sig
graph 2007

23. MERL
Veeraraghavan, Raskar, Agrawal,
Mohan & Tumblin
Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture
Differences with Plenoptic Camera
Sensor
Sensor
Microlens
array
Plenoptic Camera
Mask
Heterodyne Camera
• Micro-lens array
• Narrowband Cosine Mask
• Samples individual rays
• Samples coded comb of rays
• Needs alignment precision
• More flexible
• Some pixels wasted
• No wastage
- Half brightness, diffraction

24. Novel Sensors
• Color
– Foveon
• Dynamic Range
– HDR Camera, Log sensing
– Gradient sensing
• Identity
– Demodulation
• 3D
– ZCam, Canesta
• Motion
– Line scan Camera
– Flutter Shutter

25. Foveon: All Colors at a Single Pixel

26. High Dynamic Range
Fuji's SuperCCD S3 Pro
Sensor with high and low sensitivity sensors
per pixel location

27. Gradient Camera
• Sense Pixel Intensity Difference with
unknown locally adaptive gain
• Reconstruct image from 2D gradient field
Ramesh Raskar, MERL
Work with Jack Tumblin, Northwestern U,
Amit Agrawal, U of Maryland

28. High Dynamic Range Images
Scene
Intensity camera
saturation map
Gradient camera
saturation map
Intensity camera fail to capture range
Gradients saturate at very few isolated pixels

29. Natural Scene Properties
Intensity
Gradient
105
105
1
1
x
x
Intensity Histogram
1
105
Gradient Histogram
-105
105

30. Motion _ _

31. Line Scan Camera: PhotoFinish 2000 Hz

33. Figure 2 results
Photo with motion blur

34. Rectified Image to make motion lines parallel to scan lines.

35. Approx Cut-out
Image Deblurred by solving a linear system.

36. Fluttered Shutter Camera
[Raskar, Agrawal, Tumblin] Siggraph2006

40. Coded Exposure
Temporal 1-D
broadband code
Coded Aperture
Spatial 2-D
broadband code

41. Novel Sensors
• Color
– Foveon
• Dynamic Range
– HDR Camera, Log sensing
– Gradient sensing
• Identity
– Demodulation
• 3D
– ZCam, Canesta
• Motion
– Line scan Camera
– Flutter Shutter

42. Perspective? Or Not?
Rademacher et al, MCOP, Siggraph 1998
Agrawala et al, Long Scene Panoramas, Siggraph 2006

43. Multiperspective Camera?
[ Jingyi Yu‟ 2004 ]

44. Future ..
• „Cloth-cam‟: „Wallpaper-cam‟
– Fusion of 4D light emission and 4D capture in the surface of a cloth…
• Human Augmentation
– Cameras to replace human eyes, for blind or limited vision
– Camera on the „back‟
• More Sensors
– GPS, Compass, Temperature, fingerprint recognition, face recognition
– When, Where, What, How .. Why?
• Photo Sharing and Community:
– Photo Clip and Scene Completion
– City Scanning, Live

45. Light Sensitive Fabric
Bayindir, Fink 2004

46. Computational Photography
Novel Illumination
Light Sources
Novel Cameras
Modulators
Generalized
Optics
Generalized
Sensor
Processing
Ray
Reconstruction
Generalized
Optics
4D Incident Lighting
4D Ray Bender
Upto 4D
Ray Sampler
4D Light Field
Display
Recreate 4D Lightfield
Scene: 8D Ray Modulator

47. Fixed Color Gamut
G
R ≈ 0.0
G ≈ 0.2
R
B
B ≈ 0.8

48. Wider color gamut
G
400nm
R
550nm
700nm
λ
“Best” primaries compromise:
Wide Gamut vs. High Power
B

49. Adaptive Color
Primaries

50. Agile Spectrum Imaging
With Ankit Mohan, Jack Tumblin [Eurographics 2008]

51. Rainbow Plane inside Camera
Pinhole
Scene
Lens L2
Lens L1
C
C‟
‟
A‟
B
B‟
A
Sensor
B‟
‟
C‟
Prism or
Diffraction Grating
Rainbow
Plane
A‟
‟

52. Computational Photography
1.
2.
Epsilon Photography
– Low-level vision: Pixels
– Multi-photos by perturbing camera parameters
– HDR, panorama, …
– ‘Ultimate camera’
Coded Photography
–
–
–
–
3.
Mid-Level Cues:
•
Regions, Edges, Motion, Direct/global
•
Reversible encoding of data
Single/few snapshot
Additional sensors/optics/illum
‘Scene analysis’
Essence Photography
– High-level understanding
–
•
•
Not mimic human eye
Beyond single view/illum
‘New artform’