>A Switchable Light Field Camera Architecture with Angle SEnsitive Pixels and Dictionary-based Sparse Coding

5,669 views
5,960 views

Published on

We propose a flexible light field camera architecture that is at the convergence of optics, sensor electronics, and applied mathematics. Through the co-design of a sensor that comprises tailored, Angle Sensitive Pixels and advanced reconstruction algorithms, we show that—contrary to light field cameras today—our system can use the same measurements captured in a single sensor image to recover either a high-resolution 2D image, a low-resolution 4D light field using fast, linear processing, or a high-resolution light field using sparsity-constrained optimization.

Published in: Science, Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
5,669
On SlideShare
0
From Embeds
0
Number of Embeds
3,653
Actions
Shares
0
Downloads
54
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

>A Switchable Light Field Camera Architecture with Angle SEnsitive Pixels and Dictionary-based Sparse Coding

  1. 1. A Switchable Light Field Camera Using Angle Sensitive Pixels and Dictionary-based Space Coding Matthew Hirsch1* Sriram Sivaramakrishnan2* Suren Jayasuriya2* Albert Wang2 Alyosha Molnar2 Ramesh Raskar1 Gordon Wetzstein1 1MIT Media Lab 2Cornell University *Shared first-authorship
  2. 2. Cameras Today2DCameraCo. LightField CameraCo. FlexibleCamera!
  3. 3. Flexible Camera Vision Output
  4. 4. Flexible Camera Vision Visual Complexity ComputeTime High Resolution 2D Low Resolution Light Field High Resolution Light Field Prototype is grayscale only!
  5. 5. Computational Photography Touches... Optics Algorithms Sensors Lytro.com Dappled Photography, Veeraraghavan, et al. ?An angle-sensitive CMOS imager for single-sensor 3D photography, Wang et al.
  6. 6. Traditional Camera “Bucket of photons” Need optics to re-bin rays
  7. 7. Light Field Cameras Lens Based Mask Based barrier lenslets Veeraraghavan et al. 2007Ives, 1905Lippmann 1908Lytro 2014  Direct angle-space tradeoff  Limits Resolution  Direct angle-space tradeoff  Limits light transmission  Limits Light Field Resolution  More Flexible
  8. 8. Camera Arrays Light Field Cameras Sequential Acquisition Wilburn et al. 2002,2005 Levoy and Hanrahan 1996 Liang et al. 2008
  9. 9. Redundancy “Natural” 4D Light Field Random Light Field
  10. 10. Captured2DImage4DReconstruction Compressive Capture Marwah et al., 2013
  11. 11. Compressive Capture • Random and optimized optical codes • Multiplexing & nonlinear reconstruction Marwah et al., 2013 Image Sensor Printed Mask
  12. 12. Image Sensor Circuit Pixel Array Timing/Addressing Amplifiers ADCs Silicon Dioxide: - Insulator for metal interconnects Photo Diode Silicon Substrate Metal Transistor
  13. 13. Angle Sensitive Pixel (ASP) Phase gratings Two Interleaved Diodes 𝟏𝟎𝝁𝒎 ~𝟏𝝁𝒎 Single Pixel Silicon Dioxide: Insulator Output 1 (D1) Output 2 (D2)
  14. 14. Plane wave on grating generates periodic diffraction pattern Operating Principle: Talbot Effect Single Pixel! Phase Grating Diodes
  15. 15. Plane wave on grating generates periodic diffraction pattern Operating Principle: Talbot Effect Single Pixel!
  16. 16. Angular Response Angle (degrees) Response Amplitude (V) Single Pixel!
  17. 17. ASP model 𝝆 𝜶,𝜷 (𝛉) = 𝟏 𝟐 + 𝐦 𝟐 𝐜𝐨𝐬 𝛃𝛉 + 𝛂 m 360o β
  18. 18. Pixel Angular ResponseASP Output Conventional Pixel 𝝆- 𝝆 𝝆 𝝆+ 𝝆 𝝆 Angle Angle Angle Angle ResponseAmplitudeResponseAmplitude ResponseAmplitudeResponseAmplitude Single ASP
  19. 19. Pixel Angular ResponseASP Output 𝝆- 𝝆 𝝆 𝝆+ 𝝆 𝝆 𝐻 𝑓𝜃 𝑓𝜃𝛽 Frequency Response Angle Angle ResponseAmplitudeResponseAmplitude ResponseAmplitudeResponseAmplitude Angle Angle
  20. 20. 2D ASP Tile 𝝆 𝜶,𝜷,𝜸 (𝛉) = 𝟏 𝟐 + 𝐦 𝟐 𝐜𝐨𝐬 𝛃 𝐜𝐨𝐬 𝜸 𝛉 𝒙 + 𝛃 𝐬𝐢𝐧 𝛄 𝛉 𝒚 + 𝛂 Physical Layout Impulse Response (2D) Low 𝜷 𝟏𝟐 Med 𝜷 18 High 𝜷 24 Low 𝜷 𝟏𝟐 Med 𝜷 18 High 𝜷 24 𝜸 𝟎 𝜸 𝟎 𝜸 𝟒𝟓 𝜸 𝟒𝟓 𝜶 𝟎 𝜶 𝟎 𝜶 𝟗𝟎 𝜶 𝟗𝟎 Note: This prototype was made using a functionally similar variant of the ASP that uses a stack of metal gratings instead of phase gratings
  21. 21. ASP Sensor ASP Sensor
  22. 22. 8 8x + - + + 8 8x + - + + Row address Timing generation DataOutput 4:1 4:1 4:1 4:1 Pixel array S/H Mux Sum/Diff PGA ADC Design Layout Fabricate
  23. 23. INTUITION FROM SIGNAL PROCESSING A Fast Linear Reconstruction Method
  24. 24. Light Field Capture: 1D ASP Array 𝑓𝜃 𝑥 𝑫 𝒅 𝛽1 𝛽2 𝛽3 𝑥 Angle Amplitude Angle Amplitude Angle Amplitude Note! Space Note! Angular Frequnecy
  25. 25. Spatio-Angular Frequency Domain 𝒇 = 𝟏 𝒅 𝑭 = 𝟏 𝑫
  26. 26. LINEAR RECONSTRUCTION A Fast and Simple Reconstruction Algorithm
  27. 27. Fast/Linear Reconstruction ASP responses are approximately orthonormal wavelents Σ is used as a preconditioner for inverting capture equation • Model the image capture process: ASP Sensor Tile Impulse Response (2D)
  28. 28. NONLINEAR RECONSTRUCTION High Resolution Light Fields from a Single Photo
  29. 29. Captured2DImage = ASP Projection 4DLightField Sparse Coefficients! Compressive Light Field Reconstruction ASP Sensor Tile Impulse Response (2D)
  30. 30. Overcomplete dictionary= Light field vector = Coefficient vector s.t. is sparse Can lead to fewer non-zero coefficients Dictionary Compressive Light Field Representation Courtesy Marwah et al.
  31. 31. Training light field = is sparse Sample 1,000,000 random 4D patches from training light fields i i i for all iCoefficient vectorDictionary s.t. Dictionary Learning Courtesy Marwah et al.
  32. 32. Light Field “Atoms” in Dictionary 5,000 atoms, each 9x9 pixels and 5x5 views Light fields can be represented by only a few of these atoms Courtesy Marwah et al.
  33. 33. Captured2DImage4DReconstruction = ASP Projection 4DLightField Basis Pursuit Denoise: Sparse Coefficients! Compressive Light Field Reconstruction
  34. 34. RESULTS Simulated and Live Results from a Prototype Camera
  35. 35. Limitations 2D View Reconstructed Light Field Central View
  36. 36. Limitations • Specularities and other effects not well represented in dictionary • Sensor saturation
  37. 37. Limitations • ASP responses are non-linear in some regions ASP Tile 2D Image Light Field Center View
  38. 38. ANALYSIS Depth-of-Field, Noise, Resolution, and Failure Cases
  39. 39. Resolution, Noise Performance, Depth of Field
  40. 40. Conclusion • We built a custom, ASP sensor • New sensors, new opportunities – Shoot now compute later! – [Computation in] [quality out] • Seeking new problems – e.g. Illumination Multiplexing, Time of Flight
  41. 41. Support provided by: NSF IIS-1218411 NSF Graduate Student Fellowship (Suren) MIT Media Lab Consortia Funding NSF IIS-1116452 NSERC Postdoctural Fellowship (Gordon) Matthew Hirsch1* Sriram Sivaramakrishnan2* Suren Jayasuriya2* Albert Wang2 Alyosha Molnar2 Ramesh Raskar1 Gordon Wetzstein1 1MIT Media Lab 2Cornell University A Switchable Light Field Camera
  42. 42. Depth-of-Field ? ? ? ? ? ? ? ? ? ? ? ?
  43. 43. Resolution
  44. 44. Noise Light Field Center View

×