>A Switchable Light Field Camera Architecture with Angle SEnsitive Pixels and Dictionary-based Sparse Coding
•Download as PPTX, PDF•
1 like•8,305 views
We propose a flexible light field camera architecture that is at the convergence of optics, sensor electronics, and applied mathematics. Through the co-design of a sensor that comprises tailored, Angle Sensitive Pixels and advanced reconstruction algorithms, we show that—contrary to light field cameras today—our system can use the same measurements captured in a single sensor image to recover either a high-resolution 2D image, a low-resolution 4D light field using fast, linear processing, or a high-resolution light field using sparsity-constrained optimization.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to >A Switchable Light Field Camera Architecture with Angle SEnsitive Pixels and Dictionary-based Sparse Coding
Similar to >A Switchable Light Field Camera Architecture with Angle SEnsitive Pixels and Dictionary-based Sparse Coding (20)
Recently uploaded
Recently uploaded (20)
>A Switchable Light Field Camera Architecture with Angle SEnsitive Pixels and Dictionary-based Sparse Coding
- 1. A Switchable Light Field Camera Using Angle Sensitive Pixels and Dictionary-based Space Coding Matthew Hirsch1* Sriram Sivaramakrishnan2* Suren Jayasuriya2* Albert Wang2 Alyosha Molnar2 Ramesh Raskar1 Gordon Wetzstein1 1MIT Media Lab 2Cornell University *Shared first-authorship
- 4. Flexible Camera Vision Visual Complexity ComputeTime High Resolution 2D Low Resolution Light Field High Resolution Light Field Prototype is grayscale only!
- 5. Computational Photography Touches... Optics Algorithms Sensors Lytro.com Dappled Photography, Veeraraghavan, et al. ?An angle-sensitive CMOS imager for single-sensor 3D photography, Wang et al.
- 6. Traditional Camera “Bucket of photons” Need optics to re-bin rays
- 7. Light Field Cameras Lens Based Mask Based barrier lenslets Veeraraghavan et al. 2007Ives, 1905Lippmann 1908Lytro 2014 Direct angle-space tradeoff Limits Resolution Direct angle-space tradeoff Limits light transmission Limits Light Field Resolution More Flexible
- 8. Camera Arrays Light Field Cameras Sequential Acquisition Wilburn et al. 2002,2005 Levoy and Hanrahan 1996 Liang et al. 2008
- 9. Redundancy “Natural” 4D Light Field Random Light Field
- 11. Compressive Capture • Random and optimized optical codes • Multiplexing & nonlinear reconstruction Marwah et al., 2013 Image Sensor Printed Mask
- 12. Image Sensor Circuit Pixel Array Timing/Addressing Amplifiers ADCs Silicon Dioxide: - Insulator for metal interconnects Photo Diode Silicon Substrate Metal Transistor
- 13. Angle Sensitive Pixel (ASP) Phase gratings Two Interleaved Diodes 𝟏𝟎𝝁𝒎 ~𝟏𝝁𝒎 Single Pixel Silicon Dioxide: Insulator Output 1 (D1) Output 2 (D2)
- 14. Plane wave on grating generates periodic diffraction pattern Operating Principle: Talbot Effect Single Pixel! Phase Grating Diodes
- 15. Plane wave on grating generates periodic diffraction pattern Operating Principle: Talbot Effect Single Pixel!
- 16. Angular Response Angle (degrees) Response Amplitude (V) Single Pixel!
- 17. ASP model 𝝆 𝜶,𝜷 (𝛉) = 𝟏 𝟐 + 𝐦 𝟐 𝐜𝐨𝐬 𝛃𝛉 + 𝛂 m 360o β
- 18. Pixel Angular ResponseASP Output Conventional Pixel 𝝆- 𝝆 𝝆 𝝆+ 𝝆 𝝆 Angle Angle Angle Angle ResponseAmplitudeResponseAmplitude ResponseAmplitudeResponseAmplitude Single ASP
- 19. Pixel Angular ResponseASP Output 𝝆- 𝝆 𝝆 𝝆+ 𝝆 𝝆 𝐻 𝑓𝜃 𝑓𝜃𝛽 Frequency Response Angle Angle ResponseAmplitudeResponseAmplitude ResponseAmplitudeResponseAmplitude Angle Angle
- 20. 2D ASP Tile 𝝆 𝜶,𝜷,𝜸 (𝛉) = 𝟏 𝟐 + 𝐦 𝟐 𝐜𝐨𝐬 𝛃 𝐜𝐨𝐬 𝜸 𝛉 𝒙 + 𝛃 𝐬𝐢𝐧 𝛄 𝛉 𝒚 + 𝛂 Physical Layout Impulse Response (2D) Low 𝜷 𝟏𝟐 Med 𝜷 18 High 𝜷 24 Low 𝜷 𝟏𝟐 Med 𝜷 18 High 𝜷 24 𝜸 𝟎 𝜸 𝟎 𝜸 𝟒𝟓 𝜸 𝟒𝟓 𝜶 𝟎 𝜶 𝟎 𝜶 𝟗𝟎 𝜶 𝟗𝟎 Note: This prototype was made using a functionally similar variant of the ASP that uses a stack of metal gratings instead of phase gratings
- 22. 8 8x + - + + 8 8x + - + + Row address Timing generation DataOutput 4:1 4:1 4:1 4:1 Pixel array S/H Mux Sum/Diff PGA ADC Design Layout Fabricate
- 23. INTUITION FROM SIGNAL PROCESSING A Fast Linear Reconstruction Method
- 24. Light Field Capture: 1D ASP Array 𝑓𝜃 𝑥 𝑫 𝒅 𝛽1 𝛽2 𝛽3 𝑥 Angle Amplitude Angle Amplitude Angle Amplitude Note! Space Note! Angular Frequnecy
- 25. Spatio-Angular Frequency Domain 𝒇 = 𝟏 𝒅 𝑭 = 𝟏 𝑫
- 26. LINEAR RECONSTRUCTION A Fast and Simple Reconstruction Algorithm
- 27. Fast/Linear Reconstruction ASP responses are approximately orthonormal wavelents Σ is used as a preconditioner for inverting capture equation • Model the image capture process: ASP Sensor Tile Impulse Response (2D)
- 28. NONLINEAR RECONSTRUCTION High Resolution Light Fields from a Single Photo
- 29. Captured2DImage = ASP Projection 4DLightField Sparse Coefficients! Compressive Light Field Reconstruction ASP Sensor Tile Impulse Response (2D)
- 30. Overcomplete dictionary= Light field vector = Coefficient vector s.t. is sparse Can lead to fewer non-zero coefficients Dictionary Compressive Light Field Representation Courtesy Marwah et al.
- 31. Training light field = is sparse Sample 1,000,000 random 4D patches from training light fields i i i for all iCoefficient vectorDictionary s.t. Dictionary Learning Courtesy Marwah et al.
- 32. Light Field “Atoms” in Dictionary 5,000 atoms, each 9x9 pixels and 5x5 views Light fields can be represented by only a few of these atoms Courtesy Marwah et al.
- 33. Captured2DImage4DReconstruction = ASP Projection 4DLightField Basis Pursuit Denoise: Sparse Coefficients! Compressive Light Field Reconstruction
- 34. RESULTS Simulated and Live Results from a Prototype Camera
- 38. Limitations 2D View Reconstructed Light Field Central View
- 39. Limitations • Specularities and other effects not well represented in dictionary • Sensor saturation
- 40. Limitations • ASP responses are non-linear in some regions ASP Tile 2D Image Light Field Center View
- 41. ANALYSIS Depth-of-Field, Noise, Resolution, and Failure Cases
- 42. Resolution, Noise Performance, Depth of Field
- 43. Conclusion • We built a custom, ASP sensor • New sensors, new opportunities – Shoot now compute later! – [Computation in] [quality out] • Seeking new problems – e.g. Illumination Multiplexing, Time of Flight
- 44. Support provided by: NSF IIS-1218411 NSF Graduate Student Fellowship (Suren) MIT Media Lab Consortia Funding NSF IIS-1116452 NSERC Postdoctural Fellowship (Gordon) Matthew Hirsch1* Sriram Sivaramakrishnan2* Suren Jayasuriya2* Albert Wang2 Alyosha Molnar2 Ramesh Raskar1 Gordon Wetzstein1 1MIT Media Lab 2Cornell University A Switchable Light Field Camera
- 45. Depth-of-Field ? ? ? ? ? ? ? ? ? ? ? ?
- 46. Resolution