Downloaded 85 times
![Conclusion
• We built a custom, ASP sensor
• New sensors, new opportunities
– Shoot now compute later!
– [Computation in] [quality out]
• Seeking new problems
– e.g. Illumination Multiplexing, Time of Flight](https://image.slidesharecdn.com/iccp14upload-140509181935-phpapp01/85/A-Switchable-Light-Field-Camera-Architecture-with-Angle-SEnsitive-Pixels-and-Dictionary-based-Sparse-Coding-43-320.jpg){kind=tracking}
Uploaded byMatt Hirsch - MIT Media Lab
>A Switchable Light Field Camera Architecture with Angle SEnsitive Pixels and Dictionary-based Sparse Coding
The document discusses the development of a switchable light field camera utilizing angle sensitive pixels and dictionary-based space coding. It highlights various technical aspects and challenges of capturing high-resolution light fields and emphasizes the potential applications of this technology. The authors conclude with a vision for future advancements in sensor technology and computational photography.
More Related Content
Similar to >A Switchable Light Field Camera Architecture with Angle SEnsitive Pixels and Dictionary-based Sparse Coding
>A Switchable Light Field Camera Architecture with Angle SEnsitive Pixels and Dictionary-based Sparse Coding
- 1. A Switchable LightField Camera Using Angle Sensitive Pixels and Dictionary-based Space Coding Matthew Hirsch1* Sriram Sivaramakrishnan2* Suren Jayasuriya2* Albert Wang2 Alyosha Molnar2 Ramesh Raskar1 Gordon Wetzstein1 1MIT Media Lab 2Cornell University *Shared first-authorship
- 2.
- 3.
- 4. Flexible Camera Vision VisualComplexity ComputeTime High Resolution 2D Low Resolution Light Field High Resolution Light Field Prototype is grayscale only!
- 5. Computational Photography Touches... OpticsAlgorithms Sensors Lytro.com Dappled Photography, Veeraraghavan, et al. ?An angle-sensitive CMOS imager for single-sensor 3D photography, Wang et al.
- 6. Traditional Camera “Bucket ofphotons” Need optics to re-bin rays
- 7. Light Field Cameras LensBased Mask Based barrier lenslets Veeraraghavan et al. 2007Ives, 1905Lippmann 1908Lytro 2014 Direct angle-space tradeoff Limits Resolution Direct angle-space tradeoff Limits light transmission Limits Light Field Resolution More Flexible
- 8. Camera Arrays Light FieldCameras Sequential Acquisition Wilburn et al. 2002,2005 Levoy and Hanrahan 1996 Liang et al. 2008
- 9. Redundancy “Natural” 4D LightField Random Light Field
- 10.
- 11. Compressive Capture • Randomand optimized optical codes • Multiplexing & nonlinear reconstruction Marwah et al., 2013 Image Sensor Printed Mask
- 12. Image Sensor Circuit PixelArray Timing/Addressing Amplifiers ADCs Silicon Dioxide: - Insulator for metal interconnects Photo Diode Silicon Substrate Metal Transistor
- 13. Angle Sensitive Pixel(ASP) Phase gratings Two Interleaved Diodes 𝟏𝟎𝝁𝒎 ~𝟏𝝁𝒎 Single Pixel Silicon Dioxide: Insulator Output 1 (D1) Output 2 (D2)
- 14. Plane wave ongrating generates periodic diffraction pattern Operating Principle: Talbot Effect Single Pixel! Phase Grating Diodes
- 15. Plane wave ongrating generates periodic diffraction pattern Operating Principle: Talbot Effect Single Pixel!
- 16. Angular Response Angle (degrees) ResponseAmplitude (V) Single Pixel!
- 17. ASP model 𝝆 𝜶,𝜷 (𝛉)= 𝟏 𝟐 + 𝐦 𝟐 𝐜𝐨𝐬 𝛃𝛉 + 𝛂 m 360o β
- 18. Pixel Angular ResponseASPOutput Conventional Pixel 𝝆- 𝝆 𝝆 𝝆+ 𝝆 𝝆 Angle Angle Angle Angle ResponseAmplitudeResponseAmplitude ResponseAmplitudeResponseAmplitude Single ASP
- 19. Pixel Angular ResponseASPOutput 𝝆- 𝝆 𝝆 𝝆+ 𝝆 𝝆 𝐻 𝑓𝜃 𝑓𝜃𝛽 Frequency Response Angle Angle ResponseAmplitudeResponseAmplitude ResponseAmplitudeResponseAmplitude Angle Angle
- 20. 2D ASP Tile 𝝆𝜶,𝜷,𝜸 (𝛉) = 𝟏 𝟐 + 𝐦 𝟐 𝐜𝐨𝐬 𝛃 𝐜𝐨𝐬 𝜸 𝛉 𝒙 + 𝛃 𝐬𝐢𝐧 𝛄 𝛉 𝒚 + 𝛂 Physical Layout Impulse Response (2D) Low 𝜷 𝟏𝟐 Med 𝜷 18 High 𝜷 24 Low 𝜷 𝟏𝟐 Med 𝜷 18 High 𝜷 24 𝜸 𝟎 𝜸 𝟎 𝜸 𝟒𝟓 𝜸 𝟒𝟓 𝜶 𝟎 𝜶 𝟎 𝜶 𝟗𝟎 𝜶 𝟗𝟎 Note: This prototype was made using a functionally similar variant of the ASP that uses a stack of metal gratings instead of phase gratings
- 21.
- 22. 8 8x + - + + 8 8x + - + + Row address Timinggeneration DataOutput 4:1 4:1 4:1 4:1 Pixel array S/H Mux Sum/Diff PGA ADC Design Layout Fabricate
- 23. INTUITION FROM SIGNALPROCESSING A Fast Linear Reconstruction Method
- 24. Light Field Capture:1D ASP Array 𝑓𝜃 𝑥 𝑫 𝒅 𝛽1 𝛽2 𝛽3 𝑥 Angle Amplitude Angle Amplitude Angle Amplitude Note! Space Note! Angular Frequnecy
- 25. Spatio-Angular Frequency Domain 𝒇= 𝟏 𝒅 𝑭 = 𝟏 𝑫
- 26. LINEAR RECONSTRUCTION A Fastand Simple Reconstruction Algorithm
- 27. Fast/Linear Reconstruction ASP responsesare approximately orthonormal wavelents Σ is used as a preconditioner for inverting capture equation • Model the image capture process: ASP Sensor Tile Impulse Response (2D)
- 28. NONLINEAR RECONSTRUCTION High ResolutionLight Fields from a Single Photo
- 29. Captured2DImage = ASP Projection 4DLightField Sparse Coefficients! CompressiveLight Field Reconstruction ASP Sensor Tile Impulse Response (2D)
- 30. Overcomplete dictionary= Light field vector = Coefficientvector s.t. is sparse Can lead to fewer non-zero coefficients Dictionary Compressive Light Field Representation Courtesy Marwah et al.
- 31. Training light field =is sparse Sample 1,000,000 random 4D patches from training light fields i i i for all iCoefficient vectorDictionary s.t. Dictionary Learning Courtesy Marwah et al.
- 32. Light Field “Atoms”in Dictionary 5,000 atoms, each 9x9 pixels and 5x5 views Light fields can be represented by only a few of these atoms Courtesy Marwah et al.
- 33. Captured2DImage4DReconstruction = ASP Projection 4DLightField Basis PursuitDenoise: Sparse Coefficients! Compressive Light Field Reconstruction
- 34. RESULTS Simulated and LiveResults from a Prototype Camera
- 38. Limitations 2D View ReconstructedLight Field Central View
- 39. Limitations • Specularities andother effects not well represented in dictionary • Sensor saturation
- 40. Limitations • ASP responsesare non-linear in some regions ASP Tile 2D Image Light Field Center View
- 41. ANALYSIS Depth-of-Field, Noise, Resolution,and Failure Cases
- 42. Resolution, Noise Performance,Depth of Field
- 43. Conclusion • We builta custom, ASP sensor • New sensors, new opportunities – Shoot now compute later! – [Computation in] [quality out] • Seeking new problems – e.g. Illumination Multiplexing, Time of Flight
- 44. Support provided by: NSFIIS-1218411 NSF Graduate Student Fellowship (Suren) MIT Media Lab Consortia Funding NSF IIS-1116452 NSERC Postdoctural Fellowship (Gordon) Matthew Hirsch1* Sriram Sivaramakrishnan2* Suren Jayasuriya2* Albert Wang2 Alyosha Molnar2 Ramesh Raskar1 Gordon Wetzstein1 1MIT Media Lab 2Cornell University A Switchable Light Field Camera
- 45. Depth-of-Field ? ? ?? ? ? ? ? ? ? ? ?
- 46.
- 47.