“An Introduction to Single-Photon Avalanche Diodes—A New Type of Imager for Computer Vision,” a Presentation from the University of Wisconsin – Madison
•
0 likes•236 views
Single-photon avalanche diodes (SPADs) are novel sensors that can detect individual photons with high time resolution. SPADs allow for imaging with extreme dynamic range from low to high light conditions without saturation. They also enable minimal motion blur imaging due to their ability to precisely timestamp single photons. Recent research has demonstrated burst photography using SPAD arrays that can reconstruct non-rigid scene motion and produce almost motion-blur free images in dark environments. However, challenges remain in increasing resolution, reducing data rates and power consumption before widespread commercial applications can be realized.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to “An Introduction to Single-Photon Avalanche Diodes—A New Type of Imager for Computer Vision,” a Presentation from the University of Wisconsin – Madison
Similar to “An Introduction to Single-Photon Avalanche Diodes—A New Type of Imager for Computer Vision,” a Presentation from the University of Wisconsin – Madison (20)
More from Edge AI and Vision Alliance
More from Edge AI and Vision Alliance (20)
Recently uploaded
Recently uploaded (20)
“An Introduction to Single-Photon Avalanche Diodes—A New Type of Imager for Computer Vision,” a Presentation from the University of Wisconsin – Madison
- 1. © 2021 University of Wisconsin – Madison Introduction to Single- Photon Avalanche Diodes A New Type of Imager for Computer Vision Sebastian Bauer University of Wisconsin – Madison
- 2. © 2021 University of Wisconsin – Madison Motivation • Ultimately, computer vision algorithms are limited by the underlying data quality • Information is lost in the image acquisition process • Result: loss in speed, quality, or even system failure 2 Blurred motion Poor dynamic range Low Light https://en.wikipedia.org/wiki/Motion_blur https://www.ingentaconnect.com/content/ist/ei/2018/00 002018/00000012/art00005#
- 3. © 2021 University of Wisconsin – Madison Can we do better in terms of image acquisition? Of course! • High speed cameras, high dynamic range cameras, single photon sensitive cameras, event cameras, … • But #1: all camera types usually only good in what they are designed for • But #2: high cost Motivation 3
- 4. © 2021 University of Wisconsin – Madison • Semiconductor p-n junction reverse biased above breakdown voltage Single-Photon Avalanche Diodes (SPADs) 4 Voltage Current Avalanche photodiode (APD): linear amplifier SPAD Breakdown voltage Photon hits sensor Active avalanche quenching Bias restored
- 5. © 2021 University of Wisconsin – Madison Single-Photon Avalanche Diodes (SPADs) 5 Single photon sensitive: electron avalanche Extreme time resolution (30 … 50 ps) Silicon CMOS 1…100 ns dead time Some models: gateable Time Trigger time Detected photons Incident photons Dead time Low dark count rate
- 6. © 2021 University of Wisconsin – Madison SPADs and other Novel Sensors 6 0 2 4 6 8 10 Dynamic range Low-light Lateral resolution Power consumption Data processing Cost Time resolution SPADs CMOS Event cameras Quanta image sensors Note: this comparison is to be understood quantitatively. Even within each sensor type, there are different models with different parameters. Most importantly, SPADs, event cameras and quanta image sensors still are rather young technology, so there’s still a lot of research going on, and the numbers are volatile
- 7. © 2021 University of Wisconsin – Madison Single-Photon Avalanche Diodes (SPADs) Active imaging with laser illumination • Fluorescence lifetime imaging • 3D imaging (LiDAR) • Seeing around corners • Seeing through fog 7 Passive imaging • Extreme dynamic range • Imaging of moving objects with minimal blur Trigger Pulsed laser SPAD Object Object SPAD Ambient light
- 8. © 2021 University of Wisconsin – Madison 8 # Pixels 100 1000 10000 100000 1000000 2009 2012 2015 2018 2010 2012 2014 2016 2018 2020 SPAD Evolution 1 megapixel Research grade devices Commercial devices 106 105 104 iPhone LiDAR Data source: https://imagesensors.org/Past%20Workshops/2020%20ISSW/KazuhiroMorimoto.pdf 103
- 9. © 2021 University of Wisconsin – Madison 9 SPAD LiDAR • Automotive LiDAR: SPADs are major cost reduction factor: digital or solid-state LiDAR without moving parts • Apple: convincing AR user experience not achievable with cameras alone https://www.ifixit.com/News/45482/how-lidar-works-and- why-its-in-the-iphone-12-pro
- 10. © 2021 University of Wisconsin – Madison • High resolution sensors not yet available • Sensor data rate • Data processing effort • Power consumption • Cost • Color imaging not yet available Challenges 10
- 11. © 2021 University of Wisconsin – Madison • Array sensors: readout electronics eat a lot of space → 3D stacking Challenges 11 http://image-sensors-world.blogspot.com/2018/11/3d-stacked-spad-array.html
- 13. © 2021 University of Wisconsin – Madison Extreme Dynamic Range • Low flux 13 time Detected photons Incident photons Ingle, Atul, Andreas Velten, and Mohit Gupta. "High flux passive imaging with single-photon sensors." Proceedings CVPR. 2019. t=T Avg. # detected photons Incident photon flux
- 14. © 2021 University of Wisconsin – Madison Extreme Dynamic Range • Low flux 14 time Detected photons Incident photons Ingle, Atul, Andreas Velten, and Mohit Gupta. "High flux passive imaging with single-photon sensors." Proceedings CVPR. 2019. t=T Avg. # detected photons Incident photon flux Low photon flux: Almost all photons counted
- 15. © 2021 University of Wisconsin – Madison Extreme Dynamic Range • Moderate flux 15 time Detected photons Incident photons Ingle, Atul, Andreas Velten, and Mohit Gupta. "High flux passive imaging with single-photon sensors." Proceedings CVPR. 2019. t=T Avg. # detected photons Incident photon flux Moderate photon flux: Some photons missed
- 16. © 2021 University of Wisconsin – Madison Extreme Dynamic Range • High flux 16 time Detected photons Incident photons t=T Avg. # detected photons Incident photon flux High photon flux: Large fraction missed Ingle, Atul, Andreas Velten, and Mohit Gupta. "High flux passive imaging with single-photon sensors." Proceedings CVPR. 2019.
- 17. © 2021 University of Wisconsin – Madison Extreme Dynamic Range 17 Avg. # detected photons Incident photon flux No saturation, invertible Ingle, Atul, Andreas Velten, and Mohit Gupta. "High flux passive imaging with single-photon sensors." Proceedings CVPR. 2019.
- 18. © 2021 University of Wisconsin – Madison Extreme Dynamic Range 18 Ingle, Atul, Andreas Velten, and Mohit Gupta. "High flux passive imaging with single-photon sensors." Proceedings CVPR. 2019. saturated noisy Conventional sensor Long exposure Short exposure SPAD Single exposure +2 orders of magnitude dynamic range compared to conventional CMOS cameras
- 19. © 2021 University of Wisconsin – Madison Extreme Dynamic Range 19 Ingle, A., Seets, T., Buttafava, M., Gupta, S., Tosi, A., Gupta, M., & Velten, A. (2021). Passive Inter-Photon Imaging. arXiv preprint arXiv:2104.00059 • Experiments done by point-scanning with single-pixel SPAD • Quantization of counts limits dynamic range • Further improvement: incorporating timing information between photons • Additional 2 orders of magnitude dynamic range (~7 in total) • Point-scanning with adapted single-pixel SPAD for stable dead time
- 20. © 2021 University of Wisconsin – Madison Extreme Dynamic Range 20 Ingle, A., Seets, T., Buttafava, M., Gupta, S., Tosi, A., Gupta, M., & Velten, A. (2021). Passive Inter-Photon Imaging. arXiv preprint arXiv:2104.00059 Conventional sensor T=5 ms T=0.005 ms SPAD, timing T=5 ms SPAD, counting T=5 ms
- 22. © 2021 University of Wisconsin – Madison Minimal Blur Imaging 22 Seets, T., Ingle, A., Laurenzis, M., & Velten, A. (2021). Motion Adaptive Deblurring with Single-Photon Cameras. Proceedings IEEE/CVF WACV (pp. 1945-1954) • In each time frame, at most one photon arrival time stamp • These experiments: 32 x 32 pixels InGaAs SPAD array, 2 μs frame duration
- 23. © 2021 University of Wisconsin – Madison Minimal Blur Imaging 23 Seets, T., Ingle, A., Laurenzis, M., & Velten, A. (2021). Motion Adaptive Deblurring with Single-Photon Cameras. Proceedings IEEE/CVF WACV (pp. 1945-1954)
- 24. © 2021 University of Wisconsin – Madison Quanta Burst Photography 24 Ma, S., Gupta, S., Ulku, A. C., Bruschini, C., Charbon, E., & Gupta, M. (2020). Quanta burst photography. ACM Transactions on Graphics (TOG), 39(4), 79-1. • Similar approach. This time: 512 x 256 pixels Si array, ~100 kHz frame rate
- 25. © 2021 University of Wisconsin – Madison Quanta Burst Photography 25 Ma, S., Gupta, S., Ulku, A. C., Bruschini, C., Charbon, E., & Gupta, M. (2020). Quanta burst photography. ACM Transactions on Graphics (TOG), 39(4), 79-1. • Similar approach. This time: 512 x 256 pixels Si array, ~100 kHz frame rate
- 26. © 2021 University of Wisconsin – Madison Quanta Burst Photography 26 Ma, S., Gupta, S., Ulku, A. C., Bruschini, C., Charbon, E., & Gupta, M. (2020). Quanta burst photography. ACM Transactions on Graphics (TOG), 39(4), 79-1. • Non-rigid scene motion Naive Averaging (Long Sequence) Naive Averaging (Short Sequence)
- 27. © 2021 University of Wisconsin – Madison Quanta Burst Photography 27 Ma, S., Gupta, S., Ulku, A. C., Bruschini, C., Charbon, E., & Gupta, M. (2020). Quanta burst photography. ACM Transactions on Graphics (TOG), 39(4), 79-1. • Non-rigid scene motion: Align & merge video reconstruction
- 28. © 2021 University of Wisconsin – Madison • SPADs are highly versatile sensors • Single-photon capability and high time resolution allow for almost motion-blur free imaging in dark environments • With active illumination: LiDAR and seeing around corners • Plethora of future applications possible: industrial inspection cameras, automotive vision, night vision,… • Different methods need different SPAD parameters/capabilities • Power consumption and data processing effort are most critical problems • High-resolution arrays not yet commercially available Summary 28
- 29. © 2021 University of Wisconsin – Madison 29 More than happy to discuss, shoot me an email: sbauer8@wisc.edu SPAD operating principle: http://www.everyphotoncounts.com/spad.php Apple LiDAR: https://www.apple.com/augmented-reality/, https://ouster.com/blog/why-apple- chose-digital-lidar/, https://www.ifixit.com/News/45482/how-lidar-works-and-why-its-in-the- iphone-12-pro UW-Madison WISION lab https://wisionlab.cs.wisc.edu/ UW-Madison Computational Optics Group https://biostat.wisc.edu/~compoptics/ EPFL Aqua Lab (SPAD hardware) https://www.epfl.ch/labs/aqua/ PoLiMi SPADlab (hardware) http://www.everyphotoncounts.com/ Resources