CORNAR: Looking Around Corners using Trillion FPS Imaging
•Download as PPT, PDF•
4 likes•48,622 views
We have built a camera that can look around corners and beyond the line of sight. The camera uses light that travels from the object to the camera indirectly, by reflecting off walls or other obstacles, to reconstruct a 3D shape.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (10)
Similar to CORNAR: Looking Around Corners using Trillion FPS Imaging
Similar to CORNAR: Looking Around Corners using Trillion FPS Imaging (20)
More from Camera Culture Group, MIT Media Lab
More from Camera Culture Group, MIT Media Lab (20)
Recently uploaded
Recently uploaded (20)
CORNAR: Looking Around Corners using Trillion FPS Imaging
- 1. Raskar, Camera Culture, MIT Media Lab Computational Light Transport: CORNAR: Looking Around Corners Camera Culture using Trillion FPS Imaging Ramesh Raskar Ramesh Raskar MIT Media Lab http://raskar.info/cornar
- 2. What is around the corner ?
- 3. Can you look around the corner ?
- 4. Can you look around the corner ?
- 5. Multi-path Analysis 2nd Bounce 1st Bounce 3rd Bounce
- 8. Co-designing Optical and Digital Processing Computational Optics Light Transport Photon Hacking Displays Sensors Computational Illumination Photography Signal Processing Computer Vision Machine Learning Bit Hacking
- 9. Multi-path Analysis 2nd Bounce 1st Bounce 3rd Bounce
- 10. CORNAR: Femto-Photography FemtoFlash Trillion FPS camera With M Bawendi, MIT Chemistry Serious Sync Computational Optics •2012: 3D around a corner (NatureComm, Velten, et. al.) •2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala) •2011: DARPA Young Faculty Award •2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar) •2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009) •2008: Indirect depth (Hirsch, Raskar) •2008: Transient Light Transport (Raskar, Davis, March 2008)
- 11. Inverting Light Transport Multiple Scattering Direct/Global [Seitz , Kutulakos, Matsushita 2005] [Nayar, Raskar et al 2006] [Atcheson et al 2008] [Kutulakos, Steger 2005] Dual Photography LIDAR [Sen et al 2005]
- 12. Collision avoidance, robot navigation, …
- 14. z S x L s Occluder Streak- camera C Laser B beam Echoes of Light
- 15. z S x L s Occluder Streak- camera C Laser B beam Echoes of Light
- 16. z S x L R s Occluder Streak- camera 3rd bounce Streak Photo C Laser B beam Echoes of Light
- 17. Multi-Dimensional Light Transport 5-D Transport
- 18. Why Pico-second Resolution? ToF Diff = 0.15 mm s2 1cm s1 Occluder 3rd bounce Streak- camera C p1 p2 1st bounce Curse of Pythagoras
- 19. z S x L R s Occluder Streak- camera 3rd bounce Streak Photo C Laser B beam Echoes of Light
- 20. Trillion FPS ToF Streak Tube = Inverse of CRT Very accurate sync 1D camera: Single scan line stretched vertically in time ~2 ps resolution, 480 lines ~= 1 ns But for small samples in biochemistry
- 21. Time-Image R Time Profile for a single pixel
- 22. Time Image of a single point Time, ~2ns each row Space, 640 pixels Third Bounce (First bounce not shown)
- 25. 3D shape result from synthetic data Forward Reconstruction Invertibility Analysis Wavefront Non-linear Scene Priors Resolution and Propagation Inversion dimensions
- 26. Steady State 4D [Kajiya, 1986] [Seitz.., 2005] Impulse Response, 5D [Raskar and Davis, 2007]
- 27. Time Resolved Multi-path Imaging Scene with hidden elements Ultra fast illumination and camera 5D Raw Capture Time profiles Signal Proc. Photo, geometry, reflectance Novel light transport beyond models and inference line of sight algorithms → t 3D Time images
- 28. Third Bounce (First bounce not shown)
- 29. Third Bounce (First bounce not shown)
- 30. Third Bounce (First bounce not shown)
- 31. Third Bounce (First bounce not shown)
- 32. Third Bounce (First bounce not shown)
- 33. Photos from Streak Camera Capture Setup Hidden Scene
- 34. Photos from Streak Camera Capture Setup Hidden Scene Reconstruction Overlay
- 36. Hidden 3D Shape Space-time Photos
- 39. Motion beyond line of sight Pandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011
- 40. BRDF (reflectance) from Single Viewpoint and Time Images Naik, Zhao, Velten, Raskar, Bala, (SIGGRAPH Asia 2011)
- 42. …, cardioscopies, … Participating Media
- 46. Each frame = ~2ps = 0.6 mm of Light Travel
- 47. Camera Time
- 49. World Time
- 50. raskar.info/trillionfps Lorentz transformation: distances, velocities, ordering
- 51. FemtoFlash Trillion FPS camera Serious Sync Computational Optics Pixel Ray Wave Photons (ampli+phase) Steady State Transient Impulse/Step •2012: 3D around corner (Nature Comm, Velten et al) •2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala) •2011: DARPA Young Faculty Award •2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar) •2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009) •2008: Indirect depth (Hirsch, Raskar) •2008: Transient Light Transport (Raskar, Davis, March 2008)
- 52. • Collaborations Welcome • Dataset Available • Propose Configurations raskar(at)media.mit.edu http://www.media.mit.edu/~raskar/cornar/
- 53. http://raskar.info Femto-Photography Looking Around the Corner BRDF Detection Trillion FPS Movies Space-time Transforms
Editor's Notes
- The idea is to use the multiple bounces of light i.e. echoes of light.
- My work involves creative new ways to play with light by co-designing optical and digital processing. My work lies at the INTERSECTION of processing of photons and processing of bits. At MERL, I transformed the field of computational photography, with key papers and impact on products At Media Lab, I invented a new field ‘computational light transport’
- My idea is to use the multiple bounces of light i.e. echoes of light.
- This new form of imaging is possible by fusion of dissimilar .. A specialized camera previously used only in biochemistry labs and a new computational method that analyzes multiple bounces of light. I started the project just before I joined MIT in summer 2008. The hardware we use is borrowed and is in the lab of Prof Bawendi, MIT Chemistry, who is now a collaborator
- Here is the pipeline of how we see around corners. We have developed all the mathematical theory and now pushing into the physical experiments.
- The original formulation was in the Raskar, Davis paper in 2007.
- Here is a road map for this ambitious research project based on time-resolved imaging .. Non line of sight Looking around corner (LaC) is just one example .. Such Time resolved imaging requires one to develop a completely new set of tool for understanding our world. This is a project I started just before coming to MIT in 2008 via an NSF proposal.
- The reconstruction back in Fall 2010 was very low, about 80x80 pixels. So these are just baby steps. Top: synthetic results based on physically realistic simulations Bottom: real world results
- Top: synthetic results based on physically realistic simulations Bottom: real world results
- New results
- We can also infer reflectance and albedo Started working on a paper after a casual conversation between Raskar and Kavita Bala