Successfully reported this slideshow.

CORNAR: Looking Around Corners using Trillion FPS Imaging

4

Share

Loading in …3
×
1 of 53
1 of 53

CORNAR: Looking Around Corners using Trillion FPS Imaging

4

Share

Download to read offline

We have built a camera that can look around corners and beyond the line of sight. The camera uses light that travels from the object to the camera indirectly, by reflecting off walls or other obstacles, to reconstruct a 3D shape.

We have built a camera that can look around corners and beyond the line of sight. The camera uses light that travels from the object to the camera indirectly, by reflecting off walls or other obstacles, to reconstruct a 3D shape.

More Related Content

More from Camera Culture Group, MIT Media Lab

Related Books

Free with a 14 day trial from Scribd

See all

CORNAR: Looking Around Corners using Trillion FPS Imaging

  1. 1. Raskar, Camera Culture, MIT Media Lab Computational Light Transport: CORNAR: Looking Around Corners Camera Culture using Trillion FPS Imaging Ramesh Raskar Ramesh Raskar MIT Media Lab http://raskar.info/cornar
  2. 2. What is around the corner ?
  3. 3. Can you look around the corner ?
  4. 4. Can you look around the corner ?
  5. 5. Multi-path Analysis 2nd Bounce 1st Bounce 3rd Bounce
  6. 6. Co-designing Optical and Digital Processing Computational Optics Light Transport Photon Hacking Displays Sensors Computational Illumination Photography Signal Processing Computer Vision Machine Learning Bit Hacking
  7. 7. Multi-path Analysis 2nd Bounce 1st Bounce 3rd Bounce
  8. 8. CORNAR: Femto-Photography FemtoFlash Trillion FPS camera With M Bawendi, MIT Chemistry Serious Sync Computational Optics •2012: 3D around a corner (NatureComm, Velten, et. al.) •2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala) •2011: DARPA Young Faculty Award •2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar) •2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009) •2008: Indirect depth (Hirsch, Raskar) •2008: Transient Light Transport (Raskar, Davis, March 2008)
  9. 9. Inverting Light Transport Multiple Scattering Direct/Global [Seitz , Kutulakos, Matsushita 2005] [Nayar, Raskar et al 2006] [Atcheson et al 2008] [Kutulakos, Steger 2005] Dual Photography LIDAR [Sen et al 2005]
  10. 10. Collision avoidance, robot navigation, …
  11. 11. …, bronchoscopies, …
  12. 12. z S x L s Occluder Streak- camera C Laser B beam Echoes of Light
  13. 13. z S x L s Occluder Streak- camera C Laser B beam Echoes of Light
  14. 14. z S x L R s Occluder Streak- camera 3rd bounce Streak Photo C Laser B beam Echoes of Light
  15. 15. Multi-Dimensional Light Transport 5-D Transport
  16. 16. Why Pico-second Resolution? ToF Diff = 0.15 mm s2 1cm s1 Occluder 3rd bounce Streak- camera C p1 p2 1st bounce Curse of Pythagoras
  17. 17. z S x L R s Occluder Streak- camera 3rd bounce Streak Photo C Laser B beam Echoes of Light
  18. 18. Trillion FPS ToF Streak Tube = Inverse of CRT Very accurate sync 1D camera: Single scan line stretched vertically in time ~2 ps resolution, 480 lines ~= 1 ns But for small samples in biochemistry
  19. 19. Time-Image R Time Profile for a single pixel
  20. 20. Time Image of a single point Time, ~2ns each row Space, 640 pixels Third Bounce (First bounce not shown)
  21. 21. 3D shape result from synthetic data Forward Reconstruction Invertibility Analysis Wavefront Non-linear Scene Priors Resolution and Propagation Inversion dimensions
  22. 22. Steady State 4D [Kajiya, 1986] [Seitz.., 2005] Impulse Response, 5D [Raskar and Davis, 2007]
  23. 23. Time Resolved Multi-path Imaging Scene with hidden elements Ultra fast illumination and camera 5D Raw Capture Time profiles Signal Proc. Photo, geometry, reflectance Novel light transport beyond models and inference line of sight algorithms → t 3D Time images
  24. 24. Third Bounce (First bounce not shown)
  25. 25. Third Bounce (First bounce not shown)
  26. 26. Third Bounce (First bounce not shown)
  27. 27. Third Bounce (First bounce not shown)
  28. 28. Third Bounce (First bounce not shown)
  29. 29. Photos from Streak Camera Capture Setup Hidden Scene
  30. 30. Photos from Streak Camera Capture Setup Hidden Scene Reconstruction Overlay
  31. 31. Hidden 3D Shape Space-time Photos
  32. 32. Motion beyond line of sight Pandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011
  33. 33. BRDF (reflectance) from Single Viewpoint and Time Images Naik, Zhao, Velten, Raskar, Bala, (SIGGRAPH Asia 2011)
  34. 34. …, cardioscopies, … Participating Media
  35. 35. Space-time Photos
  36. 36. Trillion Frames Per Second Imaging http://raskar.info/trillionfps
  37. 37. Each frame = ~2ps = 0.6 mm of Light Travel
  38. 38. Camera Time
  39. 39. raskar.info/trillionfps Camera Time
  40. 40. World Time
  41. 41. raskar.info/trillionfps  Lorentz transformation: distances, velocities, ordering
  42. 42. FemtoFlash Trillion FPS camera Serious Sync Computational Optics Pixel  Ray  Wave  Photons (ampli+phase) Steady State Transient Impulse/Step •2012: 3D around corner (Nature Comm, Velten et al) •2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala) •2011: DARPA Young Faculty Award •2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar) •2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009) •2008: Indirect depth (Hirsch, Raskar) •2008: Transient Light Transport (Raskar, Davis, March 2008)
  43. 43. • Collaborations Welcome • Dataset Available • Propose Configurations raskar(at)media.mit.edu http://www.media.mit.edu/~raskar/cornar/
  44. 44. http://raskar.info Femto-Photography Looking Around the Corner BRDF Detection Trillion FPS Movies Space-time Transforms

Editor's Notes

  • The idea is to use the multiple bounces of light i.e. echoes of light.
  • My work involves creative new ways to play with light by co-designing optical and digital processing. My work lies at the INTERSECTION of processing of photons and processing of bits. At MERL, I transformed the field of computational photography, with key papers and impact on products At Media Lab, I invented a new field ‘computational light transport’
  • My idea is to use the multiple bounces of light i.e. echoes of light.
  • This new form of imaging is possible by fusion of dissimilar .. A specialized camera previously used only in biochemistry labs and a new computational method that analyzes multiple bounces of light. I started the project just before I joined MIT in summer 2008. The hardware we use is borrowed and is in the lab of Prof Bawendi, MIT Chemistry, who is now a collaborator
  • Here is the pipeline of how we see around corners. We have developed all the mathematical theory and now pushing into the physical experiments.
  • The original formulation was in the Raskar, Davis paper in 2007.
  • Here is a road map for this ambitious research project based on time-resolved imaging .. Non line of sight Looking around corner (LaC) is just one example .. Such Time resolved imaging requires one to develop a completely new set of tool for understanding our world. This is a project I started just before coming to MIT in 2008 via an NSF proposal.
  • The reconstruction back in Fall 2010 was very low, about 80x80 pixels. So these are just baby steps. Top: synthetic results based on physically realistic simulations Bottom: real world results
  • Top: synthetic results based on physically realistic simulations Bottom: real world results
  • New results
  • We can also infer reflectance and albedo Started working on a paper after a casual conversation between Raskar and Kavita Bala
  • ×