CORNAR: Looking Around Corners using Trillion FPS Imaging

64,970 views

Published on

We have built a camera that can look around corners and beyond the line of sight. The camera uses light that travels from the object to the camera indirectly, by reflecting off walls or other obstacles, to reconstruct a 3D shape.

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
64,970
On SlideShare
0
From Embeds
0
Number of Embeds
58,491
Actions
Shares
0
Downloads
206
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • The idea is to use the multiple bounces of light i.e. echoes of light.
  • My work involves creative new ways to play with light by co-designing optical and digital processing. My work lies at the INTERSECTION of processing of photons and processing of bits. At MERL, I transformed the field of computational photography, with key papers and impact on products At Media Lab, I invented a new field ‘computational light transport’
  • My idea is to use the multiple bounces of light i.e. echoes of light.
  • This new form of imaging is possible by fusion of dissimilar .. A specialized camera previously used only in biochemistry labs and a new computational method that analyzes multiple bounces of light. I started the project just before I joined MIT in summer 2008. The hardware we use is borrowed and is in the lab of Prof Bawendi, MIT Chemistry, who is now a collaborator
  • Here is the pipeline of how we see around corners. We have developed all the mathematical theory and now pushing into the physical experiments.
  • The original formulation was in the Raskar, Davis paper in 2007.
  • Here is a road map for this ambitious research project based on time-resolved imaging .. Non line of sight Looking around corner (LaC) is just one example .. Such Time resolved imaging requires one to develop a completely new set of tool for understanding our world. This is a project I started just before coming to MIT in 2008 via an NSF proposal.
  • The reconstruction back in Fall 2010 was very low, about 80x80 pixels. So these are just baby steps. Top: synthetic results based on physically realistic simulations Bottom: real world results
  • Top: synthetic results based on physically realistic simulations Bottom: real world results
  • New results
  • We can also infer reflectance and albedo Started working on a paper after a casual conversation between Raskar and Kavita Bala
  • CORNAR: Looking Around Corners using Trillion FPS Imaging

    1. 1. Raskar, Camera Culture, MIT Media Lab Computational Light Transport: CORNAR: Looking Around Corners Camera Culture using Trillion FPS Imaging Ramesh Raskar Ramesh Raskar MIT Media Lab http://raskar.info/cornar
    2. 2. What isaround the corner ?
    3. 3. Can you lookaround the corner ?
    4. 4. Can you lookaround the corner ?
    5. 5. Multi-path Analysis2nd Bounce 1st Bounce 3rd Bounce
    6. 6. Co-designing Optical and Digital Processing Computational Optics Light TransportPhoton Hacking Displays Sensors Computational Illumination Photography Signal Processing Computer Vision Machine Learning Bit Hacking
    7. 7. Multi-path Analysis2nd Bounce 1st Bounce 3rd Bounce
    8. 8. CORNAR: Femto-Photography FemtoFlash Trillion FPS camera With M Bawendi, MIT Chemistry Serious Sync Computational Optics•2012: 3D around a corner (NatureComm, Velten, et. al.)•2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala)•2011: DARPA Young Faculty Award•2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar)•2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)•2008: Indirect depth (Hirsch, Raskar)•2008: Transient Light Transport (Raskar, Davis, March 2008)
    9. 9. Inverting Light Transport Multiple Scattering Direct/Global[Seitz , Kutulakos, Matsushita 2005] [Nayar, Raskar et al 2006] [Atcheson et al 2008] [Kutulakos, Steger 2005] Dual Photography LIDAR [Sen et al 2005]
    10. 10. Collision avoidance, robot navigation, …
    11. 11. …, bronchoscopies, …
    12. 12. z S x L sOccluder Streak- camera C Laser B beam Echoes of Light
    13. 13. z S x L sOccluder Streak- camera C Laser B beam Echoes of Light
    14. 14. z S x L R s Occluder Streak- camera 3rd bounceStreak Photo C Laser B beam Echoes of Light
    15. 15. Multi-Dimensional Light Transport5-D Transport
    16. 16. Why Pico-second Resolution? ToF Diff = 0.15 mm s2 1cm s1Occluder 3rd bounce Streak- camera C p1 p2 1st bounce Curse of Pythagoras
    17. 17. z S x L R s Occluder Streak- camera 3rd bounceStreak Photo C Laser B beam Echoes of Light
    18. 18. Trillion FPS ToF Streak Tube = Inverse of CRT Very accurate sync1D camera: Single scan line stretched vertically in time ~2 ps resolution, 480 lines ~= 1 ns But for small samples in biochemistry
    19. 19. Time-ImageR Time Profile for a single pixel
    20. 20. Time Image of a single pointTime, ~2ns each row Space, 640 pixels Third Bounce (First bounce not shown)
    21. 21. 3D shape result from synthetic data Forward Reconstruction Invertibility Analysis Wavefront Non-linear Scene Priors Resolution andPropagation Inversion dimensions
    22. 22. Steady State 4D [Kajiya, 1986] [Seitz.., 2005]Impulse Response, 5D [Raskar and Davis, 2007]
    23. 23. Time Resolved Multi-path Imaging Scene withhidden elements Ultra fast illumination and camera 5D Raw Capture Time profiles Signal Proc.Photo, geometry, reflectance Novel light transport beyond models and inference line of sight algorithms → t 3D Time images
    24. 24. Third Bounce (First bounce not shown)
    25. 25. Third Bounce (First bounce not shown)
    26. 26. Third Bounce (First bounce not shown)
    27. 27. Third Bounce (First bounce not shown)
    28. 28. Third Bounce (First bounce not shown)
    29. 29. Photos from Streak CameraCapture Setup Hidden Scene
    30. 30. Photos from Streak CameraCapture Setup Hidden Scene Reconstruction Overlay
    31. 31. Hidden 3D Shape Space-time Photos
    32. 32. Motion beyond line of sightPandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011
    33. 33. BRDF (reflectance) fromSingle Viewpoint andTime ImagesNaik, Zhao, Velten, Raskar, Bala, (SIGGRAPH Asia 2011)
    34. 34. …, cardioscopies, …Participating Media
    35. 35. Space-timePhotos
    36. 36. TrillionFrames Per SecondImaginghttp://raskar.info/trillionfps
    37. 37. Each frame = ~2ps = 0.6 mm of Light Travel
    38. 38. Camera Time
    39. 39. raskar.info/trillionfpsCamera Time
    40. 40. World Time
    41. 41. raskar.info/trillionfps Lorentz transformation: distances, velocities, ordering
    42. 42. FemtoFlash Trillion FPS camera Serious Sync Computational OpticsPixel  Ray  Wave  Photons (ampli+phase) Steady State Transient Impulse/Step •2012: 3D around corner (Nature Comm, Velten et al) •2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala) •2011: DARPA Young Faculty Award •2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar) •2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009) •2008: Indirect depth (Hirsch, Raskar) •2008: Transient Light Transport (Raskar, Davis, March 2008)
    43. 43. • Collaborations Welcome• Dataset Available• Propose Configurationsraskar(at)media.mit.eduhttp://www.media.mit.edu/~raskar/cornar/
    44. 44. http://raskar.info Femto-PhotographyLooking Around the Corner BRDF Detection Trillion FPS Movies Space-time Transforms

    ×