We have built a camera that can look around corners and beyond the line of sight. The camera uses light that travels from the object to the camera indirectly, by reflecting off walls or other obstacles, to reconstruct a 3D shape.
The idea is to use the multiple bounces of light i.e. echoes of light.
My work involves creative new ways to play with light by co-designing optical and digital processing. My work lies at the INTERSECTION of processing of photons and processing of bits. At MERL, I transformed the field of computational photography, with key papers and impact on products At Media Lab, I invented a new field ‘computational light transport’
My idea is to use the multiple bounces of light i.e. echoes of light.
This new form of imaging is possible by fusion of dissimilar .. A specialized camera previously used only in biochemistry labs and a new computational method that analyzes multiple bounces of light. I started the project just before I joined MIT in summer 2008. The hardware we use is borrowed and is in the lab of Prof Bawendi, MIT Chemistry, who is now a collaborator
Here is the pipeline of how we see around corners. We have developed all the mathematical theory and now pushing into the physical experiments.
The original formulation was in the Raskar, Davis paper in 2007.
Here is a road map for this ambitious research project based on time-resolved imaging .. Non line of sight Looking around corner (LaC) is just one example .. Such Time resolved imaging requires one to develop a completely new set of tool for understanding our world. This is a project I started just before coming to MIT in 2008 via an NSF proposal.
The reconstruction back in Fall 2010 was very low, about 80x80 pixels. So these are just baby steps. Top: synthetic results based on physically realistic simulations Bottom: real world results
Top: synthetic results based on physically realistic simulations Bottom: real world results
We can also infer reflectance and albedo Started working on a paper after a casual conversation between Raskar and Kavita Bala
CORNAR: Looking Around Corners using Trillion FPS Imaging
Raskar, Camera Culture, MIT Media Lab Computational Light Transport: CORNAR: Looking Around Corners Camera Culture using Trillion FPS Imaging Ramesh Raskar Ramesh Raskar MIT Media Lab http://raskar.info/cornar
CORNAR: Femto-Photography FemtoFlash Trillion FPS camera With M Bawendi, MIT Chemistry Serious Sync Computational Optics•2012: 3D around a corner (NatureComm, Velten, et. al.)•2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala)•2011: DARPA Young Faculty Award•2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar)•2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)•2008: Indirect depth (Hirsch, Raskar)•2008: Transient Light Transport (Raskar, Davis, March 2008)
Inverting Light Transport Multiple Scattering Direct/Global[Seitz , Kutulakos, Matsushita 2005] [Nayar, Raskar et al 2006] [Atcheson et al 2008] [Kutulakos, Steger 2005] Dual Photography LIDAR [Sen et al 2005]
Time Image of a single pointTime, ~2ns each row Space, 640 pixels Third Bounce (First bounce not shown)
3D shape result from synthetic data Forward Reconstruction Invertibility Analysis Wavefront Non-linear Scene Priors Resolution andPropagation Inversion dimensions
Steady State 4D [Kajiya, 1986] [Seitz.., 2005]Impulse Response, 5D [Raskar and Davis, 2007]
Time Resolved Multi-path Imaging Scene withhidden elements Ultra fast illumination and camera 5D Raw Capture Time profiles Signal Proc.Photo, geometry, reflectance Novel light transport beyond models and inference line of sight algorithms → t 3D Time images