1. Ramesh Raskar discusses his research in computational photography and creating new types of cameras that go beyond traditional camera capabilities. 2. The goal is to develop imaging platforms that have a deeper understanding of the visual world than humans by capturing and analyzing more information. 3. Examples of this research include cameras that can capture light fields and refocus images after capture, cameras that can remove motion blur in a single photo, and techniques for capturing high-speed motion with imperceptible tags.

1. Camera Culture Ramesh Raskar Camera Culture Associate Professor, MIT Media Lab

3. Where are the ‘camera’s?

4. Where are the ‘camera’s?

5. We focus on creating tools to better capture and share visual information The goal is to create an entirely new class of imaging platforms that have an understanding of the world that far exceeds human ability and produce meaningful abstractions that are well within human comprehensibility Ramesh Raskar

6. Questions What will a camera look like in 10,20 years?

7. Cameras of Tomorrow

8. Approach Not just USE but CHANGE camera Optics, illumination, sensor, movement Exploit wavelength, speed, depth, polarization etc Probes, actuators We have exhausted bits in pixels Scene understanding is challenging Build feature-revealing cameras Process photons Technology, Applications, Society We study impact of Imaging on all fronts

9. Computational Illumination Curved Planar Non-planar Single Projector Multiple Projectors Pocket-Proj Objects 1998 1998 2002 2002 1999 2002 2003 1998 1997 Computational Camera and Photography Projector j User : T ?

10. Motion Blurred Photo

12. Flutter Shutter Camera Raskar, Agrawal, Tumblin [Siggraph2006] LCD opacity switched in coded sequence

13. Short Exposure Traditional MURA Coded Deblurred Result Captured Single Photo Shutter Banding Artifacts and some spatial frequencies are lost Dark and noisy

14. Blurring == Convolution Traditional Camera: Shutter is OPEN: Box Filter PSF == Sinc Function ω Sharp Photo Blurred Photo Fourier Transform

15. Flutter Shutter: Shutter is OPEN and CLOSED Sharp Photo Blurred Photo PSF == Broadband Function Fourier Transform Preserves High Spatial Frequencies

16. Traditional Coded Exposure Image of Static Object Deblurred Image Deblurred Image

18. Coded Exposure Temporal 1-D broadband code: Motion Deblurring Coded Aperture Spatial 2-D broadband mask: Focus Deblurring

19. Coded Aperture Camera The aperture of a 100 mm lens is modified Rest of the camera is unmodified Insert a coded mask with chosen binary pattern

20. In Focus Photo LED

21. Out of Focus Photo: Open Aperture

22. Out of Focus Photo: Coded Aperture

23. Captured Blurred Photo

24. Refocused on Person

25. Less is More Blocking Light == More Information Coding in Time Coding in Space

26. Larval Trematode Worm Coded Aperture Camera “ Photography is the Art of Exclusion”

27. Shielding Light … Larval Trematode Worm Turbellarian Worm

28. Coded Computational Photography Coded Exposure Motion Deblurring [2006] Coded Aperture Focus Deblurring [2007] Glare reduction [2008] Optical Heterodyning Light Field Capture [2007] Coded Illumination Motion Capture [2007] Multi-flash: Shape Contours [2004] Coded Spectrum Agile Wavelength Profile [2008] Epsilon-> Coded ->Essence Photography http://raskar.info

29. Computational Photography Epsilon Photography Low-level Vision: Pixels Multiphotos by bracketing (HDR, panorama) ‘ Ultimate camera’ Coded Photography Mid-Level Cues: Regions, Edges, Motion, Direct/global Single/few snapshot Reversible encoding of data Additional sensors/optics/illum Essence Photography Not mimic human eye Beyond single view/illum ‘ New artform’

30. Epsilon Photography Dynamic range Exposure braketing [Mann-Picard, Debevec] Wider FoV Stitching a panorama Depth of field Fusion of photos with limited DoF [Agrawala04] Noise Flash/no-flash image pairs [ Petschnigg04, Eisemann04] Frame rate Triggering multiple cameras [Wilburn05, Shechtman02]

31. Computational Photography Epsilon Photography Low-level Vision: Pixels Multiphotos by bracketing (HDR, panorama) ‘ Ultimate camera’ Coded Photography Mid-Level Cues: Regions, Edges, Motion, Direct/global Single/few snapshot Reversible encoding of data Additional sensors/optics/illum Essence Photography Not mimic human eye Beyond single view/illum ‘ New artform’

32. 3D Stereo of multiple cameras Higher dimensional LF Light Field Capture lenslet array [Adelson92, Ng05] , ‘3D lens’ [Georgiev05] , heterodyne masks [Veeraraghavan07] Boundaries and Regions Multi-flash camera with shadows [Raskar08] Fg/bg matting [Chuang01,Sun06] Deblurring Engineered PSF Motion: Flutter shutter [Raskar06] , Camera Motion [Levin08] Defocus: Coded aperture, Wavefront coding [Cathey95] Global vs direct illumination High frequency illumination [Nayar06] Glare decomposition [Talvala07, Raskar08] Coded Sensor Gradient camera [Tumblin05]

33. Computational Photography Epsilon Photography Low-level Vision: Pixels Multiphotos by bracketing (HDR, panorama) ‘ Ultimate camera’ Coded Photography Mid-Level Cues: Regions, Edges, Motion, Direct/global Single/few snapshot Reversible encoding of data Additional sensors/optics/illum Essence Photography Not mimic human eye Beyond single view/illum ‘ New artform’

34. Capturing the Essence of Visual Experience Exploiting online collections Photo-tourism [Snavely2006] Scene Completion [Hays2007] Multi-perspective Images Multi-linear Perspective [Jingyi Yu, McMillan 2004] Unwrap Mosaics [Rav-Acha et al 2008] Video texture panoramas [Agrawal et al 2005] Non-photorealistic synthesis Motion magnification [Liu05] Image Priors Learned features and natural statistics Face Swapping: [Bitouk et al 2008] Data-driven enhancement of facial attractiveness [Leyvand et al 2008] Deblurring [Fergus et al 2006, Jia et al 2008]

35. Computational Photography Epsilon Photography Low-level Vision: Pixels Multiphotos by bracketing (HDR, panorama) ‘ Ultimate camera’ Coded Photography Mid-Level Cues: Regions, Edges, Motion, Direct/global Single/few snapshot Reversible encoding of data Additional sensors/optics/illum Essence Photography Not mimic human eye Beyond single view/illum ‘ New artform’

36. Ramesh Raskar and Jack Tumblin Book Publishers: A K Peters

37. Mask? Sensor 4D Light Field from 2D Photo: Heterodyne Light Field Camera Full Resolution Digital Refocusing: Coded Aperture Camera Mask? Sensor Mask Sensor Mask? Sensor Mask Sensor

38. Light Field Inside a Camera

39. Lenslet-based Light Field camera [Adelson and Wang, 1992, Ng et al. 2005 ] Light Field Inside a Camera

40. Stanford Plenoptic Camera [Ng et al 2005] 4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens Contax medium format camera Kodak 16-megapixel sensor Adaptive Optics microlens array 125 μ square-sided microlenses

41. Digital Refocusing [Ng et al 2005] Can we achieve this with a Mask alone?

42. Mask based Light Field Camera [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ] Mask Sensor

43. How to Capture 4D Light Field with 2D Sensor ? What should be the pattern of the mask ?

44. Optical Heterodyning Photographic Signal (Light Field) Carrier Incident Modulated Signal Reference Carrier Main Lens Object Mask Sensor Recovered Light Field Software Demodulation Baseband Audio Signal Receiver: Demodulation High Freq Carrier 100 MHz Reference Carrier Incoming Signal 99 MHz

45. 1/f 0 Mask Tile Cosine Mask Used

46. Captured 2D Photo Encoding due to Mask

47. 2D FFT Traditional Camera Photo Heterodyne Camera Photo Magnitude of 2D FFT 2D FFT Magnitude of 2D FFT

48. Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800 2D FFT Rearrange 2D tiles into 4D planes 200*200*9*9 4D IFFT 4D Light Field 9*9=81 spectral copies 200*200*9*9

49. How to Capture 2D Light Field with 1D Sensor ? f θ f x f θ 0 f x0 Band-limited Light Field Sensor Slice Fourier Light Field Space

50. Extra sensor bandwidth cannot capture extra dimension of the light field f θ f x f θ 0 f x0 Sensor Slice Extra sensor bandwidth Fourier Light Field Space

51. Solution: Modulation Theorem Make spectral copies of 2D light field f θ f x f θ 0 f x0 Modulation Function

52. f θ Modulated Light Field f x f θ 0 f x0 Modulation Function Sensor Slice captures entire Light Field

53. 1/f 0 Mask Tile Cosine Mask Used

54. Demodulation to recover Light Field f θ f x Reshape 1D Fourier Transform into 2D 1D Fourier Transform of Sensor Signal

55. Mask Sensor Where to place the Mask? Mask Modulation Function f x f θ

56. Full resolution 2D image of Focused Scene Parts Captured 2D Photo Image of White Lambertian Plane divide

57. Coding and Modulation in Camera Using Masks Coded Aperture for Full Resolution Digital Refocusing Heterodyne Light Field Camera Mask? Sensor Mask Sensor Mask Sensor

58. Agile Spectrum Imaging Programmable Color Gamut for Sensor With Ankit Mohan, Jack Tumblin [Eurographics 2008]

59. R ≈ 0.0 G ≈ 0.2 B ≈ 0.8 Traditional Fixed Color Gamut B G R

60. Adaptive Color Primaries

61. C B A A’ B’ C’ Pinhole Lens L 1 Prism or Diffraction Grating Lens L 2 Sensor Rainbow Plane C’’ B’’ A’’ Scene Rainbow Plane inside Camera

62. Lens Flare Reduction/Enhancement using 4D Ray Sampling Captured Glare Reduced Glare Enhanced

63. Glare = low frequency noise in 2D But is high frequency noise in 4D Remove via simple outlier rejection i j x Sensor u

64. Computational Camera and Photography

65. Dependence on incident angle

66. Dependence on incident angle

67. Towards a 6D Display Passive Reflectance Field Display Martin Fuchs, Ramesh Raskar, Hans-Peter Seidel, Hendrik P. A. Lensch Siggraph 2008 1 2 1 1 1 MPI Informatik, Germany 2 MIT 2D 2D 2D

68. View dependent 4D Display

71. 6D = light sensitive 4D display

72. One Pixel of a 6D Display = 4D Display

73. Single shot visual hull Lanman, Raskar, Agrawal, Taubin [Siggraph Asia 2008]

74. Single shot 3D reconstruction: Simultaneous Projections using Masks

75. Barcode size : 3mm x 3mm Distance from camera : 5 meter Long Distance Bar-codes Woo, Mohan, Raskar, [2008]

76. Projects Lightweight Medical Imaging High speed Tomography Muscle, blood flow activity with wearable devices for patients Femto-second Analysis of Light Transport Building and modeling future ultra-high speed cameras Avoid car-crashes, analyze complex scenes Programmable Wavelength in Thermal Range Facial expressions, Healthcare Human-emotion aware computing, Fast diagnosis Second skin Wearable fabric for bio-I/O via high speed optical motion capture Record and mimic any human motion, Care for elderly, Teach a robot

77. Vicon Motion Capture High-speed IR Camera Body-worn markers Medical Rehabilitation Athlete Analysis Performance Capture Biomechanical Analysis

78. Inverse Optical Mo-Cap Traditional Device High Speed Projector + Photosensing Markers High Speed Camera + Reflecting/Emitting Markers Params Location, Orientation, Illum Location Settings Natural Settings Ambient Light Outdoors, Stage lighting Imperceptible tags Hidden under wardrobe Controlled Lighting Visible, High contrast Markers #of Tags Unlimited Space Labeling Unique Id Limited No Unique Id Marker swapping Speed Virtually unlimited Optical comm comps Limited Special high fps camera Cost Low Open-loop projectors Current: Projector/Tag=$100 High High bandwidth camera Current Camera: $10K

79. Inside of Projector The Gray code pattern Focusing Optics Gray code Slide Condensing Optics Light Source

80. Imperceptible Tags under clothing, tracked under ambient light

81. Towards Second Skin Coded Illumination Motion Capture Clothing 500 Hz with Id for each Marker Tag Capture in Natural Environment Visually imperceptible tags Photosensing Tag can be hidden under clothes Ambient lighting is ok Unlimited Number of Tags Light sensitive fabric for dense sampling Non-imaging, complete privacy Base station and tags only a few 10’s $ Full body scan + actions Elderly, patients, athletes, performers Breathing, small twists, multiple segments or people Animation Analysis

82. Coded Imaging

83. Forerunners .. Mask Sensor Mask Sensor

84. Fernald, Science [Sept 2006] Shadow Refractive Reflective Tools for Visual Computing

86. Blind Camera Sascha Pohflepp, U of the Art, Berlin, 2006

87. Cameras of Tomorrow

88. Cameras of Tomorrow Coded Exposure Motion Deblurring [2006] Coded Aperture Focus Deblurring [2007] Glare reduction [2008] Optical Heterodyning Light Field Capture [2007] Coded Illumination Motion Capture [2007] Multi-flash: Shape Contours [2004] Coded Spectrum Agile Wavelength Profile [2008] Epsilon-> Coded ->Essence Photography http://raskar.info

89. We focus on creating tools to better capture and share visual information The goal is to create an entirely new class of imaging platforms that have an understanding of the world that far exceeds human ability and produce meaningful abstractions that are well within human comprehensibility Ramesh Raskar http://raskar.info

90. Capture Cameras Everywhere Deep pervasive sensing Analysis Computer Vision Mo-cap Personalized services, tracking in real world Animation Simulation of bio/chemical/physical processes at all scales Synthesis Virtual Human, Digital Actors, Tele-avatars Exa and Zeta-scale computing: simulate every neural activity, predict weather for weeks, simulate impact of global warming Display Real world AR Realistic Displays: 6D or 8D