Raskar Paris Nov08

1,127 views

Published on

Published in: Art & Photos, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,127
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
20
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Raskar Paris Nov08

    1. 1. Camera Culture Ramesh Raskar Camera Culture Associate Professor, MIT Media Lab Computational Photography: Epsilon to Coded Imaging
    2. 3. Where are the ‘camera’s?
    3. 4. Where are the ‘camera’s?
    4. 5. <ul><li>We focus on creating tools to better capture and share visual information </li></ul><ul><li>The goal is to create an entirely new class of imaging platforms that have an understanding of the world that far exceeds human ability and produce meaningful abstractions that are well within human comprehensibility </li></ul>Ramesh Raskar
    5. 6. Questions <ul><li>What will a camera look like in 10,20 years? </li></ul>
    6. 7. Cameras of Tomorrow
    7. 8. Approach <ul><li>Not just USE but CHANGE camera </li></ul><ul><ul><li>Optics, illumination, sensor, movement </li></ul></ul><ul><ul><li>Exploit wavelength, speed, depth, polarization etc </li></ul></ul><ul><ul><li>Probes, actuators </li></ul></ul><ul><li>We have exhausted bits in pixels </li></ul><ul><ul><li>Scene understanding is challenging </li></ul></ul><ul><ul><li>Build feature-revealing cameras </li></ul></ul><ul><ul><li>Process photons </li></ul></ul><ul><li>Technology, Applications, Society </li></ul><ul><ul><li>We study impact of Imaging on all fronts </li></ul></ul>
    8. 9. Computational Illumination Curved Planar Non-planar Single Projector Multiple Projectors Pocket-Proj Objects 1998 1998 2002 2002 1999 2002 2003 1998 1997 Computational Camera and Photography Projector j User : T ?
    9. 10. Motion Blurred Photo
    10. 12. Flutter Shutter Camera Raskar, Agrawal, Tumblin [Siggraph2006] LCD opacity switched in coded sequence
    11. 13. Short Exposure Traditional MURA Coded Deblurred Result Captured Single Photo Shutter Banding Artifacts and some spatial frequencies are lost Dark and noisy
    12. 14. Blurring == Convolution Traditional Camera: Shutter is OPEN: Box Filter PSF == Sinc Function ω Sharp Photo Blurred Photo Fourier Transform
    13. 15. Flutter Shutter: Shutter is OPEN and CLOSED Sharp Photo Blurred Photo PSF == Broadband Function Fourier Transform Preserves High Spatial Frequencies
    14. 16. Traditional Coded Exposure Image of Static Object Deblurred Image Deblurred Image
    15. 18. Coded Exposure Temporal 1-D broadband code: Motion Deblurring Coded Aperture Spatial 2-D broadband mask: Focus Deblurring
    16. 19. Coded Aperture Camera The aperture of a 100 mm lens is modified Rest of the camera is unmodified Insert a coded mask with chosen binary pattern
    17. 20. In Focus Photo LED
    18. 21. Out of Focus Photo: Open Aperture
    19. 22. Out of Focus Photo: Coded Aperture
    20. 23. Captured Blurred Photo
    21. 24. Refocused on Person
    22. 25. Less is More Blocking Light == More Information Coding in Time Coding in Space
    23. 26. Larval Trematode Worm Coded Aperture Camera “ Photography is the Art of Exclusion”
    24. 27. Shielding Light … Larval Trematode Worm Turbellarian Worm
    25. 28. Coded Computational Photography <ul><li>Coded Exposure </li></ul><ul><ul><li>Motion Deblurring [2006] </li></ul></ul><ul><li>Coded Aperture </li></ul><ul><ul><li>Focus Deblurring [2007] </li></ul></ul><ul><ul><li>Glare reduction [2008] </li></ul></ul><ul><li>Optical Heterodyning </li></ul><ul><ul><li>Light Field Capture [2007] </li></ul></ul><ul><li>Coded Illumination </li></ul><ul><ul><li>Motion Capture [2007] </li></ul></ul><ul><ul><li>Multi-flash: Shape Contours [2004] </li></ul></ul><ul><li>Coded Spectrum </li></ul><ul><ul><li>Agile Wavelength Profile [2008] </li></ul></ul><ul><li>Epsilon-> Coded ->Essence Photography </li></ul>http://raskar.info
    26. 29. Computational Photography <ul><li>Epsilon Photography </li></ul><ul><ul><li>Low-level Vision: Pixels </li></ul></ul><ul><ul><li>Multiphotos by bracketing (HDR, panorama) </li></ul></ul><ul><ul><li>‘ Ultimate camera’ </li></ul></ul><ul><li>Coded Photography </li></ul><ul><ul><li>Mid-Level Cues: </li></ul></ul><ul><ul><ul><li>Regions, Edges, Motion, Direct/global </li></ul></ul></ul><ul><ul><li>Single/few snapshot </li></ul></ul><ul><ul><ul><li>Reversible encoding of data </li></ul></ul></ul><ul><ul><li>Additional sensors/optics/illum </li></ul></ul><ul><li>Essence Photography </li></ul><ul><ul><li>Not mimic human eye </li></ul></ul><ul><ul><li>Beyond single view/illum </li></ul></ul><ul><ul><li>‘ New artform’ </li></ul></ul>
    27. 30. Epsilon Photography <ul><li>Dynamic range </li></ul><ul><ul><li>Exposure braketing [Mann-Picard, Debevec] </li></ul></ul><ul><li>Wider FoV </li></ul><ul><ul><li>Stitching a panorama </li></ul></ul><ul><li>Depth of field </li></ul><ul><ul><li>Fusion of photos with limited DoF [Agrawala04] </li></ul></ul><ul><li>Noise </li></ul><ul><ul><li>Flash/no-flash image pairs [ Petschnigg04, Eisemann04] </li></ul></ul><ul><li>Frame rate </li></ul><ul><ul><li>Triggering multiple cameras [Wilburn05, Shechtman02] </li></ul></ul>
    28. 31. Computational Photography <ul><li>Epsilon Photography </li></ul><ul><ul><li>Low-level Vision: Pixels </li></ul></ul><ul><ul><li>Multiphotos by bracketing (HDR, panorama) </li></ul></ul><ul><ul><li>‘ Ultimate camera’ </li></ul></ul><ul><li>Coded Photography </li></ul><ul><ul><li>Mid-Level Cues: </li></ul></ul><ul><ul><ul><li>Regions, Edges, Motion, Direct/global </li></ul></ul></ul><ul><ul><li>Single/few snapshot </li></ul></ul><ul><ul><ul><li>Reversible encoding of data </li></ul></ul></ul><ul><ul><li>Additional sensors/optics/illum </li></ul></ul><ul><li>Essence Photography </li></ul><ul><ul><li>Not mimic human eye </li></ul></ul><ul><ul><li>Beyond single view/illum </li></ul></ul><ul><ul><li>‘ New artform’ </li></ul></ul>
    29. 32. <ul><li>3D </li></ul><ul><ul><li>Stereo of multiple cameras </li></ul></ul><ul><li>Higher dimensional LF </li></ul><ul><ul><li>Light Field Capture </li></ul></ul><ul><ul><ul><li>lenslet array [Adelson92, Ng05] , ‘3D lens’ [Georgiev05] , heterodyne masks [Veeraraghavan07] </li></ul></ul></ul><ul><li>Boundaries and Regions </li></ul><ul><ul><li>Multi-flash camera with shadows [Raskar08] </li></ul></ul><ul><ul><li>Fg/bg matting [Chuang01,Sun06] </li></ul></ul><ul><li>Deblurring </li></ul><ul><ul><li>Engineered PSF </li></ul></ul><ul><ul><li>Motion: Flutter shutter [Raskar06] , Camera Motion [Levin08] </li></ul></ul><ul><ul><li>Defocus: Coded aperture, Wavefront coding [Cathey95] </li></ul></ul><ul><li>Global vs direct illumination </li></ul><ul><ul><li>High frequency illumination [Nayar06] </li></ul></ul><ul><ul><li>Glare decomposition [Talvala07, Raskar08] </li></ul></ul><ul><li>Coded Sensor </li></ul><ul><ul><li>Gradient camera [Tumblin05] </li></ul></ul>
    30. 33. Computational Photography <ul><li>Epsilon Photography </li></ul><ul><ul><li>Low-level Vision: Pixels </li></ul></ul><ul><ul><li>Multiphotos by bracketing (HDR, panorama) </li></ul></ul><ul><ul><li>‘ Ultimate camera’ </li></ul></ul><ul><li>Coded Photography </li></ul><ul><ul><li>Mid-Level Cues: </li></ul></ul><ul><ul><ul><li>Regions, Edges, Motion, Direct/global </li></ul></ul></ul><ul><ul><li>Single/few snapshot </li></ul></ul><ul><ul><ul><li>Reversible encoding of data </li></ul></ul></ul><ul><ul><li>Additional sensors/optics/illum </li></ul></ul><ul><li>Essence Photography </li></ul><ul><ul><li>Not mimic human eye </li></ul></ul><ul><ul><li>Beyond single view/illum </li></ul></ul><ul><ul><li>‘ New artform’ </li></ul></ul>
    31. 34. Capturing the Essence of Visual Experience <ul><ul><li>Exploiting online collections </li></ul></ul><ul><ul><ul><li>Photo-tourism [Snavely2006] </li></ul></ul></ul><ul><ul><ul><li>Scene Completion [Hays2007] </li></ul></ul></ul><ul><ul><li>Multi-perspective Images </li></ul></ul><ul><ul><ul><li>Multi-linear Perspective [Jingyi Yu, McMillan 2004] </li></ul></ul></ul><ul><ul><ul><li>Unwrap Mosaics [Rav-Acha et al 2008] </li></ul></ul></ul><ul><ul><ul><li>Video texture panoramas [Agrawal et al 2005] </li></ul></ul></ul><ul><ul><li>Non-photorealistic synthesis </li></ul></ul><ul><ul><ul><li>Motion magnification [Liu05] </li></ul></ul></ul><ul><ul><li>Image Priors </li></ul></ul><ul><ul><ul><li>Learned features and natural statistics </li></ul></ul></ul><ul><ul><ul><li>Face Swapping: [Bitouk et al 2008] </li></ul></ul></ul><ul><ul><ul><li>Data-driven enhancement of facial attractiveness [Leyvand et al 2008] </li></ul></ul></ul><ul><ul><ul><li>Deblurring [Fergus et al 2006, Jia et al 2008] </li></ul></ul></ul>
    32. 35. Computational Photography <ul><li>Epsilon Photography </li></ul><ul><ul><li>Low-level Vision: Pixels </li></ul></ul><ul><ul><li>Multiphotos by bracketing (HDR, panorama) </li></ul></ul><ul><ul><li>‘ Ultimate camera’ </li></ul></ul><ul><li>Coded Photography </li></ul><ul><ul><li>Mid-Level Cues: </li></ul></ul><ul><ul><ul><li>Regions, Edges, Motion, Direct/global </li></ul></ul></ul><ul><ul><li>Single/few snapshot </li></ul></ul><ul><ul><ul><li>Reversible encoding of data </li></ul></ul></ul><ul><ul><li>Additional sensors/optics/illum </li></ul></ul><ul><li>Essence Photography </li></ul><ul><ul><li>Not mimic human eye </li></ul></ul><ul><ul><li>Beyond single view/illum </li></ul></ul><ul><ul><li>‘ New artform’ </li></ul></ul>
    33. 36. <ul><li>Ramesh Raskar and Jack Tumblin </li></ul><ul><li>Book Publishers: A K Peters </li></ul>
    34. 37. Mask? Sensor 4D Light Field from 2D Photo: Heterodyne Light Field Camera Full Resolution Digital Refocusing: Coded Aperture Camera Mask? Sensor Mask Sensor Mask? Sensor Mask Sensor
    35. 38. Light Field Inside a Camera
    36. 39. Lenslet-based Light Field camera [Adelson and Wang, 1992, Ng et al. 2005 ] Light Field Inside a Camera
    37. 40. Stanford Plenoptic Camera [Ng et al 2005] <ul><li>4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens </li></ul>Contax medium format camera Kodak 16-megapixel sensor Adaptive Optics microlens array 125 μ square-sided microlenses
    38. 41. Digital Refocusing [Ng et al 2005] Can we achieve this with a Mask alone?
    39. 42. Mask based Light Field Camera [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ] Mask Sensor
    40. 43. How to Capture 4D Light Field with 2D Sensor ? What should be the pattern of the mask ?
    41. 44. Optical Heterodyning Photographic Signal (Light Field) Carrier Incident Modulated Signal Reference Carrier Main Lens Object Mask Sensor Recovered Light Field Software Demodulation Baseband Audio Signal Receiver: Demodulation High Freq Carrier 100 MHz Reference Carrier Incoming Signal 99 MHz
    42. 45. 1/f 0 Mask Tile Cosine Mask Used
    43. 46. Captured 2D Photo Encoding due to Mask
    44. 47. 2D FFT Traditional Camera Photo Heterodyne Camera Photo Magnitude of 2D FFT 2D FFT Magnitude of 2D FFT
    45. 48. Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800 2D FFT Rearrange 2D tiles into 4D planes 200*200*9*9 4D IFFT 4D Light Field 9*9=81 spectral copies 200*200*9*9
    46. 49. How to Capture 2D Light Field with 1D Sensor ? f θ f x f θ 0 f x0 Band-limited Light Field Sensor Slice Fourier Light Field Space
    47. 50. Extra sensor bandwidth cannot capture extra dimension of the light field f θ f x f θ 0 f x0 Sensor Slice Extra sensor bandwidth Fourier Light Field Space
    48. 51. Solution: Modulation Theorem Make spectral copies of 2D light field f θ f x f θ 0 f x0 Modulation Function
    49. 52. f θ Modulated Light Field f x f θ 0 f x0 Modulation Function Sensor Slice captures entire Light Field
    50. 53. 1/f 0 Mask Tile Cosine Mask Used
    51. 54. Demodulation to recover Light Field f θ f x Reshape 1D Fourier Transform into 2D 1D Fourier Transform of Sensor Signal
    52. 55. Mask Sensor Where to place the Mask? Mask Modulation Function f x f θ
    53. 56. Full resolution 2D image of Focused Scene Parts Captured 2D Photo Image of White Lambertian Plane divide
    54. 57. Coding and Modulation in Camera Using Masks Coded Aperture for Full Resolution Digital Refocusing Heterodyne Light Field Camera Mask? Sensor Mask Sensor Mask Sensor
    55. 58. Agile Spectrum Imaging Programmable Color Gamut for Sensor With Ankit Mohan, Jack Tumblin [Eurographics 2008]
    56. 59. R ≈ 0.0 G ≈ 0.2 B ≈ 0.8 Traditional Fixed Color Gamut B G R
    57. 60. Adaptive Color Primaries
    58. 61. C B A A’ B’ C’ Pinhole Lens L 1 Prism or Diffraction Grating Lens L 2 Sensor Rainbow Plane C’’ B’’ A’’ Scene Rainbow Plane inside Camera
    59. 62. Lens Flare Reduction/Enhancement using 4D Ray Sampling Captured Glare Reduced Glare Enhanced
    60. 63. <ul><li>Glare = low frequency noise in 2D </li></ul><ul><ul><li>But is high frequency noise in 4D </li></ul></ul><ul><ul><li>Remove via simple outlier rejection </li></ul></ul>i j x Sensor u
    61. 64. Single shot visual hull: Shield Field Lanman, Raskar, Agrawal, Taubin, Siggraph Asia 2008
    62. 65. Single shot 3D reconstruction: Simultaneous Projections using Masks
    63. 66. <ul><li>Barcode size : 3mm x 3mm </li></ul><ul><li>Distance from camera : 5 meter </li></ul>Long Distance Bar-codes Woo, Mohan, Raskar, [2008]
    64. 67. Computational Camera and Photography
    65. 68. Vicon Motion Capture High-speed IR Camera Body-worn markers Medical Rehabilitation Athlete Analysis Performance Capture Biomechanical Analysis
    66. 69. Inverse Optical Mo-Cap Traditional Device High Speed Projector + Photosensing Markers High Speed Camera + Reflecting/Emitting Markers Params Location, Orientation, Illum Location Settings <ul><li>Natural Settings </li></ul><ul><ul><li>Ambient Light </li></ul></ul><ul><ul><li>Outdoors, Stage lighting </li></ul></ul><ul><ul><li>Imperceptible tags </li></ul></ul><ul><ul><li>Hidden under wardrobe </li></ul></ul><ul><li>Controlled Lighting </li></ul><ul><ul><li>Visible, High contrast Markers </li></ul></ul>#of Tags <ul><li>Unlimited </li></ul><ul><ul><li>Space Labeling </li></ul></ul><ul><ul><li>Unique Id </li></ul></ul><ul><li>Limited </li></ul><ul><ul><li>No Unique Id </li></ul></ul><ul><ul><li>Marker swapping </li></ul></ul>Speed Virtually unlimited Optical comm comps <ul><li>Limited </li></ul><ul><ul><li>Special high fps camera </li></ul></ul>Cost Low Open-loop projectors Current: Projector/Tag=$100 High High bandwidth camera Current Camera: $10K
    67. 70. Inside of Projector The Gray code pattern Focusing Optics Gray code Slide Condensing Optics Light Source
    68. 71. Imperceptible Tags under clothing, tracked under ambient light
    69. 72. Towards Second Skin Coded Illumination Motion Capture Clothing <ul><li>500 Hz with Id for each Marker Tag </li></ul><ul><li>Capture in Natural Environment </li></ul><ul><ul><li>Visually imperceptible tags </li></ul></ul><ul><ul><li>Photosensing Tag can be hidden under clothes </li></ul></ul><ul><ul><li>Ambient lighting is ok </li></ul></ul><ul><li>Unlimited Number of Tags </li></ul><ul><ul><li>Light sensitive fabric for dense sampling </li></ul></ul><ul><li>Non-imaging, complete privacy </li></ul><ul><li>Base station and tags only a few 10’s $ </li></ul><ul><li>Full body scan + actions </li></ul><ul><ul><li>Elderly, patients, athletes, performers </li></ul></ul><ul><ul><li>Breathing, small twists, multiple segments or people </li></ul></ul><ul><ul><li>Animation Analysis </li></ul></ul>
    70. 73. Coded Imaging
    71. 74. Dependence on incident angle
    72. 75. Dependence on incident angle
    73. 76. Towards a 6D Display Passive Reflectance Field Display Martin Fuchs, Ramesh Raskar, Hans-Peter Seidel, Hendrik P. A. Lensch Siggraph 2008 1 2 1 1 1 MPI Informatik, Germany 2 MIT 2D 2D 2D
    74. 77. View dependent 4D Display
    75. 80. 6D = light sensitive 4D display
    76. 81. One Pixel of a 6D Display = 4D Display
    77. 82. Forerunners .. Mask Sensor Mask Sensor
    78. 83. Camera Culture group <ul><li>New group at Media Lab </li></ul><ul><ul><li>New graduate student RA, post-doc, MEng slot open </li></ul></ul><ul><ul><li>Graduate school deadline Dec 15 th </li></ul></ul><ul><ul><li>New faculty positions </li></ul></ul><ul><li>Keywords </li></ul><ul><ul><li>Computational imaging, Signal processing, Applied optics, Computer graphics and vision, Hardware electronics, Displays, Art, Online photo collections, Visual social computing </li></ul></ul><ul><li>Acknowledgements </li></ul><ul><ul><li>My former group at MERL, Amit Agrawal, Jack Tumblin, Shree Nayar </li></ul></ul><ul><ul><li>ETVC: Frank, Evelyne </li></ul></ul><ul><ul><li>Apologies for not covering related work </li></ul></ul>http://CameraCulture.info
    79. 84. Fernald, Science [Sept 2006] Shadow Refractive Reflective Tools for Visual Computing
    80. 86. Blind Camera Sascha Pohflepp, U of the Art, Berlin, 2006
    81. 87. Cameras of Tomorrow
    82. 88. Cameras of Tomorrow <ul><li>Coded Exposure </li></ul><ul><ul><li>Motion Deblurring [2006] </li></ul></ul><ul><li>Coded Aperture </li></ul><ul><ul><li>Focus Deblurring [2007] </li></ul></ul><ul><ul><li>Glare reduction [2008] </li></ul></ul><ul><li>Optical Heterodyning </li></ul><ul><ul><li>Light Field Capture [2007] </li></ul></ul><ul><li>Coded Illumination </li></ul><ul><ul><li>Motion Capture [2007] </li></ul></ul><ul><ul><li>Multi-flash: Shape Contours [2004] </li></ul></ul><ul><li>Coded Spectrum </li></ul><ul><ul><li>Agile Wavelength Profile [2008] </li></ul></ul><ul><li>Epsilon-> Coded ->Essence Photography </li></ul>http://raskar.info
    83. 89. <ul><li>We focus on creating tools to better capture and share visual information </li></ul><ul><li>The goal is to create an entirely new class of imaging platforms that have an understanding of the world that far exceeds human ability and produce meaningful abstractions that are well within human comprehensibility </li></ul>Ramesh Raskar http://raskar.info

    ×