Raskar Coded Opto Charlotte

1,484 views
1,294 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,484
On SlideShare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
35
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • “Photography is the art of exclusion” Rockwell
  • Car result
  • Low signal to noise ratio
  • How is the blurred image formed? It’s a convolution with box filter
  • Coded exposure makes the filter broadband
  • Comparisons
  • License plate example: Blur = 60 pixels
    Can you guess what the car make is ? How many think it is the Audi ? Actually it is a Folksvagon.
  • Cannot handle edges in blur
  • 2D result
  • Reversibly encode all the information in this otherwise blurred photo
  • The glint out of focus shows the unusual pattern.
  • Difficult to argue that the worm is performing high quality deconvolution to form an image. But in our group we are setting up experiments by creating active lighting probe to understand how worms perform visual analysis.
  • Talk about limitations: Colocated artifacts, color coherency, ref can’t be obtain by subtraction
  • Stereo-pair is a simple example of coded photography.
    Many decomposition problems, direct/global, diffuse/specular,
  • Inference and perception are important. Intent and goal of the photo is important.
    The same way camera put photorealistic art out of business, maybe this new artform will put the traditional camera out of business. Because we wont really care about a photo, merely a recording of light but a form that captures meaningful subset of the visual experience. Multiperspective photos. Photosynth is an example.
  • Maybe all the consumer photographer wants is a black box with big red button. No optics, sensors or flash.
    If I am standing the middle of times square and I need to take a photo. Do I really need a fancy camera?
  • The camera can trawl on flickr and retrieve a photo that is roughly taken at the same position, at the same time of day. Maybe all the consumer wants is a blind camera.
  • Raskar Coded Opto Charlotte

    1. 1. Mitsubishi Electric Research Laboratories Raskar 2007 Media Lab, MIT Cambridge, MA Less is More:Less is More: Coded Computational PhotographyCoded Computational Photography Ramesh Raskar P r o j e c t o r T a g s P o s = 0 P o s = 2 5 5
    2. 2. MIT Media Lab Camera Culture Ramesh Raskar Camera Culture
    3. 3. Motion Blurred Photo
    4. 4. Short Exposure Traditional MURA Deblurred Result Captured Single Photo Shutter Banding Artifacts and some spatial frequencies are lost Dark and noisy
    5. 5. Blurring == Convolution Traditional Camera: Shutter is OPEN: Box Filter PSF == Sinc Function ω Sharp Photo Blurred Photo Fourier Transform
    6. 6. Flutter Shutter: Shutter is OPEN and CLOSED Preserves High Spatial Frequencies Sharp Photo Blurred Photo PSF == Broadband Function Fourier Transform
    7. 7. Flutter Shutter CameraFlutter Shutter Camera Raskar, Agrawal, Tumblin [Siggraph2006] LCD opacity switched in coded sequence
    8. 8. Traditional Coded Exposure Image of Static Object Deblurred Image Deblurred Image
    9. 9. Coded Exposure Temporal 1-D broadband code: Motion Deblurring Coded Aperture Spatial 2-D broadband mask: Focus Deblurring
    10. 10. Coded Aperture CameraCoded Aperture Camera The aperture of a 100 mm lens is modified Rest of the camera is unmodified Insert a coded mask with chosen binary pattern
    11. 11. In Focus Photo LED
    12. 12. Out of Focus Photo: Open Aperture
    13. 13. Out of Focus Photo: Coded Aperture
    14. 14. Captured Blurred Photo
    15. 15. Refocused on Person
    16. 16. Less is MoreLess is More Blocking Light == More InformationBlocking Light == More Information Coding in TimeCoding in Time Coding in SpaceCoding in Space
    17. 17. Less is More ..Less is More .. • Coded Exposure – Motion Deblurring [2006] • Coded Aperture – Focus Deblurring [2007] • Optical Heterodyning – Light Field Capture [2007] • Coded Spectrum – Agile Wavelength Profile [2008] • Coded Illumination – Motion Capture [2007] – Multi-flash: Shape Contours [2004] P r o je c t o r T a g s P o s = 0 P o s = 2 5 5
    18. 18. Computational PhotographyComputational Photography 1. Epsilon Photography – Low-level vision: Pixels – Multi-photos by perturbing camera parameters – HDR, panorama, … – ‘Ultimate camera’ 1. Coded Photography – Single/few snapshot – Reversible encoding of data – Additional sensors/optics/illum – ‘Scene analysis’ : (Consumer software?) 1. Essence Photography – Beyond single view/illum – Not mimic human eye – ‘New art form’
    19. 19. Computational PhotographyComputational Photography 1. Epsilon Photography – Low-level Vision: Pixels – Multiphotos by perturbing camera parameters – HDR, panorama – ‘Ultimate camera’ 1. Coded Photography – Mid-Level Cues: • Regions, Edges, Motion, Direct/global – Single/few snapshot • Reversible encoding of data – Additional sensors/optics/illum – ‘Scene analysis’ 1. Essence Photography – Not mimic human eye – Beyond single view/illum – ‘New artform’
    20. 20. Computational PhotographyComputational Photography 1. Epsilon Photography – Multiphotos by varying camera parameters – HDR, panorama – ‘Ultimate camera’: (Photo-editor) 1. Coded Photography – Single/few snapshot – Reversible encoding of data – Additional sensors/optics/illum – ‘Scene analysis’ : (Next software?) 1. Essence Photography – High-level understanding • Not mimic human eye • Beyond single view/illum – ‘New artform’
    21. 21. Mask? Sensor Mask SensorMask? Sensor Mask Sensor Mask? Sensor 4D Light Field from 2D Photo: Heterodyne Light Field Camera Full Resolution Digital Refocusing: Coded Aperture Camera
    22. 22. Light Field Inside a CameraLight Field Inside a Camera
    23. 23. Lenslet-based Light Field cameraLenslet-based Light Field camera [Adelson and Wang, 1992, Ng et al. 2005 ] Light Field Inside a CameraLight Field Inside a Camera
    24. 24. Stanford Plenoptic CameraStanford Plenoptic Camera [Ng et al 2005][Ng et al 2005] 4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens Contax medium format camera Kodak 16-megapixel sensor Adaptive Optics microlens array 125μ square-sided microlenses
    25. 25. Digital RefocusingDigital Refocusing [Ng et al 2005][Ng et al 2005] Can we achieve this with aCan we achieve this with a MaskMask alone?alone?
    26. 26. Mask based Light Field Camera Mask Sensor [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]
    27. 27. How to Capture 4D Light Field with 2D Sensor ? What should be the pattern of the mask ?
    28. 28. Radio Frequency HeterodyningRadio Frequency Heterodyning Baseband Audio Signal Receiver: DemodulationHigh Freq Carrier 100 MHz Reference Carrier Incoming Signal 99 MHz
    29. 29. Optical HeterodyningOptical Heterodyning Photographic Signal (Light Field) Carrier Incident Modulated Signal Reference Carrier Main LensObject Mask Sensor Recovered Light Field Software Demodulation Baseband Audio Signal Receiver: DemodulationHigh Freq Carrier 100 MHz Reference Carrier Incoming Signal 99 MHz
    30. 30. Captured 2D Photo Encoding due to Mask
    31. 31. 2D FFT Traditional Camera Photo Heterodyne Camera Photo Magnitude of 2D FFT 2D FFT Magnitude of 2D FFT
    32. 32. Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800 2D FFT Rearrange 2D tiles into 4D planes 200*200*9*94D IFFT 4D Light Field 9*9=81 spectral copies 200*200*9*9
    33. 33. A Theory of Mask-Enhanced CameraA Theory of Mask-Enhanced Camera Main LensObject Mask Sensor •Mask == Light Field Modulator •Intensity of ray gets multiplied by Mask •Convolution in Frequency domain
    34. 34. fθ fx fθ0 fx0 Band-limited Light Field Sensor Slice – Fourier Slice Theorem Photo = Slice of Light Field in Fourier Domain [Ren Ng, SIGGRAPH 2005]
    35. 35. How to Capture 2D Light Field with 1D Sensor ? fθ fx fθ0 fx0 Band-limited Light Field Sensor Slice Fourier Light Field Space
    36. 36. Extra sensor bandwidth cannot capture extra dimension of the light field fθ fx fθ0 fx0 Sensor Slice Extra sensor bandwidth
    37. 37. fθ fx ?????? ??? ???
    38. 38. Solution: Modulation Theorem Make spectral copies of 2D light field fθ fx fθ0 fx0 Modulation Function
    39. 39. fθ Modulated Light Field fx fθ0 fx0 Modulation Function Sensor Slice captures entire Light Field
    40. 40. Demodulation to recover Light Field fθ fx Reshape 1D Fourier Transform into 2D 1D Fourier Transform of Sensor Signal
    41. 41. fθ fx fθ0 fx0 Modulation Function == Sum of Impulses Physical Mask = Sum of Cosines
    42. 42. 1/f0 Mask Tile Cosine Mask Used
    43. 43. Where to place the Mask? Mask Sensor Mask Sensor Mask Modulation Function Mask Modulation Function fx fθ
    44. 44. Mask Sensor Where to place the Mask? Mask Modulation Functionfx fθ
    45. 45. Captured 2D Photo Encoding due to Cosine Mask
    46. 46. Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform 2D FFT Rearrange 2D tiles into 4D planes 200*200*9*94D IFFT 4D Light Field 9*9=81 spectral copies 200*200*9*9
    47. 47. Digital Refocusing Only cone in focus Captured Photo
    48. 48. Full resolution 2D image of Focused Scene Parts Captured 2D Photo Image of White Lambertian Plane divide
    49. 49. Coding and Modulation in Camera Using MasksCoding and Modulation in Camera Using Masks Mask? Sensor Mask Sensor Mask Sensor Coded Aperture for Full Resolution Digital Refocusing Heterodyne Light Field Camera
    50. 50. Agile Spectrum Imaging With Ankit Mohan, Jack Tumblin [Eurographics 2008]
    51. 51. C B A A’ B’ C’ Pinhole Lens L1 Prism or Diffraction Grating Lens L2 Sensor Rainbow Plane C’ ’ B’ ’ A’ ’ Scene Rainbow Plane inside Camera
    52. 52. A’ B’ C’ λ λ λ C’ ’ B’ ’ A’ ’ tS Lens L2 x I x t
    53. 53. A’ B’ C’ λ λ λ C’ ’ B’ ’ A’ ’ tR Lens L2 x I x t tS
    54. 54. Glare Reduction/Enhancement usingGlare Reduction/Enhancement using 4D Ray Sampling4D Ray Sampling Captured Glare Reduced Glare Enhanced
    55. 55. i j x Sensor u Glare = low frequency noise in 2D •But is high frequency noise in 4D •Remove via simple outlier rejection
    56. 56. i j Sensor i j x Pin-hole array mask u
    57. 57. Mask based Light Field Camera Mask Sensor
    58. 58. Glare Reduction/Enhancement usingGlare Reduction/Enhancement using 4D Ray Sampling4D Ray Sampling [Siggraph 2008][Siggraph 2008] Captured Glare Reduced Glare Enhanced
    59. 59. • 6D Display • Multi-flash Camera • Computational Photography – Online Siggraph Course Notes – Google: ‘Computational Photography’ Other ProjectsOther Projects
    60. 60. Coded Computational Photography • Coded Exposure – Motion Deblurring [2006] • Coded Aperture – Focus Deblurring [2007] – Glare reduction [2008] • Optical Heterodyning – Light Field Capture [2007] • Coded Illumination – Motion Capture [2007] – Multi-flash: Shape Contours [2004] • Coded Spectrum – Agile Wavelength Profile [2008] • Epsilon->Coded->Essence Photography http://raskar.info
    61. 61. Blind CameraBlind Camera Sascha Pohflepp, U of the Art, Berlin, 2006
    62. 62. Coded Computational Photography • Coded Exposure – Motion Deblurring [2006] • Coded Aperture – Focus Deblurring [2007] – Glare reduction [2008] • Optical Heterodyning – Light Field Capture [2007] • Coded Illumination – Motion Capture [2007] – Multi-flash: Shape Contours [2004] • Coded Spectrum – Agile Wavelength Profile [2008] • Epsilon->Coded->Essence Photography http://raskar.info

    ×