study Dappled Photography

1,386 views

Published on

study of "Dappled Photography: Mask Enhanced Cameras for
Heterodyned Light Fields and Coded Aperture Refocusing"

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,386
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
25
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Dapple 斑點Maryland, 美國馬里蘭州 ?Cambridge, 劍橋
  • 先看看這張raw sensor image. 他們設計的相機造影結果如上. 不再是單純的 2D 影像, 而是 4D 影像.簡單地說, 每一點, 又”分別”儲存了不同 ray 的結果.
  • study Dappled Photography

    1. 1. Dappled Photography: Mask Enhanced Cameras forHeterodyned Light Fields and Coded Aperture Refocusing<br />study<br />Ashok Veeraraghavan, RameshRaskar, AmitAgrawal<br />Mitsubishi Electric Research Labs (MERL), Cambridge, MA<br />Ankit Mohan, Jack Tumblin<br />Northwestern University, Evanston, IL<br />
    2. 2. Abstract<br />A theoretical framework for modulating 4D light fields using a mask betweenlens and sensors<br /><ul><li>Coded Aperture camera
    3. 3. Heterodynes Light Field camera</li></li></ul><li>Abstract<br />A theoretical framework for modulating 4D light fields using a mask betweenlens and sensors<br /><ul><li>Heterodynes Light Field camera</li></ul>Add a high-frequencymask between Lens and Sensors<br />4D light field - sense different rays from lens (u,v) in a sensor position (x,y)<br />Re-arrange light field<br /><ul><li>Low resolution with different focus settings
    4. 4. Full resolution in-focus</li></li></ul><li>Abstract<br />A theoretical framework for modulating 4D light fields using a mask betweenlens and sensors<br /><ul><li>Coded Aperture camera </li></ul>Replace the Aperture with a coded mask<br />A broadband mask enhance refocus at full resolution forLambertian scene<br /><ul><li>Heterodynes Light Field camera</li></ul>Add a high-frequencymask between Lens and Sensors<br />4D light field - sense different rays from lens (u,v) in a sensor position (x,y)<br />Re-arrange light field<br /><ul><li>Full resolution in-focus
    5. 5. Low resolution with different focus settings</li></ul>4D light field L(u,v,x,y)<br />
    6. 6. Abstract<br />A theoretical framework for modulating 4D light fields using a mask betweenlens and sensors<br /><ul><li>Heterodynes Light Field camera</li></ul>Add a high-frequencymask between Lens and Sensors<br />4D light field - sense different rays from lens (u,v) in a sensor position (x,y)<br />Rearrange light field<br /><ul><li>Low resolution with different focus settings
    7. 7. Full resolution in-focus</li></li></ul><li>Abstract<br />A theoretical framework for modulating 4D light fields using a mask betweenlens and sensors<br /><ul><li>Coded Aperture camera </li></ul>Replace the Aperture with a coded mask<br />A broadband mask enhance refocus at full resolution forLambertian scene<br />Refocusing partial<br /><ul><li>Heterodynes Light Field camera</li></ul>Add a high-frequencymask between Lens and Sensors<br />4D light field - sense different rays from lens (u,v) in a sensor position (x,y)<br />Re-arrange light field<br /><ul><li>Low resolution with different focus settings
    8. 8. Full resolution in-focus</li></li></ul><li>Heterodynes Light Field camera<br />
    9. 9. Heterodynes Light Field camera<br />Based on modulation theorem in 4D frequency domain – mask carries rays<br />= rays x mask<br />F<br />
    10. 10. Modulation Theorem<br />[Oppenheim et al. 99]<br />http://en.wikipedia.org/wiki/Amplitude_modulation<br />
    11. 11. Heterodynes Light Field camera<br />Based on modulation theorem in 4D frequency domain – mask carries rays<br />= rays x mask<br />F<br />A good mask carriers the rays ! <br />A poor mask blends the rays !<br />good mask !<br />α depends on (d,v)<br />rays<br />
    12. 12. Heterodynes Light Field camera<br />Based on modulation theorem in 4D frequency domain – mask carries rays<br />= rays x mask<br />F<br />
    13. 13. Heterodynes Light Field camera<br />Based on modulation theorem in 4D frequency domain – mask carries rays<br />recover the light field by rearranging the tiles of 2D Fourier transfer into 4D plane to get the full resolution image information for the in-focus parts of the scene<br />= rays x mask<br />F<br />Rearrange<br />F-1<br />
    14. 14. Heterodynes Light Field camera<br />Based on modulation theorem in 4D frequency domain – mask carries rays<br />recover the light field by rearranging the tiles of 2D Fourier transfer into 4D plane to get the full resolution image information for the in-focus parts of the scene<br />A raw sensor holds a modulated 4D light filed<br />= rays x mask<br />Raw sensor <br />(modulate 4D light field data)<br />In-focus at full resolution <br />(demodulated)<br />
    15. 15. Optical Heterodyning<br />Baseband Audio Signal<br />Software Demodulation<br />Main Lens<br />Object<br />Mask<br />Sensor<br />RecoveredLight Field<br />Receiver: Demodulation<br />High Freq Carrier 100.1 MHz<br />Incoming Signal<br />ReferenceCarrier<br />99 MHz<br />Incident Modulated Signal<br />Photographic Signal(Light Field)<br />Carrier <br />ReferenceCarrier<br />
    16. 16. Coded Aperture camera <br />Base on Convolution<br />Aperture as a Modulator<br />sinc function depends on θ<br />Pinhole camera has a very very broadband modulator<br />Design broadband mask<br />= rays x mask<br />
    17. 17. Outline<br />Introduction<br />Related Work<br />Theory & Framework<br />Heterodyne Light Field Camera<br />Encoded Blur Camera<br />Implements & Analysis<br />Contributions & Future Work<br />
    18. 18. Introduction<br />
    19. 19. Light Field<br />
    20. 20. Sensed image (in-focus)<br />θ<br />x<br />red : the in-focus line<br />yellow : sample<br />object<br />x<br />θ<br />Imaginary film<br />object<br />Sensor<br />Lens<br />http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt<br />
    21. 21. Sensed image (in-focus)<br />θ<br />x<br />x<br />θ<br />Imaginary film<br />Sensor<br />Lens<br />http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt<br />
    22. 22. Sensed image (in-focus)<br />θ<br />x<br />red : the in-focus line<br />yellow : sample<br />x<br />u<br />Imaginary film<br />Sensor<br />Lens<br />http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt<br />
    23. 23. Sensed image (out of focus, far)<br />θ<br />x<br />red : the in-focus line<br />yellow : sample<br />x<br />θ<br />Imaginary film<br />Sensor<br />Lens<br />http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt<br />
    24. 24. Sensed image (out of focus, far)<br />θ<br />x<br />red : the in-focus line<br />yellow : sample<br />x<br />θ<br />Imaginary film<br />Sensor<br />Lens<br />http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt<br />
    25. 25. Sensed image (out of focus, near)<br />red : the in-focus line<br />yellow : sample<br />θ<br />x<br />x<br />θ<br />Imaginary film<br />Sensor<br />Lens<br />http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt<br />
    26. 26. Sensed image (out of focus, near)<br />θ<br />x<br />red : the in-focus line<br />yellow : sample<br />x<br />θ<br />Imaginary film<br />Sensor<br />Lens<br />http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt<br />
    27. 27. Light Field Acquisition<br />Integral Photography<br /><ul><li>InteralPhotograpy [Lippmann 1908]
    28. 28. Integral camera [Okano et al. 99; Martnez-Corral et al. 04; Javidi and Okano 02]</li></ul>Light field Camera<br />Virtual viewpoint<br />[Levoy and Hanrahan 96]<br />[Gertler et al 96]<br />Virtual aperture<br />[Levoy and Hanrahan 96]<br />[Isaksen et al. 00]<br />Synthetic appearture photography (similar virtual aperture)<br />[Levoy et al. 04]<br />[Vaish et al. 04]<br />
    29. 29. Light field Camera<br />Plenoptic camera<br />Light field rendering <br />Dapped Photography <br />[Levoy and HanrahanSiggraph 96]<br />[Gortler et al 96, 06]<br /><ul><li>[Adelson et al, IEEE95]
    30. 30. [Levoy and HanrahanSiggraph 96]
    31. 31. [Gortler et al 96]
    32. 32. Hand-held light field camera [R Ng et al 05]
    33. 33. Fourier slice photography [R Ng, SIGGRAPH05]
    34. 34. The mask weights the rays </li></li></ul><li>Related work<br />Coded Imaging<br />Coded aperture imaging <br />Overcome the limit of pinhole camera [Skinner 98]<br />Coded Exposure Camera <br />[Raskar et al. 06]<br />Wavefront Coding <br />[Dowski and Cathey 95;Dowski and Johnson 99; van der Gracht et al. 96]<br />Traditional lens based [Farid and Simoncelli 98]<br />Both wavefront and coded aperture [Jahnson et al. 00]<br />Deblurring & deconvolution<br />Include extended DOF images by refocusing a light field at multiple depth and applying the digital photomontage tech. [Agarwala et al. 04]<br />Fusion of multiple blurred images [ Jaeberli 94]<br />
    35. 35. Theory & Framework<br />
    36. 36. For different focus settings, the obtained images correspond to slices at different angles, “Fourier Slice Photography ” [Ng, R. 05]<br />
    37. 37. Assumption:simulate the aperture as mask placed at lens<br />Open Aperture<br />
    38. 38. Open Aperture<br />Assumption:a planar Lambertianobject at the focus plane<br />Because no angular variations in the irradiance of rays from a Lambertian object, the content of light field is restricted to be along the fx axis<br />The sensed image is a slice of the modulated light field<br />
    39. 39. Open Aperture<br />In-focus sensor<br />The in-focus image corresponds to a slice of LA(fx, fθ) along fx(fθ=0)<br />No information lost<br />Out of focus sensor<br />The sensor image is a slanted slice<br />The slant angle depends on the degree of mis-focus<br />
    40. 40. Heterodyne Light Field Camera<br />
    41. 41. Mask as Modulator<br />
    42. 42. Mask as Modulator<br />d = v (at aperture stop, θ plane)<br />Mask affects the all rays at an angle θ in a similar way !<br />m(x, θ) = c (y = θ)<br />α = 900<br />d = 0 (at sensor, conjugate plane)<br />Mask attenuates all rays for the same x equally !<br />m(x, θ) = c (y = x)<br />α = 00<br />
    43. 43. Mask as Modulator<br />Optimal Mask Position<br />In practice, since the spatial resolution is much larger than the angular resolution, is very small, and therefore the mask needs to be placed close to the sensor<br />Optimal Mask Pattern<br /><ul><li>Harmonic sine wave
    44. 44. Boost</li></li></ul><li>Rearrange<br />
    45. 45. Notes<br />4D light field<br />Aliasing<br />When band-limit assumption is not valid in the spatial dimension, the energy in the higher spatial frequencies of the light field masquerade as energy in the lower angular dimension.<br />Post-filter the recovered light field using a Kaiser-Bessel filter with a filer width of 1.5 [Ng 05]<br />
    46. 46. Encoded Blur Camera<br />
    47. 47. Mask as modulator<br />Assumption:layered Lambertian scene<br />∵<br />Because no angular variations in the irradiance of rays from a Lambertian scene, the content of light field is restricted to be along the fx axis<br />
    48. 48. Optimal Mask for Encoding Defocus Blur<br />Blurred image is linear convolution (circularly convolution with zero padded)<br />Defocus by PSF (point spread function)<br />Coded aperture remove SNR only special cases<br />+ Star , -Natural photography<br /><ul><li>Optimal mask – continuous valued code by gradient decent optimization (Matlab, fmincon)
    49. 49. 7x7 Binary mask as initial guess
    50. 50. 10 hours of search</li></li></ul><li>Implementation & Analysis<br />
    51. 51. Heterodyne Light Field Camera<br />210 mm f/5.6<br />Nikkor-W Lens<br />Mask<br />CanoScanLiDE 70<br />scanner sensor<br />80 dots/mm<br />
    52. 52. Raw sensor image<br />
    53. 53. Scene parts which are in-focus can be recovered at full resolution<br />
    54. 54. Far Focused<br />Near Focused<br />
    55. 55. In-focus – full resolution<br />Low resolution refocused image<br />Out of focus<br />
    56. 56.
    57. 57. Analysis<br /><ul><li>Scanner sensor leading to pattern noise (horizontal /vertical lines)</li></ul>+ Easy to cover over in a conventional digital camera with a finer mask placed inside in the future<br />Computation<br />+ Computation burden is low because of computing light field and refocusing is done in Fourier domain<br />-Calibration of in-plane rotation and shift of the mask with respect to sensor <br />
    58. 58. Failure Cases<br />If Assumption of a band-limited light field is invalid, the aliasing artifacts in recovered light field<br />2D cosine mask needs to be moved away from the sensor because it results in diffraction<br />
    59. 59. Encoded Blur Camera<br />100 mm f/2.8 USM Macro Lens<br />Mask<br />Sensor<br />Canon Rebel XT SLR camera<br />
    60. 60. ISO-12233 Chart<br />
    61. 61. Modulation Transfer Function(MTF) of ISO-12233<br />MTF: low<br />MTF: high<br />
    62. 62. Full resolution digital refocusing using encoded blur camera<br />Captured photo<br />Refocused photo<br />
    63. 63. In this case, we can recover the sharp image if the blur size is larger than the occluder size<br /><ul><li>b is the vectorized blurred image
    64. 64. A is the block-Toeplitz matrix representing 2D blur
    65. 65. W is a weighting matrix which sets the weights corresponding to the occluded pixels in the blurred image to zero </li></ul>In-focus fence + blurred person<br />Deblurring without taking the occluders into account<br />Weighted deconvolution Eq.<br />Binary mask for the occluders<br />
    66. 66. Failure Cases<br />Scenes with large variation in depths and those with view dependencies can not be handle<br />Practice value<br />7x7 mask : blur size of about 20 pixels<br />Finer resolution mask can handle large defocus blur but lead to diffraction blur<br />
    67. 67. Contributions<br />= rays x mask<br />A theoretical framework of modulating 4D light fields camera working on frequency domain <br />A new class of 4D light filed camera holds full resolution modulated 4D light field<br />Don’t require additional optical elements such as lens arrays<br />Analyze defocus blur as a special case of the frequency domain re-mapping and demonstrate that a broadband mask at aperture can preserve high spatial frequencies in defocused image<br />
    68. 68. Future Work<br />Light Fields for Dynamic Scenes<br />Changing masks with time<br />Coding in time and space<br />General Ray Modulators<br />Tilted/curved/multiple masks<br />Wavelength dependent masks<br />Angular/Spatial Resolution Tradeoff<br />Applications<br />Estimating lens aberration<br />Microscopy<br />Light Field Applications<br />
    69. 69. End<br />Source<br />

    ×