study<br />Image and Deoth from a Conventional Camera with a Coded Apertrue<br />Anat Levin, Rob Fergus,    Frédo Durand, ...
Single input image<br />real objects<br />Coded Aperture<br />output #1: Depth map<br />output #2: all-infocused  image<br />
Conventional aperture and depth of field<br />Big aperture<br />Object<br />Focal plane<br />Small aperture<br />
Depth from defocus<br />Camera sensor<br />Lens<br />Point spread function<br />Focal plane<br />http://groups.csail.mit.e...
Depth from defocus<br />Camera sensor<br />Lens<br />Object<br />Point spread function<br />Focal plane<br />http://groups...
Depth from defocus<br />Camera sensor<br />Lens<br />Object<br />Point spread function<br />Focal plane<br />http://groups...
Defocus as local convolution<br />Calibrated  blur kernels at depth K<br />Local observed <br />sub-window<br />Sharp <br ...
Introduction<br />Estimation of depth – a branch of Computational Photography<br />Most challenges of  y = fk * x<br /><ul...
Related work – depth estimation<br />Active methods – additional illumination sources<br /><ul><li>Structured light method...
Related work – depth estimation<br />Passive methods – changes of focus <br /><ul><li>Depth from defocus (DFD) </li></ul>P...
Related work – depth estimation<br /><ul><li>Passive methods – changes of viewpoints
Plenoptic /light field camera</li></ul>Adelson and Wang, IEEE 92<br />	Ng et al., 05<br /><ul><li>Wavefront coding</li></u...
Overview<br />Try deconvolving local input windows with different scaled filters:<br />?<br />Larger scale <br />?<br />Co...
Challenges & contributions<br />Hard to de-convolve even when kernel is known<br />	IDEA 1: Natural images prior<br />Hard...
Deconvolution is ill posed<br />Solution 1:<br />=<br />?<br />Solution 2:<br />=<br />?<br />
IDEA 1: Natural images prior<br />What makes images special?<br />Natural<br />Unnatural<br />Image<br />gradient<br />Nat...
Deconvolution with prior<br />Convolution error<br />Derivatives prior<br />2<br />?<br />Low <br />Equal convolution erro...
Comparing deconvolution algorithms<br />Richardson-Lucy<br />Input<br />“spread” gradients<br />“localizes” gradients<br /...
Statistical Model of Images<br />“Deconvolution using natural image priors”, Levin et. al., ETAI 07<br />Spatial domain<br...
Maximum a-posteriori P(x|y)<br />likelyhood<br />Image prior <br />(gradient here) <br />Gradient operator<br />For Gaussi...
Minimize deconvolution error<br />
Deconvolution using a Gaussian prior<br />Note: solved in the frequency domain in a few seconds for MB size file<br />
Deconvolution using a sparse prior<br />Using an iterative reweighted least squares process (IRLS) [Meer 2004; Levin and W...
Iterative reweighted least squares process (IRLS)<br />
Recall: Overview<br />Try deconvolving local input windows with different scaled filters:<br />?<br />Larger scale <br />?...
IDEA 2: Coded Aperture<br />Mask (code) in aperture plane<br />Make defocus patterns different from natural images and eas...
Lens with coded aperture<br />Image of a defocused point light source<br />Aperture pattern<br />Camera sensor<br />Lens w...
Why coded ?<br />Coded aperture- reduce uncertainty in scale identification<br />Conventional<br />Coded<br />Larger scale...
Why coded ?<br />Coded aperture- reduce uncertainty in scale identification<br />Conventional<br />Coded<br />Larger scale...
Fourier transforms of 1D slide through the blur pattern <br />
Coded aperture: Scale estimation and division by zero<br />spectrum<br />spectrum<br />spectrum<br />spectrum<br />spectru...
Division by zero with a conventional aperture ?<br />spectrum<br />spectrum<br />spectrum<br />spectrum<br />spectrum<br /...
Filter Selection Criterion<br />The filter f has good depth discrimination - blurry image distributions Pk1(y) and Pk2(y) ...
Filter Design<br />Practical constrains<br />Binary filter to construct accurately<br />Cut the filter from a single piece...
Filter Design<br />Conventional <br />Conventional <br />
Blur scale identification<br />Not robust at high-frequency noise<br />Un-normalized energy term<br />λk  learn to minimiz...
Regularizing depth estimation<br />
Results<br />
Applications<br />Digital refocusing from a single image<br />e.g.  Synthesis an all-focus image<br />e.g.  Post-exposure<...
Upcoming SlideShare
Loading in …5
×

study Coded Aperture

2,242 views

Published on

study

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,242
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
41
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

study Coded Aperture

  1. 1. study<br />Image and Deoth from a Conventional Camera with a Coded Apertrue<br />Anat Levin, Rob Fergus, Frédo Durand, William Freeman<br />MIT CSAIL<br />
  2. 2. Single input image<br />real objects<br />Coded Aperture<br />output #1: Depth map<br />output #2: all-infocused image<br />
  3. 3. Conventional aperture and depth of field<br />Big aperture<br />Object<br />Focal plane<br />Small aperture<br />
  4. 4. Depth from defocus<br />Camera sensor<br />Lens<br />Point spread function<br />Focal plane<br />http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt<br />
  5. 5. Depth from defocus<br />Camera sensor<br />Lens<br />Object<br />Point spread function<br />Focal plane<br />http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt<br />
  6. 6. Depth from defocus<br />Camera sensor<br />Lens<br />Object<br />Point spread function<br />Focal plane<br />http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt<br />
  7. 7. Defocus as local convolution<br />Calibrated blur kernels at depth K<br />Local observed <br />sub-window<br />Sharp <br />sub-window<br />Input defocused image<br />Depth k=1<br />Depth k=2<br />Depth k=3<br />
  8. 8. Introduction<br />Estimation of depth – a branch of Computational Photography<br />Most challenges of y = fk * x<br /><ul><li>Hard to de-convolve even when kernel is known </li></ul>Input<br />Ringing with the traditional Richardson-Lucyalgorithm<br /><ul><li>Hard to identify correct scale:</li></ul>?<br />Larger scale <br />?<br />Correct scale <br />?<br />Smaller scale <br />
  9. 9. Related work – depth estimation<br />Active methods – additional illumination sources<br /><ul><li>Structured light methods </li></ul>Nayar et al. ICCV 95<br />Zhang and Nayar, SIGGRAPH 06<br />Projection Defocus Analysis for Capture and Image Display, Zhang and Nayar, 06<br />
  10. 10. Related work – depth estimation<br />Passive methods – changes of focus <br /><ul><li>Depth from defocus (DFD) </li></ul>Pentland, IEEE 87<br />Chaudhuri, Favaro et al. , 99<br /><ul><li>Blind Deconvolution – image prior, maximum likelyhood</li></ul>Kundur and Hatzinakos , IEEE 96<br /> Levin, NIPS 06<br /><ul><li>Coded apertures for light gathering</li></ul>Fenimore and Cannon, Optics 78<br />
  11. 11. Related work – depth estimation<br /><ul><li>Passive methods – changes of viewpoints
  12. 12. Plenoptic /light field camera</li></ul>Adelson and Wang, IEEE 92<br /> Ng et al., 05<br /><ul><li>Wavefront coding</li></ul>Cathey & Dowski, Optics 94, 95<br />1.Rays don&apos;t converge anymore<br />2.Image blur is the same for all depth<br />3.Blur spectrum does not have too many zeros<br />CompPhoto06/html/lecturenotes/25_LightField_6.pdf<br />
  13. 13. Overview<br />Try deconvolving local input windows with different scaled filters:<br />?<br />Larger scale <br />?<br />Correct scale <br />?<br />Smaller scale <br />Somehow: select best scale<br />
  14. 14. Challenges & contributions<br />Hard to de-convolve even when kernel is known<br /> IDEA 1: Natural images prior<br />Hard to identify correct scale<br /> IDEA 2: Coded Aperture<br />
  15. 15. Deconvolution is ill posed<br />Solution 1:<br />=<br />?<br />Solution 2:<br />=<br />?<br />
  16. 16. IDEA 1: Natural images prior<br />What makes images special?<br />Natural<br />Unnatural<br />Image<br />gradient<br />Natural images have sparse gradients<br />put a penalty on gradients<br />
  17. 17. Deconvolution with prior<br />Convolution error<br />Derivatives prior<br />2<br />?<br />Low <br />Equal convolution error<br />2<br />?<br />High <br />
  18. 18. Comparing deconvolution algorithms<br />Richardson-Lucy<br />Input<br />“spread” gradients<br />“localizes” gradients<br />Gaussian prior<br />Sparse prior<br />
  19. 19. Statistical Model of Images<br />“Deconvolution using natural image priors”, Levin et. al., ETAI 07<br />Spatial domain<br />Frequency domain<br />
  20. 20. Maximum a-posteriori P(x|y)<br />likelyhood<br />Image prior <br />(gradient here) <br />Gradient operator<br />For Gaussian priors<br />For sparse priors<br />
  21. 21. Minimize deconvolution error<br />
  22. 22. Deconvolution using a Gaussian prior<br />Note: solved in the frequency domain in a few seconds for MB size file<br />
  23. 23. Deconvolution using a sparse prior<br />Using an iterative reweighted least squares process (IRLS) [Meer 2004; Levin and Weiss to appear]<br />Cannot solve in frequency domain<br />Note: solved in the frequency domain around 1 hour on 2.4Ghz CPR for 2MB file<br />
  24. 24. Iterative reweighted least squares process (IRLS)<br />
  25. 25. Recall: Overview<br />Try deconvolving local input windows with different scaled filters:<br />?<br />Larger scale <br />?<br />Correct scale <br />?<br />Smaller scale <br />Somehow: select best scale<br />Challenge: smaller scale not so different than correct<br />
  26. 26. IDEA 2: Coded Aperture<br />Mask (code) in aperture plane<br />Make defocus patterns different from natural images and easier to discriminate<br />Conventional aperture<br />Our coded aperture<br />
  27. 27. Lens with coded aperture<br />Image of a defocused point light source<br />Aperture pattern<br />Camera sensor<br />Lens with coded aperture<br />Object<br />Point spread function<br />Focal plane<br />
  28. 28. Why coded ?<br />Coded aperture- reduce uncertainty in scale identification<br />Conventional<br />Coded<br />Larger scale<br />Correct scale<br />Smaller scale<br />
  29. 29. Why coded ?<br />Coded aperture- reduce uncertainty in scale identification<br />Conventional<br />Coded<br />Larger scale<br />Correct scale<br />Smaller scale<br />
  30. 30. Fourier transforms of 1D slide through the blur pattern <br />
  31. 31. Coded aperture: Scale estimation and division by zero<br />spectrum<br />spectrum<br />spectrum<br />spectrum<br />spectrum<br />Frequency<br />Frequency<br />Frequency<br />Frequency<br />Frequency<br />Estimated image<br />?<br />Observed image <br />=<br />Filter, correct scale<br />Division by zero<br />Estimated image<br />?<br /> spatial ringing<br />=<br />Filter, wrong scale<br />
  32. 32. Division by zero with a conventional aperture ?<br />spectrum<br />spectrum<br />spectrum<br />spectrum<br />spectrum<br />Frequency<br />Frequency<br />Frequency<br />Frequency<br />Frequency<br />Estimated image<br />?<br />No zero at ω !<br />Observed image <br />=<br />Filter, correct scale<br />No zero at ω !<br />Tiny value at ω<br />no spatial ringing<br />Estimated image<br />?<br />=<br />Filter, wrong scale<br />ω is zero !<br />
  33. 33. Filter Selection Criterion<br />The filter f has good depth discrimination - blurry image distributions Pk1(y) and Pk2(y) at depths k1 and k2 should not be similar<br />KL-divergence scores<br />
  34. 34. Filter Design<br />Practical constrains<br />Binary filter to construct accurately<br />Cut the filter from a single piece<br />Avoid excessive radial distortion<br />Avoid using the full aperture<br />Diffraction impose a min size on the holes in the file<br />Spec.<br />13x13 patterns with 1 mm holes<br />Each pattern, 8 different scales <br />Varying between 5~15 pixels in width <br />
  35. 35. Filter Design<br />Conventional <br />Conventional <br />
  36. 36. Blur scale identification<br />Not robust at high-frequency noise<br />Un-normalized energy term<br />λk learn to minimize the scale misclassification error on a set of traning images<br />Ek is approximate by the reconstruction error by ML solution<br />x* is the deblurred image<br />
  37. 37. Regularizing depth estimation<br />
  38. 38. Results<br />
  39. 39. Applications<br />Digital refocusing from a single image<br />e.g. Synthesis an all-focus image<br />e.g. Post-exposure<br />
  40. 40. Conclusion<br />Pros.<br /> All-infocus image and depth at a single shot<br /> No loss of image resolution (compared with Plenoptic camera) <br /> Simple modification<br />Coded aperture<br />Conventional aperture <br />Cons.<br /> 50 % light is blocked<br /> Depth is coarse<br />May need manual correction <br />

×