Upcoming SlideShare
×

# study Coded Aperture

2,242 views

Published on

study

Published in: Technology
0 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

Views
Total views
2,242
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
41
0
Likes
0
Embeds 0
No embeds

No notes for slide

### study Coded Aperture

1. 1. study<br />Image and Deoth from a Conventional Camera with a Coded Apertrue<br />Anat Levin, Rob Fergus, Frédo Durand, William Freeman<br />MIT CSAIL<br />
2. 2. Single input image<br />real objects<br />Coded Aperture<br />output #1: Depth map<br />output #2: all-infocused image<br />
3. 3. Conventional aperture and depth of field<br />Big aperture<br />Object<br />Focal plane<br />Small aperture<br />
4. 4. Depth from defocus<br />Camera sensor<br />Lens<br />Point spread function<br />Focal plane<br />http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt<br />
5. 5. Depth from defocus<br />Camera sensor<br />Lens<br />Object<br />Point spread function<br />Focal plane<br />http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt<br />
6. 6. Depth from defocus<br />Camera sensor<br />Lens<br />Object<br />Point spread function<br />Focal plane<br />http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt<br />
7. 7. Defocus as local convolution<br />Calibrated blur kernels at depth K<br />Local observed <br />sub-window<br />Sharp <br />sub-window<br />Input defocused image<br />Depth k=1<br />Depth k=2<br />Depth k=3<br />
8. 8. Introduction<br />Estimation of depth – a branch of Computational Photography<br />Most challenges of y = fk * x<br /><ul><li>Hard to de-convolve even when kernel is known </li></ul>Input<br />Ringing with the traditional Richardson-Lucyalgorithm<br /><ul><li>Hard to identify correct scale:</li></ul>?<br />Larger scale <br />?<br />Correct scale <br />?<br />Smaller scale <br />
9. 9. Related work – depth estimation<br />Active methods – additional illumination sources<br /><ul><li>Structured light methods </li></ul>Nayar et al. ICCV 95<br />Zhang and Nayar, SIGGRAPH 06<br />Projection Defocus Analysis for Capture and Image Display, Zhang and Nayar, 06<br />
10. 10. Related work – depth estimation<br />Passive methods – changes of focus <br /><ul><li>Depth from defocus (DFD) </li></ul>Pentland, IEEE 87<br />Chaudhuri, Favaro et al. , 99<br /><ul><li>Blind Deconvolution – image prior, maximum likelyhood</li></ul>Kundur and Hatzinakos , IEEE 96<br /> Levin, NIPS 06<br /><ul><li>Coded apertures for light gathering</li></ul>Fenimore and Cannon, Optics 78<br />
11. 11. Related work – depth estimation<br /><ul><li>Passive methods – changes of viewpoints
12. 12. Plenoptic /light field camera</li></ul>Adelson and Wang, IEEE 92<br /> Ng et al., 05<br /><ul><li>Wavefront coding</li></ul>Cathey & Dowski, Optics 94, 95<br />1.Rays don&apos;t converge anymore<br />2.Image blur is the same for all depth<br />3.Blur spectrum does not have too many zeros<br />CompPhoto06/html/lecturenotes/25_LightField_6.pdf<br />
13. 13. Overview<br />Try deconvolving local input windows with different scaled filters:<br />?<br />Larger scale <br />?<br />Correct scale <br />?<br />Smaller scale <br />Somehow: select best scale<br />
14. 14. Challenges & contributions<br />Hard to de-convolve even when kernel is known<br /> IDEA 1: Natural images prior<br />Hard to identify correct scale<br /> IDEA 2: Coded Aperture<br />
15. 15. Deconvolution is ill posed<br />Solution 1:<br />=<br />?<br />Solution 2:<br />=<br />?<br />
16. 16. IDEA 1: Natural images prior<br />What makes images special?<br />Natural<br />Unnatural<br />Image<br />gradient<br />Natural images have sparse gradients<br />put a penalty on gradients<br />
17. 17. Deconvolution with prior<br />Convolution error<br />Derivatives prior<br />2<br />?<br />Low <br />Equal convolution error<br />2<br />?<br />High <br />
19. 19. Statistical Model of Images<br />“Deconvolution using natural image priors”, Levin et. al., ETAI 07<br />Spatial domain<br />Frequency domain<br />
20. 20. Maximum a-posteriori P(x|y)<br />likelyhood<br />Image prior <br />(gradient here) <br />Gradient operator<br />For Gaussian priors<br />For sparse priors<br />
21. 21. Minimize deconvolution error<br />
22. 22. Deconvolution using a Gaussian prior<br />Note: solved in the frequency domain in a few seconds for MB size file<br />
23. 23. Deconvolution using a sparse prior<br />Using an iterative reweighted least squares process (IRLS) [Meer 2004; Levin and Weiss to appear]<br />Cannot solve in frequency domain<br />Note: solved in the frequency domain around 1 hour on 2.4Ghz CPR for 2MB file<br />
24. 24. Iterative reweighted least squares process (IRLS)<br />
25. 25. Recall: Overview<br />Try deconvolving local input windows with different scaled filters:<br />?<br />Larger scale <br />?<br />Correct scale <br />?<br />Smaller scale <br />Somehow: select best scale<br />Challenge: smaller scale not so different than correct<br />
26. 26. IDEA 2: Coded Aperture<br />Mask (code) in aperture plane<br />Make defocus patterns different from natural images and easier to discriminate<br />Conventional aperture<br />Our coded aperture<br />
27. 27. Lens with coded aperture<br />Image of a defocused point light source<br />Aperture pattern<br />Camera sensor<br />Lens with coded aperture<br />Object<br />Point spread function<br />Focal plane<br />
28. 28. Why coded ?<br />Coded aperture- reduce uncertainty in scale identification<br />Conventional<br />Coded<br />Larger scale<br />Correct scale<br />Smaller scale<br />
29. 29. Why coded ?<br />Coded aperture- reduce uncertainty in scale identification<br />Conventional<br />Coded<br />Larger scale<br />Correct scale<br />Smaller scale<br />
30. 30. Fourier transforms of 1D slide through the blur pattern <br />
31. 31. Coded aperture: Scale estimation and division by zero<br />spectrum<br />spectrum<br />spectrum<br />spectrum<br />spectrum<br />Frequency<br />Frequency<br />Frequency<br />Frequency<br />Frequency<br />Estimated image<br />?<br />Observed image <br />=<br />Filter, correct scale<br />Division by zero<br />Estimated image<br />?<br /> spatial ringing<br />=<br />Filter, wrong scale<br />
32. 32. Division by zero with a conventional aperture ?<br />spectrum<br />spectrum<br />spectrum<br />spectrum<br />spectrum<br />Frequency<br />Frequency<br />Frequency<br />Frequency<br />Frequency<br />Estimated image<br />?<br />No zero at ω !<br />Observed image <br />=<br />Filter, correct scale<br />No zero at ω !<br />Tiny value at ω<br />no spatial ringing<br />Estimated image<br />?<br />=<br />Filter, wrong scale<br />ω is zero !<br />
33. 33. Filter Selection Criterion<br />The filter f has good depth discrimination - blurry image distributions Pk1(y) and Pk2(y) at depths k1 and k2 should not be similar<br />KL-divergence scores<br />
34. 34. Filter Design<br />Practical constrains<br />Binary filter to construct accurately<br />Cut the filter from a single piece<br />Avoid excessive radial distortion<br />Avoid using the full aperture<br />Diffraction impose a min size on the holes in the file<br />Spec.<br />13x13 patterns with 1 mm holes<br />Each pattern, 8 different scales <br />Varying between 5~15 pixels in width <br />
35. 35. Filter Design<br />Conventional <br />Conventional <br />
36. 36. Blur scale identification<br />Not robust at high-frequency noise<br />Un-normalized energy term<br />λk learn to minimize the scale misclassification error on a set of traning images<br />Ek is approximate by the reconstruction error by ML solution<br />x* is the deblurred image<br />
37. 37. Regularizing depth estimation<br />
38. 38. Results<br />
39. 39. Applications<br />Digital refocusing from a single image<br />e.g. Synthesis an all-focus image<br />e.g. Post-exposure<br />
40. 40. Conclusion<br />Pros.<br /> All-infocus image and depth at a single shot<br /> No loss of image resolution (compared with Plenoptic camera) <br /> Simple modification<br />Coded aperture<br />Conventional aperture <br />Cons.<br /> 50 % light is blocked<br /> Depth is coarse<br />May need manual correction <br />