study

1. Francesc Moreno-NoguerComputer Vision Lab.EcolePolytechniqueFédérale de LausannePeter N. BelhumeurShree K. NayarColumbia University SIGGRAPH 2007 study Active RefocusingofImages and Videos

2. Abstract Use an activeilluminationmethod for depth estimation from a singleimage Near Far Alternate Lighting Refocused (Near) Acquired Image Computed Depth Refocused (Far)

3. Outlines Introduction Related Work Overview Projection Dot Defocus Analysis Dot Removal & Depth Estimation Realistic Refocusing Result Limits and Conclusions

4. Introduction of Refocusing Challenges of Active Refocusing Dynamic scenes  Depth Estimation be done in a single frame Active illumination Full resolution depth map Projection Dot removal Partial Occlusions captured blur kernels at depth k In-focus

5. Related work

6. Relative Work: Depth Estimation Passive Methods Active Illumination Methods Shape from shading Cannot handle depth discontinuities Coded Aperture [Levin et al. SIGGRAPH 07] Cam. H.W. modify Require Light Source Pattern Structured Light [Salvi et al. Pattern Recognition ,04] No pattern removal Projector Temporal Defocus [Zhang & Nayar SIGGRAPH06]

7. Relative Work: Digital Refocusing Refocusing Given Depth Light Field Photography Synthesis Images:Ray Tracing [Cook SIGGRAPH84] Require complete 3D model Real Images: Convolution[Photoshop; IrisFilter] Partial Occlusions Problem Light Field Camera[Ng SIGGRAPH05] Cam. H.W. modify Resolution losses Dappled Photography [Veeraraghavan SIGGRAPH07] Cam. H.W. modify Layer

8. Depth Estimation Depth Map Completion using Segmentation Dots Removal Dots Removed Acquired Image Dense Depth Matting Dots Depth Estimationby Calibration Sparse Depth Map Color Segmentation Merged Segmentation

9. Realistic Refocusing Dots Removed Depth Map Focal plane, Apertures, Window size of dots

10. Projection Dot Defocus Analysis

11. System Design Projector Camera & Projector Coaxial  have same Optical Axis

12. Blur Circle Diameter, D fc v w r D u uf with dot size w (in the projector plane)

13. Blur Circle Radiance, I fc v w r D u uf with dot size w (in the projector plane) based on Image Irradiance Equation derived in [Horn 86]

14. Camera images of dot of 3*3 pixels projected onto different depths

15. Camera images of dot of 3*3 pixels projected onto different depths

16. Dot removal and depth estimation

17. Sparse Depth Map … Depth 1 Depth 2 Calibration Patches Estimated = X

18. Sparse Depth Map … Depth 1 Depth 2 Calibration Patches Estimated

19. Sparse Depth Map … Depth 1 Depth 2 Calibration Patches Estimated

20. Depth Estimation - ux Non-textured Surface Textured Surfaces (texture by itself introduces brightness variation) based on Unsupervised Learning Alg. [Figueiredo and Jain IEEE02]

21. Depth map completion using segmentation

22. Depth Map Completion Over-Segmentation Sparse Depth Map Iterative Merging Mean-Shift [Comaniciu & Meer 02]

23. Depth Map Completion – Iterative Merging Loop: Apply Greedy alg. to group segments Merge the two most similar neighboring segments Re-computes the features of the new merged segment Iterative Merging

24. Similarity between Segments Color C Depth D Texture T Sim(i,j)=λC∙dist(Ci,Cj)+λD∙dist(Di,Dj)+λT∙dist(Ti,Tj)

25. Depth Map Completion – Refine the Depth Disc. Matting Algorithm [Wang & Cohen 05] Noisy Depth Map

26. Realistic refocusing

27. Challenge of Refocusing Partial occlusions Different parts of the lens may see different views at an object boundary  Create missing region by detecting discontinuities in depth map and extending the occluded surface using texture synthesis Foreground/background transitions Pixels at depth discontinuities may receive contributions from the fr. and bg. Blend fr./bg. images within the boundary region

28. Realistic Refocusing produces better results than existing approaches Realistic Refocusing Canon + wide aperture Photoshop - blur IrisFilter Original

29. Partial Occlusions

30. Refocusing with Alpha Maps Foreground (F) Boundary (C) Background (B) R R R R CЄF CЄB CЄF CЄB + = R * *

31. Result

32. Limitations Due to Active Illumination Translucent objects exhibit subsurface scattering Blurred dots are too weak to detect Very dark Highly inclined surface (> 70°) Poor in outdoor with strong sunlight ex: the ball and the table are assigned diff. depths due to errors on segmentation errors

33. Limitations Due to sparse dots Sparsity of the depth estimation Errors in the initial segmentation of the image ex: incorrect depth due to segmentation err.

34. Conclusions Contribution Future Work An active illumination depth estimation with single Single Frame, Complete Depth Map, Texture/Textureless scenes Projected Light Patterns are Removed High resolution refocusing of images and videos Incorporate the method into digital cameras Use intra-red source for projecting the dot patter to make the depth estimation more robust in the case of highly textured scenes

35. end