Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Multi Aperture Photography


Published on

photography presentation

Published in: Art & Photos, Business
  • Be the first to comment

Multi Aperture Photography

  1. 1. Multi-Aperture Photography Paul Green – MIT CSAIL Wenyang Sun – MERL Wojciech Matusik – MERL Frédo Durand – MIT CSAIL
  2. 2. Motivation Portrait Landscape Small Aperture Large Aperture Depth of Field Control Shallow Depth of Field Large Depth of Field
  3. 3. Depth and Defocus Blur plane of focus sensor lens defocus blur depends on distance from plane of focus subject rays from point in focus converge to single pixel circle of confusion
  4. 4. Defocus Blur & Aperture lens plane of focus defocus blur depends on aperture size aperture sensor subject circle of confusion
  5. 5. Goals <ul><li>Aperture size is a critical parameter for photographers </li></ul><ul><li>post-exposure depth of field control </li></ul><ul><li>extrapolate shallow depth of field beyond physical aperture </li></ul>
  6. 6. Outline <ul><li>Multi-Aperture Camera </li></ul><ul><ul><li>New camera design </li></ul></ul><ul><ul><li>Capture multiple aperture settings simultaneously </li></ul></ul><ul><li>Applications </li></ul><ul><ul><li>Depth of field control </li></ul></ul><ul><ul><li>Depth of field extrapolation </li></ul></ul><ul><ul><li>(Limited) refocusing </li></ul></ul>
  7. 7. Related Work <ul><li>Computational Cameras </li></ul><ul><ul><li>Plenoptic Cameras </li></ul></ul><ul><ul><ul><li>Adelson and Wang ‘92 </li></ul></ul></ul><ul><ul><ul><li>Ng et al ‘05 </li></ul></ul></ul><ul><ul><ul><li>Georgiev et al ‘06 </li></ul></ul></ul><ul><ul><li>Split-Aperture Camera </li></ul></ul><ul><ul><ul><li>Aggarwal and Ahuja ‘04 </li></ul></ul></ul><ul><ul><li>Optical Splitting Trees </li></ul></ul><ul><ul><ul><li>McGuire et al ‘07 </li></ul></ul></ul><ul><ul><li>Coded Aperture </li></ul></ul><ul><ul><ul><li>Levin et al ’07 </li></ul></ul></ul><ul><ul><ul><li>Veeraraghavan et al ’07 </li></ul></ul></ul><ul><ul><li>Wavefront Coding </li></ul></ul><ul><ul><ul><li>Dowski and Cathey ‘95 </li></ul></ul></ul><ul><li>Depth from Defocus </li></ul><ul><ul><li>Pentland ‘87 </li></ul></ul>McGuire et al ‘07 Adelson and Wang ‘92 Levin et al ’07 Veeraraghavan et al ’07 Georgiev et al‘06 Aggarwal and Ahuja ‘04
  8. 8. Plenoptic Cameras <ul><li>Capture 4D LightField </li></ul><ul><ul><li>2D Spatial (x,y) </li></ul></ul><ul><ul><li>2D Angular (u,v Aperture) </li></ul></ul><ul><li>Trade resolution for flexibility after capture </li></ul><ul><ul><li>Refocusing </li></ul></ul><ul><ul><li>Depth of field control </li></ul></ul><ul><ul><li>Improved Noise Characteristics </li></ul></ul>Lens Aperture u v Sensor (x,y) Lenslet Array Subject Lens (u,v)
  9. 9. 1D vs 2D Aperture Sampling u v Aperture 2D Grid Sampling
  10. 10. 1D vs. 2D Aperture Sampling 4 Samples Aperture 1D “Ring” Sampling 45 Samples u v Aperture 2D Grid Sampling
  11. 11. Optical Splitting Trees <ul><li>General framework for sampling imaging parameters </li></ul><ul><ul><li>Beamsplitters </li></ul></ul><ul><ul><li>Multiple cameras </li></ul></ul>Large Aperture Camera Small Aperture Camera McGuire et al ‘07 Beamsplitter Incoming light
  12. 12. Goals <ul><li>post-exposure depth of field control </li></ul><ul><li>extrapolate shallow depth of field </li></ul><ul><li>(limited) refocusing </li></ul><ul><li>1d sampling </li></ul><ul><li>no beamsplitters </li></ul><ul><li>single sensor </li></ul><ul><li>removable </li></ul>
  13. 13. Outline <ul><li>Multi-Aperture Camera </li></ul><ul><ul><li>New camera design </li></ul></ul><ul><ul><li>Capture multiple aperture settings simultaneously </li></ul></ul><ul><li>Applications </li></ul><ul><ul><li>Depth of field control </li></ul></ul><ul><ul><li>Depth of field extrapolation </li></ul></ul><ul><ul><li>Refocusing </li></ul></ul>
  14. 14. Optical Design Principles <ul><li>3D sampling </li></ul><ul><ul><li>2D spatial </li></ul></ul><ul><ul><li>1D aperture size </li></ul></ul><ul><ul><li>1 image for each “ring” </li></ul></ul>Aperture Sensor
  15. 15. <ul><li>Goal: Split aperture into 4 separate optical paths </li></ul><ul><ul><li>concentric tilted mirrors </li></ul></ul><ul><ul><li>at aperture plane </li></ul></ul>Aperture Splitting Tilted Mirrors
  16. 16. Aperture Splitting Incoming light Sensor Mirrors Focusing lenses Tilted Mirrors
  17. 17. Aperture Splitting X Ideally at aperture plane , but not physically possible! Solution: Relay Optics to create virtual aperture plane Photographic Lens Aperture Plane Relay system Aperture splitting optics New Aperture Plane
  18. 18. Optical Prototype Mirror Close-up main lens relay optics mirrors tilted mirrors lenses SLR Camera
  19. 19. Sample Data <ul><li>Raw data from our camera </li></ul>
  20. 20. <ul><li>Ideally would be rings </li></ul><ul><li>Gaps are from occlusion </li></ul>Point Spread Function Occlusion combined inner ring 1 ring 2 outer
  21. 21. Outline <ul><li>Multi-Aperture Camera </li></ul><ul><ul><li>New camera design </li></ul></ul><ul><ul><li>Capture multiple aperture settings simultaneously </li></ul></ul><ul><li>Applications </li></ul><ul><ul><li>Depth of field control </li></ul></ul><ul><ul><li>Depth of field extrapolation </li></ul></ul><ul><ul><li>Refocusing </li></ul></ul>
  22. 22. DOF Navigation
  23. 23. <ul><li>Approximate defocus blur as convolution </li></ul>DOF Extrapolation? ? Depends on depth and aperture size What is at each pixel in ? - Circular aperture blurring kernel
  24. 24. DOF Extrapolation Roadmap capture estimate blur fit model extrapolate blur Blur size Aperture Diameter Largest physical aperture I E I 1 I 2 I 0 I 3
  25. 25. Defocus Gradient Defocus blur G is slope of this line Defocus Gradient Map Defocus Gradient Blur proportional to aperture diameter Blur size Aperture Diameter D I 1 I 2 I E I 0 σ I 3 Largest physical aperture focal length aperture diameter sensor distance object distance
  26. 26. Optimization <ul><li>solve for discrete defocus gradient values G at each pixel </li></ul><ul><li>Data term </li></ul><ul><li>Graph Cuts with spatial regularization term </li></ul>Defocus Gradient Map Smallest Aperture Image
  27. 27. Depth of Field Extrapolation
  28. 28. Synthetic Refocusing <ul><li>Modify gradient labels and re-synthesize image </li></ul>gradient map “ refocused” map extrapolated f/1.8 “ refocused” synthetic f/1.8
  29. 29. Synthetic Refocusing Video
  30. 30. Depth Guided Deconvolution <ul><li>Deconvolve (deblur) with kernel given by defocus gradient map </li></ul>Before After depth-guided deconvolution Defocus gradient map Smallest aperture image
  31. 31. Discussion <ul><li>Occlusion </li></ul><ul><ul><li>Could help depth discrimination (coded aperture) </li></ul></ul><ul><li>Difficult alignment process </li></ul><ul><ul><li>Mostly because prototype </li></ul></ul><ul><li>Refocusing limited by Depth of Field </li></ul><ul><ul><li>helped by depth-guided deconvolution </li></ul></ul><ul><li>Texture required for accurate defocus gradient map </li></ul><ul><ul><li>Not critical for depth of field and refocus </li></ul></ul>
  32. 32. Summary <ul><ul><li>Multi-aperture camera </li></ul></ul><ul><ul><ul><li>1D sampling of aperture </li></ul></ul></ul><ul><ul><ul><li>Removable </li></ul></ul></ul><ul><ul><li>Post-Exposure depth of field control </li></ul></ul><ul><ul><li>Depth of field extrapolation </li></ul></ul><ul><ul><li>Limited refocusing </li></ul></ul><ul><ul><li>Depth-guided deconvolution </li></ul></ul>
  33. 33. Thanks <ul><li>People </li></ul><ul><ul><li>John Barnwell </li></ul></ul><ul><ul><li>Jonathan Westhues </li></ul></ul><ul><ul><li>SeBaek Oh </li></ul></ul><ul><ul><li>Daniel Vlasic </li></ul></ul><ul><ul><li>Eugene Hsu </li></ul></ul><ul><ul><li>Tom Mertens </li></ul></ul><ul><ul><li>Britton Bradley </li></ul></ul><ul><ul><li>Jane Malcolm </li></ul></ul><ul><ul><li>MIT Graphics Group </li></ul></ul><ul><li>Funding </li></ul><ul><ul><li>NSF CAREER award 0447561 </li></ul></ul><ul><ul><li>Ford Foundation predoctoral Fellowship </li></ul></ul><ul><ul><li>Microsoft Research New Faculty Fellowship </li></ul></ul><ul><ul><li>Sloan Fellowship </li></ul></ul>