Downloaded 122 times
![Height Map parameterization
(diffractive)
Zernike basis parameterization
(refractive)
Φ 𝑥 = [ 𝑎11, 𝑎12, … , … ] Φ[𝑥] = 𝑍𝑖
𝑗
𝑥 ∙ 𝑎𝑖𝑗](https://image.slidesharecdn.com/learningdomain-specificcameras-180925221706/85/End-to-end-Optimization-of-Cameras-and-Image-Processing-SIGGRAPH-2018-27-320.jpg)
More Related Content
![[DL輪読会]PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metr...](https://cdn.slidesharecdn.com/ss_thumbnails/181214dlpointnet-181214053349-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DL輪読会]Deep High-Resolution Representation Learning for Human Pose Estimation](https://cdn.slidesharecdn.com/ss_thumbnails/20190517hrnet-190517005504-thumbnail.jpg?width=640&height=640&fit=bounds)
![SSII2020 [O3-01] Extreme 3D センシング](https://cdn.slidesharecdn.com/ss_thumbnails/200612-ssii-extreme3dsensing-print3-200608114658-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to End-to-end Optimization of Cameras and Image Processing - SIGGRAPH 2018
More from StanfordComputationalImaging
End-to-end Optimization of Cameras and Image Processing - SIGGRAPH 2018
- 1. End-to-end Optimization ofOptics and Image Processing for Achromatic Extended Depth of Field and Super-Resolution Imaging Vincent Sitzmann* Stanford University Steven Diamond* Stanford University Yifan Peng* University of British Columbia Xiong Dun KAUST Wolfgang Heidrich KAUST Stephen Boyd Stanford University Felix Heide Stanford University Gordon Wetzstein Stanford University
- 2. Courtesy of MichaelBok Courtesy of CSIR Notes Courtesy of MicrodacCourtesy of Jeffrey Beach
- 4. Loss Optimize optics end-to-endwith higher-level processing!
- 6. Regular bi-convex lens(focus close) Optimized camera Results: Achromatic extended DOF
- 7. How do computervision pipelines work in real life?
- 8.
- 9. Step 1: Buildcamera Optimize optics to minimize aberrations: Blur/spot size, chromatic aberrations, distortions, … Point Spread Function (PSF)
- 10. Step 2: ImageSignal Processing Maximize PSNR: Demosaicking, Denoising, Deblurring, … PSF -1
- 11. Step 3: ImagePost-Processing Minimize final loss: L2, perceptual loss, classification error, … PSF
- 12.
- 13.
- 14. Teapot ? PSF -1 + 𝜖 +𝜖 + 𝜖 + 𝜖 + 𝜖
- 15. 𝜕 𝜕𝑥 𝜕 𝜕𝑥 Bunny 𝜕 𝜕𝑥 Vision: The DeepComputational Camera Optimize end-to-end PSF -1
- 16. Bunny 𝜕 𝜕𝑥 Vision: The DeepComputational Camera • Performance & robustness gains • Domain-specific hardware may reduce footprint, cost, power… • New design space: The “BunnyCam” 𝜕 𝜕𝑥 PSF 𝜕 𝜕𝑥 -1
- 17. Bunny 𝜕 𝜕𝑥 This project: Enableoptimization of optics! 𝜕 𝜕𝑥 PSF 𝜕 𝜕𝑥 -1
- 18. Prior Work Computational Cameras OpticsOptimization Differentiable ISPs Deep Computational Photography • Deep Joint Demosaicking and Denoising (Gharbi et al. 2016) • Unrolled Optimiziation with Deep Priors (Diamond et al.) • … • EDOF through wave-front coding (Dowski & Cathey, 1995) • Recovering HDR radiance maps from photographs (Debevec & Malik 1997) • … • Diffractive Achromat (Peng et al. 2016) • Lens Factory (Sun et al. 2015) • Zemax • … • HDR image reconstruction from a single exposure using deep CNNs (Eiltertsen et al. 2016) • Learning to synthesize a 4d rgbd light field from a single image (Srinivasan et al. 2017) • … -1 Co-Design, but no true joint optimization Wiener deconvolution For efficient joint optimization: need to make differentiable!
- 19. A differentiable opticsmodel 𝜕 𝜕𝑥 PSF
- 20.
- 21. How does theoptical element map to the PSF? ? * =+ 𝜕 𝜕𝑥
- 22. Wave Optics PSFsimulator PSF𝜕 𝜕𝑥
- 23. Spherical wave frompoint source
- 24. Phaseshift by opticalelement 𝑈 𝑥 = exp 𝑗𝑘 𝑥2 + 𝑧′2 + 𝑛 − 1 Φ x
- 25. Phaseshift by opticalelement 𝑈 𝑥 = exp 𝑗𝑘 𝑥2 + 𝑧′2 + 𝑛 − 1 Φ x
- 26. Phaseshift by opticalelement Φ 𝑈 𝑥 = exp 𝑗𝑘 𝑥2 + 𝑧′2 + 𝑛 − 1 Φ x
- 27. Height Map parameterization (diffractive) Zernikebasis parameterization (refractive) Φ 𝑥 = [ 𝑎11, 𝑎12, … , … ] Φ[𝑥] = 𝑍𝑖 𝑗 𝑥 ∙ 𝑎𝑖𝑗
- 28. Phaseshift by opticalelement Φ 𝑈 𝑥 = exp 𝑗𝑘 𝑥2 + 𝑧′2 + 𝑛 − 1 Φ x
- 29. Fresnel propagation tosensor 𝑈′ 𝑥 = 𝑈 𝑥 ∗ exp( 𝑗𝑘 2𝑧 𝑥2 )
- 30. Intensity measurement atsensor 𝑈′ 𝑥 2 PSF
- 31. 𝜌 𝑧′,𝜆 =exp 𝑗𝑘 𝑥2 + 𝑦2 + 𝑧′2 + 𝑛 − 1 𝜙 𝑥, 𝑦 ∗ exp 𝑗 𝑘 2𝑧 𝑥2 + 𝑦2 2 Calculating the PSF PSF
- 32. Calculating the PSF 𝜌𝑧′,𝜆 = exp 𝑗𝑘 𝑥2 + 𝑦2 + 𝑧′2 + 𝑛 − 1 𝜙 𝑥, 𝑦 ∗ exp 𝑗 𝑘 2𝑧 𝑥2 + 𝑦2 2 • Differentiable with respect to 𝜙 • Implemented as TensorFlow module • Can easily combine with other models!
- 33. Sanity check: Optimizing afocusing lens 𝜕 𝜕𝑥
- 34. Convolve natural imageswith PSF of optics simulator
- 35. Minimize L2 losswith Stochastic Gradient Descent ∙ 2
- 36. Sanity check: Optimizinga collimator lens 𝜕 𝜕𝑥 Iteration Height Map PSF
- 37. Fabrication Refractive: Single-point DiamondTurningDiffractive: Photolithography
- 39.
- 40. Problem with singlelens: Limited Depth of Field, chromatic aberrations Scene depth Focal plane
- 41. Classic EDOF: Better depthof field, but not easily invertible Scene depth Wiener deconvolution -1 Extended depth of field through wave-front coding (Dowski & Cathey, 1995) Metasurface optics for full-color computational imaging (Colburn et al 2018)
- 42. End-to-end optimization forEDOF -1 Wiener deconvolution
- 43. -1 Wiener deconvolution Add Gaussian Noise End-to-endoptimization for EDOF
- 44. -1 Wiener deconvolution Wiener deconvolution forreconstruction End-to-end optimization for EDOF
- 45. Scene depth During training: Placeinput image at random depth -1 Wiener deconvolution End-to-end optimization for EDOF
- 46. End-to-end optimized withdeconvolution: Depth-independent and easily invertible Scene depth 𝜕 𝜕𝑥 Wiener deconvolution 𝜕 𝜕𝑥 L2 loss Fresnel Lens Multifocal Lens Cubic Phase Plate Diffractive Achromat Refr. / Diffr. Hybrid Lens Ours 17.95 dB 18.32 18.33 20.20 18.92 24.30 -1
- 47. Refractive Achromatic EDOFelement • Polymethyl methacrylate (PMMA) • 5 mm aperture size • Sensor distance 35.5𝑚𝑚 • F-number 7.1 • One active optical surface • Feature size 3.69𝜇𝑚 Optical surface Caustics
- 48. Regular bi-convex lensOptimized lens Elephant (0.5m) Book (2.0m) Sensor image Test scene
- 49. Processed image Regular bi-convexlens Optimized lens Elephant (0.5m) Book (2.0m) Test scene
- 50. Regular bi-convex lens(focus close) Optimized camera Real-world capture
- 51. Diffractive Achromatic EDOFelement • Fused silica processed via 16-level photolithography • 5 mm aperture size • Sensor distance 35.5𝑚𝑚 • F-number 7.1 • One active optical surface • Feature size 2𝜇𝑚
- 52. Regular Fresnel LensOptimized Camera w. diffractive element Diffractive EDOF: Test scene
- 53.
- 54. Experimental application: Super-Resolution Pleaserefer to paper for more detail!
- 55.
- 56. -1 𝜕 𝜕𝑥 Jointly optimizing opticsand post-processing PSF 𝜕 𝜕𝑥
- 57. 𝜕 𝜕𝑥 Bunny 𝜕 𝜕𝑥 𝜕 𝜕𝑥 PSF -1 • Most optimizationalgorithms for image reconstruction can be made differentiable by unrolling • Jointly optimize CNNs with optics, image signal processing Future work: Better image reconstruction, higher-level tasks Unrolled Optimization with Deep Priors (Diamond et al.) Deep Joint Demosaicking and Denoising (Gharbi et al.) DeepISP (Schwartz et al.) FlexISP (Heide et al.) …
- 58. Bunny PSF Optical convolutional neuralnetworks with optimized diffractive optics for image classification, Chang et al., 2018 -1 𝜕 𝜕𝑥 𝜕 𝜕𝑥 𝜕 𝜕𝑥
- 59. Felix Heide Evan PengWolfgang Heidrich Stephen Boyd Gordon Wetzstein Xiong Dun
- 60. End-to-end optimization ofoptical sensing pipelines https://vsitzmann.github.io/deepoptics/ 𝜕 𝜕𝑥 Bunny 𝜕 𝜕𝑥 𝜕 𝜕𝑥 PSF -1
Editor's Notes
- #2 Welcome everyone…
- #3 In the animal kingdom, eyes have co-evolved with the brain to form a vibrant variety of domain-specific visual systems.
- #4 In contrast, while our computer vision pipelines solve widely different problems, they all follow the same fundamental approach of creating a sharp image on a sensor before processing it.
- #5 Instead, we propose to jointly optimize the camera optics with the brain of our computer vision pipelines – the algorithms.
- #6 We can then fabricate the optimized lenses with state-of-the-art manufacturing techniques such as single-point diamond turning.
- #7 We’ll show you how to use these elements to address real-world challenges such as extended depth of field.
- #8 To get started, let’s briefly think about how computer vision pipelines work in the real world.
- #9 Let’s say we wanted to classify this scene – a bunny.
- #10 The first step is to build a camera. Let‘s focus on the camera optics, characterized by its Point Spread Function, which is the impulse response of the optics. We will optimize the PSF for blur and spot size, chromatic aberrations, distortions etc., to get a sharp image with little aberrations on the sensor.
- #11 In the next step, we design image signal processing, such as Demosaicing, Denoising, Deblurring etc., to maximize the PSNR and make a nice looking image.
- #12 Lastly, we stick that image into some higher-level algorithm, such as a convolutional neural network, and optimize that for the domain-specific task we‘re trying to solve.
- #14 And then, hopefully, it works!
- #15 Except, of course, when it doesn‘t. Along each step of the pipeline, error can accumulate, and information that may be relevant to downstream tasks can be thrown away. There‘s no way for downstream tasks to feed back what information it needs!
- #16 With this work, we follow the paradimg of optimizing this whole pipeline end-to-end.
- #18 We focus on the optics with simple processing. We‘ll show you steps towards this vision.
- #20 So, let’s talk about how we arrive at a differentiable camera model.
- #21 Let’s zoom into our camera and look at what’s going on inside. The true (or latent) image is first convolved with the PSF, determined by the optics. The sensor then adds noise, so that we measure a blurred, noisy version of the true image.
- #22 We now need to find a way to compute the PSF of our optical system in a differentiable manner, so that we can compute gradients of the PSF with respect to some parameterization of the optics.
- #23 To this end, we contribute the differentiable wave optics PSF simulator.
- #24 Let’s assume there’s a light point source at a fixed distance z prime to the left. It emits a spherical wave. At distance z’, the phase of the spherical wave can be described by the squared phase term below.
- #25 The wave now hits the optical element. The optical element is represented here as a thickness function phi. Depending on its refractive index, the lens locally shifts the phase of the incoming wavefront, so that after the optical element, the wavefront is described by the term below.
- #26 The wave now hits the optical element. The optical element is represented here as a thickness function phi. Depending on its refractive index, the lens locally shifts the phase of the incoming wavefront, so that after the optical element, the wavefront is described by the term below.
- #27 The wave now hits the optical element. The optical element is represented here as a thickness function phi. Depending on its refractive index, the lens locally shifts the phase of the incoming wavefront, so that after the optical element, the wavefront is described by the term below.
- #29 The optical element can be parameterized in a variety of ways. We parameterized diffractive elements directly via a “height map”, i.e., simply a discretized grid that contains the thickness of the element at each spatial position, and a Zernike basis representation.
- #30 In the next step, the wave propagates from the aperture to the sensor. Since we are interested in an arbitrary optical element, there is no simple analytical form of this process. Instead, we have to compute the Fresnel propagation integral, which is equivalent to computing a convolution with the Fresnel propagation kernel. This is the operator to propagate light through free space.
- #31 Finally, the wavefront hits the sensor, which measures the intensity, or squared absolute, of the wavefront. Since the source of our wave was a point source at a distance z prime and of a wavelength lambda, this intensity image is equivalent to the PSF of the system for that depth and wavelength.
- #34 Now lets see how the optimization pipeline works for the simple task of learning a focusing lens.
- #35 We use a database of natural images, which we convolve with the PSF generated by the optics simulator to get the sensor output.
- #36 We apply an L2 loss to the difference between the sensor output and the input image. We optimize the pipeline for the L2 loss using stochastic gradient descent.
- #37 Here you can see the SGD algorithm running. The PSF converges to a point, the sensor output converges to a sharp image, and the optical element height map converges to a pattern like a Fresnel element.
- #38 We manufactured the lenses we optimized in simulation using two procedures: photolithography for diffractive lenses, on the left, and diamond turning for refractive lenses, on the right.
- #39 Here you can see a diffractive lens we optimized and manufactured, on the right, and a refractive lens, on the left.
- #40 I’ll now go in detail through our application of optimizing a single lens imaging system for achromatic extended depth of field.
- #41 Single lens cameras have limited depth of field, meaning the range within which an object is sharp is limited, and suffer from chromatic aberrations, or different focal points for different colors.
- #42 Classic EDOF hand-engineers a depth-independent PSF. However, the PSF engineering has no knowledge of what PSF is easily invertible, as well as the tradeoff of which changes across depth are acceptable versus which ones may break the deconvolution algorithm. This leads to deconvolution artifacts in the reconstructed image.
- #43 To tackle this problem with our pipeline, we extend the basic focusing lens case with two further steps.
- #44 We model sensor noise by adding gaussian noise to the image after PSF blur is applied.
- #45 We then use wiener deconvolution as an image reconstruction algorithm. Wiener deconvolution has an analytical solution and is thus easily differentiated through.
- #46 We place the sampled images at random depths while running the SGD algorithm. That way we optimize over all depths in expectation.
- #47 Our optimized imaging system substantially outperforms alternative approaches in simulation, measured by the PSNR of the reconstructed image.
- #48 We manufactured the optimized lens as a refractive element. You can see from the caustics that the PSF is not simple or obvious. The lens was made out of PMMA, and has an F-number of 7.1.
- #49 Here we show a test scene captured with a regular bi-convex lens, on the left, and our optimized refractive lens, on the right. The image on the right is that cast on the sensor, before reconstruction. Our optimization does not care about this image, except to the extent that it affects the final reconstructed output. That’s why the image is blurry, but with an approximately depth invariant blur.
- #50 Now you can see the reconstructed image on the right. The image is sharp and in focus over the entire depth range, and for all colors.
- #51 Here’s a real world capture with Vincent charging at the camera. On the left, we focused a bi-convex lens close to the camera. On the right, we used our optimized lens with reconstruction. Both lenses have the same F-number.
- #52 We also optimized and manufactured a diffractive element for achromatic EDOF. Again, you can see the PSF is non-trivial. The lens was made using 16 level photolithography, and has an F-number of 7.1.
- #53 Here is a test scene captured with a Fresnel lens, on the left, and the optimized diffractive element, on the right. The image on the right includes reconstruction. The output of our optimized pipeline is sharper than the Fresnel lens baseline, though both lenses suffer from haze and other effects common with diffractive elements.
- #54 We also looked at the more experimental application of super-resolution. We tested whether the optimization pipeline could learn unconventional PSFs. In this case, the pipeline learned to multiplex the image onto three different locations on the camera. You can read about the details in the paper.
- #55 We also looked at the more experimental application of super-resolution. We tested whether the optimization pipeline could learn unconventional PSFs. In this case, the pipeline learned to multiplex the image onto three different locations on the camera. You can read about the details in the paper.
- #57 With this project we enable joint optimization of camera optics with higher-level processing by building a differentiable optics simulator that can simply be placed in an end-to-end optimization pipeline.
- #58 While in this project, we focused on the optics and thus only considered simple post-processing algorithms and tasks, we are working on optimizing the whole camera pipeline end-to-end with more advanced post-processing algorithms, such as neural networks and differentiable unrolled optimization algorithms.
- #59 More recently, our colleague Julie has used differentiable optics to optimize for the first layer of a convolutional neural network, thereby saving computation and power. Check it out!
- #60 We thank our collaborators at KAUST and Stanford.
- #61 We’ve made our code publically available on Github. We welcome questions!