The document proposes a multi-aperture camera that can capture images at multiple aperture settings simultaneously. This allows for post-exposure control of depth of field and limited refocusing capabilities. The camera uses a relay system to split the aperture into separate optical paths and capture light through different sections of the aperture on a single image sensor. This enables extrapolating shallow depth of field beyond the physically largest aperture of the camera and refocusing the image after capture.
These slides use concepts from my (Jeff Funk) course entitled analyzing hi-tech opportunities to analyze how Light Field Technology is becoming economic feasible for an increasing number of applications. Light Field Cameras record all of the light fields in a picture instead of just one light field. This capability enables users to change the focus of pictures after they have been taken and to more easily record 3D data. These features are becoming economically feasible improvements because of rapid improvements in camera chips and micro-lens arrays (an example of micro-electronic mechanical systems, MEMS). These features offer alternative ways to do 3D sensing for automated vehicles and augmented reality and can enable faster data collection with telescopes.
These slides use concepts from my (Jeff Funk) course entitled analyzing hi-tech opportunities to analyze how Light Field Technology is becoming economic feasible for an increasing number of applications. Light Field Cameras record all of the light fields in a picture instead of just one light field. This capability enables users to change the focus of pictures after they have been taken and to more easily record 3D data. These features are becoming economically feasible improvements because of rapid improvements in camera chips and micro-lens arrays (an example of micro-electronic mechanical systems, MEMS). These features offer alternative ways to do 3D sensing for automated vehicles and augmented reality and can enable faster data collection with telescopes.
For the full video of this presentation, please visit:
https://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/may-2017-embedded-vision-summit-gehlhar
For more information about embedded vision, please visit:
http://www.embedded-vision.com
Jessica Gehlhar, Vision Solutions Engineer at Edmund Optics, presents the "Introduction to Optics for Embedded Vision" tutorial at the May 2017 Embedded Vision Summit.
This talk provides an introduction to optics for embedded vision system and algorithm developers. Gehlhar begins by presenting fundamental imaging lens specifications and quality metrics. She explains key parameters and concepts such as field of view, f number, working f number, NA (numerical aperture), focal length, working distance, depth of field, depth of focus, resolution, MTF (modulation transfer function), distortion, keystoning, and telecentricity and their relationships. Optical design basics and trade-offs introduced include design types, aberrations, aspheres, pointing accuracy, sensor matching, color and protective coatings, filters, temperature and environmental considerations, and their relation to sensor artifacts.
She also explores manufacturing considerations, including testing the optical components and imaging lenses in a product, and the industrial optics used for a wide range of manufacturing tests. Depending on requirements, a wide variety of tests and calibrations may be performed. These tests and calibrations become important with designs that include technologies such as multi-camera, 3D, color and NIR (near-infrared).
Millions of people worldwide need glasses or contact lenses to see or read properly. We introduce a computational display technology that predistorts the presented content for an observer, so that the target image is perceived without the need for eyewear. We demonstrate a low-cost prototype that can correct myopia, hyperopia, astigmatism, and even higher-order aberrations that are difficult to correct with glasses.
IDEAL IMAGE CHARACTERISTICS
FACTORS RELATED TO THE RADIATION BEAM
FACTORS RELATED TO THE OBJECT
FACTORS RELATED TO THE TECHNIQUE
FACTORS RELATED TO RECORDING OF THE ROENTGEN IMAGE OF THE OBJECT
DARK/ LIGHT IMAGE IDEAL IMAGE
IDEAL QUALITY CRIETRIA
Computational Displays in 4D, 6D, 8D
We have explored how light propagates from thin elements into a volume for viewing for both automultiscopic displays and holograms. In particular, devices that are typically connected with geometric optics, like parallax barriers, differ in treatment from those that obey physical optics, like holograms. However, the two concepts are often used to achieve the same effect of capturing or displaying a combination of spatial and angular information. Our work connects the two approaches under a general framework based in ray space, from which insights into applications and limitations of both parallax-based and holography-based systems are observed.
Both parallax barrier systems and the practical holographic displays are limited in that they only provide horizontal parallax. Mathematically, this is equivalent to saying that they can always be expressed as a rank-1 matrix (i.e, a matrix in which all the columns are linearly related). Knowledge of this mathematical limitation has helped us to explore the space of possibilities and extend the capabilities of current display types. In particular, we have designed a display that uses two LCD panels, and an optimisation algorithm, to produce a content-adaptive automultiscopic display (SIGGRAPH Asia 2010).
(Joint work with R Horstmeyer, Se Baek Oh, George Barbastathis, Doug Lanman, Matt Hirsch and Yunhee Kim) http://cameraculture.media.mit.edu
In other work we have developed a 6D optical system that responds to changes in viewpoint as well as changes in surrounding light. Our lenticular array alignment allows us to achieve such a system as a passive setup, omitting the need for electrical components. Unlike traditional 2D flat displays, our 6D displays discretize the incident light field and modulate 2D patterns in order to produce super-realistic (2D) images. By casting light at variable intensities and angles onto our 6D displays, we can produce multiple images as well as store greater information capacity on a single 2D film (SIGGRAPH 2008).
Ramesh Raskar joined the Media Lab from Mitsubishi Electric Research Laboratories in 2008 as head of the Lab’s Camera Culture research group. His research interests span the fields of computational photography, inverse problems in imaging and human-computer interaction. Recent inventions include transient imaging to look around a corner, next generation CAT-Scan machine, imperceptible markers for motion capture (Prakash), long distance barcodes (Bokode), touch+hover 3D interaction displays (BiDi screen), low-cost eye care devices (Netra) and new theoretical models to augment light fields (ALF) to represent wave phenomena.
In 2004, Raskar received the TR100 Award from Technology Review, which recognizes top young innovators under the age of 35, and in 2003, the Global Indus Technovator Award, instituted at MIT to recognize the top 20 Indian technology innovators worldwide. In 2009, he was awarded a Sloan Research Fellowship. In 2010, he received the Darpa Young Faculty award. He holds over 40 US patents and has received four Mitsubishi Electric Invention Awards. He is currently co-authoring a book on Computational Photography. http://raskar.info
Nityanand gopalika digital detectors for industrial applicationsNityanand Gopalika
This is a presentation by Nityanand Gopalika on Digital Radiograpgy. The presentation we given @ Digital Radiography workshop organized by GE at JFWTC, Bangalore.
Computationally Efficient Methods for Sonar Image Denoising using Fractional ...CSCJournals
Sonar images produced due to the coherent nature of scattering phenomenon inherit a multiplicative component called speckle and contain almost homogeneous as well as textured regions with relatively rare edges. Speckle removal is a pre-processing step required in applications like the detection and classification of objects in the sonar image. In this paper computationally efficient Fractional Integral Mask algorithms to remove the speckle noise from sonar images is proposed. Riemann- Liouville definition of fractional calculus is used to create Fractional integral masks in eight directions. The use of a mask incorporated with the significant coefficients from the eight directional masks and a single convolution operation required in such case helps in obtaining the computational efficiency. The sonar image heterogeneous patch classification is based on a new proposed naive homogeneity index which depends on the texture strength of the patches and despeckling filters can be adjusted to these patches. The application of the mask convolution only to the selected patches again reduce the computational complexity. The non-homomorphic approach used in the proposed method avoids the undesired bias occurring in the traditional homomorphic approach. Experiments show that the mask size required directly depends on the fractional order. Mask size can be reduced for lower fractional orders thus ensuring the computation complexity reduction for lower orders. Experimental results substantiate the effectiveness of the despeckling method. The different non reference image performance evaluation criterion are used to evaluate the proposed method.
If you are inspired by an idea 'X', how will you come up with the neXt idea? This presentation shows 6 different ways you can exercise your mind in an attempt to develop the next cool idea.
http://raskar.info
http://cameraculture.info
IMAGE ENHANCEMENT IN CASE OF UNEVEN ILLUMINATION USING VARIABLE THRESHOLDING ...ijsrd.com
Uneven illumination always affects the visual quality images which results in poor understanding about the content of the images. There is no accepted universal image enhancement algorithm or specific criteria which can fulfill user needs. The processed image may be very different with the original image in the visual effects, but it also may be similar to the original image [1]. It will be a developing tradition to integrate the advantage of various algorithms to practical application to image enhancements [2]. Zhang et al. [3] presents an adaptive image contrast enhancement method. The proposed method is based on a local gamma correction piloted by histogram analysis. In this paper , to avoid uneven Illuminance image is divided into different segments . It works locally to decrease contrast as if we perform enhancement techniques globally on portions which are already bright then this gives poor results. Enhancement techniques are applied only to those dark portions. We need accurate method that not only enhance the image but also preserve the information.
For the full video of this presentation, please visit:
https://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/may-2017-embedded-vision-summit-gehlhar
For more information about embedded vision, please visit:
http://www.embedded-vision.com
Jessica Gehlhar, Vision Solutions Engineer at Edmund Optics, presents the "Introduction to Optics for Embedded Vision" tutorial at the May 2017 Embedded Vision Summit.
This talk provides an introduction to optics for embedded vision system and algorithm developers. Gehlhar begins by presenting fundamental imaging lens specifications and quality metrics. She explains key parameters and concepts such as field of view, f number, working f number, NA (numerical aperture), focal length, working distance, depth of field, depth of focus, resolution, MTF (modulation transfer function), distortion, keystoning, and telecentricity and their relationships. Optical design basics and trade-offs introduced include design types, aberrations, aspheres, pointing accuracy, sensor matching, color and protective coatings, filters, temperature and environmental considerations, and their relation to sensor artifacts.
She also explores manufacturing considerations, including testing the optical components and imaging lenses in a product, and the industrial optics used for a wide range of manufacturing tests. Depending on requirements, a wide variety of tests and calibrations may be performed. These tests and calibrations become important with designs that include technologies such as multi-camera, 3D, color and NIR (near-infrared).
Millions of people worldwide need glasses or contact lenses to see or read properly. We introduce a computational display technology that predistorts the presented content for an observer, so that the target image is perceived without the need for eyewear. We demonstrate a low-cost prototype that can correct myopia, hyperopia, astigmatism, and even higher-order aberrations that are difficult to correct with glasses.
IDEAL IMAGE CHARACTERISTICS
FACTORS RELATED TO THE RADIATION BEAM
FACTORS RELATED TO THE OBJECT
FACTORS RELATED TO THE TECHNIQUE
FACTORS RELATED TO RECORDING OF THE ROENTGEN IMAGE OF THE OBJECT
DARK/ LIGHT IMAGE IDEAL IMAGE
IDEAL QUALITY CRIETRIA
Computational Displays in 4D, 6D, 8D
We have explored how light propagates from thin elements into a volume for viewing for both automultiscopic displays and holograms. In particular, devices that are typically connected with geometric optics, like parallax barriers, differ in treatment from those that obey physical optics, like holograms. However, the two concepts are often used to achieve the same effect of capturing or displaying a combination of spatial and angular information. Our work connects the two approaches under a general framework based in ray space, from which insights into applications and limitations of both parallax-based and holography-based systems are observed.
Both parallax barrier systems and the practical holographic displays are limited in that they only provide horizontal parallax. Mathematically, this is equivalent to saying that they can always be expressed as a rank-1 matrix (i.e, a matrix in which all the columns are linearly related). Knowledge of this mathematical limitation has helped us to explore the space of possibilities and extend the capabilities of current display types. In particular, we have designed a display that uses two LCD panels, and an optimisation algorithm, to produce a content-adaptive automultiscopic display (SIGGRAPH Asia 2010).
(Joint work with R Horstmeyer, Se Baek Oh, George Barbastathis, Doug Lanman, Matt Hirsch and Yunhee Kim) http://cameraculture.media.mit.edu
In other work we have developed a 6D optical system that responds to changes in viewpoint as well as changes in surrounding light. Our lenticular array alignment allows us to achieve such a system as a passive setup, omitting the need for electrical components. Unlike traditional 2D flat displays, our 6D displays discretize the incident light field and modulate 2D patterns in order to produce super-realistic (2D) images. By casting light at variable intensities and angles onto our 6D displays, we can produce multiple images as well as store greater information capacity on a single 2D film (SIGGRAPH 2008).
Ramesh Raskar joined the Media Lab from Mitsubishi Electric Research Laboratories in 2008 as head of the Lab’s Camera Culture research group. His research interests span the fields of computational photography, inverse problems in imaging and human-computer interaction. Recent inventions include transient imaging to look around a corner, next generation CAT-Scan machine, imperceptible markers for motion capture (Prakash), long distance barcodes (Bokode), touch+hover 3D interaction displays (BiDi screen), low-cost eye care devices (Netra) and new theoretical models to augment light fields (ALF) to represent wave phenomena.
In 2004, Raskar received the TR100 Award from Technology Review, which recognizes top young innovators under the age of 35, and in 2003, the Global Indus Technovator Award, instituted at MIT to recognize the top 20 Indian technology innovators worldwide. In 2009, he was awarded a Sloan Research Fellowship. In 2010, he received the Darpa Young Faculty award. He holds over 40 US patents and has received four Mitsubishi Electric Invention Awards. He is currently co-authoring a book on Computational Photography. http://raskar.info
Nityanand gopalika digital detectors for industrial applicationsNityanand Gopalika
This is a presentation by Nityanand Gopalika on Digital Radiograpgy. The presentation we given @ Digital Radiography workshop organized by GE at JFWTC, Bangalore.
Computationally Efficient Methods for Sonar Image Denoising using Fractional ...CSCJournals
Sonar images produced due to the coherent nature of scattering phenomenon inherit a multiplicative component called speckle and contain almost homogeneous as well as textured regions with relatively rare edges. Speckle removal is a pre-processing step required in applications like the detection and classification of objects in the sonar image. In this paper computationally efficient Fractional Integral Mask algorithms to remove the speckle noise from sonar images is proposed. Riemann- Liouville definition of fractional calculus is used to create Fractional integral masks in eight directions. The use of a mask incorporated with the significant coefficients from the eight directional masks and a single convolution operation required in such case helps in obtaining the computational efficiency. The sonar image heterogeneous patch classification is based on a new proposed naive homogeneity index which depends on the texture strength of the patches and despeckling filters can be adjusted to these patches. The application of the mask convolution only to the selected patches again reduce the computational complexity. The non-homomorphic approach used in the proposed method avoids the undesired bias occurring in the traditional homomorphic approach. Experiments show that the mask size required directly depends on the fractional order. Mask size can be reduced for lower fractional orders thus ensuring the computation complexity reduction for lower orders. Experimental results substantiate the effectiveness of the despeckling method. The different non reference image performance evaluation criterion are used to evaluate the proposed method.
If you are inspired by an idea 'X', how will you come up with the neXt idea? This presentation shows 6 different ways you can exercise your mind in an attempt to develop the next cool idea.
http://raskar.info
http://cameraculture.info
IMAGE ENHANCEMENT IN CASE OF UNEVEN ILLUMINATION USING VARIABLE THRESHOLDING ...ijsrd.com
Uneven illumination always affects the visual quality images which results in poor understanding about the content of the images. There is no accepted universal image enhancement algorithm or specific criteria which can fulfill user needs. The processed image may be very different with the original image in the visual effects, but it also may be similar to the original image [1]. It will be a developing tradition to integrate the advantage of various algorithms to practical application to image enhancements [2]. Zhang et al. [3] presents an adaptive image contrast enhancement method. The proposed method is based on a local gamma correction piloted by histogram analysis. In this paper , to avoid uneven Illuminance image is divided into different segments . It works locally to decrease contrast as if we perform enhancement techniques globally on portions which are already bright then this gives poor results. Enhancement techniques are applied only to those dark portions. We need accurate method that not only enhance the image but also preserve the information.
Getting More Precision in Videoscope Measurements While Taking Larger Measure...Olympus IMS
The challenges that go into providing accurate and precise measurements larger and from further away.Though well established, stereo measurement is often thought of as unchanging. With measurements such as distance from root, blending profiles, corrosion and area measurements requiring a greater range of measurement and precision, it is essential to understand the evolution of stereo measurement as well as other technologies available.
This presentation will focus on the basic types of measurement technologies for videoscopes, and their inherent strength and weaknesses. Reference, Shadow, Stereo, Pattern projection measurement will all be presented.
In reference to stereo measurement, recent advances and factors can improve the precision of stereo measurement compared to what existed a decade ago. What goes into the hardware and the software that translate into to a greater precision to perform greater and more reliable measurements during RVI.
Here is a Fujinon Binocular training presentation I created a few years back to explain some of the major differences and features of Fujinon Binoculars.
Basic principles of photography. David Capel. 346B IST.
Latin “Camera Obscura” = “Dark Room”
Light passing through a small hole produces an inverted image on the opposite wall
Though revolutionary in many ways, digital photography is essentially electronically implemented film photography. By contrast, computational photography exploits plentiful low-cost computing and memory, new kinds of digitally enabled sensors, optics, probes, smart lighting, and communication to capture information far beyond just a simple set of pixels. It promises a richer, even a multilayered, visual experience that may include depth, fused photo-video representations, or multispectral imagery. Professor Raskar will discuss and demonstrate advances he is working on in the areas of generalized optics, sensors, illumination methods, processing, and display, and describe how computational photography will enable us to create images that break from traditional constraints to retain more fully our fondest and most important memories, to keep personalized records of our lives, and to extend both the archival and the artistic possibilities of photography.
For the full video of this presentation, please visit:
http://www.embedded-vision.com/platinum-members/onsemi/embedded-vision-training/videos/pages/may-2016-embedded-vision-summit
For more information about embedded vision, please visit:
http://www.embedded-vision.com
Robin Jenkin, Director of Analytics, Algorithm and Module Development at ON Semiconductor, presents the "Image Sensors for Vision: Foundations and Trends" tutorial at the May 2016 Embedded Vision Summit.
Choosing the right sensor, lens and system configuration is crucial to setting you off in the right direction for your vision application. Jenkin examines fundamental considerations of image sensors that are important for embedded vision, such as pixel size, frame rate, rolling shutter vs. global shutter, back side illumination vs. front side illumination, color filter array choice and lighting, and quantum efficiency vs. crosstalk. He also explains chief ray angle, phase detect auto focus pixels, dynamic range, electron multiplied charge coupled devices, synchronization and noise, and concludes with observations on sensor trends.
A basic view of fundamentals of lens in photography. Discusses various aspects of lens, types of lens and which lens suitable for various photography moments. Hope you find it useful
This tutorial offers a step-by-step guide on how to effectively use Pinterest. It covers the basics such as account creation and navigation, as well as advanced techniques including creating eye-catching pins and optimizing your profile. The tutorial also explores collaboration and networking on the platform. With visual illustrations and clear instructions, this tutorial will equip you with the skills to navigate Pinterest confidently and achieve your goals.
Hadj Ounis's most notable work is his sculpture titled "Metamorphosis." This piece showcases Ounis's mastery of form and texture, as he seamlessly combines metal and wood to create a dynamic and visually striking composition. The juxtaposition of the two materials creates a sense of tension and harmony, inviting viewers to contemplate the relationship between nature and industry.
Explore the multifaceted world of Muntadher Saleh, an Iraqi polymath renowned for his expertise in visual art, writing, design, and pharmacy. This SlideShare delves into his innovative contributions across various disciplines, showcasing his unique ability to blend traditional themes with modern aesthetics. Learn about his impactful artworks, thought-provoking literary pieces, and his vision as a Neo-Pop artist dedicated to raising awareness about Iraq's cultural heritage. Discover why Muntadher Saleh is celebrated as "The Last Polymath" and how his multidisciplinary talents continue to inspire and influence.
Fashionista Chic Couture Maze & Coloring Adventures is a coloring and activity book filled with many maze games and coloring activities designed to delight and engage young fashion enthusiasts. Each page offers a unique blend of fashion-themed mazes and stylish illustrations to color, inspiring creativity and problem-solving skills in children.
3. Depth and Defocus Blur plane of focus sensor lens defocus blur depends on distance from plane of focus subject rays from point in focus converge to single pixel circle of confusion
4. Defocus Blur & Aperture lens plane of focus defocus blur depends on aperture size aperture http://photographertips.net sensor subject circle of confusion
5.
6.
7.
8.
9. 1D vs 2D Aperture Sampling u v Aperture 2D Grid Sampling http://photographertips.net
10. 1D vs. 2D Aperture Sampling 4 Samples Aperture 1D “Ring” Sampling 45 Samples u v Aperture 2D Grid Sampling http://photographertips.net
17. Aperture Splitting X Ideally at aperture plane , but not physically possible! Solution: Relay Optics to create virtual aperture plane Photographic Lens Aperture Plane Relay system Aperture splitting optics New Aperture Plane
18. Optical Prototype Mirror Close-up main lens relay optics mirrors tilted mirrors lenses SLR Camera
24. DOF Extrapolation Roadmap capture estimate blur fit model extrapolate blur Blur size Aperture Diameter Largest physical aperture I E I 1 I 2 I 0 I 3
25. Defocus Gradient Defocus blur G is slope of this line Defocus Gradient Map Defocus Gradient Blur proportional to aperture diameter Blur size Aperture Diameter D I 1 I 2 I E I 0 σ I 3 Largest physical aperture focal length aperture diameter sensor distance object distance
Today I will be talking about how to provide useful Depth of field controls to photographers. Depth of field is the term photographers use to describe the range of distances in the scene that are sharp in the final image. shallow DOF, as shown in the top row of images, emphasizes the subject and removes distracting backgrounds, and is often used in portrait photography . Large DOF is necessary when where there are large distances between points of interest in the scene, for example in landscape and outdoor photography. DOF of field is directly controlled by the size of the lens aperture. Large Apertures produce shallow DOF, and Small apertures produce large DOF A photographer must the choose aperture size before taking each photo. And We would like to alleviate some of this burden by allowing post-exposure depth of field control, by capturing multiple aperture settings in one exposure. Now I will go into some details of DoF
The Job of a lens is to take all the rays leaving a point on an subject located at the plane of focus, and converge them into a single point at the sensor. However, if the subject is moved away from the focus plane, the rays no longer converge to a single point, and instead produce a blurry spot. The further the subject is from the focus plane, the larger the spot will be. [CLICK] We call the size of the blurred spot, the circle of confusion or the defocus blur.
In addition to depending on depth, defocus blur also depends on the size of the aperture. [click] the smaller the aperture, the smaller the blur. So by controlling the aperture size, we can control the defocus blur, and hence DOF.
Its for this reason that aperture size is a critical parameter for a photographer to set before taking each photo, and also a parameter that takes experience to control well. we want to facilitate post-exposure DOF control by allowing the photographer to explorer the aperture settings after taking the photo . Additionally, the amount of defocus blur is limited by the physical size of the aperture, so to get the Shallow depth of field that many photographers want, it is necessary to use expensive, large aperture lenses. We would like to allow extrapolation of shallow DOF beyond the constraints of the physical aperture on your lens.
In this talk I will be presenting a new camera design that enables the capture of multiple aperture settings in a single exposure. I will also discuss some of the applications of the multi-aperture camera, with an emphasis on post-exposure depth of field control and depth of field extrapolation. And I will also describe a limited refocusing method.
We build on many previous works in the area of passive computational cameras that allow DOF control. I will go into the details and differences of several, and discuss how they ultimately shaped our design. Also, In the second half of the talk, I will describe the software applications of our camera, many of which build on depth from defocus.
The first body of work that really shaped our design is Plenoptic, or Lightfield Cameras that can capture the 4d lightfield entering the camera, by Sampling 2 spatial dimensions, and 2 angular dimensions. The general idea of these cameras is to trade some spatial resolution to record angular information instead. For example, The design of Ng and colleagues uses a lenslet array placed just over the camera sensor to direct the light that comes from different regions of the aperture to different pixels on the sensor. For example, The purple ray that passes through the center of the aperture is captured at one pixel, while rays that passes through the edges of the aperture are recorded at different pixels. In a sense, what you get is a 4D function, that has the standard 2 spatial dimensions, and 2 extra angular dimensions that sample the aperture in a grid like pattern that I show here. This extra information allows them to perform refocusing, and DOF control after the exposure is taken, and produces very high quality images. Unfortunately the image resolution is reduced by as much as 100 times, and adding the lenslet array has permanently modified your camera.
Going back to the particular application of DOF control, we remember that the main controllable parameter that affects DOF is the aperture size. The set of images on the right show how the aperture size changes in a normal camera. We can mimic the same behavior with a light field camera by summing different portions of the 2d grid of samples. For example, to create an image with a large depth of field, we would only use the central sample. [click] Now to synthesize the image from slightly larger aperture, we sum the central sample plus the next “ring” of samples. And so on…
lightfield cameras are extremely general, but if what we are really interested in is changing the aperture, then we could have saved a lot of resolution if we had stored the “rings” directly instead of the grid of samples. In other words we have stored a 2d sampling, for something that is essentially 1d. This is the key idea that shaped our camera design.
One alternative to using a lightfield camera that can capture the 1D space of apertures is to use a network of beamsplitters and cameras, for example as described in the Optical Splitting Trees work by McGuire and colleagues. The main drawback to this approach is that you loose light. In order to get variations in the aperture settings, each camera must use a physical aperture that blocks light. It is also difficult to image onto a single sensor, which would preclude it from being used with standard SLR cameras.
our review of previous work has added a few more goals. [CLICK] We want to capture the 1d space of aperture rings directly without using beamsplitters we want to use a single Image Sensor so that it can be used with consumer photography and finally, we would like it to be removable, so that you can take a normal photograph if you want.
Now I will go into the details of our particular optical design
To restate our design principles, we want to record a 3D sampling of the light entering the camera: {PAUSE} capturing 2 spatial dimensions, and 1 aperture dimension. In practice, we will capture 4 aperture rings onto the 4 quadrants of the sensor [click] For example the light that enters the smallest green aperture should form an image in the lower left corner on the sensor. [click] So the main task of our optical system is to divert the light from each aperture ring along different paths to form 4 separate images.
Our strategy to split the light arriving at the aperture into 4 paths, is to place a set of tilted mirror surfaces at the [click] Aperture plane, With Each surface having a different orientation
The tilted surfaces are oriented such that light striking different regions of the aperture is diverted along different directions. For example, light that strikes the center green mirror is reflected downwards, [click] While, light that hits the blue mirror is reflected upwards. Note, the mirrors are colored here for illustrative purposes, but are normal first surface mirrors. [click] Once split, we use mirrors to fold the light back towards the sensor, and [click] lenses to form an image on the sensor.
Ideally, we would place the aperture splitting mirrors at the aperture plane of the photographic lens. [click] Unfortunately, In practice we can’t, [click] As the aperture lies inside of the lens. Our solution is to use a relay system to create a virtual aperture plane outside of the photographic lens, and place our mirrors there. A relay system is really just more lenses that make an image the aperture.
I have brought our optical prototype to give some sense of its scale. [hold up camera as prop]. Our prototype uses an off the shelf photographic lens [click] attached to relay optics. [click] The relay optics produce an image of the aperture onto the central splitting mirrors. [click] The light is then reflected onto 1 of 4 folding mirrors [click] depending on its position in the aperture. And finally focused through lenses [click] onto the ccd sensor of a SLR camera [click]
Our design successfully splits the aperture into four regions. Here is an example of the data we capture with our camera; where Each quadrant is an image through one of the rings.
Here I am showing the Point Spread Functions of each of the rings for a point light source placed off the plane of focus. The ring shape of the PSFs indicates that our optical design does indeed split the light at the aperture into rings as intended. Ideally each would be perfectly circular and not contain any gaps or occlusions. These gaps in the PSF, which are particularly pronounced in the larger rings, are caused by the reflected light being occluded by bases of the other mirrors. The rightmost image, shows the 4 other images summed together, and is an illustration of the PSF of the reconstructed full aperture image.
Now I will discuss some of the new applications and algorithms possible with our multi-aperture camera
The first thing we can do with this data is to adjust the depth of field between the smallest and largest apertures captured. By successively summing the images from different rings, we can create images as if taken through larger apertures. [CLICK] The more rings that we sum together, the shallower the DOF becomes
So we were able to construct a sequence of images taken with the different aperture sizes. But what if we want to synthesize an image as if taken from a larger aperture? What does this extrapolated image look like? We know that it should be similar to the other images, only blurrier, because we are using a larger aperture. But exactly how much blurrier? We can approximate the defocus blur at each pixel as a convolution with a kernel that depends on the aperture size, and the depth at each pixel. This allows to relate the blur observed already in our captured images, to the extrapolated image. assuming that I0 is the smallest aperture image, We can express the blur in the other images as a convolution of image I0. For example, this pixel in I1 is a sum of some disc shaped area in I0. The corresponding pixel in I2, is also sum, but of a slightly larger area. The blur will be even larger for the extrapolated image. The only issue is to figure out exactly how large it should be. And then do this for every pixel in the extrapolated image.
The general roadmap of what we would like to do to extrapolate blur is as follows: First we capture data with our multi-aperture camera, as I’ve described earlier. From this data we have 4 images, each taken with a different aperture setting. Next, we would like to estimate the amount of blur, at each pixel, in each of the 4 images. Then, we would like to fit a function to the blur samples that allows us to extrapolate. Finally, we evaluate the extrapolation function for the new aperture size, and synthesize a new blurred image using the extrapolated blur size. In fact aperture size has a simple linear relation to blur size, and we can use this model to combine the estimation of blur and fitting the extrapolation function for a more robust estimation. Next, I’ll briefly discuss the linear model of aperture and blur size that we will use as our extrapolation function.
Looking at our lens diagram geometrically, a similar triangles argument can be used to see that there is a linear relationship between aperture diameter and blur size. For example, if the aperture size is halved, the blur should also be halved. [CLICK] More formally, we can derive an equation relating the blur size, sigma, to many of the imaging parameters such as focal length, lens aperture diameter, object and sensor distances. The details of this equation really aren’t important because we are going to group all of these terms together, and replace them with one number which we call the defocus gradient. This substitution helps make the linear form become more apparent. We call G, the defocus gradient because it describes the rate at which defocus blur changes with respect to aperture diameter. [CLICK] and From this we see that G is the slope of our extrapolation line. The key benefit of Performing this substitution is that it has reduced our task to essentially fitting a line to several data points. And then to extrapolate, we just evaluate the line at the desired aperture size. [CLICK]We call the estimate of the defocus gradient at each pixel A defocus gradient map. The DGM is related to a depth map, and infact we could have solved for depth and then converted it into a defocus gradient map. But this is a simple method that directly solves for the quantity that we are interested in, specifically How does defocus change with aperture size.
To solve for the defocus gradient map, we construct a graph problem and use graph cuts optimization. Our objective function includes a data fitting term as well as a regularization term to enforce spatial smoothness among neighbors. The data term searches for the defocus gradient value that best explains the observed blur in different aperture images. Please see our paper for more details of the optimization.
Here is an example of our DOF extrapolation technique where the depth of field is extrapolated beyond the largest aperture size.
We can also perform a type of synthetic refocusing. By shifting the labels in the defocus gradient map, we can synthetically move the apparent plane of focus. And then blur using the DOF extrapolation technique just described. Darker values indicate depths closer to the plane of focus This method only works if you originally focused on the nearest or furthest object in the scene. Otherwise there is an ambiguity about whether a point is in front of or behind the original plane of focus. Depth from Defocus methods often have similar restrictions.
Here I am showing a video where the focus plane is being moved from doll to doll
One issue with our technique, particularly for refocusing, is that the refocusing ability is limited by the depth of field present in the smallest aperture image. We can improve the sharpness by using deconvolution techniques, where the size of the kernel used to deconvolve the image Is varied at each pixel according to our estimate of the local amount of blur, taken from the defocus gradient map. The bottom two images show a close-up comparison of the original captured image data, and after our depth guided deconvolution. We used richardson-lucy deconvolution, with the kernel determined by our DGM.
A main limitation of our optical design is the occlusion that the central splitting mirror produces. This occlusion is inherent to our design, but we believe it can be reasonably well minimized. Also, there is the potential to use is as an advantage, in for example a coded-aperture type method, which you will hear about in the next two talks. Also, there is a difficult and tedious alignment process to get the images to route correctly to the sensor. Part of the difficulty of the alignment is because our system is only a prototype, which was built entirely from custom parts made in our in-house machine shop. The main issue with our DOF extrapolation and refocusing methods is that the DOF of the output image is dependent on the DOF present in the smallest aperture image. But image sharpness can be improved with our depth guided deconvolution method. Also, similarly to many Depth from defocus methods, we need texture in the image to compute an accurate defocus gradient map. Fortunately, for DOF extrapolation, its isn’t critical to have Accurate defocus gradients in textureless regions because we are blurring them, and a textureless area looks very similar blurred or unblurred.
In conclusion We have presented the design for a multi-aperture camera that captures a 1d sampling of aperture sizes, instead of a 2d grid. Our system doesn’t use beamsplitters which loose light And is completely removable. Our Multi-Aperture camera enables post-exposure DOF control, And in particular DOF extrapolation. In addition we can perform limited refocusing. And Finally we presented a depth guided deconvolution method to improve sharpness.