This document describes a project that uses photometric stereo to reconstruct 3D surfaces from images taken under different lighting conditions on a computer screen. Photometric stereo uses variations in pixel intensities across images to estimate surface normals and reconstruct the 3D shape. The project creates a MATLAB program that performs photometric stereo in real-time by flashing different light patterns on a screen and capturing images with a webcam. By using singular value decomposition, the program can reconstruct surfaces without knowing the exact positions of the light sources, overcoming a limitation of traditional photometric stereo. The reconstruction contains noise but demonstrates photometric stereo in a less controlled environment. Further work could explore tradeoffs of the method and improve efficiency.
3D Reconstruction from Multiple uncalibrated 2D Images of an ObjectAnkur Tyagi
3D reconstruction is the process of capturing the shape and appearance of real objects. In this project we are using passive methods which only use sensors to measure the radiance reflected or emitted by the objects surface to infer its 3D structure.
Image segmentation techniques
More information on this research can be found in:
Hussein, Rania, Frederic D. McKenzie. “Identifying Ambiguous Prostate Gland Contours from Histology Using Capsule Shape Information and Least Squares Curve Fitting.” The International Journal of Computer Assisted Radiology and Surgery ( IJCARS), Volume 2 Numbers 3-4, pp. 143-150, December 2007.
Stereo Correspondence Algorithms for Robotic Applications Under Ideal And Non...CSCJournals
The use of visual information in real time applications such as in robotic pick, navigation, obstacle avoidance etc. has been widely used in many sectors for enabling them to interact with its environment. Robotics require computationally simpler and easy to implement stereo vision algorithms that will provide reliable and accurate results under real time constraint. Stereo vision is a less expensive, passive sensing technique, for inferring the three dimensional position of objects from two or more simultaneous views of a scene and there is no interference with other sensing devices if multiple robots are present in the same environment. Stereo correspondence aims at finding matching points in the stereo image pair based on Lambertian criteria to obtain disparity. The correspondence algorithm will provide high resolution disparity maps of the scene by comparing two views of the scene under the study. By using the principle of triangulation and with the help of camera parameters, depth information can be extracted from this disparity .Since the focus is on real-time application, only the local stereo correspondence algorithms are considered. A comparative study based on error and computational costs are done between two area based algorithms. Evaluation of Sum of absolute Difference algorithm, which is less computationally expensive, suitable for ideal lightening condition and a more accurate adaptive binary support window algorithm that can handle of non-ideal lighting conditions are taken for this study. To simplify the correspondence search, rectified stereo image pairs are used as inputs.
An evaluation of two popular segmentation algorithms, the mean shift-based segmentation algorithm and a graph-based segmentation scheme. We also consider a hybrid method which combines the other two methods.
3D Reconstruction from Multiple uncalibrated 2D Images of an ObjectAnkur Tyagi
3D reconstruction is the process of capturing the shape and appearance of real objects. In this project we are using passive methods which only use sensors to measure the radiance reflected or emitted by the objects surface to infer its 3D structure.
Image segmentation techniques
More information on this research can be found in:
Hussein, Rania, Frederic D. McKenzie. “Identifying Ambiguous Prostate Gland Contours from Histology Using Capsule Shape Information and Least Squares Curve Fitting.” The International Journal of Computer Assisted Radiology and Surgery ( IJCARS), Volume 2 Numbers 3-4, pp. 143-150, December 2007.
Stereo Correspondence Algorithms for Robotic Applications Under Ideal And Non...CSCJournals
The use of visual information in real time applications such as in robotic pick, navigation, obstacle avoidance etc. has been widely used in many sectors for enabling them to interact with its environment. Robotics require computationally simpler and easy to implement stereo vision algorithms that will provide reliable and accurate results under real time constraint. Stereo vision is a less expensive, passive sensing technique, for inferring the three dimensional position of objects from two or more simultaneous views of a scene and there is no interference with other sensing devices if multiple robots are present in the same environment. Stereo correspondence aims at finding matching points in the stereo image pair based on Lambertian criteria to obtain disparity. The correspondence algorithm will provide high resolution disparity maps of the scene by comparing two views of the scene under the study. By using the principle of triangulation and with the help of camera parameters, depth information can be extracted from this disparity .Since the focus is on real-time application, only the local stereo correspondence algorithms are considered. A comparative study based on error and computational costs are done between two area based algorithms. Evaluation of Sum of absolute Difference algorithm, which is less computationally expensive, suitable for ideal lightening condition and a more accurate adaptive binary support window algorithm that can handle of non-ideal lighting conditions are taken for this study. To simplify the correspondence search, rectified stereo image pairs are used as inputs.
An evaluation of two popular segmentation algorithms, the mean shift-based segmentation algorithm and a graph-based segmentation scheme. We also consider a hybrid method which combines the other two methods.
LOCAL DISTANCE AND DEMPSTER-DHAFER FOR MULTI-FOCUS IMAGE FUSION sipij
This work proposes a new method of fusion image using Dempster-Shafer theory and local variability (DST-LV). This method takes into account the behaviour of each pixel with its neighbours. It consists in calculating the quadratic distance between the value of the pixel I (x, y) of each point and the value of all the neighbouring pixels. Local variability is used to determine the mass function defined in DempsterShafer theory. The two classes of Dempster-Shafer theory studied are : the fuzzy part and the focused part. The results of the proposed method are significantly better when comparing them to results of other methods.
International Journal of Engineering Research and Applications (IJERA) aims to cover the latest outstanding developments in the field of all Engineering Technologies & science.
International Journal of Engineering Research and Applications (IJERA) is a team of researchers not publication services or private publications running the journals for monetary benefits, we are association of scientists and academia who focus only on supporting authors who want to publish their work. The articles published in our journal can be accessed online, all the articles will be archived for real time access.
Our journal system primarily aims to bring out the research talent and the works done by sciaentists, academia, engineers, practitioners, scholars, post graduate students of engineering and science. This journal aims to cover the scientific research in a broader sense and not publishing a niche area of research facilitating researchers from various verticals to publish their papers. It is also aimed to provide a platform for the researchers to publish in a shorter of time, enabling them to continue further All articles published are freely available to scientific researchers in the Government agencies,educators and the general public. We are taking serious efforts to promote our journal across the globe in various ways, we are sure that our journal will act as a scientific platform for all researchers to publish their works online.
A STUDY AND ANALYSIS OF DIFFERENT EDGE DETECTION TECHNIQUEScscpconf
In the first study [1], a combination of K-means, watershed segmentation method, and Difference In Strength (DIS) map were used to perform image segmentation and edge detection
tasks. We obtained an initial segmentation based on K-means clustering technique. Starting from this, we used two techniques; the first is watershed technique with new merging
procedures based on mean intensity value to segment the image regions and to detect their boundaries. The second is edge strength technique to obtain accurate edge maps of our images without using watershed method. In this technique: We solved the problem of undesirable over segmentation results produced by the watershed algorithm, when used directly with raw data images. Also, the edge maps we obtained have no broken lines on entire image. In the 2nd study level set methods are used for the implementation of curve/interface evolution under various forces. In the third study the main idea is to detect regions (objects) boundaries, to isolate and extract individual components from a medical image. This is done using an active contours to detect regions in a given image, based on techniques of curve evolution, Mumford–Shah functional for segmentation and level sets. Once we classified our images into different intensity regions based on Markov Random Field. Then we detect regions whose boundaries are not necessarily defined by gradient by minimize an energy of Mumford–Shah functional forsegmentation, where in the level set formulation, the problem becomes a mean-curvature which will stop on the desired boundary. The stopping term does not depend on the gradient of the image as in the classical active contour. The initial curve of level set can be anywhere in the image, and interior contours are automatically detected. The final image segmentation is one
closed boundary per actual region in the image.
Study on Contrast Enhancement with the help of Associate Regions Histogram Eq...IJSRD
Histogram equalization is an uncomplicated and extensively used image distinction enhancement technique. The crucial drawback of histogram equalization is it transforms the brightness of the image. To overcome this drawback, different histogram Equalization methods have been projected. These methods protect the brightness on the result image but, do not have a usual look. Therefore this paper is an attempt to bridge the gap and results after the processed Associate regions are collected into one image. The mock-up result explains that the algorithm can not only improve image information successfully but also remain the imaginative image luminance well enough to make it likely to be used in video arrangement directly.
At the end of this lesson, you should be able to;
define segmentation.
Describe edge based in segmentation.
describe thresholding and its properties.
apply edge detection and thresholding as segmentation techniques.
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
Developing 3D Viewing Model from 2D Stereo Pair with its Occlusion RatioCSCJournals
We intend to make a 3D model using a stereo pair of images by using a novel method of local matching in pixel domain for calculating horizontal disparities. We also find the occlusion ratio using the stereo pair followed by the use of The Edge Detection and Image SegmentatiON (EDISON) system, on one the images, which provides a complete toolbox for discontinuity preserving filtering, segmentation and edge detection. Instead of assigning a disparity value to each pixel, a disparity plane is assigned to each segment. We then warp the segment disparities to the original image to get our final 3D viewing Model.
An Efficient Approach for Image Enhancement Based on Image Fusion with Retine...ijsrd.com
Aiming at problems of poor contrast and blurred edges in degraded images, a novel enhancement algorithm is proposed in present research. Image fusion refers to a technique that combines the information from two or more images of a scene into a single fused image.The Algorithm uses Retinex theory and gamma correction to perform a better enhancement of images. The algorithm can efficiently combine the advantages of Retinex and Gamma correction improving both color constancy and intensity of image.
LOCAL DISTANCE AND DEMPSTER-DHAFER FOR MULTI-FOCUS IMAGE FUSION sipij
This work proposes a new method of fusion image using Dempster-Shafer theory and local variability (DST-LV). This method takes into account the behaviour of each pixel with its neighbours. It consists in calculating the quadratic distance between the value of the pixel I (x, y) of each point and the value of all the neighbouring pixels. Local variability is used to determine the mass function defined in DempsterShafer theory. The two classes of Dempster-Shafer theory studied are : the fuzzy part and the focused part. The results of the proposed method are significantly better when comparing them to results of other methods.
International Journal of Engineering Research and Applications (IJERA) aims to cover the latest outstanding developments in the field of all Engineering Technologies & science.
International Journal of Engineering Research and Applications (IJERA) is a team of researchers not publication services or private publications running the journals for monetary benefits, we are association of scientists and academia who focus only on supporting authors who want to publish their work. The articles published in our journal can be accessed online, all the articles will be archived for real time access.
Our journal system primarily aims to bring out the research talent and the works done by sciaentists, academia, engineers, practitioners, scholars, post graduate students of engineering and science. This journal aims to cover the scientific research in a broader sense and not publishing a niche area of research facilitating researchers from various verticals to publish their papers. It is also aimed to provide a platform for the researchers to publish in a shorter of time, enabling them to continue further All articles published are freely available to scientific researchers in the Government agencies,educators and the general public. We are taking serious efforts to promote our journal across the globe in various ways, we are sure that our journal will act as a scientific platform for all researchers to publish their works online.
A STUDY AND ANALYSIS OF DIFFERENT EDGE DETECTION TECHNIQUEScscpconf
In the first study [1], a combination of K-means, watershed segmentation method, and Difference In Strength (DIS) map were used to perform image segmentation and edge detection
tasks. We obtained an initial segmentation based on K-means clustering technique. Starting from this, we used two techniques; the first is watershed technique with new merging
procedures based on mean intensity value to segment the image regions and to detect their boundaries. The second is edge strength technique to obtain accurate edge maps of our images without using watershed method. In this technique: We solved the problem of undesirable over segmentation results produced by the watershed algorithm, when used directly with raw data images. Also, the edge maps we obtained have no broken lines on entire image. In the 2nd study level set methods are used for the implementation of curve/interface evolution under various forces. In the third study the main idea is to detect regions (objects) boundaries, to isolate and extract individual components from a medical image. This is done using an active contours to detect regions in a given image, based on techniques of curve evolution, Mumford–Shah functional for segmentation and level sets. Once we classified our images into different intensity regions based on Markov Random Field. Then we detect regions whose boundaries are not necessarily defined by gradient by minimize an energy of Mumford–Shah functional forsegmentation, where in the level set formulation, the problem becomes a mean-curvature which will stop on the desired boundary. The stopping term does not depend on the gradient of the image as in the classical active contour. The initial curve of level set can be anywhere in the image, and interior contours are automatically detected. The final image segmentation is one
closed boundary per actual region in the image.
Study on Contrast Enhancement with the help of Associate Regions Histogram Eq...IJSRD
Histogram equalization is an uncomplicated and extensively used image distinction enhancement technique. The crucial drawback of histogram equalization is it transforms the brightness of the image. To overcome this drawback, different histogram Equalization methods have been projected. These methods protect the brightness on the result image but, do not have a usual look. Therefore this paper is an attempt to bridge the gap and results after the processed Associate regions are collected into one image. The mock-up result explains that the algorithm can not only improve image information successfully but also remain the imaginative image luminance well enough to make it likely to be used in video arrangement directly.
At the end of this lesson, you should be able to;
define segmentation.
Describe edge based in segmentation.
describe thresholding and its properties.
apply edge detection and thresholding as segmentation techniques.
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
Developing 3D Viewing Model from 2D Stereo Pair with its Occlusion RatioCSCJournals
We intend to make a 3D model using a stereo pair of images by using a novel method of local matching in pixel domain for calculating horizontal disparities. We also find the occlusion ratio using the stereo pair followed by the use of The Edge Detection and Image SegmentatiON (EDISON) system, on one the images, which provides a complete toolbox for discontinuity preserving filtering, segmentation and edge detection. Instead of assigning a disparity value to each pixel, a disparity plane is assigned to each segment. We then warp the segment disparities to the original image to get our final 3D viewing Model.
An Efficient Approach for Image Enhancement Based on Image Fusion with Retine...ijsrd.com
Aiming at problems of poor contrast and blurred edges in degraded images, a novel enhancement algorithm is proposed in present research. Image fusion refers to a technique that combines the information from two or more images of a scene into a single fused image.The Algorithm uses Retinex theory and gamma correction to perform a better enhancement of images. The algorithm can efficiently combine the advantages of Retinex and Gamma correction improving both color constancy and intensity of image.
Wavelet-Based Warping Technique for Mobile Devicescsandit
The role of digital images is increasing rapidly in
mobile devices. They are used in many
applications including virtual tours, virtual reali
ty, e-commerce etc. Such applications
synthesize realistic looking novel views of the ref
erence images on mobile devices using the
techniques like image-based rendering (IBR). Howeve
r, with this increasing role of digital
images comes the serious issue of processing large
images which requires considerable time.
Hence, methods to compress these large images are v
ery important. Wavelets are excellent data
compression tools that can be used with IBR algorit
hms to generate the novel views of
compressed image data. This paper proposes a framew
ork that uses wavelet-based warping
technique to render novel views of compressed image
s on mobile/ handheld devices. The
experiments are performed using Android Development
Tools (ADT) which shows the proposed
framework gives better results for large images in
terms of rendering time.
Comparative Study and Analysis of Image Inpainting TechniquesIOSR Journals
Abstract: Image inpainting is a technique to fill missing region or reconstruct damage area from an image.It
removes an undesirable object from an image in visually plausible way.For filling the part of image, it use
information from the neighboring area. In this dissertation work, we present a Examplar based method for
filling in the missing information in an image, which takes structure synthesis and texture sysnthesis together.
In exemplar based approach it used local information from an image to patch propagation.We have also
implement Nonlocal Mean approach for exemplar based image inpainting.In Nonlocal mean approach it find
multiple samples of best exemplar patches for patch propagation and weight their contribution according to
their similarity to the neighborhood under evaluation. We have further extended this algorithm by considering
collaborative filtering method to synthesize and propagate with multiple samples of best exemplar patches. We
have to preformed experiment on many images and found that our algorithm successfully inpaint the target
region.We have tested the accuracy of our algorithm by finding parameter like PSNR and compared PSNR
value for all three different approaches.
Keywords: Texture Synthesis, Structure Synthesis, Patch Propagation ,imageinpainting ,nonlocal approach,
collabrative filtering.
Dense Visual Odometry Using Genetic AlgorithmSlimane Djema
Our work aims to estimate the camera motion mounted on the head of a mobile robot or a moving object from RGB-D images in a static scene. The problem of motion estimation is transformed into a nonlinear least squares function. Methods for solving such problems are iterative. Various classic methods gave an iterative solution by linearizing this function. We can also use the metaheuristic optimization method to solve this problem and improve results. In this paper, a new algorithm is developed for visual odometry using a sequence of RGB-D images. This algorithm is based on a genetic algorithm. The proposed iterative genetic algorithm searches using particles to estimate the optimal motion and then compares it to the traditional methods. To evaluate our method, we use the root mean square error to compare it with the based energy method and another metaheuristic method. We prove the efficiency of our innovative algorithm on a large set of images.
Efficient 3D stereo vision stabilization for multi-camera viewpointsjournalBEEI
In this paper, an algorithm is developed in 3D Stereo vision to improve image stabilization process for multi-camera viewpoints. Finding accurate unique matching key-points using Harris Laplace corner detection method for different photometric changes and geometric transformation in images. Then improved the connectivity of correct matching pairs by minimizing
the global error using spanning tree algorithm. Tree algorithm helps to stabilize randomly positioned camera viewpoints in linear order. The unique matching key-points will be calculated only once with our method.
Then calculated planar transformation will be applied for real time video rendering. The proposed algorithm can process more than 200 camera viewpoints within two seconds.
MULTIPLE REGION OF INTEREST TRACKING OF NON-RIGID OBJECTS USING DEMON'S ALGOR...cscpconf
In this paper we propose an algorithm for tracking multiple ROI (region of interest) undergoing non-rigid transformations. Demon's algorithm based on the idea of Maxwell's demon, has been applied here to estimate the displacement field for tracking of multiple ROI. This algorithm works on pixel intensities of the sequence of images thus making it suitable for tracking objects/regions undergoing non-rigid transformations. We have incorporated a pyramid-based approach for demon's algorithm computations of displacement field, which leads to significant reduction in the convergence speed and improvement in the accuracy. This algorithm is applied for tracking non-rigid objects in laproscopy videos which would aid surgeons in Minimal Invasive Surgery (MIS).
Multiple region of interest tracking of non rigid objects using demon's algor...csandit
In this paper we propose an algorithm for tracking multiple ROI (region of interest) undergoing
non-rigid transformations. Demon's algorithm based on the idea of Maxwell's demon, has been
applied here to estimate the displacement field for tracking of multiple ROI. This algorithm
works on pixel intensities of the sequence of images thus making it suitable for tracking
objects/regions undergoing non-rigid transformations. We have incorporated a pyramid-based
approach for demon's algorithm computations of displacement field, which leads to significant
reduction in the convergence speed and improvement in the accuracy. This algorithm is applied
for tracking non-rigid objects in laproscopy videos which would aid surgeons in Minimal
Invasive Surgery (MIS).
A NOVEL APPROACH TO SMOOTHING ON 3D STRUCTURED ADAPTIVE MESH OF THE KINECT-BA...csandit
3-dimensional object modelling of real world objects in steady state by means of multiple point
cloud (pcl) depth scans taken by using sensing camera and application of smoothing algorithm
are suggested in this study. Polygon structure, which is constituted by coordinates of point
cloud (x,y,z) corresponding to the position of 3D model in space and obtained by nodal points
and connection of these points by means of triangulation, is utilized for the demonstration of 3D
models. Gaussian smoothing and developed methods are applied to the mesh consisting of
merge of these polygons, and a new mesh simplification and augmentation algorithm are
suggested for the over the 3D modelling. Mesh consisting of merge of polygons can be
demonstrated in a more packed, smooth and fluent way. In this study is shown that applied the
triangulation and smoothing method for 3D modelling, perform to a fast and robust mesh
structures compared to existing methods therewithal no remeshing is necessary for refinement
and reduction.
A NOVEL APPROACH TO SMOOTHING ON 3D STRUCTURED ADAPTIVE MESH OF THE KINECT-BA...
Final
1. Photometric Stereo Using Computer Screen Lighting Configurations
Richard Kim with the support of Mostafa Shuva, Hana Baker, John Lipor, David Hong, Professor Laura Balzano
Abstract
Traditional methods of photography have been about capturing location, color, and
intensity of light. However, depth information about parts of the image is not obtainable
from such methods. Photometric stereo is a method of depth computation from images
taken under different lighting conditions. Along with image stitching algorithms,
photometric stereo constructs a 3D depth image of an object. This is fundamentally
possible due to the concept found in linear algebra called “least squares”, where by
looking at the trend of similarity between images, the depth information can be extracted.
Currently, the applications of photometric stereo are vast in areas such as computer
vision, or even the movie industry, as it can be employed to capture a better 3D computer
model of an object used later for special effects.
The project attempts to create a MATLAB program that would be able to retrieve a
real-time 3D model of an object seen from a webcam, as the computer screen flashes
different patterns of lights. For preliminary testing, we used a pre-existing dataset and
algorithm provided in EECS 453/551 through MATLAB. As photometric stereo requires at
least three distinct pictures, we created an algorithm that would flash four different light
patterns. The computer then processes the images captured by the webcam in order to
get a 3D depth model in real time. This project aims to push the application of
photometric stereo in to an uncertain and real time environments. The effort was in part to
capture the interest of incoming engineering students and show them the applicability of
mathematics in interesting computer vision problems.
Objective
This project attempts to perform Photometric Stereo in less certain, real-time
environments. The traditional method assumes that the exact position of the light sources
relative to the object are known. However, the lighting parameters are much more
generalized in the case of computer screen display, as the exact distances and direction
from the computer light screen and the object are not known. Figure 1a [1] shows
traditional photometric stereo, with lights at different positions assumed to be a point
source, while Figure 1b shows the different lightings from the screen in our set up.
Methods
Photometric stereo aims to reconstruct a 3D surface of an object, given multiple
pictures of the same object in the same angle under different lighting conditions. We used
the data files from EECS 453 to become familiarized with photometric stereo. The data
package contained 12 images of a cat under different lighting conditions along with the
unit vectors describing the position of the lights relative to the object for each of the
pictures, as shown in Figure 2.
The 3D reconstruction is possible because the intensity of the light at every point of
the object can be decomposed into an equation in which we can extract the unit-norm
surface normal vector of the object, given in Equation 1 below. Highlighted in blue are
the known values, and highlighted in red the unknown values.
𝐼 𝑥, 𝑦 = 𝛼(𝑥, 𝑦)(𝑙 𝑇 𝑛 𝑥, 𝑦 )
Matrix L ∈ 𝑅3
, containing vectors 𝑙, represent the unit vectors of the light source
from the object, 𝛼(𝑥, 𝑦) the scaling constant called the surface albedo, and 𝑛(𝑥, 𝑦) the
unit-norm surface normal vector of the object at (𝑥, 𝑦). Since the inherent reflective
characteristic of the object isn’t known, the equation can be further simplified to
Equation 2, in matrix form.
𝐼 𝑥𝑦 = 𝐿 𝑇 𝜌(𝑥, 𝑦)
where 𝜌(𝑥, 𝑦) = 𝛼 𝑥, 𝑦 𝑛 𝑥, 𝑦
This assumption is valid since the albedo, or 𝛼 𝑥, 𝑦 , is the scaling constant, which
won’t affect the direction and therefore the unit-norm surface normal vector of the object
𝑛 𝑥, 𝑦 , which is found through Equation 3, and represented in Figure 3 below.
𝑛 𝑥, 𝑦 =
𝜌 𝑥,𝑦
𝜌(𝑥,𝑦) 2
Least Squares
In solving for 𝜌 𝑥, 𝑦 , an understanding of least squares is needed, which a basic
application of is shown in Figure 4 and Equation 4 below.
Given measurements Y = y1 … yn
T
and their corresponding
locations X = x1 … xn
T, least square solves
𝑚 = 𝑚
arg 𝑚𝑖𝑛
𝑌 − 𝑋𝑚 2
2
This concept of least squares is the main mathematical model driving photometric
stereo as it gets the most accurate 𝜌(𝑥, 𝑦) value representing the object. In solving for
𝜌(𝑥, 𝑦), Equation 5 presents itself in a similar fashion to the Equation 4 above.
Given Intensities 𝐼 𝑥𝑦 = 𝐼1 𝑥, 𝑦 … 𝑙d(𝑥, 𝑦) T
of each picture and their corresponding light direction
vectors L = 𝑙1 … 𝑙d , solves
𝜌 = 𝜌
arg 𝑚𝑖𝑛
𝐼 − 𝐿 𝑇 𝜌 2
2
Using this equation, we look at the same pixel location of each photographs, where
by comparing the different photographs at the same pixel location, an oversolved system
is presented. Each picture shows a different intensity value at the same location due to
the differing light locations (i.e. different light direction vectors). However, the surface
normal recovered are similar according to Equation 2. Least squares allows for the best
representative surface normal.
Surface Reconstruction
Unit-norm surface normal vectors that were found using least squares are needed for
surface reconstruction as it helps determine the partial derivatives
𝜕
𝜕𝑥
and
𝜕
𝜕𝑦
of the surface
function 𝑓(𝑥, 𝑦) characterizing the object, as shown in Equation 6.
𝜕𝑓
𝜕𝑥
= −
𝑛1 𝑥,𝑦
𝑛3 𝑥,𝑦
and
𝜕𝑓
𝜕𝑦
= −
𝑛2 𝑥,𝑦
𝑛3 𝑥,𝑦
Having each of the x and y partial derivatives are important as it represents the
magnitude of the change in depth of the object as it moves along x and y direction
respectively. Graphing the partial derivatives results in Figure 5.
We can also define the relationship between the object surface function 𝑓(𝑥, 𝑦) and
the partial derivatives
𝜕𝑓
𝜕𝑥
and
𝜕𝑓
𝜕𝑦
by using a finite differences approximation of the
derivative. Equation 7 displays this relationship, where the “+1” compares the difference
between the values of pixels next to each other.
𝜕𝑓
𝜕𝑥
≈
𝑓 𝑥+1,𝑦 −𝑓 𝑥,𝑦
𝑥+1 −𝑥
and
𝜕𝑓
𝜕𝑦
≈
𝑓 𝑥,𝑦+1 −𝑓 𝑥,𝑦
𝑦+1 −𝑦
Equation 7 can be represented in matrix quantities by
𝐷 𝑛 ⨂𝐼 𝑚
𝐼 𝑛 ⨂𝐷 𝑚
𝑓. With the rest of the
parameters known, the surface function 𝑓 can be solved for by using the method of least
squares.
𝑓 𝑥, 𝑦 = arg min 𝑏 − 𝐴𝑓 2
2
where 𝑏 =
𝜕𝑓
𝜕𝑥
𝜕𝑓
𝜕𝑦
and A =
𝐷 ⨂𝐼 𝑚
𝐼 𝑛 ⨂𝐷 𝑚
After solving for 𝑓 𝑥, 𝑦 , plotting the result returns the reconstructed 3D surface
shown in Figure 6 below.
Surface Reconstruction from Computer Screen Lighting
The aim here was to create a program in MATLAB that would extend photometric stereo
to a more uncertain, real-time environment Since photometric stereo requires multiple
lighting conditions, the computer screen would then display four different lighting
conditions while the web camera connected to the computer would take the pictures, as
demonstrated in Figure 7.
In the ideal photometric stereo set up, the L matrix containing unit-vectors from the
camera to the object was provided in order to find the unit-norm surface normal vectors
𝑛 𝑥, 𝑦 . However in our set up, it would not be clear what L matrix would be as the lighting
from computer screen could not be assumed as point-source. Another obstacle was that
the object would not always be fixed in the same location. To overcome the challenge, a
paper from Georgia Institute of Technology [3] has found a relationship between matrix 𝐼 𝑥𝑦
and 𝑛 𝑥, 𝑦 that is present by performing single value decomposition (SVD) on matrix 𝐼 𝑥𝑦.
This relationship can be used to readily recover the surface normal vectors 𝑛 𝑥, 𝑦 without
having to solve Equation 5. After being able to recover the unit-norm surface normal
vectors, a surface can then be constructed in the same fashion as in traditional
photometric stereo. The resulting 3D surface is shown in Figure 8 below.
Figure 1a: Traditional lighting parameter [1]
Figure 2: Sample Images with unique lighting condition for Photometric Stereo through EECS 453
Figure 3: Representation of a Least Squares solution
(2)
(4)
(3)
(5)
(6)
Figure 5: Plot of partial derivatives of x and y with respect to the surface function f(x,y)
(7)
(8)
Figure 6: The depth map and 3D surface reconstructed from the object
Figure 7: Web camera taking pictures under different screen lighting conditions
Figure 8: 3D surface reconstruction of a hand using the MATLAB program
Limits and Unresolved Questions
Currently, the surface reconstruction from our MATLAB code contains much more
noise compared to the traditional methods of Photometric Stereo. This may be due to
lighting condition, where the object’s environment has minimal to no lighting. Also, having
reflective or light objects could cause unwanted artifacts in the surface reconstruction.
Having more unique screen lighting configurations could also lessen the noise of the
3D reconstruction from the MATLAB program. Photometric Stereo requires a minimum of
3 pictures with unique lighting conditions. Using more unique patterns could lessen the
noise as there are more data points to interpolate the surface-norm vectors. However, this
would result in a longer delay for the reconstruction, as calculations would need to be
done with more data.
Since the program was made to perform construction in real time, the program
performs the reconstruction through the most recent four images captured. Therefore, if
an object were to be moved while the program was running, there would be a ghosting
effect between the reconstructions, as the position of the object in the four images would
not align.
Conclusion
We were able to create a program that used methods of Photometric Stereo to get
back the 3D surface with less known parameters. This was possible by using singular
value decomposition, which relates the picture intensities matrix 𝐼 𝑥𝑦 with the unit-norm
surface normal matrix 𝑁. The direction matrix 𝐿, containing the unit-vectors of the position
of light sources relative to the object, was not needed in our reconstruction. The limits of
our program is that the surface contains a lot of noise, possibly due to a variety of factors
such as the lighting environment, the minimal amount of pictures used, and the
reconstruction of moving objects. Overall, the project was a success in pushing
photometric stereo to real time, as the traditional computation on MATLAB takes over a
minute. Also, the set-up of the scene is much less demanding, as obtaining the exact
direction vectors of the light relative to the object in the traditional methods would be
difficult without the right measuring tools.
In the future, we would like to explorer the exact tradeoffs of utilizing the SVD
reconstruction method instead of the traditional reconstruction method. We would also like
to find ways to make the program efficient. Currently the speed of the program is
bottlenecked at the computation of least squares. Finding or establishing an
approximation algorithm that could return values that are close to the values found
through least squares may be an option in utilizing more unique lighting conditions and
higher resolution photos for reconstruction.
Figure 1b: Experimental lighting parameter
𝑦1 ≈ 𝑚𝑥1
𝑦2 ≈ 𝑚𝑥2
𝑦3 ≈ 𝑚𝑥3
𝑦4 ≈ 𝑚𝑥4
𝑦5 ≈ 𝑚𝑥5
[1] Photo courtesy of Wikepedia. “Least Squares”. 2016.
[2] Photo courtesy of Wikipedia. “Surface Normals”. 2016.
[3] G. Schindler, 2008. [Online]. Available:
http://www.cc.gatech.edu/conferences/3DPVT08/Program/Papers/paper209.pdf [Apr. 3, 2016]
References
Figure 3: Surface Normal [2]
(1)