2. Motivation
• View morphing has tremendously been used
in special effects.
• It allows us to combine the salient features of
different images into one.
• In surveillance and security systems, where
cameras are used to capture the footage of
people coming in and out. It is tough to
capture the perfect frontal face of any person.
3. Motivation
• We often get faces from certain angle.
• In such scenarios it becomes very difficult to
perform face recognition on all the faces seen
in the footage and get the correct results.
4. Motivation
• Face recognition technology is
rapidly growing as evidenced by
the new surveillance system.
• Here is an example of
‘surveillance system that can
recognize a face from 36 million
other faces in one second’, from
Hitachi.
• It processes faces as it records
and keep a pre indexed gallery of
faces rather than a messy
footage.
• Super speed is achieved as it
compares the faces in real time
Source : Hitachi Face Recognition System in Surveillance Systems
by pre grouping faces with http://mashable.com/2012/03/23/hitachi-face-recognition/
similar features, so that it can
narrow down the field of search
quickly.
5. Motivation
• Limitations
– The software assumes that people are looking
right into the camera or facing not more than 30
degrees of the centre.
– Also it requires the people to be close to the
camera. Anything smaller than 40 x 40 pixel
square face is not recognizable at all.
6. Research Problem
Can we use View Morphing in Recognition
systems ?
• As we see, for any surveillance system we
need a frontal face to perform recognition.
• We investigate through our study that
whether we can generate a face given the
footage images which can be used for face
recognition.
7. Research Problem
• Primary focus of the research was to be able
to perform view morphing on face images
with different poses and expressions to
generate intermediate pose and expression
images which capture all the salient features
of the face.
8. Datasets Used
• For pose variant images we used MULTI PIE database.
Sample images from the database
9. Dataset Used
• For expression variant images we used JAFFE database.
Sample images from the database
10. Introduction
• View morphing is a technique used for
transforming one image into another.
• Hollywood films and Disney uses morphing for
effects and animation.
• Algorithms has been used to generate morphing
between images of face of different people as
well as between images of face of individuals.
• Warping and Cross Fading are the two
techniques involved in any kind of morphing.
11. Introduction
• Cross Fading is simple but does not produce
smooth transitions, major problem is to warp.
• Cross Fading generate a series of frames that
are linear combinations of the original images.
Example of cross fading,
Source : http://odonnell-
wiki.cs.uchicago.edu/inde
x.php/TenStepCrossfade
12. Introduction
• For smoother transitions it is necessary to use
warping.
• Warping transforms shape where as cross
dissolve transform color only color.
• We will now demonstrate the two different
approaches that were taken during the project
involving different kinds of warping and cross
fade techniques.
14. Feature Based Field Morphing
Outline of the process
Input Image 1
Input Image1
& & image 2
image 2
Pre Extracting Coordinate Cross
Processing Features Transform Dissolve
15. Feature Based Field Morphing
• Pre Processing
– Estimating Fundamental Matrix (F) using 8 Point
Algorithm.
– Finding Eigen vectors of F , the epipoles.
– Rotate the epipolar vectors and compute the
transform.
– Apply the transform to get the prewarped image.
16. Estimating F using 8-point algorithm
• To calculate F, the user first inputs
corresponding feature points from both the
images.
• These points are used to perform a mapping
between the two images.
17. Estimating F using 8-point algorithm
• The fundamental matrix F is defined by
x' Fx 0
for any pair of matches x and x’ in two images.
f11 f12 f13
• Let x=(u,v,1)T and x’=(u’,v’,1)T, F f 21 f 22 f 23
f 31 f 32 f 33
each match gives a linear equation
uu' f11 vu ' f12 u ' f13 uv' f 21 vv' f 22 v' f 23 uf31 vf 32 f 33 0
18. 8-point algorithm
f11
f12
f13
u1u1´ v1u1´ u1´ u1v1´ v1v1´ v1´ u1 v1 1
f 21
u 2 u 2 ´ v2 u 2 ´ u 2 ´ u 2 v2 ´ v2 v2 ´ v2 ´ u 2 v2 1
f 22 0
f 23
u n u n ´ vn u n ´ u n ´ u n vn ´ vn vn ´ vn ´ u n vn 1
f 31
f 32
f 33
• In reality, instead of solving Af 0, we seek f
to minimize Af , least eigenvector of A A.
19. 8-point algorithm – Problem?
• F should have rank 2
• To enforce that F is of rank 2, F is replaced by F’ that
minimizes F F' subject to the rank constraint.
• This is achieved by SVD. Let F UΣV, where
1 0 0 1 0 0
Σ 0 2 0 , let Σ' 0 2 0
0 0 3 0 0 0
then F' UΣ' V is the solution.
20. Image Transformation
• After Computing F we now have variables that
can be used to transform the image and
compute the pre-warped image.
• Following is an example from the experiment.
The first two images are the original images with features marked for estimating
F. The last two images are the resultant pre-warped images.
21. Feature Morphing Algorithm
• Step 1
– Input corresponding feature LINES.
– For every line PQ in source image there is a
matching line P’Q’ in the destination image.
30. Mesh warping
• Mesh warping algorithms relates features with
non-uniform mesh in the source and
destination images.
• The original images are first broken down into
small regions then these regions are mapped
onto each other to produce a morph.
32. Mesh Warping Algorithm
• Two step process
– In the first step, we generate the mapping by the
process of scattered data interpolation. The
interpolation technique that we have attempted to
use is the biharmonicsplineinterpolation, built in the
Matlab Programming Environment.
– In the second step, the mapping is applied
incrementally to shift the features from the first image
to the second and vice versa.
– The two sequences are combined via a linear cross
fade to form the final morph.
33. Process of Generating Mapping Arrays
1. Input 2 arrays of size nx 2 containing the coordinates of n
corresponding points in the initial and final images: Riand
Rf.
2. Compute the difference array Subtract Ri fromRfand split
the resultant array into column vectors [Hx, Hy] = Rf - Ri.
3. Now create 2 arrays equal in size to the original image: Mx
and My.
4. Create data points in the arrays Mx and My using Hx and Hy
1. Mx(Rx,i, Ry,i) = Hxi
2. My(Rx,i, Ry,i) = Hyi
3. Now Apply the interpolation (cubic or biharmonicspline)
to assign values in Mx and My.
4. Save Mx and My.
34. The Interpolation Process
1. Input number of frames over, which morph should
occur, N+1. Also, mapping arrays Mx and Myand initial and
final images Ii and If.
2. Create array Fn of equal size to images.
3. Define ‘frame weighting’
W = (n−1)/(N-1)
W’= (1-W)
4. For all values of (x,y) within Fn.
1. Define a = Mx(x,y)*W ,
b= My(x,y)*W
2. Set Fn(x,y) = W*I(x+a,y+b) + W’*I(x-a,y-b)
5. Compile all the frames from F1to Fn.
36. Interface
• The user then selects the critical points
between the two images and saves them into
respective .mat files.
37. Resultant images
The images on the top left corner and
bottom right corner are nearly the
original images.
30 intermediate frames were
generated between the two faces with
different expressions.
The images shows 12 of the 30
intermediate frames.
38. Resultant images
The images on the top left corner and
bottom right corner are nearly the
original images.
30 intermediate frames were
generated between the two faces with
different expressions.
The images shows9of the 30
intermediate frames.
39. Resultant images
The images on the top left corner and
bottom right corner are nearly the
original images.
30 intermediate frames were
generated between the two faces with
different expressions.
The images shows9of the 30
intermediate frames.
40. Experiments – Face Recognition
• 4 experiments were performed on the resultant
images.
• For face recognition we used Local Binary Pattern
and Chi Square Distance measure.
• LBP is a feature vector used for classification in
computer feature.
• It divides the image into cells and computes a 8
digit binary number for each cell using the
neighboring cells.
• A histogram over the cells are computed using
the frequency of the each number occurring.
41. Experiment 1
Field Morphing Case 1
Scenario Before Morphing
Data Set Multi Pie
Number of Subjects in gallery 30
Number of images per subject 2
Difference of Pose
Angle difference 30 degrees
Number of intermediate images 0
generated
Total number of images in gallery 60
Number of test images 30
Number of correct matches 22
Percentage Accuracy 73.3
42. Experiment 2
Field Morphing Case 2
Scenario After Morphing
Data Set Multi Pie
Number of Subjects in gallery 30
Number of images per subject 2
Difference of Pose
Angle difference 30 degrees
Number of intermediate images 20
generated
Total number of images in gallery 600
Number of test images 30
Number of correct matches 20
Percentage Accuracy 66.6
43. Result of Face Recognition
After morphing
Top row shows an example of
mismatch
Bottom row shows an
example of correct match.
44. Experiment 3
Mesh Warping Case 1
Scenario Before Morphing
Data Set JAFFE
Number of Subjects in gallery 10
Number of images per subject 20
Difference of Expression
Angle difference 0
Number of intermediate images 0
generated
Total number of images in gallery 200
Number of test images 20
Number of correct matches 15
Percentage Accuracy 75
45. Experiment 4
Mesh Warping Case 2
Scenario After Morphing
Data Set JAFFE
Number of Subjects in gallery 10
Number of images per subject 20
Difference of Expression
Angle difference 0
Number of intermediate images 30
generated
Total number of images in gallery 300
Number of test images 20
Number of correct matches 12
Percentage Accuracy 60
46. Result of Face Recognition
After Morphing
Top row shows an example of
mismatch
Bottom row shows an
example of correct match.
47. Analysis – Feature Morphing
1. All the salient features of the face were restored in all the
frames.
2. There were few distortions but those were not important
if we see from a face recognition point of view.
3. While performing warping, the shape and size of the
images change, due to which the LBP features of the
similar frames change, leading to low accuracy.
4. Also the test images have to warped differently using one
of the gallery image so that the feature vector can be
compared, which can lead to another variant of shape.
5. LBP uses unwanted information from the images to
compute the feature vector.
48. Analysis – Mesh Warping
• For JAFFE database and mesh warping
algorithm there was no such problem of shape
and size of the images.
• The only anticipated reason for reduced
accuracy was the not so perfect test images.
49. Conclusion
• From the above discussion and analysis we
conclude that view morphing which was never
used for face recognition purpose before is a
valid tool, with few shortcomings which can be
removed.
• Morphing is a common technique in special
effects and is now becoming and important part
of computer vision and pattern recognition.
• Face recognition can be seen as a new application
of the existing view morphing algorithms.
50. References
• View Morphing - Seitz and Dyer, 1996
• View Interpolation - Chen and Williams, 1993
• Image Morphing - Beier and Neely, 1992
• Image Morphing - Mullens and Notley, 2006
• Image Morphing: A survey - G. Wolberg, 1998
Editor's Notes
The two images are first represented as meshes. Then corresponding feature points are selected Using these feature points the two images are mapped from source to destination.
There are two types of interpolation available – cubic and biharmonicThese are (cubic) used to find the smoothest curve or surface that passesThrough a set of irregularly spaced data points.
The mapping is applied incrementally to shift the features in the original image to those in the final image and vice versa.
