Using Morphable Models for 3D Face Reconstruction

Computer Vision and Remote Sensing
Technical University Berlin, Germany
Department of Electrical Engineering
IIT Roorkee, India
Using Morphable Face Model to Improve
Stereo Reconstruction and Visualising the
Model on a Smartphone
HARDIK JAIN
Under the Guidance of
Prof. Olaf HELLWICH
Prof. RS ANAND

IIT Roorkee, India
PRESENTATION OUTLINE
 Introduction
 Motivation
 Research Methodology
 Results and Evaluation
 Visualisation
 Conclusion and Scope for Future Work
 Further Reading
1 of 25

IIT Roorkee, India
INTRODUCTION
Face Reconstruction
 Multiple Image Reconstruction
Visual structure from motion
 Stereo Reconstruction
 Single Image Reconstruction
3D Morphable Model
2 of 25

IIT Roorkee, India
3D Morphable Model
 Statistical model of face meshes which are in dense
correspondence.
 Principal component analysis (PCA) is performed on
these set of 𝑀 meshes.
 Mean shape ҧ𝑠, 𝑀 − 1 Principal Components 𝑆𝑖 and 𝜎𝑠,𝑖
2
eigen values
 Shape parameter vector cs = 𝛼1, … , 𝛼 𝑀−1
𝑇
iisi
M
i
model SsS 2
,
1
1=
= 


INTRODUCTI
ON
Single Image Reconstruction
Morphable
Model
3 of 25

IIT Roorkee, India
MOTIVATION Stereo Reconstruction
Stereo Model
Stereo
Model
High
Quality
Face Scan
Stereo Model and High
Quality Scan Cloud
Compare
4 of 25

IIT Roorkee, India
Face Model
MOTIVATION Single Image Reconstruction
Face Image Face Model
Face Model Cloud
Compare
5 of 25

IIT Roorkee, India
MOTIVATION
Single
Image
Reconstruc
tion
Stereo
Model
Deformed Face
Model
Shape
Information
Texture &
Smoothness
Deformation
6 of 25

IIT Roorkee, India
RESEARCH METHODOLOGY
Morphable
Model
Stereo Pair
Image
Deformed Face
Model
Single Image
Landmarking
Pose Estimation
Shape Fitting
Texture Extraction
Global& Local
Deformation
Stereo
Reconstruction
Face Model
7 of 25
Method Overview

IIT Roorkee, India
Landmarking Annotation
 To find ෠𝐅 = 𝑥1 𝑦1, … , 𝑥 𝑛 𝑦𝑛
𝑇
∈ ℝ2𝑛
 Obtain Face Bounding Box
 Cascade based Regression Method
 Initial estimate is centred to the Bounding box
 Regressor rt 𝐼, ෠F(t)
෠F(t+1) = ෠F(t) + rt 𝐼, ෠F(t)
RESEARCH
METHODOLOGY
Landmark Annotated
Face Image
8 of 25

IIT Roorkee, India
Pose Estimation
Camera Orientation Matrix, 𝐏
 𝐱 𝒊 = 𝐏𝐗 𝒊 , where 𝐱 𝒊∈ ℝ2 and 𝐗 𝒊∈ ℝ3
 Affine Camera Model
 Gold Standard Algorithm of Hartley & Zisserman
RESEARCH
METHODOLOGY
9 of 25
2D Landmark
Points
3D Points
on MM

IIT Roorkee, India
Shape Fitting
Estimation of shape Parameter 𝐜s
 Probabilistic Approach
 Minimise 𝐸 = σ𝑖=1
3𝑁 𝑦 𝑚𝑜𝑑𝑒𝑙2𝐷,𝑖 −𝑦 𝑖
2
2 𝜎 2𝐷,𝑖
2 + 𝐜s
2
Face Image Face Shape
Model
RESEARCH
METHODOLOGY
iisi
M
i
model SsS 2
,
1
1=
= 


10 of
25

IIT Roorkee, India
Texture Extraction
 Obtained from Face Image
 Isomap Algorithm
 Retaining Geodesic Distance
Texture Map
RESEARCH
METHODOLOGY
Face Image
11 of
25

IIT Roorkee, India
STEREO RECONSTRUCTION
 Obtain stereo pair images
 Camera Calibration
 Keypoint Detection
 Triangulation
 Dense Reconstruction
RESEARCH
METHODOLOGY
Stereo
Model
12 of
25

IIT Roorkee, India
DEFORMATION
Motivation
Surface Registration
Approach
Two-step deformation
Global and Local transformation
෥𝒗𝑗 = Φ𝑙𝑜𝑐𝑎𝑙 ∘ Φ 𝑔𝑙𝑜𝑏𝑎𝑙 (𝒗 𝑗)
RESEARCH
METHODOLOGY
13 of
25

IIT Roorkee, India
Global Deformation
Using few control points
Radial Basis Function 𝜙 𝑥, 𝑋𝑐 = 𝜙 𝑥 − 𝑋𝑐
Weighted Combination of RBF
𝑔 𝑥 = ෍
𝑘=1
𝑁
𝜆 𝑐 𝜙 𝑥 − 𝑋𝑐
Gaussian Kernel 𝜙𝑖,𝑐 𝑥 = 𝑒
−
𝑥 𝑖− 𝑋 𝑐
2
2𝜎2
RESEARCH
METHODOLOGY
Deformation
14 of
25

IIT Roorkee, India
 𝜆 𝑐 = 𝜙−1 𝑔 applied on Sourcepartial

𝜆 𝑟,1
⋮
𝜆 𝑟,𝑁
=
𝜙1,1 ⋯ 𝜙1,𝑁
⋮ ⋱ ⋮
𝜙 𝑉,1 ⋯ 𝜙 𝑉,𝑁
−1
𝑔 𝑟,1
⋮
𝑔 𝑟,𝑉
, for r = (x,y,z)
 Face unspecific results for N=25
 Few 100 milliseconds on intel quad core
computer
RESEARCH
METHODOLOGY
Deformation Global Deformation
Weight Coefficients (𝜆 𝑐)
15 of
25

IIT Roorkee, India
Local Deformation
 Smoothen the overshoots of RBF
 Influence of nearby neighbors
 Non-rigid transformation by k nearest
neighbouring vertices
 Procrustes Analysis
 Affine transformation (𝐁𝑖 and 𝐭 𝑖)
RESEARCH
METHODOLOGY
Deformation
16 of
25

IIT Roorkee, India
 Φlocal 𝑣 = w0 𝑣 𝐁0 𝑣 + 𝐭0 + σi=1
𝐾
wi 𝑣 𝐁i 𝑣 + 𝐭i
 wi 𝑣 =
1
𝐾
d− 𝑣 𝑖− 𝑣
d
, 𝑑 = σi=1
𝐾
𝑣𝑖 − 𝑣
 w0(𝑣) =
1
𝐾
 𝐾 = 12
 Requires Few seconds
RESEARCH
METHODOLOGY
DEFORMATION
Local Deformation
1
5
4
2
3
6
17 of
25

IIT Roorkee, India
DEFORMED FACE MODEL
RESULTS AND
EVALUATION
Face Image Face Model Deformed Face Model
18 of
25

IIT Roorkee, India
EVALUATION
RESULTS AND
EVALUATION
19 of
25
3.5532 3.3073 2.8738

IIT Roorkee, India
EVALUATION
RESULTS AND
EVALUATION
20 of
25
4.2734 2.9985 2.0133

IIT Roorkee, India
EVALUATION
RESULTS AND
EVALUATION
21 of
25
2.9982 2.5212 2.174

IIT Roorkee, India
VISUALISATION
 Virtual Reality
 Using Smartphone and Cardboard Viewer
 Android phones with support of OpenGL ES 3.1
 Implemented on SDK provided by Google
22 of
25

IIT Roorkee, India
Smartphone View
VISUALISATION
23 of
25

IIT Roorkee, India
CONCLUSION AND SCOPE
FOR FUTURE WORK
 Improved reconstruction after information
fusion
 Technique Could be used for various
other objects
 Cheap alternative visualisation platform
 Smartphone visualisation Environment
Improvement
24 of
25

IIT Roorkee, India
FURTHER READING
 V. Blanz and T. Vetter, “A morphable model for the synthesis of 3D faces,”
in Proceedings of the 26th annual conference on Computer graphics and
interactive techniques. ACM Press/Addison-Wesley Publishing Co., 1999,
pp. 187–194.
 P. Huber, G. Hu, R. Tena, P. Mortazavian, W. P. Koppen, W. Christmas, M.
Ratsch, and J. Kittler, “A multiresolution 3D morphable face model and
fitting framework,” in 11th International Joint Conference on Computer
Vision, Imaging and Computer Graphics Theory and Applications, February
2016.
P. Huber, Z.-H. Feng, W. Christmas, J. Kittler, and M. Rätsch, “Fitting 3D
morphable models using local features,” arXiv preprint arXiv:1503.02330,
2015.
 V. Kazemi and J. Sullivan, “One millisecond face alignment with an
ensemble of regression trees,” in Computer Vision and Pattern Recognition
(CVPR), 2014 IEEE Conference on. IEEE, 2014, pp. 1867– 1874.
 R. W. Sumner, J. Schmid, and M. Pauly, “Embedded deformation for shape
manipulation,” ACM Transactions on Graphics (TOG), vol. 26, no. 3, p. 80,
2007.
 “Cardboard.” https://developers.google.com/cardboard/overview, 2016.
25 of
25

IIT Roorkee, India
FURTHER READING
Thank You

Using Morphable Models for 3D Face Reconstruction

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Using Morphable Models for 3D Face Reconstruction

Similar to Using Morphable Models for 3D Face Reconstruction (20)

Using Morphable Models for 3D Face Reconstruction