 Developed a system that generates a 3D face using a video of single frontal face as the input  Implemented Principal Component Analysis and Active Shape Model (ASM) for facial feature point detection  Applied the POSIT algorithm to estimate the head pose for realistic shading  Modified the classic ASM and critically appraised my approach by comparing it with alternative approaches

1. Generating 3D Faces by Tracking and Pose
Estimation in Video Streams
Qichen Wang
December 2017

2. Objectives
• to develop a system that tracks the face in video streams on-the-fly, with the
assumption that the video streams contain only one frontal face.
• to estimate the orientation of the face, so that the face can be rendered with
realistic shading.
• to explore ways to improve accuracy, robustness and efficiency of the system.

3. Active Shape Model
Search for the cartilage on an MR image of the knee Search for a face
Developed by Tim Cootes at University of Manchester in 1995

4. Statistical Shape Model
• Model the distribution of N shapes aligned into the same coordinate system, each shape is
represented with a 2n-dimensional vector
• Apply Principal Component Analysis (PCA) to reduce the dimensionality
• Use the model to generate arbitrary shapes
If a shape is originally represented as
x = (𝑥%, … , 𝑥(, 𝑦%, … . , 𝑦(),
It can also be represented using the singular vectors as
x ≈ x. + 𝑃𝑏, 𝑃 = (𝑝%, … , 𝑝3),
where x. is the mean shape in the distribution; 𝑝%, … , 𝑝3 are the eigenvectors.
Therefore, the parameters b that represent x in the k-dimensional space are
𝑏 = 𝑃,
(x − x.)

5. Profile Model
The iterative approach to improve the fit of a shape
instance x, to an image object is listed as follows.
1. Examine a region of the image around each
point xi to find the best nearby match
2. Update the parameters for translation, rotation,
scaling and the parameters of the principle
components to best fit the newly found shape
3. Apply constraints to the parameters of principal
components, to ensure the newly fitted shape
is valid, then update x
4. Repeat from Step 1 until the result converges
Image Object
model
boundary
Nearest point on the edge
of object in normal
direction
profile normal
to the
boundary point on the
shape model

6. Sampled Profile
Profile Model
Cost of Fit
Level 2
Level 1
Level 0
Profile Model
Gaussian Image Pyramid

7. samples from the MUCT database samples from the person-specific data set
Training Set
3755 images, 276 subjects, 76 feature points 91 images, 1 subject, 76 feature points

8. Not Aligned Translation Aligned Procrustes Aligned
Procrustes Analysis

9. Not Aligned Translation Aligned
Procrustes Analysis
The training set of faces is a
collection of N shapes, each of
which is presented as a set of n
points in the 2D plane.
Rigid transformation is applied
to each shape by translate its
center of mass (𝑥̅, 𝑦.) to the
origin of image plane to get its
translation aligned shape 𝑃7 ,
where
𝑥̅ =
%
(
∑ 𝑥77 , 𝑦. =
%
(
∑ 𝑦77

10. Translation Aligned Procrustes Aligned
Procrustes Analysis
The Procrustes aligned shapes and their
canonical C can be iteratively obtained in the
following steps:
1. Set the canonical shape C as the unit
vector of the average of aligned faces,
𝐶 =
;
|;|
, 𝐹 = |
%
>
∑ 𝑃7|7
2. For each 𝑃7, update it by using least
square method to find the scale and
rotation to best align it to C.
3. Update C as in Step 1, after taking a
record of old C.
4. If the deviation from C to old C is not
greater than the convergence
threshold, yield the Procrustes aligned
shapes P1, P2... PN, and canonical shape
C; otherwise go to step 2.

11. Procrustes Analysis
Intuitively the alignment in Step 2 consists of rotation and scaling, which provides two variables 𝜃 and s
and the transformation matrix is
𝑇 =
𝑠 B 𝑐𝑜𝑠𝜃 −𝑠 B 𝑠𝑖𝑛𝜃
𝑠 B 𝑠𝑖𝑛𝜃 𝑠 B 𝑐𝑜𝑠𝜃
, define 𝑎 = 𝑠 B 𝑠𝑖𝑛𝜃, 𝑏 = 𝑠 B 𝑐𝑜𝑠𝜃
The optimization problem in Step 2 is minimizing the residual R defined as
𝑅 = min
L,M
∑ 𝑎 −𝑏
𝑏 𝑎
𝑥7
𝑦7
−
𝑥N,7
𝑦N,7
O
(
7P% , reformatted as 𝑅 = min
L,M
∑
𝑥7 −𝑦7
𝑦7 𝑥7
𝑎
𝑏
−
𝑥N,7
𝑦N,7
O
(
7P%
The value of a and b that minimize R can be found by making the partial derivative of R on a and b,
equals to zero. In matrix format, it is
𝑥% −𝑦%
𝑦% 𝑥%
⋮ ⋮
𝑥7 −𝑦7
𝑦7 𝑥7
⋮ ⋮
𝑎
𝑏
−
𝑥N,%
𝑦N,%
⋮
𝑥N,7
𝑦N,7
⋮
⋮
=
0
⋮
⋮
0
⋮
The result is
𝑎
𝑏
=
%
∑ (ST
UV WT
U)X
TYZ
∑
𝑥7 𝑥N,7 + 𝑦7 𝑦N,7
𝑥7 𝑦N,7 − 𝑦7 𝑥N,7
(
7P%

12. without mirrored images with mirrored images
MUCT database
Person-specific
training set
Canonical Face
Use mirrored images to get
a symmetrical face.

13. Rigid Basis
The rigid transformation to get an arbitrary face shape X that is similar to C can
be written in the following format, where X represents the result of first rotate
and scale one face shape then translate it; columns of matrix B are components
of X; and p is the corresponding coefficients.
𝑋 =
𝑎 −𝑏
𝑏 𝑎
𝑥N,%
𝑦N,%
+
𝑡S
𝑡W
⋮ ⋮ ⋮
𝑎 −𝑏
𝑏 𝑎
𝑥N,(
𝑦N,(
+
𝑡S
𝑡W
=
𝑥N,% −𝑦N,% 1 0
𝑦N,% 𝑥N,% 0 1
⋮ ⋮ ⋮ ⋮
𝑥N,( −𝑦N,( 1 0
𝑦N,( 𝑥N,( 0 1
𝑎
𝑏
𝑡S
𝑡W
= 𝐵𝑝
This equation shows that a linear combination of 4 column vectors produces a
shape that is similar to C. Hence, the rigid basis can be obtained by applying
Gram-Schmidt ortho-normalization to B to get the rigid basis R.
The canonical shape C = [𝑥N,%, 𝑦N,%, …, 𝑥N,(, 𝑦N,(],
, is the first dimension of R.

14. Non-rigid basis / Shape Model
Use PCA to reduce the dimensionality. The non-rigid basis can be obtained in the following steps:
1. Subtract the rigid components from the Procrustes aligned shapes (𝑃%, 𝑃O... 𝑃>), to get their
non-rigid components Y = (𝑃′%, 𝑃′O ... 𝑃′>) using the following equation,
𝑃′7= 𝑃7 − 𝑅𝑅,
𝑃7
2. Apply principal component analysis (PCA) on Y. First apply the singular value decomposition on
Y. The column vectors in its left-singular vector matrix U contains the principal components
sorted in descending order. The 𝛴 matrix contains singular values, which are the importance of
each column vectors in U.
𝑌O(×> = 𝑈S 𝑉,
= 𝑢% 𝑢O … 𝑢g
𝜎% 0 … … … … 0
0 𝜎O 0 … … … 0
⋮ ⋱ ⋱ ⋱ … … ⋮
0 0 0 𝜎g 0 … 0
𝑣%
,
𝑣O
,
⋮
𝑣k
,
3. Yield the first k column vectors 𝑢%, 𝑢O ... 𝑢3 in U, whose 𝜎 values add up to a predefined
threshold.

15. Non-rigid basis / Shape Model
The threshold is 95% of the sum of all singular values.

16. 1 2 3 54 96 7 8

17. before
clamping
after
clamping
Clamping
Procrustes Align the shape with the canonical shape, then for each non-rigid
dimension, restrict the component to a range of 6 times the standard deviation of
the distribution on this dimension observed in the training set.

18. patch
search window
area of interest
feature point
Patch Model

19. an arbitrary image can be warped so that the feature is centered at the search window. If the
patch being trained is for the j’th feature in the model using image X and its shape 𝑃l, and
the patch’s width and height are W and H respectively, consider the warp in the inverse of
the following steps:
1. 𝑇% = −
m
O
, −
n
O
The center of the patch is translated to the origin.
2. RS ←calc_simil(𝑃l), RS is the rotation and scale from the reference shape C’, which is
created by scaling C by specifying the width of face in pixel, to shape 𝑃l for training
current feature patch.
3. 𝑇O = 𝑥p, 𝑦p Translate the feature to align it with its original copy in image X.
T1 RS T2
Inverse(𝑇O S R 𝑇%)
Patch Model - aligning

20. Patch Model - training
train a patch using stochastic gradient descent
• I : area of interest
• Ideal R: ideal response, a
Gaussian Distribution
• R: real response
• dP: the difference between
R and ideal R times the
vector representing the
area in I covered by P

21. Patch Model
Face width = 100 pixels
10 X 10 pixels 20 X 20 pixels 40 X 40 pixels

22. Patch Model - tracking
𝑅 𝑥, 𝑦 =
∑ 𝑇(𝑥′, 𝑦′) B 𝐼(𝑥 + 𝑥r, 𝑦 + 𝑦′)Sr,Wr
∑ 𝑇(𝑥′, 𝑦′)O B ∑ 𝐼(𝑥 + 𝑥r, 𝑦 + 𝑦′)O BSr,WrSr,Wr

23. Experiment Setup
For the first frame in the video, use Haar Filter to find the position of the face, use a scaled
canonical face as the initial face shape.
For the rest frames, use the result from the previous frame as the initial face shape.
The proposed approach:
reference face width is 100 pixels; patch model 20 X 20 pixels with three
levels of search windows 20X20, 10X10, 5X5 pixels; person-specific patch
model; don’t have convergence requirement for shape fitting
Alternative approaches:
1. have convergence requirement
2. use models trained from MUCT database
3. use 1D profile model of 20 pixels with three levels of
search ranges 20, 10, 5 pixels
Input:
a 720p video captured at 30 frames per second

24. Experiment Result

25. using profile model
using MUCT data
the proposed approach
the proposed approach
with convergence criteria
Experiment Result

26. Head Pose Estimation
• Transformation from object space to camera space is
unknown
• the POSIT algorithm, can iteratively calculate T; if the focal
length and at least 4 non-coplanar vertices in object space
and their projection on image plane is known
𝑇 =
𝑖s 𝑖t 𝑖u −𝑋v
𝑗s 𝑗t 𝑗u −𝑌v
𝑘s 𝑘t 𝑘u −𝑍v
0 0 0 1

27. Shading
Delaunay
Triangulation
• Apply Delaunay triangulation to the tracked
feature points to generate a triangle mesh
• Measure the vertex normals of vertices of a 3D
head model correspond to the feature points
• Measure the coordinates of 4 non-coplanar
vertices of the 3D head model in object space
• Use a composite face as the texture
• For each frame, run POSIT and apply the
rotation to vertex normals

29. DEMO