EPFL_presentation

Internship Summary
Ian Dewancker
August 25, 2011
Ian Dewancker Internship Summary

Problem

Problem
Realtime Performance-Based Facial Animation

Problem
“We demonstrate that compelling
3D facial dynamics can be
reconstructed in realtime without
the use of face markers, intrusive
lighting, or complex scanning
hardware.”

Problem
Oﬄine Preproccesing

Problem
Oﬄine Preproccesing
“Manually selected feature
correspondences guide the
reconstruction of user-speciﬁc
expressions”

Problem
Goal : Simplify generation of user-speciﬁc expressions
(while maintaining quality)

Approach
Looking for model(s) that deﬁne how face mesh vertices should
deform to ﬁt scan data.

Approach
Several existing techniques :
Thin-Shell Deformation
Deformation Graph
Principal Component Analysis (PCA)
As-Rigid-as-possible
Radial Basis Functions
PriMo

Approach
Several existing techniques :
Thin-Shell Deformation
Deformation Graph
Principal Component Analysis (PCA)
As-Rigid-as-possible
Radial Basis Functions
PriMo
Here we focus on investigating the use of PCA models for
generating user speciﬁc expressions

PCA Overview
Idea :
From a set of mean adjusted observations, produce a set of
orthogonal basis vectors, ordered by decreasing variance.

PCA Overview
Idea :
From a set of mean adjusted observations, produce a set of
orthogonal basis vectors, ordered by decreasing variance.
How :
Take the singular value decomposition (SVD) of the mean adjusted
data matrix. Select the ﬁrst n left singular vectors as an
approximate orthogonal basis. These are the principal components
and represent vectors along the axes of maximum variance

PCA Overview
In 2D:
X =
x1 x2 · · · xm
y1 y2 · · · ym

PCA Overview
In 2D:
X =
x1 x2 · · · xm
y1 y2 · · · ym
µ =
xµ
yµ

PCA Overview
In 2D:
X =
x1 x2 · · · xm
y1 y2 · · · ym
µ =
xµ
yµ
G = X−µ 1 1 · · · 1
G = UΣV ∗

PCA Overview
In 2D:
X =
x1 x2 · · · xm
y1 y2 · · · ym
µ =
xµ
yµ
G = X−µ 1 1 · · · 1
G = UΣV ∗
U =
b1x b2x
b1y b2y
Σ =
3 0
0 1

PCA Identity Model
How does this help us with ﬁtting faces?

PCA Identity Model
Ni =










xi1
yi1
zi1
.
.
.
xik
yik
zik










∈ R
3k
Suppose we treat a collection of m ﬁtted neutral face meshes as
data observations and generate the following:

PCA Identity Model
Ni =










xi1
yi1
zi1
.
.
.
xik
yik
zik










∈ R
3k
Suppose we treat a collection of m ﬁtted neutral face meshes as
data observations and generate the following:
X =

 N1 N2 N3 · · · Nm

 µN =
1
m
m
i=1
Ni

PCA Identity Model
As before in the example, we compute the matrix G and its SVD:
G = X − µN 1 1 1 · · · 1 G = UΣV ∗

PCA Identity Model
G = X − µN 1 1 1 · · · 1 G = UΣV ∗
In this case, U now holds the principal components of the diﬀerence
vectors from the average neutral face to those in the data set.
U =

 B1 B2 B3 · · · B3k



PCA Identity Model
G = X − µN 1 1 1 · · · 1 G = UΣV ∗
In this case, U now holds the principal components of the diﬀerence
vectors from the average neutral face to those in the data set.
U =

 B1 B2 B3 · · · B3k


We expect that any neutral face (provided training set is representative)
can be reasonably approximated by adding a linear combination of the
PCs to the mean neutral face:
Fneutral = µN + ω1B1 + ω2B2 + ω3B3 + · · · + ωpBp p 3k

PCA Expression Model

Consider deformations from neutral face to expressions like “mouth open”.

Raw Diﬀerence Model
X =

 D1 D2 D3 · · · Dm

 Di =

Raw Diﬀerence Model
X =

 D1 D2 D3 · · · Dm

 Di =
µD = Mean of training data
Bi = Principal component of training data
FE = Expression face mesh to be ﬁt (e.g. mouthOpen)
FN = Neutral face mesh
FE = FN + s[µD + ω1B1 + ω2B2 + · · · + ωpBp]

Deformation Transfer Model
X =

 L1 L2 L3 · · · Lm

 Li =

Deformation Transfer Model
X =

 L1 L2 L3 · · · Lm

 Li =
µL = Mean of deform transferred training data
DTBi = Deformed transferred PCs of training data
DTF(FN →µL) = Neutral face deform transferred to µL
FE = Expression face mesh to be ﬁt (e.g. mouthOpen)
FN = Neutral face mesh (provided)
FE = FN+s[(DTF(FN →µL)−FN)+ω1DTB1+ω2DTB2+· · ·+ωpDTBp]

Training Data
Training data is absolutely necessary for the PCA models.
A larger number of training examples will likely be more
representative of the expression space and thus, our model will be
better able to approximate arbitrary faces
In our case, we require the following to generate training data:
3D Scans Fitted Meshes

3D Scans
We used the structured light scanner to record everyone in the
LGG + Keenan (12 people) in 4 diﬀerent expressions ( Neutral,
MouthOpen, Smile, and Kiss ) for a total of 48 scans.
The scanner also provides an image that can be used for 2D
texture constraints when ﬁtting the mesh

Face Fitting
We needed a way to eﬃciently ﬁt the template or neutral face
mesh to the point clouds produced by the scanner.

Face Fitting
We needed a way to eﬃciently ﬁt the template or neutral face
mesh to the point clouds produced by the scanner.
FaceFitting Application

Face Fitting
Addition of UI to more easily control deformation parameters and
perform other operations.
Fitting HD version of loki quad mesh (27k vertices, 54k faces)

Face Fitting
One of the most tedious parts of ﬁtting is setting the 2D image
constraints (what we are trying to shield the user from). App can
save and load markers, camera matrices and images easily.
Added ability to create 2D constraints using rendered image
instead of one from real camera

Face Fitting
Included a heat map visualization to show distances between
correspondences

Building PCA Models
With our collection of ﬁtted meshes we can now generate PCA
expression models

Building PCA Models
expression models
Somewhat tedious to type various commands to generate PCA
models, especially for deformation transfer PCA. Also, if we want
to evaluate our PCA model, the person we are ﬁtting should not
be included in the model.

Building PCA Models
expression models
Somewhat tedious to type various commands to generate PCA
models, especially for deformation transfer PCA. Also, if we want
to evaluate our PCA model, the person we are ﬁtting should not
be included in the model.
Python script generates PCA expression models for each model
type (2) and expression (3), with 1 model including all faces and
12 with one person from the set discluded, giving a total of 78
PCA models.

Results
The two PCA expression models were compared using the same ﬁtted
meshes as training data.
Neutral faces were ﬁt until convergence using only the expression model
for deformation

Results
for deformation
Raw Diﬀerence PCA
Expression Mean Error (mm) Mean Error (90%) (mm)
smile 0.906 0.574
mouthOpen 0.948 0.558
kiss 1.037 0.593

Results
for deformation
Raw Diﬀerence PCA
smile 0.906 0.574
kiss 1.037 0.593
Deformation Transfer PCA
smile 0.904 0.582
kiss 0.970 0.596

Results
The following screen captures are taken from the faceshift app
after pressing the ﬁt (automagic) button
Using raw diﬀerence PCA expression model, scans from kinect

Results

Future Work
Smile and Mouth Open expressions look pretty good, however Kiss
expression is not that great.
Curling of lip not properly modeled. Lip vertices get contorted or
ﬂattened unnaturally.

Farewell
Thanks everyone for an awesome term!

EPFL_presentation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (10)

Similar to EPFL_presentation

Similar to EPFL_presentation (20)

EPFL_presentation