ABSTRACT: In this work, we present a computer aided system for interactive extraction of anthropometrical points (landmarks) on 3D human face meshes, and a methodology for a statistical analysis of the anthropometrical points. In the developed software, we employed real time rendering techniques and interactive picking through collisions detection and haptic feedback, to allow intuitive user interaction with the virtual model. We also exploit the geometric information of the meshes by computing and displaying the curvatures and shape index, to give to the user a better understanding of the 3D data by using color maps. The proposed method was tested on a database of 35 faces from healthy Mexican individuals, obtained with a low cost structured light stereovision system. One of the objectives of this study, was to determine the statistical variability of a set of 19 face landmarks, which define facial features of the eyes, mouth, nose, cheeks and chin. To validate the reliability of the hand-extracted landmarks, a Technical Error of Measurement (TEM) analysis was performed. After the points were extracted, a rigid registration of the landmarks, to those from a reference head model, was applied by determining an optimal rigid transformation consisting of a unit quaternion and a translation vector obtained from the cross-covariance matrix. To obtain the mean landmarks set and the modes of variation, a Principal Components Analysis (PCA) based on the covariance matrix was employed. Finally, we approximated the average facial shape of the population under study, by deforming the reference head model through cage-based registration. In such approach, the high resolution model is attached to a rough mesh or cage, which encloses the detailed model, by using Mean Value Coordinates (MVC) each vertex of the detailed mesh is represented as a linear combination of the cage mesh vertices, which allow detail preserving deformation of the high polygonal mesh by displacement of the vertices from the coarse mesh, then the cage is iteratively deformed through Laplacian deformation in order to minimize the squared distance between corresponding landmarks. Besides working with points, and linear and angular measurements, the developed software also allows interactively to select paths and contours on the mesh surface, obtaining geodesic distances and areas; and even working on images. Finally, our work can be extended for the analysis of other complex anatomical structures, and potentially it may have other applications such as facial recognition on forensics, random face generation for avateering in virtual environments, and building compact 3D face databases.

1. Computational Tools for
Extracting, Representing and
Analyzing Facial Features
Saúl Heredia, Miguel Padilla, Alfonso Gastelum, Patrice Delmas, Jorge Márquez
saulhp91@hotmail.com
jorge.marquez@ccadet.unam.mx
1

2. Introduction
• The use of landmarks or feature points, has been extensively
applied to face recognition and searches on facial data bases
(Scheenstra05).
• Existing automated methods for facial landmark extraction
(Gupta10, Kaushik10) only focus on a determined set of
landmarks for specific applications.
• It is necessary an interactive, computer-assisted method as
an alternative to direct anthropometry (Enciso03), for the
analysis of facial features on ethnical groups (Carnicky06).
2

3. Objectives
• In this work we develop a software for interactive extraction of
anthropometrical landmarks, in order to determine the statistical
variation of a 19 landmarks set, over a population of 35 healthy
Mexican individuals.
• We also analyze the validity of the landmarks set, as a way to
represent the facial features over the population, via deformable
registration of a reference shape.
• The complexity of the facial surfaces requires also visualization
tools for interactive display of 3D parametric-images of curvature
distributions, among several anthropometry and shape
descriptors.
3

4. Materials
• A set of 19 craniofacial points
(Kolar97, George07)
representing facial salient
features.
• A reference model of the head
made with the MakeHuman
software†.
4
1. Nasion (n)
2. Endocanthionright (enr)
3. Endocanthioinleft (enl)
4. Exocanthionright (exr)
5. Exocanthioinleft (exl)
6. Alareright (alr)
7. Alareleft (all)
8. Subnasale (sn)
9. Labiale superius (ls)
10.Stomion (sto)
11.Labiale inferius (li)
12.Gnathion (gn)
13.Cheilionright (chr)
14.Cheilionleft (chl)
15.Sublabiale (sl)
16.Pogonion (pg)
17.Pronasale (prn)
18.Zygionright (zyr)
19.Zygionleft (zyl)
† www.makehuman.org

5. • A database of 35 3D
heads obtained with a
low cost stereovision
system with structured
light, from Mexican
subjects with the
following requirements:
Amerindian or mestizo,
20 – 40 years old
(López15).
5

6. Software Features
• Interactive Picking for
landmarks selection, based on
an octree representation.
• Measurement of distances,
angles, paths, contours and
ratios.
• Estimation and visualization of
mesh curvatures.
• Force feedback.
6

7. Interactive Picking
• To determine which point in the
scene has been selected by the
user in real time.
• A ray vs mesh collision problem:
test every single triangle, take
closest intersection point in
linear time.
• Our approach: Build octree
during startup, and traverse cells
in logarithmic time.
7

8. Force feedback
• A PHANTOM OMNI† haptic
device was employed.
• Virtual coupling was performed
using a PID controller.
• Collision detection was realized
through a point inside mesh test.
• The ideal position is the nearest
point over the mesh surface.
8
Ideal
positionF=?
Collision
detection
† http://www.dentsable.com

9. Statistical Analysis of Facial Features
• A set of 19 landmarks were
selected on each model.
• A rigid registration (Besl92)
towards a reference head
model was first applied.
• A Principal Components
Analysis and Technical
Measurement of Error (TEM)
was performed.
• Mean shape and modes of
variation were obtained. 9

10. Results
• Points with the highest TEM
are the Zygion landmarks (18,
19).
• The landmarks with highest
variance are the left and right
Zygion (18, 19), Gnathion (12),
Nasion (1), and Pogonion (16).
• The landmarks with the lowest
variance are the Stomion (10),
Labiale superius (9), Subnasale
(8) and Labiale inferius (11). 10

11. Deformable registration of 3D facial meshes
• By using Mean Value Coordinates (Ju05, Floater05) the detailed mesh is
deformed as result of deforming the coarse mesh or cage.
• The cage is iteratively deformed while minimizing the distance between
corresponding points (Savoye13).
• Laplacian Coordinates (Alexa03) are used for low distortion of the mesh.
11

12. Cage-Based Mesh Deformation
• Each vertex on the high resolution mesh is represented as a weighted
sum of the vertices of a enclosing low resolution mesh (Cage).
• By deforming the cage mesh the high resolution mesh is deformed as
result.
iv
jc
1
m
i ij j
j
w

 v c
j
c
i
v
1
m
i ij j
j
w

  v c
V CW
 V WC
1 11 12 1 1
2 21 22 2 2
1 2
m
m
n n n nm m
w w w
w w w
w w w
    
    
    
    
    
    
v c
v c
v c
1
1
m
ij
j
w

  
1
0
m
ij j i
j
w

  c v
(Ju et al., 2005) 12

13. Mean Value Coordinates
• Let v be a point inside a closed mesh, and T = vivjvk a oriented face of
the mesh.
• We project the triangle T on the unit sphere centered at v.
,
1
i
i i
vi
w
r


  T
T
,
2
jk ij ij jk ki ki jk
i
i jkn
  

   


T
n n n n
e (Floater et al., 2005)
13

14. Laplacian Coordinates
• Each vertex of the mesh is represented respecting to its neighbors
coordinates, rather than to a global coordinate frame.
14
(Alexa, 2003)
i i i δ v viv
iv
   
1
j
i j
ii 
 v
v v
NN
V ΔL
Laplace
operator
Laplacian
Coordinates
Mesh
Vertices
1
 I D AL
Identity
matrix
Diagonal matrix
Adjacency
matrix
 iid i N
iδ
One-ring neighborhood
 iN

15. Laplacian Deformation
• Solving LV=D for V
is not possible in a
naïve way: L is
singular, very large
and sparse.
• Constraint some
vertices inside a ROI
and solve for free
vertices by least
squares.
15
(Alexa, 2003)
2 2
1
argmin
n
i i i i
i i

  
 
      
 
 v Q
V v q vL v
User constraintsLaplacian coordinates
of deformed mesh
Laplacian coordinates
of original mesh

16. Iterative Cage-based Registration
16
Target position
  : , ,k k ks k  qS
Vertex index
Weight
 1
2
2
, , 1 1
argmin
t t t
m k
m m
t t t t
j j k k kj j
j js
w  
 
 
 
 
   
 
 
  c c
c δ q c
S
L
Distortion to the
source shape
Distance between
deformed source and target
  0.001
max 0.85, min 0.99 ,1.0t t
e 

  0.01
max 0.0, min 1, 0.8t t
e  
(Savoye, 2013)
Fitting term
Distortion term

17. Our approach
17
Rigid registration Affine registration
Iterative cage-based registration
Reference Acquisition
Rigid +
deformable
registration
Affine +
deformable
registration
t t t t
t
t t
 
 
   
   
   
Δ
C
Q
L
W
Target vertices
Cage Laplacian
coordinates
Source vertices
Mean Value
Coordinates
Cage Laplace
operator
Cage
Vertices

18. Validation of the Alignment of the Meshes
• Four conditions were tested:
rigid and affine registration,
alone and plus deformation.
• The Euclidean Distance
Transform was determined
(Marquez08).
18
Rigid
Rigid + Deformable Affine Affine + Deformable
2.56e+01
2.05e+01
1.54e+01
1.02e+01
5.12e+00
1.91e-06
-5.12e+00
-1.02e+01
-1.54e+01
-2.05e+01
-2.56e+01
Distance [mm]
     , | sgn min
1
sgn
1
A
c
D A d d
A
A


   
 
 
 
q
p p q p
p
p

19. Results
• The lowest alignment error corresponds to affine registration plus
deformable registration.
• The original model from the acquisition can be approximated by
deformable registration while matching corresponding landmarks.
19
Landmarks Alignment Error
RMSError(mm)
Rigid Rigid + Deformable Affine +
Deformable
Affine

20. Additional Results
• Average face model.
• Facial morphing.
• Random face generation.
20
Random Generated Landmarks

21. Conclusions
• A set of interactive software tools were developed and validated for
landmark-based anthropometrics of facial models.
• A haptic (force feedback) device allows “feeling” landmark picking.
• Morphometric results, surface features such as curvature and intersubject
similarity evaluation are displayed with several scientific visualization
techniques.
• Non-linear registration, model average extraction and shape morphing
have been incorporated, using Principal Component Analysis for assessing
the modes of variation of a population.
• The system can be applied for studying other 3D models of complex
structures.
21

22. References
• Alexa, M. (2003). "Differential coordinates for local mesh morphing and deformation." The Visual Computer 19(2-3): 105-114.
• Besl, P. J. and N. D. McKay (1992). "A method for registration of 3-D shapes." Pattern Analysis and Machine Intelligence, IEEE Transactions on 14(2): 239-
256.
• Carnicky, J. and D. C. Jr. (2006). "Three-dimensional measurement of human face with structured-light illumination." Measurement Science Review 6(2): 1.
• Enciso, R., A. Shawa, U. Neumann and J. Mah (2003). "3D head anthropometric analysis."
• Floater, M. S., G. Kós and M. Reimers (2005). "Mean value coordinates in 3D." Computer Aided Geometric Design 22(7): 623-631.
• George, R. M. (2007). Facial Geometry: Graphic Facial Analysis for Forensic Artists, Charles C. Thomas.
• Gupta, S., M. Markey and A. Bovik (2010). "Anthropometric 3D Face Recognition." International Journal of Computer Vision 90(3): 331-349.
• Ju, T., S. Schaefer and J. Warren (2005). "Mean value coordinates for closed triangular meshes." ACM Trans. Graph. 24(3): 561-566.
• Kaushik, V. D., V. K. Pathak and P. Gupta (2010). "Geometric Modeling of 3D-Face Features and Its Applications." JOURNAL OF COMPUTERS 5(9): 1-10.
• Kolar, J. C. and E. M. Salter (1997). Craniofacial Anthropometry Practical Measuremet of the Head and Face for Clinical, Surgical and Research Use.
Springfield, Illinois, U.S.A., Charles C Thomas Publisher, LTD.
• López, L. (2015). Morfometría facial en poblaciones sanas mediante un sistema de estereovisión. Master, Universidad Nacional Autónoma de México.
• Marquez, J. A. (2008). Enhancing watershed segmentation of touching and weakly-connected features in biomedical images. Engineering in Medicine and
Biology Society, 2008. EMBS 2008. 30th Annual International Conference of the IEEE.
• Savoye, Y. (2013). Iterative cage-based registration from multi-view silhouettes. Proceedings of the 10th European Conference on Visual Media Production.
London, United Kingdom, ACM: 1-10.
• Scheenstra, A. (2005). 3D Facial Image Comparison Using Landmarks - A study to the discriminating value of the characteristics of 3D facial landmarks and
their automated detection. Master, Utrecht University.
22