The document discusses recent advances in facial modeling and animation, focusing on various methodologies including geometric modeling, 3D reconstruction, and animation frameworks. It highlights the use of image-based approaches for realistic character modeling and the significance of data acquisition technologies for enhancing productivity and realism. Additionally, it addresses the challenges and future developments in automating facial animation processes and achieving more accurate character representations.

1. Keynote :Keynote :
«New approaches«New approaches
in the field of facial modeling and animation»in the field of facial modeling and animation»
«Nouvelles approches«Nouvelles approches
de modélisation et d’animation faciale»de modélisation et d’animation faciale»
Speaker :Speaker :
Cédric GuiardCédric Guiard

2. Technological state of artTechnological state of art
Since 1972 [Parke], numerous research works
related to various thematic :
- Geometric modeling (NURBS, M-R, …)
- Physics based muscle models (Spline, FFD, FEM…),
- Anthropometrics modeling/parameterization,
- Skin deformations, wrinkles, aging
- Skin texture and rendering effects (sub-surface scattering)
- lips/speech synchronization
…
Multidisciplinary domain.
Diversity of approaches.

3. « Image-based » approach« Image-based » approach
Modeling and Animation approach based on the
acquisition and analysis of real data.
Important developments related to :
• Progress and democratization of
data acquisition, storage and treatment systems
• Methods and techniques stemming
from Computer Vision

4. Presentation frameworkPresentation framework
Set of technologies and production tools related to
3D character modeling and animation.
Objectives :
• Productivity gains
• Enhance realism
• To rationalize the production pipeline,
• To favor the creation while offering no restrictions
nor constraints in the using of traditional tools
and approaches.

5. Presentation FrameworkPresentation Framework
Technological expertise involved :
• Data matching and Model correspondence
(interactive/semi-automatic/fully-automatic)
• Definition of relevant parameterization/search spaces
(data mining, signal processing, numerical analysis)
Major concerns :
• To permit the selection/validation of data which directly or
indirectly impact upon the final results quality.
• To enable a manual edition
in accordance with traditional approaches.

6. FacialFacial modelingmodeling
With similarity constraintsWith similarity constraints
Two problematic / Two approaches :
Without similarity constraintsWithout similarity constraints
Reconstruction
of
faces
Instantiation
of
faces

7. Modeling with similarity constraintsModeling with similarity constraints
Reconstruction based on
2D images
(set of photos, pictures)
and/or
3D acquisitions
(3D scans, stereo-vision)

8. Reconstruction from 2D imagesReconstruction from 2D images

9. structured-light 3D acquisitionstructured-light 3D acquisition
Eyetronics , 3D Metrics, Inspeck

10. 3D Reconstruction3D Reconstruction

11. Modeling without similarity constraintModeling without similarity constraint
FaceSpace :FaceSpace :
Definition ofDefinition of
a continuous representation spacea continuous representation space
enabling theenabling the
intuitive parameterization/characterizationintuitive parameterization/characterization
of the appearance of a faceof the appearance of a face

12. FaceSpace DefinitionFaceSpace Definition
Computation of the correspondence between
a set of 3D acquisitions / a generic face model

13. « Average Face »
PC 1
PC 2
PC 3 PC n
Data statistical analysis
Each point of this space defines the appearance of a particular face
Principal Component
Analysis
FaceSpace definition :
• Determination of the average face
• Characterization of the minimal
framework enabling the
representation of the initial data set.
• Association of a probabilistic low
defining the likelihood of a considered
face.

14. Example
AM 1
AM 2
AM n
Definition of privileged meaningful directions
Each point of this space defines the appearance of a particular face
Characterization of different
morphological attributes
The data are scored
regarding a set of natural criteria.
The statistical analysis of these
values enables to define
meaningful directions within this
representation space.

15. Benefices of this approach in the field of image synthesisBenefices of this approach in the field of image synthesis
Facial modeling through the simple specification
of the desired appearance for the considered model.
Original
+Male
+Female
Hoked nose+Weight
Caricature

16. The same approach can be extended to Human BodyThe same approach can be extended to Human Body
Body modeling through the specification of
morphological attributes reguling the model’s appearance
Zoran Popovic, University of Washington

17. Benefices of this approach in the fields of image analysisBenefices of this approach in the fields of image analysis
Reconstructed
model
Nearest
appearance
model
Back projection in
the FaceSpace
Applicative domains :
• Biometry
• Sécurity, defense
• Beauty, cosmetic
• Reconstructive,
Plastic surgery
…
Enable to measure the similarity between two faces

18. Rest of the workflow…Rest of the workflow…
• Automatic integration of complementary elements
(eyes, eyelashes, inner-mouth, tongue, tooth),
• Animation skeleton with anatomical constraints,
• Junction and adaptation of a pre-skinned body
in order to form a whole model ready for animation,
• Parameterization of all elements.
Definition of a whole character ready for animation :
Set of technologies enabling to capitalize
through the reuse and adaptation
of predefined models

19. M-R geometric databaseM-R geometric database

20. Integration and parameterizationIntegration and parameterization

21. Hairstyles modeling and animationHairstyles modeling and animation

22. Derivation process to define low-def modelsDerivation process to define low-def models
Low-def polygonal model and oriented texture generation
based on identical hair flow definition

23. Facial animationFacial animation
Evolution in time of the degree of freedom
associated to the considered face model
Difficulties :
• Topologic/geometric complexity of the model
• Time sampling-rate
• Character Design
• Dynamics
Abstraction mechanisms :
Ex : BlendShapes { - TargetShapes ( geometry )
- Interpolation ( time )

24. State of artState of art
Normalization / Standardization :
• Facial Action Coding System (44 FAUs)
• SNHC/MPEG-4 (68 FAPs)
Several aspects remain unsolved :
• Choice/production of the different TargetShapes
• Production of the data that drive the animation
• Interpolation functions/basis between keys
• Blending of different facial expressions
…
• Reuse/transfer of the defined animation ?

25. Animation SpaceAnimation Space
Definition of representation space
devoted to facial animation
through the analysis of real dynamical sequences
Goals :
• Precise and compact characterization
of the various facial expressions,
• To enable their edition in a natural way,
• To properly and unambiguously define
the facial deformations between animation keys.

26. 3D Dynamical acquisitions3D Dynamical acquisitions
Acquisition process :
Digital vidéo camera + flash projector
Unstructured data
encompassing interferences and noise

27. Correspondence computation :Correspondence computation :
At each frame / for each acquisition :
Computation of a dense correspondence
with the generic face model

28. Bootstrapping : training phaseBootstrapping : training phase
Semi-automatic correspondence :
Low-resolution FaceMask :
Lower part of the face
124 Points
Reconstruction of 145
acquisitions
considering markers

29. Principal components analysis
of samples in correspondence
PC 1
PC 2
PC 3 PC n
Objectives :
• To characterize the number of D.o.F.
required to express the various states.
• To constraint the possible values
on each of these dimensions.
Bootstraping: definition of a search spaceBootstraping: definition of a search space
Tracking without markers
Noise reduction

30. IC 1
IC 2
IC 3 IC n
Representation Space dedicated to animationRepresentation Space dedicated to animation
Independent Component
Analysis
This decomposition enables to
de-correlate the d.o.f underlying
the model deformations in time.
The shapes deriving from this
decomposition permit a natural
interpretation close to a
muscular model.
This framework insures an
intuitive edition.

31. Selection of visemes and emotional expressionsSelection of visemes and emotional expressions
Viseme : [visual + phoneme]
52 Phonemes
• 2*6 consonants (co-articulation)
• 7 monophtongues
• 1 silence
Emotional Expressions : • surprise
• Fear
• Disgust
• Anger
• Joy
• Sadness
Selection, acquisition, correspondence computation
characterisation/back-projection in the I.C. framework
In each case :

32. IC 1
IC 2
IC 3 IC n
Animation approach within this spaceAnimation approach within this space
Benefits :
• To automate a consistent transitions
between the animation keys.
(BSpline interpolation on the I.C.s)
• To permit two different edition types
(Visemes, I.C.)
Facial animation amounts to
define the correct trajectory
within this space of visemes.

33. Integration in the production pipelineIntegration in the production pipeline
• Interactive animation
• Tracking
• Animation based on phoneme
decomposition/alignment
(Voice Analysis, TTS, …)

34. Illustration in 3D animation productionIllustration in 3D animation production
AttitudeStudio©

35. RetargetingRetargeting
Retargeting :
Techniques enabling to
(directly or indirectly)
reuse/adapt pre-defined animation sequences
Objectives :
• To make-up for morphological differences
• To gain in productivity
• To drastically reduce the amount of information required
for the definition and transmission of animations
(web3D/online games)
• To offer new exploitation possibilities
(the possibility to choose a model at runtime for instance)

36. ContextsContexts
Depend on the considered problematic and production constraints
Objectives:
• Production of FAUs, TargetShapes for facial animation
• Exploitation/transfer of the defined animation
Information types:
• Displacement fields
• Animation channels/parameters
Production contexts:
• Do the models share a common topology?
• Is it possible to standardize/normalize the model?
• Are the data required for animation already defined?
• Amount of interactions allowed during the correspondence process?

37. Three innovative approachesThree innovative approaches
Faces based on a generic model
and/or MPEG-4 compliant
Direct
reuse/transfer
of animation sequences
Faces modeled
through the use of a FaceSpace
Automatic instantiation
of the required viseme and
emotional expressions
Faces of
any topology / geometry
Semi-automatic
correspondence with
morphological adaptation

38. Animation library
based on a generic model
Result on
the reconstructed model
1) Generic face model1) Generic face model

39. MPEG-4: principle of morphological adaptationMPEG-4: principle of morphological adaptation
MW0
MNS0
ENS0
ES0
IRISD0
68
« Facial Animation
Parameters »
Expressed considering a set
of intrinsic morphological measurements
« Facial Animation Parameters Units »

40. 2) Semi-automatic correspondence2) Semi-automatic correspondence
Enable to report displacements from one model to an other
(whatever the considered topology and geometry)
taking into accounts the morphological differences among them
Modeling / production of data required for animation / animation

41. 3) Automatic generation of facial expressions3) Automatic generation of facial expressions
Dynamic FaceSpace :
Definition of a VisemeSpace for each of the faces
implied in the FaceSpace definition.
Morphological analysis :
Characterization
within the static
FaceSpace

42. Conclusions :Conclusions :
Future directions of development :
Correspondence characterization :
• To fully automate the process
• To take into account complex structures (body for inst.)
Reduce the amount of a priori information
• 3D tracking without markers for any faces
• Video-tracking
…
General panorama of the technological foundations
Multiples interesting combinations

43. Thanks !Thanks !
HumanHuman
Factory /Factory /
Duran-Duran-
DuboiDuboi
Gilles GambierGilles Gambier
Gilles RenautGilles Renaut
Magali AgutMagali Agut
Mark ChouadraMark Chouadra
Olivier DegrandOlivier Degrand
Patrick KrolPatrick Krol
Remi TemmosRemi Temmos
MESH /MESH /
ISTIST
EuropeanEuropean
ProjectProject
Amaury AubelAmaury Aubel
Curzio BassoCurzio Basso
Daniel ThalmannDaniel Thalmann
Dirk CallaertsDirk Callaerts
Gregor KalbererGregor Kalberer
Lorna HerdaLorna Herda
Luc van GoolLuc van Gool
Pascal MuellerPascal Mueller
Marc ProesmansMarc Proesmans
Nadia ThalmannNadia Thalmann
Pascal FuaPascal Fua
Sunil HadapSunil Hadap
Thomas VetterThomas Vetter
Volker BlanzVolker Blanz