This document summarizes an internship project to generate 3D avatar animations from latent representations. The intern aimed to manipulate robots through natural language by utilizing 3D avatar motion data provided by DeNA. Experiments using autoencoders and principal component analysis (PCA) found that PCA performed best, able to compress the high-dimensional motion data into 30 dimensions with less than 1% error. The conclusion is that linear dimension reduction methods like PCA can highly compress motion data, and each dimension of the resulting latent representations may correspond to meaningful motions like floating or squatting.

1. Generation of 3D-avatar
animation from latent
representations
PFN internship Fukumu Tsutsumi
(Mentors: Unno-san, Fukuda-san)
1

2. Introduction
• We aimed at manipulating robots by natural language.
• This time DeNA provides us 3D-avatar data (containing model and
motion data), so we utilize them to generate 3D-avatar motions.
• There are many kinds of motions:
• Motion without targets (e.g. “jump”) ← We focus on this
• Motion with targets (e.g. “throw”)
• Motion related to places (e.g. “ride”)
• Motion by tools with targets (e.g. “eat”)
• We intend to use unsupervised way to parametrize motions.
2

3. Previous works
• Graph-based approach (Lee+, 2006) (Zhao+, 2009)
• Regard a motion as a sequence of frames (avatar poses)
• Reduce the “similar” vertices into one vertex, so that make new, natural
transitions
• Synthetic way by timeline-like annotation (Arikan+, 2003)
• Timeline-like UI, by which users can mark given features on timelines
• System synthesize motions based on human annotations
3

4. Settings / Methods
• [Settings]
• Find “good” (= reconstructing well, smaller) dimension reduction models
• Generate motions from latent representations
• [Methods]
• Auto Encoder
• Principal Component Analysis (PCA)
• PCA, Kernel PCA, Sparse PCA, etc.
4
original
reconstructed
reduced
Forward propagation
dim=5480 dim=5480transform
inverse
transform
Generation phase

5. Datasets
• DeNA provides us both model and motion data
• Includes 100k+ motions
• Model
• Set of avatar bones, which consists of translation, y/z-axis, defining
model coordinates system
• One model has about 500 ～ 600 bones
• Motion
• Each motion data consists of 36 frames
• Each frame consists of translation, rotation-axis, rotation-angle,
scale of each bone, which come to be 10 dimensions
• In general, one avatar has about 5k ～ 6k dimensions!
5

6. Experiment #1 Auto Encoder
6
• Input data are normalized.
• Minimize the error between
original and reconstructed vector.
• Let 𝑥 be the original vector, then
reconstructed vector is
𝑥′
= 𝑊2 𝜎 𝑊1 𝑥
• Even human-shaped avatar
cannot be obtained
Reconstructed vector
Original vector
Latent vector
dim=5480
dim=100
dim=5480
Forwardpropagation
W1
W2

7. Experiment #2 PCA
• No normalization
• (linear) PCA gives error less
than 1e+01 with 30 dimensions.
• Kernel PCA does not perform
well than linear PCA.
7
Reconstructed vector
Original vector
Latent vector
dim=5480
dim=30
dim=5480
Forwardpropagation
PCA
Inv.
PCA

8. Conclusion
• Motion data can be highly compressed by dimension
reduction, such as PCA.
• At this time linear methods perform well than non-linear
methods.
• We can parametrize motion data by a meaningful way.
• Each dimension of latent representations can be regarded as some
motion, such as “float”, “squat”, “camera zoom-in”, etc.
• Show demo
Acknowledgements
I appreciate for the motion data provided by DeNA.
8