Inverse kinematics (IK) technology helps game developers create natural character animations but its complexity makes it time consuming to implement across a large cast. This talk details how the Deep Learning Deployment Toolkit (DLDT) allows game developers to quickly deploy deep-learning algorithms to solve character IK problems and to yield better results

1. Yoon Dongwon (Game AI Team, Game AI Lab, AI Center, NCSOFT)
Jang Hanyoung (Motion AI Team, Game AI Lab, AI Center, NCSOFT)

2. @IntelSoftware @IntelGraphics 2
Agenda
• Introduction
• Deep learning based Inverse Kinematics
• Optimization using Intel® OpenVINO™ toolkit
• Result
• Conclusion

3. @IntelSoftware @IntelGraphics 3
The Team
Game AI Lab
Motion AI
Team
Game AI
Team
Reinforcement
Learning Team

5. @IntelSoftware @IntelGraphics 5
Inverse Kinematics (IK)
• Compute joint angle that makes end-effector move to
the target.
Target
Joint angle 3
Joint angle 1
Joint angle 2
End-effector
?
?
?
Catch the target

6. @IntelSoftware @IntelGraphics 6
Inverse Kinematics in Games
• Generate animations that interact with the
surrounding environment
ex) Solving the foot skating problem ex) Generate animation to catch some object

7. @IntelSoftware @IntelGraphics 7
Goal
• Large number of characters climb rugged cliff with IK

8. @IntelSoftware @IntelGraphics 8
Challenges
Traditional methods are not suitable for the objective
Numerical approach Analytical approach
Quality Good Bad
Performance Slow Fast
Numerical approach Analytical approach Deep Learning
Quality Good Bad Good
Performance Slow Fast Fast
Solution

10. @IntelSoftware @IntelGraphics 10
Approach
“Input”
• Current Hands and Feet Positions
• Target Hands and Feet Positions
“Output”
• Climbing motion
• Number of Frames : 60
• Length : 2 seconds
Deep Learning based IK Solver
R
R

11. @IntelSoftware @IntelGraphics 11
Architecture [1/2]
Trajectory
Network
60 Pose
Networks
R
R
5 Trajectories
Deep Learning based IK Solver

12. @IntelSoftware @IntelGraphics 12
Architecture [2/2]
Trajectory
Network
60 Pose
Networks
Curve
fitting
R
R
R
R
5 Trajectories
Smoothened
5 Trajectories
Deep Learning based IK Solver

13. @IntelSoftware @IntelGraphics 13
Gathering Training Data
1. Produce manually created “reference motion”.
2. Then randomize “start and end targets” based on “reference motion”.
3. Finally for each “start and end targets”, generate motion data using
“numerical approach”.
Random targets
Reference motion Generated motion using numerical approach

15. @IntelSoftware @IntelGraphics 15
Overview
Training server Infers IK Animation
Deploy
Trained Network
“How to optimize client-side inference?”
• The process of Deep-learning based IK
Client

16. @IntelSoftware @IntelGraphics
① Small-sized tasks
16
CPU vs GPU [1/3] : Requirements
Inference for IK
• Small-sized neural networks
• Tasks are performed for each
game loop
• Small batch size
• Require quick response
( low–latency )
• GPU is busy rendering.
② Inference on Game-Client
Client

17. @IntelSoftware @IntelGraphics 17
CPU vs GPU [2/3] : ① Small-sized tasks
Small sized neural networks and
small batch size
Large sized neural networks and
large batch size

18. @IntelSoftware @IntelGraphics 18
CPU vs GPU [3/3] : ② Inference overhead on Game-Client
Without Inference CPU usage GPU usage Game frame rate Inference latency
No Game - - - -
Overwatch 22 % 97 % 146 fps -
PUBG 38 % 97 % 141 fps -
Assassin Creed : Odyssey 53 % 96 % 69 fps -
Inference on CPU using
Openvino CPU usage GPU usage Game frame rate Inference latency
No Game - - - 50.63ms
Overwatch 68 % ( 309.1 % ) 98 % ( 101.0 % ) 145 fps ( 99.3 % ) 57.97 ms ( 114 % )
PUBG 75 % ( 197.4 % ) 96 % ( 99.0 % ) 137 fps ( 97.2 % ) 59.92 ms ( 118 % )
Assassin Creed : Odyssey 84 % ( 158.5 % ) 96 % ( 100.0 % ) 69 fps ( 100.0 % ) 82.87 ms ( 164 % )
CPU : Intel i7-6700K GPU : Nvidia GTX 1080 ( ) : Rate of change
Inference on GPU using
Openvino
CPU usage GPU usage Game frame rate Inference latency
No Game - - - 43.87ms
Overwatch 38 % ( 172.7 % ) 97 % ( 100.0 % ) 119 fps ( 81.5 % ) 77.51 ms ( 177 % )
PUBG 66 % ( 173.7 % ) 97 % ( 100.0 % ) 113 fps ( 80.1 % ) 76.01 ms ( 173 % )
Assassin Creed : Odyssey 66 % ( 124.5 % ) 96 % ( 100.0 % ) 58 fps ( 84.1 % ) 88.93 ms ( 203 % )

19. @IntelSoftware @IntelGraphics 19
Choosing Solution for Deep Learning Inference
423.48 ms
34.75 ms
20.34 ms
2.21 ms
0
50
100
150
200
250
300
350
400
450
Naive cpp Numpy Tensorflow DL Inference Engine
Average Latency on CPU
(Lower numbers indicate better performance)
12X
faster
20X
faster
191X
faster

20. @IntelSoftware @IntelGraphics
Game Client
IK Solver
20
Introduction to Intel OpenVINO
MKL-DNN Plug-in
Inference engine
Model
Optimizer
IRTrain a
model
CPU
Deep Learning Deployment Toolkit
Deploy Pre-Trained Model
Tensorflow, Caffe, MXNet, ONNX

21. @IntelSoftware @IntelGraphics 21
Optimization : Cubic Hermite Spline [1/2]
! = 0
! =1
$%
$&
'%
'&
$ ! = 2!) − 3!, + 1 $% + !) − 2!, + ! '% + −2!) + 3!, $& + (!) − !,)'&

22. @IntelSoftware @IntelGraphics 22
Optimization : Cubic Hermite Spline [2/2]
! " = 2"% − 3"( + 1 !+ + "% − 2"( + " ,+ + −2"% + 3"( !- + ("% − "(),-
! " = "% "( " 1
2 −2 1 1
−3 3 −2 −1
0 0 1 0
1 0 0 0
!+
!-
,+
,-
Matrix calculations can be accelerated by Intel Openvino .
in matrix form

23. @IntelSoftware @IntelGraphics 23
Optimization : Batch inference
Serial computation
Parallel computation

24. @IntelSoftware @IntelGraphics
IK Solver
24
Optimization for multiple characters [1/2]
Character
Character
Character
Character
Character
Motion timer
updates frame number.
Trajectory
Network
Pose
Network
Next frame motion data
Batch Manager
Request next frame
motion data
Next motion frame

25. @IntelSoftware @IntelGraphics 25
Frequency of used batch size
(10, 14.5 )
(20, 15.3 )
(30, 9.0 )
(50, 7.7 )
(70, 5.5 )
0
2
4
6
8
10
12
14
16
18 1
3
5
7
10
12
15
20
25
29
40
50
54
70
90
110
130
150
170
190
210
230
250
270
300
370
400
520
540
1000
Frequencyofuse
Used Batch Size

26. @IntelSoftware @IntelGraphics 26
Finding optimal batch size that achieves maximum
throughput while maintaining limited latency
Set bs to initial batch size
Repeat until we obtain a reliable max latency
Start measuring latency
If number of inference > bs
infer multiple times
Else If number of inference <= bs
infer once and ignore unused space
Finish measuring latency
Update max latency
End
0
2
4
6
8
10
12
14
16
18
5 10 15 20 25 30 35 40 45 50
Batch size

28. @IntelSoftware @IntelGraphics 28
Quality Comparison
Our ApproachNumerical Approach

29. @IntelSoftware @IntelGraphics 29

31. @IntelSoftware @IntelGraphics 31
Deep Learning based Inverse Kinematics
§ Character animation becomes more realistic by solving IK.
§ Conventional approaches for full-body IK are complex or computationally
expensive.
§ Solving full-body IK using Intel Openvino is cost efficient and also flexible.
§ Process of “Climbing IK Solver”
1. Trajectory network
2. Curve fitting
3. Pose network
§ Gathered training data from manually crafted reference motion.

32. @IntelSoftware @IntelGraphics 32
Optimization using Intel® OpenVINO™ toolkit
§ CPU is suitable for computing inference on Game-Client.
§ Deep Learning Deployment Toolkit is high performance framework for computing
inference on a CPU.
§ Model Optimizer converts and optimizes the trained model to IR. On Game-Client,
Inference Engine loads this IR and it can now run inference on CPU using optimized
hardware plugin (MKL-DNN plugin).
§ Improved curve fitting performance by computing matrix form of cubic Hermit spline
with Inference Engine.
§ Batching inference task improves performance by parallelizing inference task by the
Inference Engine.
§ Finding optimal batch size to meet limited latency

33. Questions?
33
Yoon Dongwon
E-mail : dongwonyoon@ncsoft.com

34. @IntelSoftware @IntelGraphics
Reference
https://software.intel.com/en-us/openvino-toolkit
https://software.intel.com/en-us/articles/intel-software-engineers-assist-with-
unreal-engine-419-optimizations
https://intel.github.io/mkl-dnn/understanding_memory_formats.html
https://software.intel.com/en-us/articles/OpenVINO-Inference-Engine-
Optimization-Guide