The document presents an overview of data-driven AI applications in entertainment and healthcare, focusing on advancements in computer graphics, human modeling, and medical image analysis. It highlights the integration of machine learning techniques for creating realistic virtual humans and improving medical diagnostics, specifically in lung cancer detection. The conclusion emphasizes the potential for further innovation through the combination of data-driven methods and model-based analysis in both fields.

1. Data-Driven AI
for Entertainment and Healthcare
Demetri Terzopoulos
UCLA Distinguished Professor & Chancellor’s Professor of Computer Science
Co-Founder & Chief Scientist, VoxelCloud, Inc.

2. Visual Computing
• Computer graphics (synthesis)
• Computer vision (analysis)
Talk Overview
1. AI/ML in computer graphics
- Human Modeling and Animation
2. AI/ML in computer vision
- Medical Image Analysis
3. The future & questions
 Images & Videos
 Mathematical models
Computer
Models
Images /
Videos
Computer
Vision
Computer
Graphics

3. “Final Fantasy: The Spirits Within”
(Square Pictures, Inc., 2001)
Virtual Humans in Movies and Games
These characters are neither autonomous nor intelligent
“Metal Gear Solid” game

4. Motion Capture Technology
3D tracking of body-attached IR reflectors

5. Physics
The Artificial Life Approach
Comprehensive computational models of humans and animals
• Modeling the body and mind
Biomechanics / Locomotion
Perception
Behavior
Learning
Cognition
The Artificial Life Modeling Pyramid
Artificial
Intelligence

6. Realistic Biomechanical Modeling of the Human Body
• Almost all the articular bones and skeletal muscles
– 75 bones (165 DOFs), 846 muscles
• Volumetric finite element soft tissue model
– 354K tetrahedral elements

7. 3D Musculature Model

8. Biomechanical Actuator Model

9. Upper Body Actuators

10. Biomechanical Simulation

11. Biomechanical Simulation

12. How Can we Control Complex, State-of-the-Art
Biomechanical Human Models Like These?
The natural way is through neuromuscular control
• The advanced virtual human models can learn to control themselves
like real humans do!
– This is accomplished using massive quantities of training data
– The training data are synthesized by the human models themselves

13. Anatomical Structure of the Neck

14. Skeletal System
• 7 cervical vertebrae and a skull
coupled by 3-DOF joints
• Ligaments/disks
passive joint springs
• Equations of motion
0)qb(q,qM(q)  
moment
arm matrix
active
muscle
force
neural
inputmass
a),q(q,P(q)f)qb(q,qM(q) c
 
gravity, Coriolis,
passive elastic
forces

15. Biomechanical Neck Model
Total of 72 anatomically-based muscle actuators
in 3 layers
48 deep muscles
(16 longus colli, 16 erector, 16 rotator)
6 muscles at each joint increase controllability
12 intermediate muscles
(scalerius: 4 anterior, 4 posterior, 4 capitis)
12 superficial muscles
(2 sternomastoid, 2 cleidooccipital, 8 trapezius)
The big challenge is co-actuation and control

16. What Would Leonardo da Vinci Think of This?

17. Neuromuscular Control of the Musculoskeletal Model
muscles
muscle
contraction
forces
skeletal
system
environment
gravity,
applied
force
bio-
mechanical
face
head pose
voluntary
controller
feedfwd
signal
setpoint
signal
proprioceptive feedback
(pose, velocity of head)
reflex
controller
muscle
activation
levels
muscle feedback
(strain/strain rate)
ts
tf
tu
tt s
Trained Deep Neural Networks

18. • Set random target pose
Training the Neural Networks

19. • Using inverse kinematics, compute desired muscle lengths
• Using inverse dynamics, compute muscle activations to
achieve desired muscle lengths (under gravity)
Training the Neural Networks

20. Training the Neural Networks
Target Pose
Activations
• Repeat with about 20K random target poses

21. Learned Neck-Head-Face-Eye Behavior

22. Realistic Animation of Swimming
Biomechanical human model immersed in fluid
• Advanced multi-physics simulation

23. Learning to Swim From Examples

24. Close-up View of the Biomechanical
Swimmer Model

25. Medical Image Analysis
Deformable models: A powerful, model-based MIA
approach
• Segmentation
• Registration
• Shape reconstruction and modeling
• Motion estimation and analysis

26. Tongue Tracking in Ultrasound
[Kambhamettu et al]
Using an Active Contour Model

27. Interactive Serial Segmentation of
Biomedical Images
EM neuronal tissue sections

28. Retinal Angiogram Segmentation

29. Level-Set Active Contour Model
Gland Segmentation

30. 𝑝1
𝑝2
𝑝3
𝑝4
𝑝5
𝑝 𝑁
Data-Driven Learning of
Statistical Deformable Models
Training data: Numerous expert-segmented images
2N
p
.
x1
yN
0
...
.
.
.......
.
.... ....
...
....
𝑝1
𝑝2
𝑝3
𝑝4
𝑝5𝑝 𝑁

31. Active Shape Model (ASM)
[Cootes & Taylor, 1992]
Labeled model Trained model Fitting the model
in training image
PC 1
PC 2

32. Intelligent, Model-Based Medical Image Analysis:
Deformable Organisms
Corpus Callosum Organism
fornix
2
3N-2
1
genu
spleniumrostrum
body
NN-1
upper/right
lower/left
fornix
2
3N-2
1
genu
spleniumrostrum
body
NN-1
upper/right
lower/left

33. Deformable Organism’s AI Planner

34. Multiple Deformable Organisms

35. Driving Forces in Medical Imaging
Exploding data volumes
• Procedure volume growth
• Imaging technology advances
• Manpower shortage
Evidence-based diagnosis
• Early-stage disease screening
• Longitudinal tracking
• Experience shortage
500
25,000
2012 2020
Medical Data (Petabytes)*
14,400+
Possible diagnoses (WHO)
12,000,000+
Misdiagnoses in the US per year**
**Singh et al 2012

36. Data-Driven,
Machine Learning Approaches
Deep Learning is “revolutionizing”
computer vision and other fields
• It is playing an increasingly important role
in Medical Image Analysis

37. AI and Deep Learning in Medicine
Automated, accelerated, and accurate insight from
massive medical data
• Lower cost
• Higher efficiency
• Fewer misdiagnoses

38. Case Study: Lung Cancer
Deadliest cancer worldwide
• One in five cancer deaths is from lung cancer
Early detection is critical
18,000,000+
New lung cancer cases per year
15,000,000+
Deaths from lung cancer per year
49% 45%
30% 31%
14%
5% 1%
0%
25%
50%
75%
IA IB IIA IIB IIIA IIIB IV
5-yr survival by stage
>85%
Diagnosed at late stage
>50%
Die within one year of diagnosis

39. Deep Convolutional Neural Networks
Applied to Lung Nodule Detection/Analysis

40. Imaging-Based Medical Analytics and Diagnostics
AI in Medicine
NewMargin
Ventures
Sequoia
CapitalTencent Holdings

41. Lung Cancer Screening Platform
Imaging
Medical data
Pathology
Web based interface
Secure data storage
Cloud AI engine
Reports and insights
Local archive

42. Malignancy Assessment with AI
Performance trajectory
• 04/2016: >90% consistency with expert panel
• 06/2016: >85% accuracy vs. ground truth information
• 12/2017: pushing the limits of CT-based diagnosis

43. Additional Use Cases
Coronary heart disease
Diabetic retinopathy

44. Conclusion
Data-driven AI has enormous potential in the
entertainment and healthcare industries
• Much initial success of data-driven machine learning approaches to
control advanced biomechanical models of humans and other animals
• The most promising avenue of future innovation in medical imaging is
data-driven machine learning methods working in combination with
powerful model-based image analysis methods
• Much more research must be done to realize the full potential in real-
world industrial applications

45. Thank You !
Terzopoulos.com