Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Data-Driven AI for Entertainment and Healthcare
1. Data-Driven AI
for Entertainment and Healthcare
Demetri Terzopoulos
UCLA Distinguished Professor & Chancellor’s Professor of Computer Science
Co-Founder & Chief Scientist, VoxelCloud, Inc.
2. Visual Computing
• Computer graphics (synthesis)
• Computer vision (analysis)
Talk Overview
1. AI/ML in computer graphics
- Human Modeling and Animation
2. AI/ML in computer vision
- Medical Image Analysis
3. The future & questions
Images & Videos
Mathematical models
Computer
Models
Images /
Videos
Computer
Vision
Computer
Graphics
3. “Final Fantasy: The Spirits Within”
(Square Pictures, Inc., 2001)
Virtual Humans in Movies and Games
These characters are neither autonomous nor intelligent
“Metal Gear Solid” game
5. Physics
The Artificial Life Approach
Comprehensive computational models of humans and animals
• Modeling the body and mind
Biomechanics / Locomotion
Perception
Behavior
Learning
Cognition
The Artificial Life Modeling Pyramid
Artificial
Intelligence
6. Realistic Biomechanical Modeling of the Human Body
• Almost all the articular bones and skeletal muscles
– 75 bones (165 DOFs), 846 muscles
• Volumetric finite element soft tissue model
– 354K tetrahedral elements
12. How Can we Control Complex, State-of-the-Art
Biomechanical Human Models Like These?
The natural way is through neuromuscular control
• The advanced virtual human models can learn to control themselves
like real humans do!
– This is accomplished using massive quantities of training data
– The training data are synthesized by the human models themselves
14. Skeletal System
• 7 cervical vertebrae and a skull
coupled by 3-DOF joints
• Ligaments/disks
passive joint springs
• Equations of motion
0)qb(q,qM(q)
moment
arm matrix
active
muscle
force
neural
inputmass
a),q(q,P(q)f)qb(q,qM(q) c
gravity, Coriolis,
passive elastic
forces
15. Biomechanical Neck Model
Total of 72 anatomically-based muscle actuators
in 3 layers
48 deep muscles
(16 longus colli, 16 erector, 16 rotator)
6 muscles at each joint increase controllability
12 intermediate muscles
(scalerius: 4 anterior, 4 posterior, 4 capitis)
12 superficial muscles
(2 sternomastoid, 2 cleidooccipital, 8 trapezius)
The big challenge is co-actuation and control
17. Neuromuscular Control of the Musculoskeletal Model
muscles
muscle
contraction
forces
skeletal
system
environment
gravity,
applied
force
bio-
mechanical
face
head pose
voluntary
controller
feedfwd
signal
setpoint
signal
proprioceptive feedback
(pose, velocity of head)
reflex
controller
muscle
activation
levels
muscle feedback
(strain/strain rate)
ts
tf
tu
tt s
Trained Deep Neural Networks
18. • Set random target pose
Training the Neural Networks
19. • Using inverse kinematics, compute desired muscle lengths
• Using inverse dynamics, compute muscle activations to
achieve desired muscle lengths (under gravity)
Training the Neural Networks
20. Training the Neural Networks
Target Pose
Activations
• Repeat with about 20K random target poses
25. Medical Image Analysis
Deformable models: A powerful, model-based MIA
approach
• Segmentation
• Registration
• Shape reconstruction and modeling
• Motion estimation and analysis
26. Tongue Tracking in Ultrasound
[Kambhamettu et al]
Using an Active Contour Model
35. Driving Forces in Medical Imaging
Exploding data volumes
• Procedure volume growth
• Imaging technology advances
• Manpower shortage
Evidence-based diagnosis
• Early-stage disease screening
• Longitudinal tracking
• Experience shortage
500
25,000
2012 2020
Medical Data (Petabytes)*
14,400+
Possible diagnoses (WHO)
12,000,000+
Misdiagnoses in the US per year**
**Singh et al 2012
36. Data-Driven,
Machine Learning Approaches
Deep Learning is “revolutionizing”
computer vision and other fields
• It is playing an increasingly important role
in Medical Image Analysis
37. AI and Deep Learning in Medicine
Automated, accelerated, and accurate insight from
massive medical data
• Lower cost
• Higher efficiency
• Fewer misdiagnoses
38. Case Study: Lung Cancer
Deadliest cancer worldwide
• One in five cancer deaths is from lung cancer
Early detection is critical
18,000,000+
New lung cancer cases per year
15,000,000+
Deaths from lung cancer per year
49% 45%
30% 31%
14%
5% 1%
0%
25%
50%
75%
IA IB IIA IIB IIIA IIIB IV
5-yr survival by stage
>85%
Diagnosed at late stage
>50%
Die within one year of diagnosis
41. Lung Cancer Screening Platform
Imaging
Medical data
Pathology
Web based interface
Secure data storage
Cloud AI engine
Reports and insights
Local archive
42. Malignancy Assessment with AI
Performance trajectory
• 04/2016: >90% consistency with expert panel
• 06/2016: >85% accuracy vs. ground truth information
• 12/2017: pushing the limits of CT-based diagnosis
44. Conclusion
Data-driven AI has enormous potential in the
entertainment and healthcare industries
• Much initial success of data-driven machine learning approaches to
control advanced biomechanical models of humans and other animals
• The most promising avenue of future innovation in medical imaging is
data-driven machine learning methods working in combination with
powerful model-based image analysis methods
• Much more research must be done to realize the full potential in real-
world industrial applications
