This document discusses the data storage challenges of training artificial intelligence systems for autonomous vehicles. It notes that autonomous vehicles require massive amounts of sensor data from training vehicles to develop accurate models. Enterprise storage systems struggle to meet the scale, throughput, and affordability needs of autonomous vehicle AI training workloads. In contrast, Cray's ClusterStor storage solution is designed for these extreme AI workloads, scaling to hundreds of petabytes with sustained high throughput performance. ClusterStor provides a reliable, efficient, and scalable platform for autonomous vehicle customers to develop AI models.

1. © 2019 Cray Inc.
TRAINING AI FOR
AUTONOMOUS VEHICLES:
W H E R E E N T E R P R I S E S T O R A G E H I T S T H E W A L L
Per Nyberg I Uli Plechschmidt I London, March 28 2019

2. © 2019 Cray Inc.
Pharma Energy
Autonomous
Technologies Insurance National AI Initiative Scientific Research
Top 5 Global
Pharma
Major Integrated
Oil and Gas
Company
Fortune 20 Global
Technology
Company
Leading Personal
Lines Property and
Casualty Insurer
Alan Turing
Institute
(UK National Data Science
and Artificial Intelligence
Center)
National Energy
Research Scientific
Computing Center
Cray CS-Storm Dense
GPU Cluster
Cray CS-Storm and
XC50 w. Urika-XC
Cray CS-Storm Dense
GPU Cluster
Cray CS-Storm Dense
GPU Cluster
Cray Urika-GX
Cray XC40 “Cori”
Supercomputer
Supporting core
research and
development in areas
including
cheminformatics and
large machine learning
workloads such as
CryoEM
Applying machine and
deep learning
techniques to identify
previously undetected
features in subsurface
seismic imaging
Machine and deep
learning workloads to
develop systems for
connected cars and
autonomous
technologies
Application of deep
learning for the
automation of claims
processing
Enabling the
development of
advanced applications
across a number of
fields including
engineering and
technology, defense and
security, and smart
cities
Supporting machine and
deep learning across a
broad set of science
disciplines
Record breaking deep
learning results in
Climate, High Energy
Physics and Astronomy
Industry Academia Government
AI IS PERVASIVE ACROSS OUR CUSTOMER BASE

3. © 2019 Cray Inc.
WHAT IS AUTONOMOUS DRIVING?
3

4. © 2019 Cray Inc.
Massive Simulation Using AI/ML @ Extreme Scale
MASSIVE SIMULATION TASK
* www.rand.org/content/dam/rand/pubs/research_reports/RR1400/RR1478/RAND_RR1478.pdf
“Autonomous vehicles need to be driven more than
11 billion miles to be 20% better than humans.
With a fleet of 100 vehicles, 24 hours a day, 365 days a
year, at 25 miles per hour, this would take 518 years*.”
As we cannot wait
518 years
4

5. © 2019 Cray Inc.
HIGH-LEVEL DEVELOPMENT PROCESS
Sensor data from
training cars
Model
Engineering
Autonomous
VehiclesIngest BIG Data Trained Model
Data logger in
training car
AI infrastructure in
data center
Control unit in
self-driving car
Data scientists
5

6. © 2019 Cray Inc.
AUTONOMOUS VEHICLES: AI WORKFLOW
Search
analysis
Training
data
Report
results
Test
data
Data
preprocessing
Ingest
data
Model
training
Re-
simulation
Model
testing
Model
deployment
Train Test Loop
Data-intensive
Compute-intensive
6

7. © 2019 Cray Inc.
AUTONOMOUS VEHICLES: THE TIP OF THE PYRAMID
Extreme
Moderate
StorageRequirements
Level 1: Workgroup AI
Level 2: Departmental AI
Level 3: Divisional AI
Level 4:Training AI for Autonomous Vehicles
7

8. © 2019 Cray Inc.
EXAMPLES OF TRAINING CARS
Up to 5 Terabyte/hour per training car running 8 hours x 100 training cars =
4 Petabyte per day
a
100 training cars running 5 working days per week =
20 Petabyte per week
8

9. © 2019 Cray Inc.
EXTREME STORAGE REQUIREMENTS
Example: European Car Manufacturer
Base
Case
Scenario
100% Growth
“Buffer”
Translation to
Storage
Requirements
Useable capacity in
single namespace
130
Petabyte
260
Petabyte
About 170 PB (340 PB) raw capacity
Re-simulation of 100%
of the data in ….. hours
100
hours
50
hours
• 130 PB in 100 hours = 361 GB/sec
• 130 PB in 50 hours = 722 GB/sec
• 260 PB in 100 hours = 722 GB/sec
• 260 PB in 50 hours = 1.4 TB/sec
9
~ 470 HD movies
per second
~ 87 million
HD movies

10. © 2019 Cray Inc.
STORAGE CHALLENGE TRIANGLE
Scalability
(in Petabyte)
Throughput
(in Gigabyte/second)
Affordability
(%age of storage spending of total Autonomous Driving budget)
10

11. © 2019 Cray Inc.
ENTERPRISE STORAGE IS CHALLENGED
Scalability
(to 100’s of
Petabyte capacity)
Throughput
(of 100’s of
Gigabyte/sec)
Affordability
(achieve the requirements
at lowest possible TCO)
Classic Block/File Storage
(e.g. Dell EMC PowerMax, NetApp FAS, HPE
3PAR)
Scale-Out NAS
(e.g. Dell EMC Isilon, NetApp Clustered
ONTAP)
Distributed Object Storage
(e.g. Scality, Cloudian, Dell EMC ECS, IBM
COS, Pure Storage Flashblade etc.)
Hadoop-based Storage
(e.g. Cloudera/Hortonworks, MapR, Oracle Big
Data Appliance etc.)
Public Cloud Storage
(e.g. Amazon AWS S3, Microsoft Azure,
Google Cloud Storage etc.)
11

12. © 2019 Cray Inc. 12
SUPERCOMPUTING STORAGE WORKS!
Storage choices of the Top 100* Global Supercomputers
27
20
14
10
8 7
5 5
3
1
0
5
10
15
20
25
30
Cray
ClusterStor
DDN
EXAScaler
NetApp Chinese Lenovo
DSS-G
DDN
GRIDScaler
Fujitsu
FEFS
IBM ESS Unknown Supermicro
#1
*November 2018 top500.org list
#1File System in the Top 100
with 76% share
LUSTRE
OTHERS
Storage System in the Top 100
with 27% share

13. © 2019 Cray Inc.
WHY DID UBER ATG SWITCH TO LUSTRE?
http://wiki.lustre.org/images/e/ed/LUG2018-Lustre_at_Uber-Cobb_Xiong.pdf 13

14. © 2019 Cray Inc. 14
HIGH LEVEL SOLUTION ARCHITECTURE
Hadoop CPU nodes
for data pre-processing
CPU & dense GPU nodes
for deep learning
Hardware-in-the-loop (HIL)
simulators
(data access with standard CIFS/NFS)Lustre client Lustre client
90%+ READ 90%+ READ
Scalable CIFS/NFS Gateway
READWRITE
Cray ClusterStor L300 Lustre Global Namespace
Scaling to 100’s of Petabyte and Terabyte/second

15. © 2019 Cray Inc.
ENGINEERED RACK-SCALE HPC STORAGE
Data Network Switches (IB, OPA, GbE)
Management Network Switches
System Management Unit (SMU)
Metadata Management Unit (MMU)
Base Rack Expansion
Rack
Expansion
Rack
Expansion
Rack
Expansion
Rack
Scalable Storage Units (SSU)
with embedded HA OSS*:
5U84 5U84 2U24
L300NL300 L300F
All HHD:
Large,
sequential
I/O
All Flash:
Small,
random,
I/O
Hybrid
(SSD/HDD):
Mixed I/O
*Object Storage Servers
Mix & Match
Ex
15

16. © 2019 Cray Inc.
CLUSTERSTOR FOR AUTONOMOUS VEHICLES
Performance Efficiency
Achieve requirements with the most efficient architecture
requiring less storage gear (drives, enclosures, racks)
More budget left to spend on data science
Engineered Solution
Pre-integrated, tested, tuned,
and shipped ready to deploy.
Days instead of weeks
to implementation
Reliability
No single point of failure
architecture and parity
de-clustered RAID
Less downtime and
(much) faster rebuilds
Scalability
Sustained linear performance
when adding capacity up to
hundreds of Petabyte
Predictable application
performance @ scale
Management & Support
Integrated system management, System Snapshot
Analyzer (SSA)/Call home, hardware monitoring
with health alerts, API, HPC storage analytics
on job level with View for ClusterStor
Less downtime and faster time-to-problem
resolution
Unique
Values
1
34
5 2
16

17. © 2019 Cray Inc.
THANK YOU
W h e n e v e r y t h i n g e l s e b r e a k s ,
i t i s t i m e t o c o n s i d e r y o u r f i r s t C r a y s y s t e m !