AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Data & AI Platform Tech Lead, @Alluxio) - Siyuan Sheng (Senior Software Engineer, @Alluxio) Speed and efficiency are two requirements for the underlying infrastructure for machine learning model development. Data access can bottleneck end-to-end machine learning pipelines as training data volume grows and when large model files are more commonly used for serving. For instance, data loading can constitute nearly 80% of the total model training time, resulting in less than 30% GPU utilization. Also, loading large model files for deployment to production can be slow because of slow network or storage read operations. These challenges are prevalent when using popular frameworks like PyTorch, Ray, or HuggingFace, paired with cloud object storage solutions like S3 or GCS, or downloading models from the HuggingFace model hub. In this presentation, Lu and Siyuan will offer comprehensive insights into improving speed and GPU utilization for model training and serving. You will learn: - The data loading challenges hindering GPU utilization - The reference architecture for running PyTorch and Ray jobs while reading data from S3, with benchmark results of training ResNet50 and BERT - Real-world examples of boosting model performance and GPU utilization through optimized data access

1. Improve Speed and GPU
Utilization for Model
Training & Serving
Lu Qiu, Siyuan Sheng

2. AI Platform Tech Lead
& Open Source PMC Maintainer
@ Alluxio
linkedin.com/in/luqiu-ai
Lu Qiu
2
Senior Software Engineer
@ Alluxio
www.linkedin.com/in/siyuan-sheng
Siyuan Sheng
2

3. Open Source Started From UC Berkeley AMPLab
1000+ contributors &
growing
4000+ Git Stars
Apache 2.0 Licensed
Million+ Download;
GitHub’s Top 100 Most
Valuable Repositories
Out of 96 Million
Join the conversation
on Slack
slackin.alluxio.io
#9
Most critical open
source Java projects
(Google OpenSSF)

4. 4
Alluxio Data Platform
High Performance data access, unified global view

5. ALLUXIO 5
COMPANIES USING ALLUXIO
INTERNET
PUBLIC CLOUD PROVIDERS
GENERAL
E-COMMERCE
OTHERS
TECHNOLOGY FINANCIAL SERVICES
TELCO & MEDIA
LEARN MORE

6. Inefficiencies in AI infrastructure
● Prolonged AI Model Lifecycle
● Underutilized GPUs
GPU is waiting for data to be ready
AI is all about getting information
from Data
6

7. Waiting for Data to be Ready for AI
7
Data Loading
Preprocessing
Training
t=0
GPU IDLE
Low GPU Utilization Rate + Long Model Lifecycle
t=2 t=4 t=6

8. Ray/PyTorch: Streamlined Operations
t=0
GPU IDLE
Increase GPU utilization + Faster Model Lifecycle
t=2 t=4 t=6
Data Loading
Preprocessing
Training

9. ● Separation of Compute and Storage
● Large Data Volumes
● Crowded Networks
● Slow Data Transfer
● Storage Request Rate Limit or Outages
Data Loading Bottleneck

10. Data Loading become the Bottleneck
10
Data Loading
Preprocessing
Training
t=0
GPU IDLE
Low GPU Utilization Rate + Long Model Lifecycle
t=2 t=4 t=6

11. While using Ray/PyTorch for training…
Performance & Cost Implications are
● You might load the entire dataset again and again for each epoch
● You cannot cache the hottest data among multiple training jobs
automatically
● You might be suffering from a cold start every time
Data Loading Bottleneck

12. Data Loading Bottleneck for Each Epoch
12
t=0
GPU IDLE
t=2 t=4 t=6 t=8 t=10 t=12
EPOCH 0 EPOCH 1

13. Storage CPU Machine GPU Machine
Alluxio Ray
Distributed Caching Data Preprocessing
Storage
Reduce data transfer
& storage cost
Speed up data loading & preprocessing for AI training
Data
Speed up data loading
& Preprocessing
Compute
Increase GPU
Utilization Rate
Ray

14. Storage Ray
Ray + Alluxio: Speed up data loading
Alluxio
FASTER IS
BETTER
Often
Only when necessary
TIME CONSUMING

15. Ray in I/O bottlenecked workloads
15
Data Loading
Preprocessing
Training
t=0
GPU IDLE
Low GPU Utilization Rate + Long Model Lifecycle
t=2 t=4 t=6

16. Ray + Alluxio: Speed up data loading
t=0
GPU IDLE
Increase GPU utilization + Faster Model Lifecycle
t=2 t=4 t=6
Data Loading from Alluxio
Preprocessing
Training

17. High Scalability
● Cache 10 billion+ objects with an architecture that scales-out
horizontally without single node dependency
Performance
● Single-node storage with 50+ million objects per node
● Workload-specific optimizations for ML training & inference
○ Low latency - < 1ms
○ High throughput - hundreds of GB/s per Alluxio cluster
Stability and Reliability
● Automatic fallback to data lake storage for masking any
failures to due to capacity or other reasons
Alluxio Design

18. Ray + Alluxio Fsspec: Easy Usage
# Import fsspec & alluxio fsspec implementation
import fsspec
import s3fs
from alluxiofs import AlluxioFileSystem
alluxio = AlluxioFileSystem(etcd_host=host, target_protocol= "s3")
# Ray read data from Alluxio using S3 URL
ds = ray.data.read_images("s3://ai-ref-arch/imagenet-full/train",
filesystem=alluxio)
See more in: https://github.com/fsspec/alluxiofs
Using Alluxiofs instead of S3fs
Original S3 URL

19. 19
Cloud Native Distributed Caching System
Read
Write
Local
Cache
Worker Selection
Consistent
Hash Ring
ETCD
Membership Service
Alluxio Worker 0
Alluxio Worker 1
Alluxio Worker 2
s3fs / gcsfs / huggingfacefs …..
Alluxio FSSpec + Alluxio System Cache
Stateless, Easy scaling, Fault Tolerant & Highly Available

20. Ray + Alluxio + Parquet - Multi-node
● Comparison
○ Ray + Same Region S3
○ Ray + Alluxio + Same Region S3
● Dataset
○ 200 MiB files, adds up to 60 GiB
○ Images in Parquet Format
● Script
○ Ray nightly multi-node train
benchmark
○ 28 train workers
With Alluxio
Without Alluxio

21. Twitter.com/alluxio
Linkedin.com/alluxio
Website
www.alluxio.io
Slack
https://alluxio.io/slack
@
Social Media
Github
https://github.com/Alluxio
Lu Qiu
www.linkedin.com/in/luqiu-ai
Siyuan Sheng
www.linkedin.com/in/siyuan-sheng
Scan QR code to access resources ->