This document summarizes PyTorch's approach to distributed data parallel training. It discusses the challenges of mathematical equivalence, a non-intrusive API, and high performance. It describes how PyTorch uses allreduce primitives and gradient bucketing to average gradients across processes. It evaluates the performance of distributed data parallel training with PyTorch using NCCL and Gloo backends on ResNet50 and BERT models.

1. PyTorch Distributed:
Experiences on Accelerating
Data Parallel Training
Shen Liy Yanli Zhaoy Rohan Varmay Omkar Salpekary
Pieter Noordhuis Teng Liy Adam Paszkez
Jeff Smithy Brian Vaughany Pritam Damaniay Soumith Chintalay
fshenli, yanlizhao, rvarm1, osalpekarg@fb.com,
pcnoordhuis@gmail.com, tengli@fb.com, adam.paszke@gmail.com,
fjeffksmith, bvaughan, pritam.damania, soumithg@fb.com
yFacebook AI zUniversity of Warsaw
A presentation by Rëza Habibi

2. Outlines
1- INTRODUCTION
2- PyTorch and Data Parallelism
3- SYSTEM DESIGN
4- EVALUATION
5-CONCLUSION
2

3. Introduction
3
• PyTorch
• Data parallelism
• Deep Neural Networks

4. Introduction
4
To provide a general distributed data parallel package, the challenges are
three-fold:
• Mathematical equivalence
• Non-intrusive and interceptive API
• High Performance

5. What is PyTorch?
5

6. 6
Data Parallelism
• DataParallel for single-process.
• Multithread data parallel training using multiple GPUs on the same
machine.
• Distributed DataParallel(DDP) for multi-process data parallel
training across GPUs and machines.
• RPC for general distributed model parallel training.

7. 7
AllReduce
• AllReduce is the primitive communication API used by
DistributedDataParallel to compute gradient summation across all
processes.
• It is supported by multiple communication libraries, including NCCL,
Gloo, and MPI.

8. SYSTEM DESIGN
8

9. 9
What is the requirements for API?
• Non-intrusive:
The API must be non-intrusive to applications.
• Interceptive:
The API needs to allow the implementation to
intercept various signals and trigger appropriate
algorithms promptly.

10. 10
What is the requirements for API?
10

11. Gradient Reduction
11

12. Gradient Bucketing
12

13. 13
Collective Communication

14. 14

15. EVALUATION
15

16. CONCLUSION
16
This paper explained the design and implementation of
the distributed data parallel module in PyTorch v1.5, and
conducted performance evaluations on NCCL and Gloo back-end using ResNet50 and
BERT models.