PR-129: Horovod: fast and easy distributed deep learning in TensorFlow
•Download as PPTX, PDF•
2 likes•576 views
- Title: Horovod: fast and easy distributed deep learning in TensorFlow - Paper: https://arxiv.org/abs/1802.05799 - Youtube: https://youtu.be/8zQECRiONAo Taekmin Kim, http://github.com/tantara
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to PR-129: Horovod: fast and easy distributed deep learning in TensorFlow
Similar to PR-129: Horovod: fast and easy distributed deep learning in TensorFlow (20)
Recently uploaded
Recently uploaded (20)
PR-129: Horovod: fast and easy distributed deep learning in TensorFlow
- 1. Horovod: fast and easy distributed deep learning in Tensorflow PR-129 Taekmin Kim Dec 23, 2018
- 2. On single machine ● Training ResNet-50 on TPU ○ Batch Size: 1024 ○ Accuracy: 76%(Top-1) ○ Training Time: 17hours ● Training Faster RCNN on 8GPUs ○ Batch Size: 8~16 ■ 1~2 per GPU
- 3. Why large-scale training? ● Better accuracy ○ E.g. Object detection ○ Group Normalization(ECCV 2018) ● Fast training ○ ResNet-50 ■ 6.6 minutes, 75.8%(Top-1) ■ 64k per mini-batch, 2048 GPUs Group Normalization
- 4. Distributed Tensorflow: How to use https://www.tensorflow.org/deploy/distributed
- 5. Distributed Tensorflow: How to use https://www.tensorflow.org/deploy/distributed
- 9. Distributed Tensorflow ● Parameter-Worker Architecture ● Issues ○ need to decide # parameter servers, workers ○ difficult to edit configurations ■ Codes, ...
- 10. Issues ● Communication Cost ○ Average gradients ○ Update weights ● Others ○ GPU/CPU ○ Network ○ Storage
- 11. ring-allreduce http://research.baidu.com/bringing-hpc-techniques-deep-learning Add Add 2*(N-1) iterations ● N-1: Add ● N-1: Send & Receive Send & Receive Send & Receive
- 12. Horovod ● Stand-alone package ○ pip install horovod ● ring-allreduce ○ NCCL ● Horovod Timeline ○ Chrome extension ● Tensor Fusion ● MPI
- 13. Horovod Timeline
- 14. Tensor Fusion ● 65% improvement in performance ● Algorithm:
- 15. Example
- 16. Example
- 17. Results: Inception V3, ResNet-101
- 18. Results: with RDMA networking ● Support RDMA(Remote Direct Memory Access) networking ○ e.g. InfiniBand
- 19. Current Horovod ● Support ○ Tensorflow ■ Estimator API ○ Keras ○ PyTorch https://github.com/uber/horovod
- 20. Summary ● Communication ○ ring-allreduce ● Libraries ○ NCCL, MPI ● Benefits ○ Large-scale training(e.g. AutoML) ● Future work ○ PyTorch 1.0 ■ torch.distributed ○ Tensorflow 2.0
- 21. Related Work ● Training ○ Communication Cost ■ Deep Gradient Compression ■ Training ImageNet in Four Minutes ● using mixed precision ○ RNN, RL ■ Dynamic Control Flow ● Inference ○ Low Latency RNN Inference with Cellular Batching
Editor's Notes
- In the ring-allreduce algorithm, shown on Figure 4, each of N nodes communicates with two of its peers 2 ∗ (N − 1) times. During this communication, a node sends and receives chunks of the data buffer. In the first N − 1 iterations, received values are added to the values in the node’s buffer. In the second N − 1 iterations, received values replace the values held in the node’s buffer. Patarasuk and Yuan in [9] suggest that this algorithm is bandwidth-optimal, meaning that if the buffer is large enough, it will optimally utilize the available network. In addition to being network-optimal, the allreduce approach is much easier to understand and adopt. Users utilize a Message Passing Interface (MPI) [10] implementation such as Open MPI [11] to launch all copies of the TensorFlow program. MPI then transparently sets up the distributed infrastructure necessary for workers to communicate with each other. All the user needs to do is modify their program to average gradients using an allreduce() operation.
- we found that RDMA did not significantly improve our performance and only achieved a three to four percent increase over TCP networking. RDMA, however, did help Horovod exceed 90 percent scaling efficiency on both mode the VGG-16 model experienced a significant 30 percent speedup when we leveraged RDMA networking.