Introduction into distributed training of AI/ML algorithms Hannes Fassold, JOANNEUM RESEARCH

1. 1
Distributed training
and frameworks
AI4Media – WP3 Workshop, 2021-05-04
Hannes Fassold, JOANNEUM RESEARCH - DIGITAL

2. Introduction & Motivation
• Motivation
• State-of-the-art DL models get bigger, as well as the datasets
on which they are trained on
• GPT-3 model (SoA text processing / NLP model)
• model size is 175 billion parameters
• trained on 500 billion tokens
• So training time gets up and up …
• Doing a single training is usally not sufficient
• Hyperparameter tuning is usually done
• Adapt learning rate, momentum, ..
• Might want to experiment with the network architecture
• Network architecture search
2

3. Definition
• In distributed training the workload to train a model is split up
and shared among multiple processors (workers / nodes)
• Can be a “cluster” of few workers or up to several hundreds
• Usually, each worker is equipped with 2-8 GPUs
• Optimal case is linear scaling
• Training time is inversely proportionally to number of workers
• Usually not achieved due to adverse affects for larger clusters
• Serial parts (which can not be paralllized) get more prominent
• Communication cost (between workers) may rise dis-proportionally
• Distributed training allows training “from scratch” on a huge dataset in minutes
• E.g. Image classification model can be trained in 1.5 minutes
on ImageNet dataset, employing 512 GPUs
3

4. Data parallelism versus Model parallelism
• Data parallelism
• Training data is split into chunks
• Each worker processes a chunk
and updates model
• Advantages
• Can be applied to any model
• Disadvantages
• Each worker must have enough (GPU)
memory to hold the whole model
• Updated model must be communicated
regularly to all workers
4

5. Data parallelism versus Model parallelism
• Model parallelism
• Model is split into several parts
• Each worker processes
its respective model part
• Advantages
• Support for large models which
do not fit in GPU memory (e.g. NLP models)
• Disadvantages
• One has to find an efficient split of the model,
depends on model structure and number of workers
5

6. System architecture
• System architecture describes how the model parameter
updates of the different workers are performed
• Centralized system architecture
• Workers periodically report their model
updates to one (or more) parameter servers
• Decentralized system architecture
• Workers exchange the model updates
directly via an allreduce operation
• Topology of the allreduce operation is critical
• Fully connected => Communication cost O(n^2) !
• Usually using high-performance topologies like
ring, tree, butterfly etc.
6

7. Synchronization strategies
• Different strategies to synchronize the model parameters between all workers
• Synchronous
• Sync of model parameters is done after each iteration (mini-batch)
• Prone to straggler problem (slowest worker delays all workers)
• Bounded asynchronous
• Workers may train on model parameters which are ‘a few iterations’ old
• Asynchronous (e.g. Hogwild algorithm)
• Workers update their model completely independent from others
• Difficult to reason about model convergence
• Lost update problem: new parameters written by
worker A could be overwritten by worker B
7

8. Distributed training frameworks
• Main DL frameworks (PyTorch, TensorFlow, MXNet)
• Provide mainly support for a single node (but using multiple GPUs)
• Horovod (Uber)
• PyTorch, TensorFlow, Keras, MXNet
• Data parallelism and limited model parallelism
• Fairscale (Facebook)
• PyTorch
• Data parallelism and limited model/pipeline parallelism
• Deepspeed (Microsoft)
• PyTorch
• Data parallelism and model/pipeline parallelism
• Gradient compression (1-bit Adam/LAMB), …
8

9. 9

10. 10

11. 11

12. 12

13. 13

14. 14