Dmitry Spodarets, Сloud Сomputing (AWS) Lead at Provectus and CEO в FlyElephant.

1. Training Deep Learning Models
on Multi-GPUs Systems
Dmitry Spodarets
Odessa / Machine Learning Meetup / 27.02.2019

3. Multi-GPUs Systems

4. Distributed training
Data parallel vs model parallel
Faster or larger models?
Distributed TensorFlow training
https://www.youtube.com/watch?v=bRMGoPqsn20

5. Distributed training
Asynchronous vs Synchronous
Fast or precise?
Keras - multi-GPU training is not automatic :(
https://keras.io/utils/#multi_gpu_model

6. Bottlenecks
RAM / CPU I/O
Connections

7. NVLink (200 GB/sec)

8. NVSwitches (300 GB/sec)

9. Distributed training framework for
TensorFlow, Keras, PyTorch, and MXNet

10. Horovod Stack
● Plugs into TensorFlow via custom op mechanism
● Uses MPI for worker discovery and reduction coordination
● Uses NVIDIA NCCL for actual reduction on the server and across servers

11. Horovod Example - Keras
import keras
from keras import backend as K
import tensorflow as tf
import horovod.keras as hvd
# Initialize Horovod.
hvd.init()
# Pin GPU to be used to process local rank (one GPU per process)
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
config.gpu_options.visible_device_list = str(hvd.local_rank())
K.set_session(tf.Session(config=config))
# Build model…
model = ...
opt = keras.optimizers.Adadelta(1.0)
# Add Horovod Distributed Optimizer.
opt = hvd.DistributedOptimizer(opt)
model.compile(loss=keras.losses.categorical_crossentropy, optimizer=opt, metrics=['accuracy'])
# Broadcast initial variable states from rank 0 to all other processes.
callbacks = [hvd.callbacks.BroadcastGlobalVariablesCallback(0)]
model.fit(x_train, y_train, callbacks=callbacks, epochs=10, validation_data=(x_test, y_test))

12. Running Horovod
To run on a machine with 4 GPUs:
$ mpirun -np 4
-H localhost:4
-bind-to none -map-by slot
-x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH
-mca pml ob1 -mca btl ^openib
python train.py
To run on 4 machines with 4 GPUs each:
$ mpirun -np 16
-H server1:4,server2:4,server3:4,server4:4
-bind-to none -map-by slot
-x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH
-mca pml ob1 -mca btl ^openib
python train.py

13. Running Horovod
https://eng.uber.com/horovod/

14. Which GPU to Get for Deep Learning
http://timdettmers.com/2018/11/05/which-gpu-for-deep-learning/

15. Dmitry Spodarets
d.spodarets@flyelephant.net
https://flyelephant.net/gpu-dedicated-servers