In this talk, we evaluate training of deep recurrent neural networks with half-precision floats on Pascal and Volta GPUs. We implement a distributed, data-parallel, synchronous training algorithm by integrating TensorFlow and CUDA-aware MPI to enable execution across multiple GPU nodes and making use of high-speed interconnects. We introduce a learning rate schedule facilitating neural network convergence at up to O(100) workers. Strong scaling tests performed on GPU clusters show linear runtime scaling and logarithmic communication time scaling for both single and mixed precision training modes. Performance is evaluated on a scientific dataset taken from the Joint European Torus (JET) tokamak, containing multi-modal time series of sensory measurements leading up to deleterious events called plasma disruptions. Half-precision significantly reduces memory and network bandwidth, allowing training of state-of-the-art models with over 70 million trainable parameters while achieving a comparable test set performance as single precision.

1. Alexey Svyatkovskiy (Microsoft/Princeton U/PPPL),
Julian Kates-Harbeck (Harvard U/PPPL),
William Tang (Princeton U/PPPL)
Training neural networks
with low precision floats
#DL2SAIS

2. Outline
• Training neural networks with reduced floating-point precision
– Motivation and challenges
– Half-precision and mixed precision training approaches
• Enabling FP16 training on Pascal and Volta GPUs
– Compression of weights and gradients
– Master copy of weights, loss scaling
– TensorCores
• Fusion Recurrent Neural Net (FRNN) analysis framework
– The physics problem we are trying to solve
– Training LSTMs on long sequences of variable length
– Distributed training at the largest scale
– Results on TSUBAME 3.0 and OLCF Summit supercomputers
• Conclusions
2#DL2SAIS

3. Training with reduced floating-point precision: motivation
• Models are getting larger
– Optimize memory and network bandwidths
• Need faster training
– Improve maximum computational throughput, use larger mini-batch size
– Minimize communication overhead during distributed training
NN architecture Num. layers Num. trainable
parameters
GFLOPS
AlexNet 8 60 million 1.4
GoogLeNet 22 4 million 4
Inception V3 48 24 million 9
ResNet50 152 26 million 23
Deep Speech 2 11 38 millions 465
#DL2SAIS 3

4. FP16 training: motivation and challenges
#DL2SAIS 4

5. FP16 training: motivation and challenges
Challenges
• IEEE 754 FP16 has a narrow numerical range
– Overflow problem: Inf/NaN cause irreversible damage to training (quite rare)
– “Vanishing gradient“ (quite common)
• Small valued gradients
• Large ratio of weight to weight gradient
Subnormal range: [2-24
,2-14
] Normal range: [214
,65504]
float
half
-127 -24-14 0 15 128
#DL2SAIS 5

6. Mixed precision with master copy of weights
• Store weights, activations, gradients in FP16; perform weight update in FP32
– Solves “vanishing gradient” problem
– Disadvantage: conversion between FP32 and FP16 may be slow; extra memory to keep
an extra copy of weights in FP32
Mixed arithmetic precision
• FP16 matrix multiply, FP32 accumulate,
either FP32/FP16 for activations
• Volta TensorCores enable this on the
hardware level
– Solves “vanishing gradient” problem
– Programmable matrix-multiply-and-accumulate units.
8 TensorCores per streaming multiprocessor
Enabling FP16 training (I)
#DL2SAIS 6

7. Enabling FP16 training (III)
Loss scaling: shift gradients to occupy higher bins of FP16 representable range, which are not
used
– Apply a scalar factor to the loss function before backpropagation step
– Unscale weight gradients before the update step to maintain the magnitude of updates the
same as for FP32; alternatively re-tune the hyperparameters
• Scaling factors will depend on the neural network/dataset in hand
FP16 representable range FP16 representable rangeScale factor of 10 shifts
by 3 exponent values right
Everything below is irrelevant
for training
#DL2SAIS 7

8. Enabling FP16 training (II)
Weight and gradient sharing
• Quantize weight and gradient matrices into
N bins via clustering
– Weights in the same bin share the same value
– Store index of shared weights in integer matrix
Dynamic fixed-point representation
• Train in full precision (float)
• Quantize weight and activation
– Gather statistics about weight and activation (min, max
for each layer)
– Convert each float value to fixed point format (Google TPU uses int8 equivalent
representation and arithmetic for that)
S. Han et. al. “Deep compression”
#DL2SAIS 8

9. Single GPU results
• Volta, no loss-scaling required for accurate convergence
• Volta, loss-scaling required for accurate convergence
9#DL2SAIS
Model FP32 baseline Mixed precision
AlexNet 56.77 % 56.93 %
VGG-D 65.40 % 65.43 %
GoogLeNet 68.33 % 68.43 %
Inception V3 73.85 % 74.13 %
Resnet50 75.92 % 76.04 %
Model FP32 baseline MP no loss scaling MP loss sclaing
Faster R-CNN 69.1 % 68.6 % 69.7 %
Multibox SSD 76.9 % diverges 77.1 %
Result from: P. Micikevicius et. al. Mixed Precision
Training, https://arxiv.org/abs/1710.03740

10. OLCF Summit: hardware and setup
• Summit is an IBM system located at the Oak Ridge Leadership Computing Facility.
Having theoretical peak-performance of 200 PF, it is one of the “smartest” computers in the
world for deep learning applications with
mixed-precision capability in excess of 3 EF
– To be ranked as #1 supercomputing system
worldwide this summer
– 4600 compute nodes, two IBM POWER9,
PPC64LE processors and
6 Volta V100 accelerators per node, NVlink
– Deploy using Singularity containers
– LSF scheduler, jsrun to execute jobs
– AlpineTDS IBM Spectrumscale filesystem
– Globus DTN, HPSS
#DL2SAIS 10

11. Physics problem we are trying to solve
• Fusion energy: a clean source of energy that can be produced on Earth in a tokamak-style
fusion reactor
• Tokamak: a type of design for a nuclear fusion device. It is a torus-shaped vacuum chamber
surrounded by magnetic coils
• Existing tokamak-style fusion reactors: JET, DIII-D, NSTX
• Plasma disruptions are large macroscopic instabilities resulting in:
– Loss of confinement – ends fusion reaction
– Intense radiation – damaging
concentration in small areas
– Current quench – high magnetic forces
• Prediction and avoidance of disruptions is
a crucial aspect of tokamak operation,
especially in high-performance regime
#DL2SAIS 11

12. • Scalar time series: plasma current, lock-mode amplitude
• Vector time series
• All as a function of ρ (normalized flux surface)
• 1D current, electron temperature, radiation profiles
ρ = 0
ρ = 1
0.7 0.1 1.5 0.05
0.7 0.1 1.5 0.05 1.1 2.3 0.1
1.1 2.3 0.1
2.3 2.2 2.0 1.7 … 0.2 0.0
!" #$% & '$
1D convolution and activation
Max Pooling
Raw profile
Salient features
Full feature vector
Data: scalar and vector time series
#DL2SAIS 12

13. Fusion Recurrent Neural Net (FRNN)
Signals
Output
> Threshold?
Output: Disruption coming?
RNN Architecture:
• LSTM, 3 layers
• 300 hidden units per cell
• Stateful, returns sequences
Output Output
Internal
State
T = t
0D signals 1D
1D data
CNN
CNN architecture:
• Number of conv. filters: 10
• Size of conv. filters: 3
• Number of conv. layers: 2
• Pool size: 2
Time-distributed FC layer
Time-distributed FC layer
• apply to every temporal slice on
LSTM output
Alarm Alarm Alarm
LSTMLSTMLSTM
Signals Signals
0D signals 0D signals1D 1D
1D data
CNN
1D data
CNN
T = 0 T = 1#DL2SAIS 13

14. • Time series lengths in our problem vary by orders of magnitude: challenge for sequence-based
learning tasks
• We approximate gradient computation by truncated backprop through time (BPTT)
– Gradients are computed over a subsequence of time series, the internal states are saved,
and then the next subsequence is fed to the RNN while using the last internal states from
the previous subsequence as the initial internal states of the current subsequence
• Traditional approach of bucketing would not work as the sequence length is strongly correlated
with whether shots are disruptive or non-disruptive and thus individual batches would be biased
• We implement custom approach resetting internal state of individual examples within mini batch
– Since there is a persistent internal state between successive subsequences in time, it is not
possible to use more than one subsequence from a given shot in a given mini-batch
14#DL2SAIS
Training LSTM on sequences of variable lengths (I)

15. Training LSTM on sequences of variable lengths (II)
• To train batch-wise we need M independent (stemming from different shots) time slices of equal
length to feed to the GPU
– Maintain a buffer of M separate shots. At every training
step, the first T time slices of the buffer are fed as the
next batch, then shift the buffer by T time steps
• When shot is finished in the buffer, a new shot
is loaded and the RNN internal states of the
corresponding batch index are reset for training
• Internal states of the other batch indices are
maintained. Thus, the internal state is persisted
during learning for the entire length of any given shot
– Allow the RNN to learn temporal patterns much longer than unrolling length T and potentially as long as
the entire shot
• Random offsets of shots against each other and random shuffling of train set provides a mixture
of disruptive and non-disruptive samples for the network at every batch to stabilize training
15#DL2SAIS

16. Introducing FRNN
• FRNN represents a combined workflow implemented in Python
– Keras for layer definitions
– TensorFlow optimizers
– Custom parameter averaging, global weights update in CUDA-aware MPI
– Not a general purpose tool: geared toward Fusion energy scientific
community, but certain parts can be useful for a wider community
– https://github.com/PPPLDeepLearning/plasma-python
• We demonstrate how to enable FP16 for distributed training with this
combined Keras/TensorFlow/MPI based workflow
– Weight update is sensitive to floating point precision
– Volta/FP16 optimized algorithms are disabled by default in TF, chosen by
CuDNN autotuner
16#DL2SAIS

17. Distributed training flow
(Step 3) Update optimizer internal state and global
weights using the global gradient
Replica 1
GPU
worker
Replica 2
GPU
worker
Replica 3
GPU
worker
Replica 4
GPU
worker
Replica N
GPU
worker
Average local gradients
collected from workers
(Step 2) Aggregate local
gradients from each worker
using MPI_Allreduce, then
average them
(Step 4)
MPI_Broadcast
global parameters
after update
back to workers,
repeat (Step 1)
!"#$%&#
'
= !"#$%&#
'
− *+!"#$%&#
'
(Step 0) Initialize network parameters randomly, based on the model configuration
(Step 1) Perform fprop
and bprop on each
worker
17

18. No loss in training accuracy
• Use validation level area under the ROC and AUC (area under curve) to characterize
the quality of FRNN and applicability of the half-precision training
• Validation level AUC as a function of epoch show similar shapes for both FP16 and
FP32
– Reaching the plateau at around AUC=0.87 by the epoch 6
– SGD optimizer with momentum, loss scaling factor=10
The test set AUC=0.96
for
both the FP16 and
FP32
Base learning
rate
λ0 = 0.0004
Base learning rate
λ0 = 0.001
#DL2SAIS 18

19. Performance and scaling properties (I)
• FRNN shows nearly-linear runtime scaling and logarithmic communication
0.3 0.4 0.5 0.6 0.7 0.8
Validation AUC achieved
102
103
104
Speedupoverserial
Ideal
Measured
10 0 10 1 10 2 10 3 10 4
Number of GPUs
10 −1
10 0
10 1
10 2
10 3
10 4
Timeperepoch(s)
Data
Scaling model
Ideal scaling
1
G
PU
102
G
PU
s
(paralleltraining)
102
G
PU
s
(paralleltuning)
104
G
PU
s
10 −1
10 0
10 1
10 2
10 3
10 4
Runtime(hours)
2 4 6 8 10 12 14 16
Scaled walltime (hours)
10−2
10−1
100
Trainingloss
NGPU = 4
NGPU = 16
NGPU = 64
NGPU = 128
NGPU = 256
0.4
0.5
0.6
0.7
0.8
0.9
ValidationAUCachieved
10−2 10−1 100 101
Time to disruption (s)
0.0
0.2
0.4
0.6
Fractionofdetected
0.03 0.07 0.2 0.5 1.0
Time to disruption (s)
0.5
0.6
0.7
0.8
0.9
TestAUCachieved
cb
#DL2SAIS 19

20. Performance and scaling properties (II)
• Results for hyperparameter tuning with10000 GPUs
with parallel random search across
100 models, trained on 100 GPUs each.
• The time to solution for finding a model of given
validation AUC is compared to the scenario
of using a single GPU.
– The inset shows the time required for finding the best
model in the scenarios of using 1, 100 and
10000 total GPUs.
– For 100 GPUs, we distinguish training the 100 models
in serial, but using 100 GPUs for each model
(“parallel training”) vs. running 100 models in parallel,
trained on 1 GPU each (“parallel tuning”),
which both achieve nearly 100x speedup.
0.3 0.4 0.5 0.6 0.7 0.8
Validation AUC achieved
102
103
104
Speedupoverserial
Ideal
Measured
10 0 10 1 10 2 10 3 10 4
Number of GPUs
10 −1
10 0
10 1
10 2
10 3
10 4
Timeperepoch(s)
Data
Scaling model
Ideal scaling
1
G
PU
102
G
PU
s
(paralleltraining)
102
G
PU
s
(paralleltuning)
104
G
PU
s
10 −1
10 0
10 1
10 2
10 3
10 4
Runtime(hours)
2 4 6 8 10 12 14 16
Scaled walltime (hours)
10−2
10−1
100
Trainingloss
NGPU = 4
NGPU = 16
NGPU = 64
NGPU = 128
NGPU = 256
0.4
0.5
0.6
0.7
0.8
0.9
ValidationAUCachieved
10−2 10−1 100 101
Time to disruption (s)
0.0
Fracti
0.03 0.07 0.2 0.5 1.0
Time to disruption (s)
0.5
cb
#DL2SAIS 20

21. Conclusions
• As neural network models are getting larger, reduced floating point precision training
approaches become crucial
– Ability to optimize computational throughput and reduce memory and network bandwidths,
allowing for faster training
• Develop a domain-specific deep learning framework integrating Keras and TensorFlow with
custom parameter averaging and global weight update routines implemented with CUDA-aware
MPI intended for disruption forecasting in tokamaks
– Strong nearly linear runtime scaling on GPU clusters
– Hyperparameter tuning with O(10000) of nodes
• Evaluated training of deep RNNs with half-precision floats on a real scientific dataset
– Use scientific JET plasma disruption time series dataset
– Use loss scaling approach to facilitate model convergence at FP16
– Obtained comparable validation level performances at the end of each epoch for half and
single floating point precisions
21#DL2SAIS

22. Microsoft Cloud+AI
SMART Data Science
Team is hiring!
ML-Enhanced Coding
Assistant based on a
Markov Chain Model
Automatic Code
Reviews
Generation through
Deep Learning
Answering Documentation
Questions using a Seq-to-
Seq Deep Learning Model
AI Infused
Analytics
Code Risk
Analysis using
Graph Learning
#DL2SAIS 22
Contact: neels@microsoft.com

23. Backup
23#DL2SAIS

24. NVIDIA Volta architecture: TensorCores
• What are TensorCores?
– Programmable matrix-multiply-and-accumulate units. 8 TensorCores per
streaming multiprocessor
– Each tensor core can perform operations on small 4x4 matrices
– 1 matrix-multiply-and-accumulate operation per GPU clock
– During program execution, multiple Tensor Cores are used concurrently by a full
warp of execution in GPU
https://devblogs.nvidia.com/tensor-core-ai-performance-milestones/
#DL2SAIS 24

25. Training flow
Forward pass: mostly
matrix and element-wise
multiplications
Backprop: chain-rule
differentiation,
element-wise multiplications
Optimizer weight update:
multiply gradients by learning
rate (e.g. SGD)
!"
= $" !"%&
!' = $' !&
()
(!"%&
=
()
(!"
$"
()
($"
=
()
(!"
!"%&
!& = $& *
X
!&
!'
Forward pass on the next
mini-batch of data…
!"
= $" !"%&
!& = $& *
X
!&
()
(*
=
()
(!&
$&
()
($&
=
()
(!&
*
()
(!"%&
$" = $" − ,
()
($"
$& = $& − ,
()
($&
()
($"
$"
$&()
($&
-.// )!"-.// )!"
()
(!"
#DL2SAIS 25

26. Recurrent Neural Nets: basic description
• RNNs are a family of neural networks to process sequential data
• Feed forward equations are recurrent:
! " = $ + &ℎ " − 1 + *+(")
ℎ " = tanh ! "
2 " = 3 + 4ℎ(")
L
o
h
x
y
U
WV
Unfold
Notations:
x – input sequence,
U – is the input to hidden weight matrix,
W - hidden to hidden,
V – hidden to output weights
b,c are the biases
tanh() is the activation function (non-linearity)
o – output sequence
Loss L and target values are denoted as y
U U
h(t-1) h(t) h(…)h(…)
W W W
x(t-1) x(t)
o(t-1) o(t)
L(t-1) L(t)
y(t-1) y(t)
#DL2SAIS 26

27. Model convergence at large N
• Learning rate schedule is crucial to facilitate model convergence during
distributed training when the effective mini-batch size is large
• We use exponential learning rate decay with adjustable base learning rate
• Effective batch size is multiplied by the
number of workers
• In a distributed regime, the base
learning rate is reduced as the number
of workers N is increased
• Learning rate is clipped
!" = !$ %"
!$ (', )) =
!$
1.0 + '/)
#DL2SAIS 27

28. Half-precision training (I)
Half-precision training:
– Store weights, activations and gradients in FP16
– Perform matrix and element-wise multiplications, reduction ops in FP16 (including during fprop, bprop and weight
update)
• You already know it is possible to perform FP16 math, arithmetics and comparisons on NVIDIA GPUs with compute capability
5.3 and greater
– CUDA half intrinsics (round to nearest even): http://docs.nvidia.com/cuda/cuda-math-
api/group__CUDA__MATH__INTRINSIC__HALF.html#group__CUDA__MATH__INTRINSIC__HALF
– TensorFlow has DT_HALF registered type:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/types.cc
– We add a custom MPI datatype of 2 contiguous bytes: https://github.com/PPPLDeepLearning/plasma-
python/blob/master/plasma/primitives/ops.py
• This talk:
– Review techniques to enable FP16 and mixed precision training
– Introduce FRNN framework for disruption forecasting in tokamak fusion plasmas , discuss strong scaling
– Evaluate FP16 training on FRNN framework and a real scientific dataset from JET experiment
– Evaluate FP16 training on the benchmark IMDB dataset
#DL2SAIS 28
Half-precision training (I)
Half-precision training:
– Store weights, activations and gradients in FP16
– Perform matrix and element-wise multiplications, reduction ops in FP16 (including during fprop, bprop and weight
update)
• You already know it is possible to perform FP16 math, arithmetics and comparisons on NVIDIA GPUs with compute capability
5.3 and greater
– CUDA half intrinsics (round to nearest even): http://docs.nvidia.com/cuda/cuda-math-
api/group__CUDA__MATH__INTRINSIC__HALF.html#group__CUDA__MATH__INTRINSIC__HALF
– TensorFlow has DT_HALF registered type:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/types.cc
– We add a custom MPI datatype of 2 contiguous bytes: https://github.com/PPPLDeepLearning/plasma-
python/blob/master/plasma/primitives/ops.py
• This talk:
– Review techniques to enable FP16 and mixed precision training
– Introduce FRNN framework for disruption forecasting in tokamak fusion plasmas , discuss strong scaling
– Evaluate FP16 training on FRNN framework and a real scientific dataset from JET experiment
– Evaluate FP16 training on the benchmark IMDB dataset
#DL2SAIS 28

29. RNN Predictions
True positive example
TTD [ms]
Current Increase
Locked Mode Spike
Density Drop
Internal Inductance Spike
Stored Diamagnetic Energy Drop
FRNN Signal Spike
ArbitraryUnits