Wen Phan explained the need for GPUs in machine learning at DC.

1. Wen Phan
Introduction to GPUs for
Machine Learning

2. Agenda
• Context and Why GPUs?
– Matrix Multiplication Example
• CUDA
• GPU and Machine Learning
– Deep Learning
– Parallel Computing: GBM, GLM
• GPU-Enabled H2O
• Getting Started

3. Need for More Compute
• Lots of Data
• Complex Architectures
• Many Models

4. Historic Ways for More Compute
• Faster Clock Rates
• Multi-Core
• Distributed Computing

5. CPU Trends
Original data collected and plotted by M. Horowitz, F. Labonte, O. Shacham, K. Olukotun, L. Hammond, and C. Batten,
dotted line extrapolations by C. Moore

6. Distributed Computing

7. Why GPUs?

8. GPU Accelerated Computing
GPUCPU
nVIDIA

9. GPUs for Parallel Tasks
Traditional CPUs are
not economically feasible
2.3 PFlops 7000 homes
7.0
Megawatts
7.0
Megawatts
CPU
Optimized for
Serial Tasks
GPU Accelerator
Optimized for Many
Parallel Tasks
10x performance/socket
> 5x energy efficiency
Era of GPU-accelerated
computing is here
nVIDIA

10. GPU Devotes More Transistors to Data Processing
CUDA C Programming Guide

11. CPU VS GPU
https://videocardz.com/39721/nvidia-geforce-gtx-titan-released

12. Latency Versus Throughput
• Latency: Time to do a task.
• Throughput: Number of tasks per unit time.
• Fictitious Example:
– CPU
• Latency: 1 ns per task
• Throughput: (1 task per ns) x (6 cores) = 6 task per ns
– GPU
• Latency: 10 ns per task
• Throughput: (0.1 task per ns) x (2000 cores) = 200 task per ns
• CPUs are latency optimized; GPUs are throughput optimized

14. NVIDIA Latest GPUs
http://www.anandtech.com/show/11172/nvidia-unveils-geforce-gtx-1080-ti-next-week-699

15. CUDA C Programming Guide

16. Matrix Multiplication

17. Matrix Multiplication
2
4 A
3
5
m ⇥ k
2
4 B
3
5
k ⇥ n
=
2
4 C
3
5
m ⇥ n

18. Matrix Multiplication
2
6
6
6
6
6
4
a1,1 a1,2 a1,3 . . . a1,k
a2,1 a2,2 a2,3 . . . a2,k
a3,1 a3,2 a3,3 . . . a3,k
...
...
...
...
...
am,1 am,2 am,3 . . . am,k
3
7
7
7
7
7
5
A
2
6
6
6
6
6
4
b1,1 b1,2 b1,3 . . . b1,n
b2,1 b2,2 b2,3 . . . b2,n
b3,1 b3,2 b3,3 . . . b3,n
...
...
...
...
...
bk,1 bk,2 bk,3 . . . bk,n
3
7
7
7
7
7
5
B
=
2
6
6
6
6
6
4
c1,1 c1,2 c1,3 . . . c1,n
c2,1 c2,2 c2,3 . . . c2,n
c3,1 c3,2 c3,3 . . . c3,n
...
...
...
...
...
cm,1 cm,2 cm,3 . . . cm,n
3
7
7
7
7
7
5
C
ci,j =
kX
h=1
ai,hbh,j

19. CUDA

20. CUDA
• Historically, GPUs were used for, well, graphics processing. But, people realized that the fine-
grained parallelism inherently in GPU architecture could be exploited for general purpose
computing.
• CUDA (Compute Unified Device Architecture)
– Parallel computing platform
– Programming model and API
– Allows enabled GPUs for general purpose processing

21. Speed Up Parallelizable Code
Application Code
GPU
Use GPU to
Parallelize
Compute-Intensive
Functions
CPU
Rest of Sequential
CPU Code
nVIDIA

22. Matrix Multiplication
CUDA C Programming Guide

23. CUDA Matrix Multiplication
CUDA C Programming Guide
CPU
GPU
CPU
Memory
GPU
Memory

24. Allocate Memory and Copy to GPU (Device)
CUDA C Programming Guide
CPU
GPU
CPU
Memory
GPU
Memory

25. Invoke Kernel (GPU Program)
CUDA C Programming Guide
CPU
GPU
CPU
Memory
GPU
Memory

26. CUDA C Programming Guide

27. CUDA C Programming Guide

28. CUDA C Programming Guide

29. CUDA C Programming Guide

30. Matrix Multiplication Kernel
CUDA C Programming Guide

31. Free Memory
CUDA C Programming Guide
CPU
GPU
CPU
Memory
GPU
Memory

32. CUDA Ecosystem

33. SC12 Demo: Using CUDA Library to accelerate applications

34. SC12 Demo: Using CUDA Library to accelerate applications

35. SC12 Demo: Using CUDA Library to accelerate applications

36. GPUs and Machine
Learning

37. GPUs and Machine Learning
• Poster Child: Deep Learning
• Parallel Computing
– Model Parallelism
– Data Parallelism
– Training Parallelism

38. Deep Learning

39. Multi-Layer Perceptron Neural Network
Efron and Hastie. Computer Age Statistical Inference.

40. MNIST
MNIST Database

41. Image as a Tensor
TensorFlow

42. Training
TensorFlow

43. Supervised Learning
TensorFlow

44. Neuron

45. Activation Functions
Efron and Hastie. Computer Age Statistical Inference.

46. Layer of Neurons

47. Layer of Neurons

48. Matrix Multiplication!

49. Hidden Layers

50. Convolutional Neural Networks
• Leverages the fact that data has spatial structure
– Add idea of locality
• Tremendous success with computer vision tasks
• “Put deep learning on the map”

51. Frobenius Inner Product
X =
2
4
2 2 1
2 0 1
2 1 2
3
5 , K =
2
4
1 1 1
1 1 1
1 1 1
3
5
hX, KiF =
X
i,j
xi,jki,j
= (2 ⇤ 1) + (2 ⇤ 1) + (1 ⇤ 1) + (2 ⇤ 1) + (0 ⇤ 1) + (1 ⇤ 1) + (2 ⇤ 1) + (1 ⇤ 1) + (2 ⇤ 1)
= 2 + 2 + 1 2 + 0 1 2 + 1 + 2
= 3

52. Convolution
X =
2
6
6
6
6
6
6
6
6
4
0 0 0 0 0 0 0
0 0 2 1 0 2 0
0 1 2 2 1 0 0
0 0 2 0 1 1 0
0 1 2 1 2 1 0
0 2 0 0 2 1 0
0 0 0 0 0 0 0
3
7
7
7
7
7
7
7
7
5
K =
2
4
1 1 1
1 1 1
1 1 1
3
5
A =
2
6
6
6
6
4
1 4 0 4 1
3 4 1 4 1
4 9 3 5 0
3 0 2 5 0
5 2 3 5 2
3
7
7
7
7
5

53. Convolution
X
d
X
i,j
xi,j,dki,j,d
Andrej Karpathy. CS231n Convolutional Neural Networks for Visual Recognition.

54. Convolution Layer
Andrej Karpathy. CS231n Convolutional Neural Networks for Visual Recognition.

55. Convolutional Layer
w
d
h
g
0
@
X
d
X
i,j
xi,j,dki,j,d + b
1
A
X
Kf
X
d
X
i,j
xi,j,dki,j,d
¯
b
g(·)

56. ¯
Convolutional Layer
w
d
h
X
d
X
i,j
xi,j,dki,j,d
b
g
0
@
X
d
X
i,j
xi,j,dki,j,d + b
1
A
g(·)
f
X
K

57. Convolutional Layer
w
d
h
X
Input Image Activation Maps
Convolution

58. Convolutional Layer
• f = receptive field
(filter size)
• p = padding
• s = stride
• m = number of filters
Input Volume Output Volume
Convolution
wI
hI
dI
wO
dO
hO
wO =
wI f + 2p
s
+ 1
hO =
wI f + 2p
s
+ 1
dO = m

59. Example: LeNet

60. Inception ResNet V2

61. ImageNet

62. cuDNN

63. cuDNN Optimization

64. CUDA and cuDNN Speed Up

65. ImageNet Results
http://kaiminghe.com/ilsvrc15/ilsvrc2015_deep_residual_learning_kaiminghe.pdf

66. ImageNet Entries Using GPUs
https://devblogs.nvidia.com/parallelforall/nvidia-ibm-cloud-support-imagenet-large-scale-visual-recognition-challenge/

67. Deep Water: Next-Gen Distributed Deep Learning
One Interface - GPU Enabled - Significant Performance Gains
Inherits All H2O Properties in Scalability, Ease of Use and Deployment
Recurrent Neural Networks
enabling natural language
processing, sequences, time series,
and more
Convolutional Neural Networks
enabling Image, video, speech
recognition
Hybrid Neural Network Architectures
enabling speech to text translation,
image captioning, scene parsing and
more
H2O integrates with existing GPU
backends for significant performance
gains
H2O Deep Learning Algo

69. Parallel Computing

70. Parallel Computing
• Model Parallelism: Split up a single model

71. Random Forest
T1(x) T2(x) T3(x) TB(x)
ˆy = f(x; T1, . . . , TB)

72. Random Forest
T1(x) T2(x) T3(x) TB(x)

73. Deep Learning Model Parallelism
Large Scale Distributed Deep Networks. J. Dean, et. al.

74. Parallel Computing
• Model Parallelism: Split up a single model
• Data Parallelism: Split up data to train a single model

75. Deep Learning Data Parallelism
Large Scale Distributed Deep Networks. J. Dean, et. al.

76. H2O Deep Learning Architecture

77. Gradient Boosting Machine (GBM)
T1(x) T2(x) T3(x) TM (x)
fM (x) =
MX
i=1
Ti(x)

78. Gradient Boosting Machine (GBM)
T1(x) T2(x) T3(x) TM (x)
fi(x) = fi 1(x) + Ti(x; ˆ⇥i)

79. Gradient Boosting Machine (GBM)
T1(x) T2(x) T3(x) TM (x)

80. Decision Tree
R1
R2
Hastie, Tibshirani, Friedman. Elements of Statistical Learning

81. GBM Data Parallelism
{Xi; ti}?

82. GBM Data Parallelism
1
2
K
X = {X1, . . . , XK}
math (X1)
math (X2)
math (XK)
{Xi; ti} = f(math (X1) , . . . , math (XK))

83. GBM Data Parallelism
1
2
K
X = {X1, . . . , XK}
math (X1)
math (X2)
math (XK)
{Xi; ti} = f(math (X1) , . . . , math (XK))
Full Data Parallelism for Each Level of Tree Growth!

84. CPU Cluster

85. GPU Cluster
• Level GPUs to accelerate processing and fine-grain parallelism on each node
CPU
GPU
math (X1) math (X2) math (XK)math (X3)

86. GBM on GPU
T1(x) T2(x) T3(x) TM (x)
Application Code
GPU
Use GPU to
Compute-Intensive
Functions
CPU
Rest of Sequential
CPU Code

87. H2O XGBoost

88. Parallel Computing
• Model Parallelism: Split up a single model
• Data Parallelism: Split up data to train a single model
• Training Parallelism: Split up different parts of the training process
– Ensemble Base Learners
– Cross-Validation
– Hyperparameters

89. Linear Regression
Hastie, Tibshirani, Friedman. Elements of Statistical Learning
X =
2
6
6
6
4
x1,1 x1,2 . . . x1,p
x2,1 x2,2 . . . x2,p
...
...
...
...
xn,1 xn,2 . . . xn,p
3
7
7
7
5
minimize
nX
i=1
0
@yi 0
pX
j=1
xi,j j
1
A
2
minimize ||y X ||2
2

90. Ridge Regression
Hastie, Tibshirani, Wainwright. Statistical Learning with Sparsity
minimize
0,
||y X ||2
2
subject to || ||2
2  t

91. Ridge Regression
Hastie, Tibshirani, Wainwright. Statistical Learning with Sparsity
minimize ||y X ||2
2 + || ||2

92. Lasso Regression
Hastie, Tibshirani, Wainwright. Statistical Learning with Sparsity
minimize
0,
||y X ||2
2
subject to || ||1  t

93. Lasso Regression
Hastie, Tibshirani, Wainwright. Statistical Learning with Sparsity
minimize ||y X ||2
2 + || ||1

94. Elastic Net Regression
minimize ||y X ||2
2 +
✓
↵|| ||1 +
1
2
(1 ↵)|| ||2
2
◆

95. Single Node GPU
CPU
GPU

96. Single Node Multi-GPU
CPU
GPU

97. Elastic Net Regression Training Parallelism
J( ; , ↵) = ||y X ||2
2 +
✓
↵|| ||1 +
1
2
(1 ↵)|| ||2
2
◆
minimize J( ; i, ↵1) minimize J( ; i, ↵2) minimize J( ; i, ↵G)
GPU1 GPU2 GPUG
i 2 {0, . . . , 1}

98. Elastic Net Regression Training Parallelism

99. GPU-Enabled H2O

100. DGX-1

104. GPU Open Analytics Initiative

106. GTC 2017

107. GTC 2017

108. Getting Started

109. CUDA
https://developer.nvidia.com/cuda-downloads

110. cuDNN
https://developer.nvidia.com/cudnn

111. Try It Today

112. Questions?