Graphics Processing Units (GPUs) are becoming popular for achieving high performance of computation intensive workloads. The GPU offers thousands of cores for floating point computation. This is beneficial to machine learning algorithms that are computation intensive and are parallelizable on the Spark platform. While the current execution strategy of Spark is to execute computations for the workload across nodes, only CPUs on each node execute computation. If Spark could use GPUs on each node, users benefit from GPUs that can reduce the execution time of the algorithm and reduce the number of nodes in a cluster. Recently, while application programmers use DataFrame APIs for their application, machine learning algorithms work with RDDs that keep data across nodes for distributed computation on CPU cores. A RDD keeps data as a Scala collection class in a row-based format. The computation model of GPU can achieve high performance for contiguous data in a column-based format. For enabling efficient GPU computation on Spark, we present a column-based RDD that can keep data as an array. When we execute them on the GPU, our implementation simply copies data in the column-based RDD to the GPU device memory. Then, each GPU cores execute operations faster on the device memory. CPU cores can execute existing functions on the column-based RDD. In this session, we will give the following contribution to the Spark community: (1) we give a brief design overview of transparent GPU exploitations from programmers (2) we show our APIs to build a GPU-accelerated library using column-based RDD and show the performance gain of some programs (3) we discuss current work for transparent GPU code generation from DataFrame APIs The package for (2) is available at http://github.com/IBMSparkGPU/GPUEnabler

1. Kazuaki Ishizaki, IBM Research
Madhusudanan Kandasamy, IBM Systems
Transparent GPU Exploitation
on Apache Spark
#Res9SAIS

2. About Me – Madhusudanan Kandasamy
• STSM(Principal Engineer) at IBM Systems
• Working for IBM Power Systems over 15 years
– AIX & Linux OS Development
– Apache Spark Optimization for Power Systems
– Distributed ML/DL Framework with GPU & NVLink
• IBM Master Inventor (20+ Patents, 18 disclosure publications)
• Committer of GPUEnabler
– Apache Spark Plug-in to execute GPU code on Spark
• https://github.com/IBMSparkGPU/GPUEnabler
• Github: https://github.com/kmadhugit
• E-mail: madhusudanan@in.ibm.com
2Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS

3. What You Will Learn from This Talk
• Why accelerate your workloads using GPU on Spark
– GPU/CUDA Overview
– Spark + GPU for ML workloads
• How to program GPUs in Spark
– Invoke Hand-tuned GPU program in CUDA
– Translate DataFrame program to GPU code automatically
• What are key factors to accelerate program
– Parallelism in a program
– Data format on memory
3Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS

4. GPU/CUDA Overview
• GPGPU - Throughput
• CUDA, which is famous,
requires programmers
to explicitly write
operations for
– allocate/deallocate
device memories
– copying data between
CPU and GPU
– execute GPU kernel
4Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
// code for GPU
__global__ void GPUMultiplyBy2(
float* d_a, float* d_b, int n) {
int i = threadIdx.x;
if (n <= i) return;
d_b[i] = d_a[i] * 2.0;
}
void fooCUDA(N, float *A, float *B, int N) {
int sizeN = N * sizeof(float);
cudaMalloc(&d_A, sizeN); cudaMalloc(&d_B, sizeN);
cudaMemcpy(d_A, A, sizeN, HostToDevice);
GPUMultiplyBy2<<<N, 1>>>(d_A, d_B, N);
cudaMemcpy(B, d_B, sizeN, DeviceToHost);
cudaFree(d_B); cudaFree(d_A);
}

5. Spark + GPU for ML workloads
• Spark provides efficient ways to parallelize jobs across
cluster of nodes
• GPUs provide thousands of cores for efficient way to
parallelize job in a node.
• GPUs provide up to 100x processing over CPU *
• Combining Spark + GPU for lightning fast processing
– We will talk about two approaches
5Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
* https://blogs.nvidia.com/blog/2009/12/16/whats-the-difference-between-a-cpu-and-a-gpu/

6. Outline
• Why accelerate your workloads using GPU on Spark
• How to program GPUs in Spark
– Invoke Hand-tuned GPU program in CUDA
– Translate DataFrame program to GPU code automatically
• Toward faster GPU code
• How two frameworks work?
6Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS

7. Invoke Hand-tuned GPU program in CUDA
• GPUEnabler to simplify development
• Implemented as Spark package
– Can be drop-in into your version of Spark
• Easily Launch hand coded GPU kernels from map() or
reduce() parallel function in RDD, Dataset
• manages GPU memory, copy data between GPU and CPU,
and convert data format
• Available at https://github.com/IBMSparkGPU/GPUEnabler
7Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS

8. Example - hand tuned CUDA kernel in Spark
8Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
CUDA is a programming language
for GPU defined by NVIDIA
PTX is an assembly language file
that can be generated by a CUDA file
Step 1: Write CUDA kernels (without memory management and data copy)
Step 2: Write Spark program
Step 3: Compile and submit
Object SparkExample {
val mapFunction = new CUDAFunction("multiplyBy2", Seq(“value”), “example.ptx”)
val output = sc.parallelize(1 to 65536, 24).cache
.mapExtFunc(x => x*2, mapFunction).show }
__global__ void multiplyBy2(int *in, int *out, long size) {
long i = threadIdx.x + blockIdx.x * blockDim.x;
if (size <= i) return;
out[i] = in[i] * 2;
}
$ nvcc example.cu -ptx
$ mvn package
$ bin/spark-submit --class SparkExample SparkExample.jar
--packages com.ibm:gpu-enabler_2.11:1.0.0

9. Performance Improvements of GPU
program over Parallel CPU
• Achieve 3.6x for CUDA-based mini-batch logistic regression
using one P100 card over POWER8 160 SMT cores
9Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
Relative execution time over GPU version
IBM Power System S822LC for High Performance Computing “Minsky”, at 4 GHz with 512GB memory, one P100 card, Fedora 7.3, CUDA 8.0,
IBM Java pxl6480sr4fp2-20170322_01(SR4 FP2), 128GB heap, Apache Spark 2.0.1, master=“local[160]”, GPU Enabler as 2017/5/1, N=112000,
features=8500, iterations=15, mini-batch size=10, parallelism(GPU)=8, parallelism(CPU)=320
Shorter is better for GPU

10. Outline
• Why accelerate your workloads using GPU on Spark
• How to program GPUs in Spark
– Invoke Hand-tuned GPU program in CUDA (GPUEnabler)
– Translate DataFrame program to GPU code automatically
• Toward faster GPU code
• How two frameworks work?
10Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS

11. “Transparent” GPU Exploitation
• Enhanced Spark by modifying Spark source code
• Accept expression in select(), selectExpr(), and
reduce() in DataFrame
• Automatically generate CUDA code from DataFrame
program
• Automatically manage GPU memory and copy data
between GPU and CPU
• No data format conversion is required
11Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS

12. Example - “Transparent” GPU Exploitation
• Write Spark program in DataFrame
• Compile and submit them
12Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
$ mvn package
$ bin/spark-submit --class SparkExample SparkExample.jar
Object SparkExample {
val output = sc.parallelize(1 to 65536, 24).toDF(“value”).cache
.select($”value” * 2).cache.show }

13. Performance Improvements of Spark
DataFrame program over Parallel CPU
• Achieve 1.7x for Spark vector multiplication using one P100
card over POWER8 160 SMT cores
13Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
Relative execution time over GPU version
IBM Power System S822LC for High Performance Computing “Minsky”, at 4 GHz with 512GB memory, one P100 card, Fedora 7.3, CUDA 8.0,
IBM Java pxl6480sr4fp2-20170322_01(SR4 FP2), 128GB heap, Based on Apache Spark master (id:657cb9), master=“local[160]”, N=480, vector
length=1600, parallelism(GPU)=8, parallelism(CPU)=320
Shorter is better for GPU

14. Outline
• Why accelerate your workloads using GPU on Spark
• How to program GPUs in Spark
– Invoke Hand-tuned GPU program in CUDA (GPUEnabler)
– Translate DataFrame program to GPU code automatically
• Toward faster GPU code
• How two frameworks work?
14Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS

15. About Me – Kazuaki Ishizaki
• Researcher at IBM Research in compiler optimization
• Working for IBM Java virtual machine over 20 years
– In particular, just-in-time compiler
• Active Contributor of Spark since 2016
– 98 commits, #37 in the world (25 commits, #8 in 2018)
• Committer of GPUEnabler
• Homepage: http://ibm.biz/ishizaki
• Github: https://github.com/kiszk, Twitter: @kiszk
15Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS

16. Toward Faster GPU code
• Assign a lot of parallel computations into GPU cores
• Reduce # of memory transactions to GPU memory
16Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS

17. Assign A Lot of Parallel Computations into
GPU Cores
• Achieve high utilization of GPU
17Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
Achieve high performanceAchieve low performance
Busy idle Busy

18. Reduce # of Memory Transactions
• Depends on memory layout, # of memory transactions are
different in a program
18Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
x1 y1 2 61 5x3 y3
Assumption: 4 consecutive data elements
can be coalesced by GPU hardware
4 v.s. 2
memory transactions to
GPU device memory
Row-oriented layout Column-oriented layout
Pt(x: Int, y: Int)
Load Pt.x1
Load Pt.x2
Load Pt.x3
Load Pt.x4
Load Pt.y1
Load Pt.y2
Load Pt.y3
Load Pt.y4
x2 y2 x4 y4 43 87
x1 x2 x3 x4
GPU
cores
Load Pt.x Load Pt.yLoad Pt.x Load Pt.y
1 231 2 4
y1 y2 y3 y4x1 x2 x3 x4 y1 y2 y3 y4
Memory
access
GPU
cores
Memory
transaction
Memory
access
Memory
transaction
Memory
access

19. Toward Faster GPU Code
• Assign a lot of parallel computations into GPU cores
– Spark program has been already written by a set of parallel
operations
• e.g. map, join, …
• Reduce # of memory transactions
– Column-oriented layout achieves better performance
• The paper* reports 3.3x performance improvement of GPU kernel
execution of kmeans over row-oriented layout
19Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
* Che, Shuai and Sheaffer, Jeremy W. and Skadron, Kevin.
“Dymaxion: Optimizing Memory Access Patterns for Heterogeneous Systems”, SC’11

20. Questions
• How can we write a parallel program for GPU on Spark?
• How can we use column-oriented storage for GPU in
Spark?
20Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS

21. Questions
• How can we write a parallel program for GPU on Spark?
– Thanks to Spark programming model!!
• How can we use column-oriented storage for GPU in
Spark?
21Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS

22. Outline
• Why accelerate your workloads using GPU on Spark
• How to program GPUs in Apache Spark
– Invoke hand-tuned GPU program in CUDA (GPUEnabler)
– Translate DataFrame program to GPU code automatically
• Toward faster GPU code
• How two frameworks work?
22Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS

23. Data Movement with GPUEnabler
• Data in RDD is moved into off-heap as column-oriented
storage
23Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
Java Heap
GPU
Off-heap
CPU
Data copy
map
Data copy
Format
conversion
Format
conversion
RDD1 RDD3val rdd1.cache
val rdd2=rdd1.mapExtFunc(…)
val rdd3=rdd2.mapExtFunc(…)
Row-
oriented
storage
Column-
oriented
storage
Row-
oriented
storage
map
Row-
oriented
storage
Row-
oriented
storage
Column-
oriented
storage

24. Data Movement with GPUEnabler
• Data in RDD is moved into off-heap as column-oriented
storage
24Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
Java Heap
GPU
Off-heap
CPU
Data copy
map
Data copy
Format
conversion
Format
conversion
RDD1 RDD3val rdd1.cache
val rdd2=rdd1.mapExtFunc(…)
val rdd3=rdd2.mapExtFunc(…)
Row-
oriented
storage
Column-
oriented
storage
Column-
oriented
storage
map
Column-
oriented
storage
Row-
oriented
storage
Column-
oriented
storage
Conversions and copies
are automatically done

25. How To Write GPU Code in GPUEnabler
• Write a GPU kernel corresponds to map()
25Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
val rdd2 = rdd1
.mapExtFunc(p => Point(p.x*2, p.y*2), mapFunction)
__global__ void multiplyBy2(int *inx, int *iny,
int *outx, int *outy, long size) {
long i = threadIdx.x + blockIdx.x * blockDim.x;
if (size <= i) return;
outx[i] = inx[i] * 2; outy[i] = iny[i] * 2;
}
GPU Kernel
Spark Program

26. Data Movement with DataFrame
• Cache in DataFrame already uses column-oriented storage
• We enhanced to create cache for DataFrame in off-heap
– Reduce conversion and data copy between Java heap and Off-heap
26Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
Java Heap
GPU
Off-heap
CPU
Data copy Data copy
DataFrame’s cache DataFrame’s cache
val df1.cache
val df2=df1.map(…)
val df3=df2.map(…).cache
Column-
oriented
storage
Column-
oriented
storage
map + map
Column-
oriented
storage
Column-
oriented
storage

27. Data Movement with DataFrame
• Cache in DataFrame already uses column-oriented storage
• We enhanced to create cache for DataFrame in off-heap
– Reduce conversion and data copy between Java heap and Off-heap
27Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
Java Heap
GPU
Off-heap
CPU
Data copy Data copy
DataFrame’s cache DataFrame’s cache
val df1.cache
val df2=df1.map(…)
val df3=df2.map(…).cache
Column-
oriented
storage
Column-
oriented
storage
map + map
Column-
oriented
storage
Column-
oriented
storage
copy is
automatically done

28. How To Generate GPU Code from
DataFrame
• Added a new path to generate CUDA code into Catalyst
28Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS
Derived Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Michael Armbrust
DataFrame
program
Unresolved
logical plan
Logical
Plan
Optimized
Logical Plan
Physical
Plan
Java
CUDA
for kernel
Analysis Optimizations Planning
Code
generation
Newly
added
Catalyst

29. Takeaway
• Why accelerate your workloads using GPU on Spark
– Achieved up to 3.6x over 160-CPU-thread parallel execution
• How to use GPUs on Spark
– Invoke Hand-tuned GPU program in CUDA (GPUEnabler)
– Translate DataFrame program to GPU code automatically
• How two approaches execute program on GPUs
– Address easy programming for many non-experts, not the
state-of-the-art performance by small numbers of top-notch
programmers
29Transparent GPU Exploitation on Apache Spark, Ishizaki & Kandasamy, #Res9SAIS