Jeremy Eder documented an all-in-one Kubernetes installation on an HPE Apollo 6500 server with 8 NVIDIA Tesla V100 GPUs. The installation included RHEL7, Kubernetes 1.8, CUDA 9, and an NVIDIA GPU device plugin to enable scheduling of pods that request GPU resources. Benchmark tests were run on the cluster that utilized all 8 GPUs and set new performance records for a prominent risk analytics benchmark.

1. Jeremy Eder
2018-02-15

2. ●
●
FSI

3. FSI

4. ●
●
●
●

5. ●
●
●
●
●
●

8. ●
○
■
■
■
○
●

9. ●
○
●
●
●

10. StorageNodes
RoutersRegistriesMasters
Gluster
Registry
Storage
GPU GPU
FPGA FPGA
master
and
etcd
master
and
etcd
master
and
etcd
snowflakes

11. ● All-in-One Kubernetes Installation
● (hack/local-up-cluster.sh)
● Node labeled
● Containers:
○ RHEL7+CUDA9
○ RHEL7+CUDA9+-DEVICE-PLUGIN
○ RHEL7+CUDA9+STAC-A2
● CUDA 9
● 8 x NVIDIA Tesla V100 (Volta) GPUs
● HPE Apollo 6500 w/XL270d Gen9
● Red Hat Enterprise Linux 7.4
● Kubernetes 1.8 (setup info)
● nvidia-smi
--applications-clocks=877,1380
● https://rhelblog.redhat.com/2017/11/21/red-hat-and-partners-deliver-new-performance-records-on-promi
nent-risk-analytics-benchmark/
● https://news.developer.nvidia.com/a-new-stac-a2-record/

12. Volta GPU Kubelet
Device Plugin
(daemonset)
Kube Scheduler
Volta GPU
Volta GPU
Volta GPU
Volta GPU
Volta GPU
Volta GPU
Volta GPU
Benchmark (pod)
resources:
limits:
nvidia.com/gpu: 8
kubectl create

13. Benchmark (pod)
resources:
limits:
nvidia.com/gpu: 8
Volta GPU Kubelet
Device Plugin
(daemonset)
Kube Scheduler
Volta GPU
Volta GPU
Volta GPU
Volta GPU
Volta GPU
Volta GPU
Volta GPU
kubectl create

14. Recent GPU-related activity
● Red Hat + NVIDIA Partnership
● Machine-Learning OpenShift Commons
● radanalytics templates for ML-workflow on OpenShift
● Early KubeFlow involvement