Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DCSF 19 Accelerating Docker Containers with NVIDIA GPUs


Published on

Using the NVIDIA Container Runtime, many developers and enterprises have been developing, benchmarking and deploying deep learning (DL) frameworks, HPC and other GPU accelerated containers at scale for the last two years. In this talk, we will go over the architecture of the NVIDIA Container Runtime and discuss our recent close collaboration with Docker. The result of our collaboration with Docker is a seamless native integration of the runtime enabling Docker Engine 19.03 CE and the forthcoming Docker Enterprise release to run GPU accelerated containers. We will also highlight containerized NVIDIA drivers. This new feature eliminates the overhead of provisioning GPU machines and brings GPU support on container optimized operating systems, which either lack package managers for installing software or require all applications to run in containers. In this session, you will learn how GPU accelerated containers can be easily built and deployed through the use of driver containers and native support for GPUs in Docker 19.03. The session will include a demo of running a GPU accelerated deep learning container using the new CLI options in Docker 19.03 and containerized drivers. Running NVIDIA GPU accelerated containers with Docker has never been this easy!

Published in: Software
  • Be the first to comment

DCSF 19 Accelerating Docker Containers with NVIDIA GPUs

  1. 1. Pramod Ramarao | Swati Gupta Accelerating Docker Containers with NVIDIA GPUs
  2. 2. Why GPUs? Unlock massive parallel compute power 1980 1990 2000 2010 2020 GPU-Computing perf 1.5X per year Original data up to the year 2010 collected and plotted by M. Horowitz, F. Labonte, O. Shacham, K. Olukotun, L. Hammond, and C. Batten New plot and data collected for 2010-2015 by K. Rupp 102 103 104 105 106 107 Single-threaded perf 1.5X per year 1.1X per year APPLICATIONS SYSTEMS ALGORITHMS CUDA ARCHITECTURE
  3. 3. An Example 2x CPU Single Node 1X P100 Single Node 1X V100 DGX-1 8x V100 At scale 2176x V100 Relative Time to Train (ResNet-50) ResNet-50, 90 epochs to solution | CPU Server: dual socket Intel Xeon Gold 6140 Sony 2176x V100 record on <4 Minutes 3.3 Hours 25 Days 30 Hours 4.8 Days Train a deep neural net in < 4 mins
  4. 4. Workloads Accelerated by GPUs AI Training & Inference Gaming Media & Entertainment Science Speech Visual Search Video AnalysisRobotics
  5. 5. How GPU Acceleration Works with CUDA Application Code GPU CPU 5% of Code 95% of Execution Compute-Intensive Functions Rest of Sequential CPU Code 95% of Code 5% of Execution +
  6. 6. Why GPU Containers? I want to get started with DL & NVIDIA GPUs Data Scientist / Developer
  7. 7. Why GPU Containers? I want to get started with DL & NVIDIA GPUs Data Scientist / Developer >= 13.0 <= 12.0 ├─ NVIDIA Drivers 410 ├─ CUDA 10.0 ├─ NVIDIA cuDNN 7.4 ├─ Bazel 0.19.2 ├─ NVIDIA Drivers 384 ├─ CUDA 9.0 ├─ NVIDIA cuDNN 7.2 ├─ Bazel 0.15.0 “This framework requires installing N different dependencies from sources” 1
  8. 8. Why GPU Containers? I want to deploy my model on a GPU cluster at scale Data Scientist / Developer ├─ NVIDIA Drivers 410 ├─ CUDA 10.0 ├─ NVIDIA cuDNN 7.4 ├─ Bazel 0.19.2 ├─ Multi-GPU Communications Libraries (NCCL) ├─ Horovod ├─ OpenMPI ├─ Mellanox OFED Drivers ├─ NVIDIA DALI “I need to install a number of other components for performance” 2
  9. 9. GPU Support in Docker: 1.0  CLI wrapper around the Docker CLI  Used a plugin for volume mounts to expose devices and NVIDIA driver libraries $ NV_GPU=0 nvidia-docker run -ti nvidia/cuda nvidia-docker http + unix docker dockerd nvidia-docker-plugin NVIDIA Driver (CUDA, NVML) http + unix container process http GPU information
  10. 10. GPU Support in Docker: 1.0 Open-source Project on GitHub since 2016 >2MM downloads >7.5K stars >13 MM pulls of CUDA images from DockerHub Enabled Various Use-cases ► NGC optimized containers from NVIDIA ► Adopted by major deep learning frameworks
  11. 11. Evolution of NVIDIA Runtime ► New application areas for GPU containers e.g. graphics (GL, Vulkan) and HPC (Infiniband) ► Support GPUs across other CPU architectures – POWER, ARM64 ► New CUDA platform features ► Container orchestration matured, needing better integration (e.g. Kubernetes, Swarm)
  12. 12. GPU Support in Docker: 2.0 NVML NVIDIA Driver CUDAlibnvidia-container nvidia-container-runtime-hook Components OCI Runtime Interface Containerized Applications ► Integrates Linux container internals instead of wrapping Docker ► Includes runtime library, headers, CLI tools ► Support new use-cases - HPC, DL, Graphics
  13. 13. Native GPU Support in Docker 19.03  Run GPU containers using “--gpus” option in the Docker CLI  Simplified installation to enable GPU support  Fewer NVIDIA packages to install  No custom runtimes or registration required docker grpc + unix dockerd docker-containerd + shim http(s)(+unix) runc nvidia-container- runtime-hook libnvidia-container NVIDIA Driver (CUDA, NVML) container process $ docker run -it --gpus all nvidia/cuda DeviceRequest
  14. 14. NVIDIA Runtime Library ► Configures containers to leverage NVIDIA hardware (accessible via CLI) ► Based on kernel primitives ► Meant to be agnostic of the container runtime ► Exposes GPU devices, capabilities (e.g. graphics, compute) and enforces CUDA compatibility checks $ nvidia-container-cli info NVRM version: 410.104 CUDA version: 10.0 Device Index: 0 Device Minor: 1 Model: GeForce GT 710 GPU UUID: GPU-1ebf0405-77f6-1bdc-7635-036be63e98b3 Bus Location: 00000000:01:00.0 Architecture: 3.5 Device Index: 1 Device Minor: 0 Model: TITAN V GPU UUID: GPU-7ac63497-2c78-9e7b-93bc-fe984e0ed79a Bus Location: 00000000:02:00.0 Architecture: 7.0
  15. 15. Demo Time: Run CUDA Containers ► Use --gpus to launch a basic CUDA container and showcase sample CUDA app (deviceQuery) ► Showcase HPC + GL (visualization) container
  16. 16. Containerized Drivers Simplifying Deployment of NVIDIA Software  Using NVIDIA drivers is as easy as start/stop container  Benefits  Speed  Container operating systems  Portability  Beta available now on DockerHub; productization in 2H 2019 CUDA Container NVIDIA Driver Container persistenced Host Linux OS Distribution OS kernel driver kmods NVIDIA + Docker 19.03
  17. 17. Containerized Drivers Driver Container nvidia-installer Host Linux OS Distribution OS kernel driver kmods Link and load kernel modules or recompile (e.g. nvidia-drm.ko, nvidia- uvm.ko, nvidia.ko, nvidia-modeset.ko) NVIDIA + Docker 19.03 1
  18. 18. Containerized Drivers CUDA Container Driver Container persistenced Host Linux OS Distribution OS kernel driver kmods NVIDIA + Docker 19.03 GPU containerization from driver container (i.e. mount driver, devices…) 3 nvidia-installer Run persistence daemon (driver container lifetime is tied to nvidia-persistenced process) 2
  19. 19. Demo Time: Inference on GPUs ► Deep learning inference on GPUs with containerized drivers
  20. 20. Future Work ► Use Docker plugins to setup the NVIDIA runtime ► Using --gpus in Kubernetes, Swarm ► Support ARM64 $ docker plugin install nvidia
  21. 21. Try Docker 19.03 with NGC Ready GPU optimized containers from NVIDIA ► Gets started easily with Docker 19.03 and 50+ GPU optimized containers from NVIDIA NGC ► Visit NGC 50+ Containers DL, ML, HPC Pre-trained Models NLP, Classification, Object Detection & more Industry Workflows Medical Imaging, Intelligent Video Analytics Model Training Scripts NLP, Image Classification, Object Detection & more
  22. 22. Pramod Ramarao | Swati Gupta Accelerating Docker Containers with NVIDIA GPUs