SlideShare a Scribd company logo
1 of 33
Download to read offline
Containerizing GPU
Applications with Docker
for Scaling to the Cloud
FUTURE OF PACKAGING APPLICATIONS
SUBBU RAMA
CPU
CPU
CPU
GPU
GPU
GPU
GPU
Mem
MemMemMem
CPU CPU CPU CPU
GPU GPU GPU GPU
GPU GPU GPU GPU
Mem Mem Mem Mem
Mem Mem Mem Mem
Data Center
Virtual
Supercomputer
GPU
GPU
Mem
GPUGPU
CPU
MemMem
Mem
Turns Discrete Computing Resources into a Virtual Supercomputer
What problems are we trying to solve?
Hardware is Stuck:
proper setup and optimization can take days
code portability
Software is Stuck
4
Operating system requirements
Library dependencies
Drivers
Interoperability between tools
Proper installation can take days
Hardware is Stuck
5
Code portability
Performance portability
Resource provisioning
Proper setup and optimization can take days
Goal
Given:
Applications from different vendors
Systems of different capabilities
Heterogeneous hardware
Compose a workflow that:
Works: individual components work, thus workflow works
Is Portable: workload can be migrated across infrastructure
Is Performant: has the ability to take advantage of GPU hardware
Is Secure: individual components can be easily audited
Current Solutions
Current solutions revolve around a common denominator:
Operating system that works for all tools in chain
Compute nodes which satisfy the most memory hungry application
Need GPUs? Must deploy on top of GPU only nodes
Cost sensitive? Must deploy on low-end CPU only nodes
Common denominator shortcoming: Inefficiencies
Poor utilization / over provisioning
Non-performant
Solution
Containerize all applications
◦ Create GPU/CPU versions
Assemble containers into workflow templates
◦ To represent particular use cases and pipelines
Use workflow templates to create virtual clusters
◦ Optimize performance / budget via virtual clusters
Containers are nothing new
Part of Linux for last 10 years
LXC, FreeBSD Jails, Solaris Containers, etc.
What is new are APIs
◦ Docker
◦ Rocket
◦ Etc.
Specifically
◦ A complete runtime environment: OS, application, libraries, dependencies, binaries, and configuration files
◦ Can be quickly deployed on a set of container hosts when needed
Containers vs. VMs (Stack Comparison)
Why Containers?
Easy Deployment
◦ Avoid hours of environment / application setup
◦ Fast environment spin-up / tear-down
Flexibility
◦ Applications use preferred version of OS, libs language versions, etc.
◦ Move data to application, or move Application to data
Reproducibility / Reliability / Scaling
◦ Workflow steps start with clean and immutable images
◦ Reliability through easy migration and checkpointing
DevOps Hell
Containerizing GPU Applications with Docker for Scaling to the Cloud
GPU Containers the NVIDIA Way
Much easier that it used to be
◦ One no longer has to fully reinstall the NVIDIA
driver within the container
◦ No more container vs. host system driver
matching conflicts - container works with host
OS driver - there is still a driver and toolkit
dependency
◦ https://github.com/NVIDIA/nvidia-docker
Requirements
◦ Host has NVIDIA Drivers
◦ Host has Docker installed
Shell Scripts
Chef
Puppet
Ansible
GPU Container Getting Started (CAFFE)
Create a Dockerfile
◦ Very small, easy to re-build/update container if needed
◦ Reproducible builds
◦ Specify Operating System
◦ Install Operating System basics
◦ Install Application Dependencies
◦ Install Application
Once Dockerfile is done:
Build Container, Test Container, Store Container in Repo
Quickly spin up and container where and when
needed
Enables “fire-and-forget” GPU applications
What about data?
Long answer: we’ll get to that in a bit
Short: Put it somewhere else, keep containers small
Dockerfile Code
Demo 1: Deploy GPU Container Across Clouds
Demo will show:
Launching container on Cloud #1 and execute application
Taking exact same container and launch on Cloud #2 and execute application
containers run on any cloud or datacenter or OS (even Windows)
containers use different types of GPUs and drivers
and everything works transparently
“fire and forget” GPU applications on GPU hardware you need wherever it may be
Container Performance
People are sceptical about container performance vs bare-metal
◦ There are special cases where performance can be an issue, but in general performance is on par, and
better than VMs
◦ Docker versus Bare Metal is within 10% performance
W. Felter, A. Ferreira, R. Rajamony, and J. Rubio. An Updated Performance Comparison of Virtual Machines and Linux
Containers.Technology, 28:32, 2014. (IBM)
So what about Data?
In general, avoid storing data in containers.
◦ Container ought to be immutable
◦ bring it up, perform a task, return the result, shut it down
◦ Containers ought to be small
◦ size of containers impacts startup times
◦ size of containers impacts time it takes to pull container from repository
Use Data Volume Containers
Application Flow Pipelines & Scheduling
Sophisticated tool flows rarely consist of a single application
◦ Some steps may only run on CPUs
◦ Some steps may execute on a CPU or a GPU
Challenge is how to schedule these flow efficiently to either obtain faster turnaround times or better
overall throughput, while maintaining reproducible results
Example: HPC workflow
Semiconductor Circuit Design
Example: HPC workflow
Semiconductor Circuit Design
CPU/GPU App
Containers and Schedulers
In general several assumptions can be made about today’s clusters
◦ # CPU nodes >> # GPU nodes
◦ GPU nodes have a fixed #GPUs in them
◦ Best machine for an application is usually determined by
◦ amount of memory
◦ amount and type of CPUs
◦ amount and type of GPUs
How can containers help with scheduling give this constraint vs. regular schedulers
Schedulers
What if we can break Physical Machine
Limitations?
Most cloud service provider and data centers are limited by physical limitations
◦ Example: Largest machines has 2 GPUs (Softlayer), 4 GPUs (AWS)
◦ Rack can only have max amount of GPUs due to power constraints
What if we could create virtual machines and clusters and present them to applications as a single
virtual machines?
How would this change the clusters and schedulers?
* Elastic Containers or Elastic Machines via Containers (grow or shrink)
Introduce Bitfusion Boost Containers
We can:
◦ Combine Bitfusion Boost and Containers -> create magic!
What things can we build?
◦ Create a machine which has 16 or more virtual GPUs!
◦ Run an application across these GPUs without having to setup MPI, SPARK, HADOOP!
◦ Run GPU applications on non-GPU machines by automatically offloading to GPU machines in the cluster
◦ All of the above can be done WITHOUT CODE CHANGES for GPU enabled applications!
Boost Container Building Blocks
Boost Server Container
◦ Boost provisioned container with Boost Server
◦ Runs on any GPU provisioned host
◦ Can act as a client at the same time
Boost Client Container
◦ Boost provisioned container with Boost Client and End User Application
◦ Runs on any-type of instance including CPU only instances
Boost Container Architecture
Demo 2: Build Virtual GPU Instances in the
Cloud
Demo will show the following using containers:
How in minutes we can create virtual GPU cluster configurations
How we can provision GPU machines which don’t exist in the physical world
How we can run GPU applications on non-GPU machines
How we can execute applications across these configurations without changing a single line of code!
On
cloud.bitfusion.io
Thank You
Visit Bitfusion Booth #731
@Bitfusionio | @subburama | subbu@bitfusion.io

More Related Content

What's hot

Kubernetes and Hybrid Deployments
Kubernetes and Hybrid DeploymentsKubernetes and Hybrid Deployments
Kubernetes and Hybrid DeploymentsSandeep Parikh
 
Cloud spanner architecture and use cases
Cloud spanner architecture and use casesCloud spanner architecture and use cases
Cloud spanner architecture and use casesGDG Cloud Bengaluru
 
Kubernetes laravel and kubernetes
Kubernetes   laravel and kubernetesKubernetes   laravel and kubernetes
Kubernetes laravel and kubernetesWilliam Stewart
 
Learn kubernetes in 90 minutes
Learn kubernetes in 90 minutesLearn kubernetes in 90 minutes
Learn kubernetes in 90 minutesLarry Cai
 
Kubernetes in Docker
Kubernetes in DockerKubernetes in Docker
Kubernetes in DockerDocker, Inc.
 
Introduction kubernetes 2017_12_24
Introduction kubernetes 2017_12_24Introduction kubernetes 2017_12_24
Introduction kubernetes 2017_12_24Sam Zheng
 
Kubernetes training
Kubernetes trainingKubernetes training
Kubernetes trainingDes Drury
 
Kubernetes architecture
Kubernetes architectureKubernetes architecture
Kubernetes architectureJanakiram MSV
 
Gordon's secret session kubernetes on windows
Gordon's secret session   kubernetes on windowsGordon's secret session   kubernetes on windows
Gordon's secret session kubernetes on windowsDocker, Inc.
 
GlueCon kubernetes & container engine
GlueCon kubernetes & container engineGlueCon kubernetes & container engine
GlueCon kubernetes & container enginebrendandburns
 
Going deep (learning) with tensor flow and quarkus
Going deep (learning) with tensor flow and quarkusGoing deep (learning) with tensor flow and quarkus
Going deep (learning) with tensor flow and quarkusRed Hat Developers
 
Distributed tensorflow on kubernetes
Distributed tensorflow on kubernetesDistributed tensorflow on kubernetes
Distributed tensorflow on kubernetesinwin stack
 
KubeCon EU 2016: Bringing an open source Containerized Container Platform to ...
KubeCon EU 2016: Bringing an open source Containerized Container Platform to ...KubeCon EU 2016: Bringing an open source Containerized Container Platform to ...
KubeCon EU 2016: Bringing an open source Containerized Container Platform to ...KubeAcademy
 
2016 - Continuously Delivering Microservices in Kubernetes using Jenkins
2016 - Continuously Delivering Microservices in Kubernetes using Jenkins2016 - Continuously Delivering Microservices in Kubernetes using Jenkins
2016 - Continuously Delivering Microservices in Kubernetes using Jenkinsdevopsdaysaustin
 
Production sec ops with kubernetes in docker
Production sec ops with kubernetes in dockerProduction sec ops with kubernetes in docker
Production sec ops with kubernetes in dockerDocker, Inc.
 
Revolutionizing WSO2 PaaS with Kubernetes & App Factory
Revolutionizing WSO2 PaaS with Kubernetes & App FactoryRevolutionizing WSO2 PaaS with Kubernetes & App Factory
Revolutionizing WSO2 PaaS with Kubernetes & App FactoryImesh Gunaratne
 
How to integrate Kubernetes in OpenStack: You need to know these project
How to integrate Kubernetes in OpenStack: You need to know these projectHow to integrate Kubernetes in OpenStack: You need to know these project
How to integrate Kubernetes in OpenStack: You need to know these projectinwin stack
 

What's hot (20)

Kubernetes and Hybrid Deployments
Kubernetes and Hybrid DeploymentsKubernetes and Hybrid Deployments
Kubernetes and Hybrid Deployments
 
Kubernetes 101 Workshop
Kubernetes 101 WorkshopKubernetes 101 Workshop
Kubernetes 101 Workshop
 
Cloud spanner architecture and use cases
Cloud spanner architecture and use casesCloud spanner architecture and use cases
Cloud spanner architecture and use cases
 
Kubernetes laravel and kubernetes
Kubernetes   laravel and kubernetesKubernetes   laravel and kubernetes
Kubernetes laravel and kubernetes
 
Learn kubernetes in 90 minutes
Learn kubernetes in 90 minutesLearn kubernetes in 90 minutes
Learn kubernetes in 90 minutes
 
My kubernetes toolkit
My kubernetes toolkitMy kubernetes toolkit
My kubernetes toolkit
 
Kubernetes in Docker
Kubernetes in DockerKubernetes in Docker
Kubernetes in Docker
 
Introduction kubernetes 2017_12_24
Introduction kubernetes 2017_12_24Introduction kubernetes 2017_12_24
Introduction kubernetes 2017_12_24
 
Kubernetes training
Kubernetes trainingKubernetes training
Kubernetes training
 
Kubernetes architecture
Kubernetes architectureKubernetes architecture
Kubernetes architecture
 
Gordon's secret session kubernetes on windows
Gordon's secret session   kubernetes on windowsGordon's secret session   kubernetes on windows
Gordon's secret session kubernetes on windows
 
GlueCon kubernetes & container engine
GlueCon kubernetes & container engineGlueCon kubernetes & container engine
GlueCon kubernetes & container engine
 
Going deep (learning) with tensor flow and quarkus
Going deep (learning) with tensor flow and quarkusGoing deep (learning) with tensor flow and quarkus
Going deep (learning) with tensor flow and quarkus
 
Distributed tensorflow on kubernetes
Distributed tensorflow on kubernetesDistributed tensorflow on kubernetes
Distributed tensorflow on kubernetes
 
KubeCon EU 2016: Bringing an open source Containerized Container Platform to ...
KubeCon EU 2016: Bringing an open source Containerized Container Platform to ...KubeCon EU 2016: Bringing an open source Containerized Container Platform to ...
KubeCon EU 2016: Bringing an open source Containerized Container Platform to ...
 
"Kubernetes as Driver of Generic IT Automation"
"Kubernetes as Driver of Generic IT Automation""Kubernetes as Driver of Generic IT Automation"
"Kubernetes as Driver of Generic IT Automation"
 
2016 - Continuously Delivering Microservices in Kubernetes using Jenkins
2016 - Continuously Delivering Microservices in Kubernetes using Jenkins2016 - Continuously Delivering Microservices in Kubernetes using Jenkins
2016 - Continuously Delivering Microservices in Kubernetes using Jenkins
 
Production sec ops with kubernetes in docker
Production sec ops with kubernetes in dockerProduction sec ops with kubernetes in docker
Production sec ops with kubernetes in docker
 
Revolutionizing WSO2 PaaS with Kubernetes & App Factory
Revolutionizing WSO2 PaaS with Kubernetes & App FactoryRevolutionizing WSO2 PaaS with Kubernetes & App Factory
Revolutionizing WSO2 PaaS with Kubernetes & App Factory
 
How to integrate Kubernetes in OpenStack: You need to know these project
How to integrate Kubernetes in OpenStack: You need to know these projectHow to integrate Kubernetes in OpenStack: You need to know these project
How to integrate Kubernetes in OpenStack: You need to know these project
 

Similar to Containerizing GPU Applications with Docker for Scaling to the Cloud

OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...NETWAYS
 
Enabling Cognitive Workloads on the Cloud: GPUs with Mesos, Docker and Marath...
Enabling Cognitive Workloads on the Cloud: GPUs with Mesos, Docker and Marath...Enabling Cognitive Workloads on the Cloud: GPUs with Mesos, Docker and Marath...
Enabling Cognitive Workloads on the Cloud: GPUs with Mesos, Docker and Marath...Indrajit Poddar
 
Week 8 lecture material
Week 8 lecture materialWeek 8 lecture material
Week 8 lecture materialAnkit Gupta
 
Open shift and docker - october,2014
Open shift and docker - october,2014Open shift and docker - october,2014
Open shift and docker - october,2014Hojoong Kim
 
Detailed Introduction To Docker
Detailed Introduction To DockerDetailed Introduction To Docker
Detailed Introduction To Dockernklmish
 
Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications OpenEBS
 
Docker dev ops for cd meetup 12-14
Docker dev ops for cd meetup 12-14Docker dev ops for cd meetup 12-14
Docker dev ops for cd meetup 12-14Simon Storm
 
Introduction to PaaS and Heroku
Introduction to PaaS and HerokuIntroduction to PaaS and Heroku
Introduction to PaaS and HerokuTapio Rautonen
 
.NET Cloud-Native Bootcamp
.NET Cloud-Native Bootcamp.NET Cloud-Native Bootcamp
.NET Cloud-Native BootcampVMware Tanzu
 
Meteor South Bay Meetup - Kubernetes & Google Container Engine
Meteor South Bay Meetup - Kubernetes & Google Container EngineMeteor South Bay Meetup - Kubernetes & Google Container Engine
Meteor South Bay Meetup - Kubernetes & Google Container EngineKit Merker
 
Docker for the enterprise
Docker for the enterpriseDocker for the enterprise
Docker for the enterpriseBert Poller
 
Docker & aPaaS: Enterprise Innovation and Trends for 2015
Docker & aPaaS: Enterprise Innovation and Trends for 2015Docker & aPaaS: Enterprise Innovation and Trends for 2015
Docker & aPaaS: Enterprise Innovation and Trends for 2015WaveMaker, Inc.
 
DCEU 18: Use Cases and Practical Solutions for Docker Container Storage on Sw...
DCEU 18: Use Cases and Practical Solutions for Docker Container Storage on Sw...DCEU 18: Use Cases and Practical Solutions for Docker Container Storage on Sw...
DCEU 18: Use Cases and Practical Solutions for Docker Container Storage on Sw...Docker, Inc.
 
Docker - A high level introduction to dockers and containers
Docker - A high level introduction to dockers and containersDocker - A high level introduction to dockers and containers
Docker - A high level introduction to dockers and containersDr Ganesh Iyer
 
Kubernetes is all you need
Kubernetes is all you needKubernetes is all you need
Kubernetes is all you needVishwas N
 
Introduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OSIntroduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OSSteve Wong
 
create auto scale jboss cluster with openshift
create auto scale jboss cluster with openshiftcreate auto scale jboss cluster with openshift
create auto scale jboss cluster with openshiftYusuf Hadiwinata Sutandar
 

Similar to Containerizing GPU Applications with Docker for Scaling to the Cloud (20)

OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...
 
Where should I run my code? Serverless, Containers, Virtual Machines and more
Where should I run my code? Serverless, Containers, Virtual Machines and moreWhere should I run my code? Serverless, Containers, Virtual Machines and more
Where should I run my code? Serverless, Containers, Virtual Machines and more
 
Enabling Cognitive Workloads on the Cloud: GPUs with Mesos, Docker and Marath...
Enabling Cognitive Workloads on the Cloud: GPUs with Mesos, Docker and Marath...Enabling Cognitive Workloads on the Cloud: GPUs with Mesos, Docker and Marath...
Enabling Cognitive Workloads on the Cloud: GPUs with Mesos, Docker and Marath...
 
Week 8 lecture material
Week 8 lecture materialWeek 8 lecture material
Week 8 lecture material
 
Open shift and docker - october,2014
Open shift and docker - october,2014Open shift and docker - october,2014
Open shift and docker - october,2014
 
Detailed Introduction To Docker
Detailed Introduction To DockerDetailed Introduction To Docker
Detailed Introduction To Docker
 
Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications
 
Docker dev ops for cd meetup 12-14
Docker dev ops for cd meetup 12-14Docker dev ops for cd meetup 12-14
Docker dev ops for cd meetup 12-14
 
Introduction to PaaS and Heroku
Introduction to PaaS and HerokuIntroduction to PaaS and Heroku
Introduction to PaaS and Heroku
 
.NET Cloud-Native Bootcamp
.NET Cloud-Native Bootcamp.NET Cloud-Native Bootcamp
.NET Cloud-Native Bootcamp
 
Meteor South Bay Meetup - Kubernetes & Google Container Engine
Meteor South Bay Meetup - Kubernetes & Google Container EngineMeteor South Bay Meetup - Kubernetes & Google Container Engine
Meteor South Bay Meetup - Kubernetes & Google Container Engine
 
Docker for the enterprise
Docker for the enterpriseDocker for the enterprise
Docker for the enterprise
 
Containers and Docker
Containers and DockerContainers and Docker
Containers and Docker
 
56k.cloud training
56k.cloud training56k.cloud training
56k.cloud training
 
Docker & aPaaS: Enterprise Innovation and Trends for 2015
Docker & aPaaS: Enterprise Innovation and Trends for 2015Docker & aPaaS: Enterprise Innovation and Trends for 2015
Docker & aPaaS: Enterprise Innovation and Trends for 2015
 
DCEU 18: Use Cases and Practical Solutions for Docker Container Storage on Sw...
DCEU 18: Use Cases and Practical Solutions for Docker Container Storage on Sw...DCEU 18: Use Cases and Practical Solutions for Docker Container Storage on Sw...
DCEU 18: Use Cases and Practical Solutions for Docker Container Storage on Sw...
 
Docker - A high level introduction to dockers and containers
Docker - A high level introduction to dockers and containersDocker - A high level introduction to dockers and containers
Docker - A high level introduction to dockers and containers
 
Kubernetes is all you need
Kubernetes is all you needKubernetes is all you need
Kubernetes is all you need
 
Introduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OSIntroduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OS
 
create auto scale jboss cluster with openshift
create auto scale jboss cluster with openshiftcreate auto scale jboss cluster with openshift
create auto scale jboss cluster with openshift
 

Recently uploaded

GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncObject Automation
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 
Unleashing the power of AI in UiPath Studio with UiPath Autopilot.
Unleashing the power of AI in UiPath Studio with UiPath Autopilot.Unleashing the power of AI in UiPath Studio with UiPath Autopilot.
Unleashing the power of AI in UiPath Studio with UiPath Autopilot.DianaGray10
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
Plant tissue culture pharmacongosy-1 Semester 4
Plant tissue culture pharmacongosy-1 Semester 4Plant tissue culture pharmacongosy-1 Semester 4
Plant tissue culture pharmacongosy-1 Semester 4Nandakishor Deshmukh
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Reference Domain Ontologies and Large Medical Language Models.pptx
Reference Domain Ontologies and Large Medical Language Models.pptxReference Domain Ontologies and Large Medical Language Models.pptx
Reference Domain Ontologies and Large Medical Language Models.pptxChimezie Ogbuji
 
Monitoring Java Application Security with JDK Tools and JFR Events.pdf
Monitoring Java Application Security with JDK Tools and JFR Events.pdfMonitoring Java Application Security with JDK Tools and JFR Events.pdf
Monitoring Java Application Security with JDK Tools and JFR Events.pdfAna-Maria Mihalceanu
 
GDG Cloud Southlake 31: Santosh Chennuri and Festus Yeboah: Empowering Develo...
GDG Cloud Southlake 31: Santosh Chennuri and Festus Yeboah: Empowering Develo...GDG Cloud Southlake 31: Santosh Chennuri and Festus Yeboah: Empowering Develo...
GDG Cloud Southlake 31: Santosh Chennuri and Festus Yeboah: Empowering Develo...James Anderson
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Dev Dives: Master advanced authentication and performance in Productivity Act...
Dev Dives: Master advanced authentication and performance in Productivity Act...Dev Dives: Master advanced authentication and performance in Productivity Act...
Dev Dives: Master advanced authentication and performance in Productivity Act...UiPathCommunity
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
ServiceNow Integration with MuleSoft.pptx
ServiceNow Integration with MuleSoft.pptxServiceNow Integration with MuleSoft.pptx
ServiceNow Integration with MuleSoft.pptxshyamraj55
 
AI Health Agents: Longevity as a Service in the Web3 GenAI Quantum Revolution
AI Health Agents: Longevity as a Service in the Web3 GenAI Quantum RevolutionAI Health Agents: Longevity as a Service in the Web3 GenAI Quantum Revolution
AI Health Agents: Longevity as a Service in the Web3 GenAI Quantum RevolutionMelanie Swan
 

Recently uploaded (20)

GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 
Unleashing the power of AI in UiPath Studio with UiPath Autopilot.
Unleashing the power of AI in UiPath Studio with UiPath Autopilot.Unleashing the power of AI in UiPath Studio with UiPath Autopilot.
Unleashing the power of AI in UiPath Studio with UiPath Autopilot.
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
Plant tissue culture pharmacongosy-1 Semester 4
Plant tissue culture pharmacongosy-1 Semester 4Plant tissue culture pharmacongosy-1 Semester 4
Plant tissue culture pharmacongosy-1 Semester 4
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Reference Domain Ontologies and Large Medical Language Models.pptx
Reference Domain Ontologies and Large Medical Language Models.pptxReference Domain Ontologies and Large Medical Language Models.pptx
Reference Domain Ontologies and Large Medical Language Models.pptx
 
Monitoring Java Application Security with JDK Tools and JFR Events.pdf
Monitoring Java Application Security with JDK Tools and JFR Events.pdfMonitoring Java Application Security with JDK Tools and JFR Events.pdf
Monitoring Java Application Security with JDK Tools and JFR Events.pdf
 
GDG Cloud Southlake 31: Santosh Chennuri and Festus Yeboah: Empowering Develo...
GDG Cloud Southlake 31: Santosh Chennuri and Festus Yeboah: Empowering Develo...GDG Cloud Southlake 31: Santosh Chennuri and Festus Yeboah: Empowering Develo...
GDG Cloud Southlake 31: Santosh Chennuri and Festus Yeboah: Empowering Develo...
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Dev Dives: Master advanced authentication and performance in Productivity Act...
Dev Dives: Master advanced authentication and performance in Productivity Act...Dev Dives: Master advanced authentication and performance in Productivity Act...
Dev Dives: Master advanced authentication and performance in Productivity Act...
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
ServiceNow Integration with MuleSoft.pptx
ServiceNow Integration with MuleSoft.pptxServiceNow Integration with MuleSoft.pptx
ServiceNow Integration with MuleSoft.pptx
 
AI Health Agents: Longevity as a Service in the Web3 GenAI Quantum Revolution
AI Health Agents: Longevity as a Service in the Web3 GenAI Quantum RevolutionAI Health Agents: Longevity as a Service in the Web3 GenAI Quantum Revolution
AI Health Agents: Longevity as a Service in the Web3 GenAI Quantum Revolution
 

Containerizing GPU Applications with Docker for Scaling to the Cloud

  • 1. Containerizing GPU Applications with Docker for Scaling to the Cloud FUTURE OF PACKAGING APPLICATIONS SUBBU RAMA
  • 2. CPU CPU CPU GPU GPU GPU GPU Mem MemMemMem CPU CPU CPU CPU GPU GPU GPU GPU GPU GPU GPU GPU Mem Mem Mem Mem Mem Mem Mem Mem Data Center Virtual Supercomputer GPU GPU Mem GPUGPU CPU MemMem Mem Turns Discrete Computing Resources into a Virtual Supercomputer
  • 3. What problems are we trying to solve? Hardware is Stuck: proper setup and optimization can take days code portability
  • 4. Software is Stuck 4 Operating system requirements Library dependencies Drivers Interoperability between tools Proper installation can take days
  • 5. Hardware is Stuck 5 Code portability Performance portability Resource provisioning Proper setup and optimization can take days
  • 6. Goal Given: Applications from different vendors Systems of different capabilities Heterogeneous hardware Compose a workflow that: Works: individual components work, thus workflow works Is Portable: workload can be migrated across infrastructure Is Performant: has the ability to take advantage of GPU hardware Is Secure: individual components can be easily audited
  • 7. Current Solutions Current solutions revolve around a common denominator: Operating system that works for all tools in chain Compute nodes which satisfy the most memory hungry application Need GPUs? Must deploy on top of GPU only nodes Cost sensitive? Must deploy on low-end CPU only nodes Common denominator shortcoming: Inefficiencies Poor utilization / over provisioning Non-performant
  • 8. Solution Containerize all applications ◦ Create GPU/CPU versions Assemble containers into workflow templates ◦ To represent particular use cases and pipelines Use workflow templates to create virtual clusters ◦ Optimize performance / budget via virtual clusters
  • 9. Containers are nothing new Part of Linux for last 10 years LXC, FreeBSD Jails, Solaris Containers, etc. What is new are APIs ◦ Docker ◦ Rocket ◦ Etc. Specifically ◦ A complete runtime environment: OS, application, libraries, dependencies, binaries, and configuration files ◦ Can be quickly deployed on a set of container hosts when needed
  • 10. Containers vs. VMs (Stack Comparison)
  • 11. Why Containers? Easy Deployment ◦ Avoid hours of environment / application setup ◦ Fast environment spin-up / tear-down Flexibility ◦ Applications use preferred version of OS, libs language versions, etc. ◦ Move data to application, or move Application to data Reproducibility / Reliability / Scaling ◦ Workflow steps start with clean and immutable images ◦ Reliability through easy migration and checkpointing
  • 14. GPU Containers the NVIDIA Way Much easier that it used to be ◦ One no longer has to fully reinstall the NVIDIA driver within the container ◦ No more container vs. host system driver matching conflicts - container works with host OS driver - there is still a driver and toolkit dependency ◦ https://github.com/NVIDIA/nvidia-docker Requirements ◦ Host has NVIDIA Drivers ◦ Host has Docker installed
  • 16. GPU Container Getting Started (CAFFE) Create a Dockerfile ◦ Very small, easy to re-build/update container if needed ◦ Reproducible builds ◦ Specify Operating System ◦ Install Operating System basics ◦ Install Application Dependencies ◦ Install Application Once Dockerfile is done: Build Container, Test Container, Store Container in Repo Quickly spin up and container where and when needed Enables “fire-and-forget” GPU applications What about data? Long answer: we’ll get to that in a bit Short: Put it somewhere else, keep containers small
  • 18. Demo 1: Deploy GPU Container Across Clouds Demo will show: Launching container on Cloud #1 and execute application Taking exact same container and launch on Cloud #2 and execute application containers run on any cloud or datacenter or OS (even Windows) containers use different types of GPUs and drivers and everything works transparently “fire and forget” GPU applications on GPU hardware you need wherever it may be
  • 19. Container Performance People are sceptical about container performance vs bare-metal ◦ There are special cases where performance can be an issue, but in general performance is on par, and better than VMs ◦ Docker versus Bare Metal is within 10% performance W. Felter, A. Ferreira, R. Rajamony, and J. Rubio. An Updated Performance Comparison of Virtual Machines and Linux Containers.Technology, 28:32, 2014. (IBM)
  • 20. So what about Data? In general, avoid storing data in containers. ◦ Container ought to be immutable ◦ bring it up, perform a task, return the result, shut it down ◦ Containers ought to be small ◦ size of containers impacts startup times ◦ size of containers impacts time it takes to pull container from repository Use Data Volume Containers
  • 21. Application Flow Pipelines & Scheduling Sophisticated tool flows rarely consist of a single application ◦ Some steps may only run on CPUs ◦ Some steps may execute on a CPU or a GPU Challenge is how to schedule these flow efficiently to either obtain faster turnaround times or better overall throughput, while maintaining reproducible results
  • 23. Example: HPC workflow Semiconductor Circuit Design CPU/GPU App
  • 24. Containers and Schedulers In general several assumptions can be made about today’s clusters ◦ # CPU nodes >> # GPU nodes ◦ GPU nodes have a fixed #GPUs in them ◦ Best machine for an application is usually determined by ◦ amount of memory ◦ amount and type of CPUs ◦ amount and type of GPUs How can containers help with scheduling give this constraint vs. regular schedulers
  • 26. What if we can break Physical Machine Limitations? Most cloud service provider and data centers are limited by physical limitations ◦ Example: Largest machines has 2 GPUs (Softlayer), 4 GPUs (AWS) ◦ Rack can only have max amount of GPUs due to power constraints What if we could create virtual machines and clusters and present them to applications as a single virtual machines? How would this change the clusters and schedulers? * Elastic Containers or Elastic Machines via Containers (grow or shrink)
  • 27. Introduce Bitfusion Boost Containers We can: ◦ Combine Bitfusion Boost and Containers -> create magic! What things can we build? ◦ Create a machine which has 16 or more virtual GPUs! ◦ Run an application across these GPUs without having to setup MPI, SPARK, HADOOP! ◦ Run GPU applications on non-GPU machines by automatically offloading to GPU machines in the cluster ◦ All of the above can be done WITHOUT CODE CHANGES for GPU enabled applications!
  • 28. Boost Container Building Blocks Boost Server Container ◦ Boost provisioned container with Boost Server ◦ Runs on any GPU provisioned host ◦ Can act as a client at the same time Boost Client Container ◦ Boost provisioned container with Boost Client and End User Application ◦ Runs on any-type of instance including CPU only instances
  • 30. Demo 2: Build Virtual GPU Instances in the Cloud Demo will show the following using containers: How in minutes we can create virtual GPU cluster configurations How we can provision GPU machines which don’t exist in the physical world How we can run GPU applications on non-GPU machines How we can execute applications across these configurations without changing a single line of code!
  • 31. On
  • 33. Thank You Visit Bitfusion Booth #731 @Bitfusionio | @subburama | subbu@bitfusion.io