SlideShare a Scribd company logo
1
Docker Volume Isolator In Mesos
Jie Yu, Gilbert Song, Guangya Liu
2
Gilbert Song
Distributed Systems Engineer
Mesosphere
Guangya LiuJie Yu
Distributed Systems Engineer
Apache Mesos PMC & Committer
Mesosphere
Senior Software Engineer
IBM
3
If your containers are “cattle”, you have nothing to
lose...
The test: I don’t need persistence beyond the life of
the container instance or host.
But what to do if your containers are pet, if you
containers need to persist data?
Motivation
4
Stateful Service:
A service that needs to persist data/state
A typical stateful service
● Use external volume services
● Stateless to Mesos
Stateful services (like databases) need
durable and scalable state
Motivation
5
There are many docker
images on DockerHub are
databases...
Motivation
6
Current state of storage solution in `MesosContainerizer`
Local
● Persistent Volume
Master
Mesos Agent
Aurora Executor
Tasks
mysqld
datanode
Hadoop Executor
Tasks
Operating System
7
But local is not enough!!
Local
● Availability Risk
○ Migrate container to another host –
your storage is gone
○ Host goes down – your service goes
down
● Scale Limitation
○ Need more storage than the host has?
Sorry…
8
What we would like?
External Distributed Volume for stateful service
Master
Mesos Agent
Aurora Executor
Tasks
mysqld
datanode
Hadoop Executor
Tasks
Operating System
9
The challenges
+ others
Give operator the freedom to choose different external storage platforms.
10
Docker with External Volume
IBM Spectrum Scale AWS GCE OpenStack
Docker Volume Driver
11
Mesos with External Volume
IBM Spectrum Scale AWS GCE OpenStack
Docker Volume Driver
Docker Volume Isolator
Docker Volume Isolator
IBM Spectrum Scale AWS GCE OpenStack
Docker Volume Driver
● A driver based model
● Docker Volume Isolator takes care of
business intelligence:
○ Mount with driver name, volume name and
some options.
○ Unmount with driver name and volume name.
12
13
Docker Volume Isolator Architecture
HTTP/JSON UNIX/TCP SOCKETS
for Docker Volume API
Docker Volume Driver Plugin
(Convoy, Rex-ray, Flocker)
Docker Volume
Driver Client
Docker Volume
Isolator
dvdcli
Docker Daemon
(Optional)
Docker Volumes
Mount: dvdcli mount –volumedriver=convoy –volumename=vol1 --volumeopts=size=5 --volumeopts=iops=150
Unmount: dvdcli unmount --volumedriver=convoy --volumename=vol1
DeviceMapper NFS Amazon EBS GlusterFS Ceph
14
Some internal details
How does it works (with Rex-ray)?
Mesos Agent docker/volume dvdcli Rex-ray
Container
1) Framework launch a task
2) Provision the container
3) Get the mount point from dvdcli
4) Mount the external volume to mesos container
EBS
16
● Have `dvdcli` installed on each agent
● Volume Lifecycle management (create, delete)
● Use the same Docker volume in both the DockerContainerizer and the
MesosContainerizer simultaneously is strongly discouraged
○ MesosContainerizer has its own reference counting to decide when to unmount a Docker
volume.
○ It would be problematic if a Docker volume is unmounted by MesosContainerizer but the
DockerContainerizer is still using it.
Limitations
17
● Get rid of `dvdcli`
Future Work
18
● Part of DC/OS
● Create Volumes with your normal Marathon App Definition
● Track via Marathon UI
● Reuse the same volume after app restart
External Volumes in Marathon and DC/OS
19
Demo Time
DEMO Time!!
20
Thank you!
Questions?

More Related Content

What's hot

Hands On Gluster with Jeff Darcy
Hands On Gluster with Jeff DarcyHands On Gluster with Jeff Darcy
Hands On Gluster with Jeff Darcy
Gluster.org
 
Integration of Glusterfs in to commvault simpana
Integration of Glusterfs in to commvault simpanaIntegration of Glusterfs in to commvault simpana
Integration of Glusterfs in to commvault simpana
Gluster.org
 
CoreOS Intro
CoreOS IntroCoreOS Intro
CoreOS Intro
Isaac Johnston
 
Kubernetes Architecture and Introduction – Paris Kubernetes Meetup
Kubernetes Architecture and Introduction – Paris Kubernetes MeetupKubernetes Architecture and Introduction – Paris Kubernetes Meetup
Kubernetes Architecture and Introduction – Paris Kubernetes Meetup
Stefan Schimanski
 
Docker Architecture (v1.3)
Docker Architecture (v1.3)Docker Architecture (v1.3)
Docker Architecture (v1.3)
rajdeep
 
Openvz booth
Openvz boothOpenvz booth
Openvz booth
OpenVZ
 
Solr on Docker - the Good, the Bad and the Ugly
Solr on Docker - the Good, the Bad and the UglySolr on Docker - the Good, the Bad and the Ugly
Solr on Docker - the Good, the Bad and the Ugly
Sematext Group, Inc.
 
Kubernetes introduction
Kubernetes introductionKubernetes introduction
Kubernetes introduction
Dongwon Kim
 
Docker Overview
Docker OverviewDocker Overview
Docker Overview
Rohit Jnagal
 
Gluster and Kubernetes
Gluster and KubernetesGluster and Kubernetes
Gluster and Kubernetes
Gluster.org
 
Wanting distributed volumes - Experiences with ceph-docker
Wanting distributed volumes - Experiences with ceph-dockerWanting distributed volumes - Experiences with ceph-docker
Wanting distributed volumes - Experiences with ceph-docker
Ewout Prangsma
 
Xen virtualization and multi-brick enviornment experiences
Xen virtualization and multi-brick enviornment experiencesXen virtualization and multi-brick enviornment experiences
Xen virtualization and multi-brick enviornment experiences
mrpetersisl
 
CoreOS Overview
CoreOS OverviewCoreOS Overview
CoreOS Overview
Nikolay Yurin
 
OSv – The OS designed for the Cloud
OSv – The OS designed for the CloudOSv – The OS designed for the Cloud
OSv – The OS designed for the Cloud
Yandex
 
OSv presentation from Linux Foundation Collaboration Summit
OSv presentation from Linux Foundation Collaboration SummitOSv presentation from Linux Foundation Collaboration Summit
OSv presentation from Linux Foundation Collaboration Summit
Don Marti
 
CloudStack Automated Integration Testing with Marvin
CloudStack Automated Integration Testing with Marvin CloudStack Automated Integration Testing with Marvin
CloudStack Automated Integration Testing with Marvin
NetApp
 
Guaranteeing CloudStack Storage Performance
Guaranteeing CloudStack Storage Performance Guaranteeing CloudStack Storage Performance
Guaranteeing CloudStack Storage Performance
NetApp
 
CoreOS @Codetalks Hamburg
CoreOS @Codetalks HamburgCoreOS @Codetalks Hamburg
CoreOS @Codetalks Hamburg
Timo Derstappen
 
Docker 1.11 @ Docker SF Meetup
Docker 1.11 @ Docker SF MeetupDocker 1.11 @ Docker SF Meetup
Docker 1.11 @ Docker SF Meetup
Docker, Inc.
 
Docker internals
Docker internalsDocker internals
Docker internals
Rohit Jnagal
 

What's hot (20)

Hands On Gluster with Jeff Darcy
Hands On Gluster with Jeff DarcyHands On Gluster with Jeff Darcy
Hands On Gluster with Jeff Darcy
 
Integration of Glusterfs in to commvault simpana
Integration of Glusterfs in to commvault simpanaIntegration of Glusterfs in to commvault simpana
Integration of Glusterfs in to commvault simpana
 
CoreOS Intro
CoreOS IntroCoreOS Intro
CoreOS Intro
 
Kubernetes Architecture and Introduction – Paris Kubernetes Meetup
Kubernetes Architecture and Introduction – Paris Kubernetes MeetupKubernetes Architecture and Introduction – Paris Kubernetes Meetup
Kubernetes Architecture and Introduction – Paris Kubernetes Meetup
 
Docker Architecture (v1.3)
Docker Architecture (v1.3)Docker Architecture (v1.3)
Docker Architecture (v1.3)
 
Openvz booth
Openvz boothOpenvz booth
Openvz booth
 
Solr on Docker - the Good, the Bad and the Ugly
Solr on Docker - the Good, the Bad and the UglySolr on Docker - the Good, the Bad and the Ugly
Solr on Docker - the Good, the Bad and the Ugly
 
Kubernetes introduction
Kubernetes introductionKubernetes introduction
Kubernetes introduction
 
Docker Overview
Docker OverviewDocker Overview
Docker Overview
 
Gluster and Kubernetes
Gluster and KubernetesGluster and Kubernetes
Gluster and Kubernetes
 
Wanting distributed volumes - Experiences with ceph-docker
Wanting distributed volumes - Experiences with ceph-dockerWanting distributed volumes - Experiences with ceph-docker
Wanting distributed volumes - Experiences with ceph-docker
 
Xen virtualization and multi-brick enviornment experiences
Xen virtualization and multi-brick enviornment experiencesXen virtualization and multi-brick enviornment experiences
Xen virtualization and multi-brick enviornment experiences
 
CoreOS Overview
CoreOS OverviewCoreOS Overview
CoreOS Overview
 
OSv – The OS designed for the Cloud
OSv – The OS designed for the CloudOSv – The OS designed for the Cloud
OSv – The OS designed for the Cloud
 
OSv presentation from Linux Foundation Collaboration Summit
OSv presentation from Linux Foundation Collaboration SummitOSv presentation from Linux Foundation Collaboration Summit
OSv presentation from Linux Foundation Collaboration Summit
 
CloudStack Automated Integration Testing with Marvin
CloudStack Automated Integration Testing with Marvin CloudStack Automated Integration Testing with Marvin
CloudStack Automated Integration Testing with Marvin
 
Guaranteeing CloudStack Storage Performance
Guaranteeing CloudStack Storage Performance Guaranteeing CloudStack Storage Performance
Guaranteeing CloudStack Storage Performance
 
CoreOS @Codetalks Hamburg
CoreOS @Codetalks HamburgCoreOS @Codetalks Hamburg
CoreOS @Codetalks Hamburg
 
Docker 1.11 @ Docker SF Meetup
Docker 1.11 @ Docker SF MeetupDocker 1.11 @ Docker SF Meetup
Docker 1.11 @ Docker SF Meetup
 
Docker internals
Docker internalsDocker internals
Docker internals
 

Similar to Docker volume-isolator-in-mesos

SQL Server in DevOps Town Hall Webinar
SQL Server in DevOps Town Hall WebinarSQL Server in DevOps Town Hall Webinar
SQL Server in DevOps Town Hall Webinar
Travis Wright
 
Containers and Nutanix - Acropolis Container Services
Containers and Nutanix - Acropolis Container ServicesContainers and Nutanix - Acropolis Container Services
Containers and Nutanix - Acropolis Container Services
NEXTtour
 
Introduction to Docker storage, volume and image
Introduction to Docker storage, volume and imageIntroduction to Docker storage, volume and image
Introduction to Docker storage, volume and image
ejlp12
 
Real-World Docker: 10 Things We've Learned
Real-World Docker: 10 Things We've Learned  Real-World Docker: 10 Things We've Learned
Real-World Docker: 10 Things We've Learned
RightScale
 
Michigan IT Symposium 2017 - Container BOF
Michigan IT Symposium 2017 - Container BOFMichigan IT Symposium 2017 - Container BOF
Michigan IT Symposium 2017 - Container BOF
Jeffrey Sica
 
Docker at Cloud9 IDE
Docker at Cloud9 IDEDocker at Cloud9 IDE
Docker at Cloud9 IDE
lennartkats
 
Accelerate your development with Docker
Accelerate your development with DockerAccelerate your development with Docker
Accelerate your development with Docker
Andrey Hristov
 
Accelerate your software development with Docker
Accelerate your software development with DockerAccelerate your software development with Docker
Accelerate your software development with Docker
Andrey Hristov
 
Docker and kubernetes
Docker and kubernetesDocker and kubernetes
Docker and kubernetes
Dongwon Kim
 
A Gentle Introduction to Docker and Containers
A Gentle Introduction to Docker and ContainersA Gentle Introduction to Docker and Containers
A Gentle Introduction to Docker and Containers
Docker, Inc.
 
Docker from A to Z, including Swarm and OCCS
Docker from A to Z, including Swarm and OCCSDocker from A to Z, including Swarm and OCCS
Docker from A to Z, including Swarm and OCCS
Frank Munz
 
Cassandra and docker
Cassandra and dockerCassandra and docker
Cassandra and docker
Ben Bromhead
 
Introduction to Docker and Monitoring with InfluxData
Introduction to Docker and Monitoring with InfluxDataIntroduction to Docker and Monitoring with InfluxData
Introduction to Docker and Monitoring with InfluxData
InfluxData
 
The internals and the latest trends of container runtimes
The internals and the latest trends of container runtimesThe internals and the latest trends of container runtimes
The internals and the latest trends of container runtimes
Akihiro Suda
 
codemotion-docker-2014
codemotion-docker-2014codemotion-docker-2014
codemotion-docker-2014
Carlo Bonamico
 
Why everyone is excited about Docker (and you should too...) - Carlo Bonamic...
Why everyone is excited about Docker (and you should too...) -  Carlo Bonamic...Why everyone is excited about Docker (and you should too...) -  Carlo Bonamic...
Why everyone is excited about Docker (and you should too...) - Carlo Bonamic...
Codemotion
 
Docker 101
Docker 101 Docker 101
Docker 101
Kevin Nord
 
Docker handons-workshop-for-charity
Docker handons-workshop-for-charityDocker handons-workshop-for-charity
Docker handons-workshop-for-charity
Yusuf Hadiwinata Sutandar
 
Improving Development and Deployment with Docker
Improving Development and Deployment with DockerImproving Development and Deployment with Docker
Improving Development and Deployment with Docker
Irfad Razick
 
Containerize! Between Docker and Jube.
Containerize! Between Docker and Jube.Containerize! Between Docker and Jube.
Containerize! Between Docker and Jube.
Henryk Konsek
 

Similar to Docker volume-isolator-in-mesos (20)

SQL Server in DevOps Town Hall Webinar
SQL Server in DevOps Town Hall WebinarSQL Server in DevOps Town Hall Webinar
SQL Server in DevOps Town Hall Webinar
 
Containers and Nutanix - Acropolis Container Services
Containers and Nutanix - Acropolis Container ServicesContainers and Nutanix - Acropolis Container Services
Containers and Nutanix - Acropolis Container Services
 
Introduction to Docker storage, volume and image
Introduction to Docker storage, volume and imageIntroduction to Docker storage, volume and image
Introduction to Docker storage, volume and image
 
Real-World Docker: 10 Things We've Learned
Real-World Docker: 10 Things We've Learned  Real-World Docker: 10 Things We've Learned
Real-World Docker: 10 Things We've Learned
 
Michigan IT Symposium 2017 - Container BOF
Michigan IT Symposium 2017 - Container BOFMichigan IT Symposium 2017 - Container BOF
Michigan IT Symposium 2017 - Container BOF
 
Docker at Cloud9 IDE
Docker at Cloud9 IDEDocker at Cloud9 IDE
Docker at Cloud9 IDE
 
Accelerate your development with Docker
Accelerate your development with DockerAccelerate your development with Docker
Accelerate your development with Docker
 
Accelerate your software development with Docker
Accelerate your software development with DockerAccelerate your software development with Docker
Accelerate your software development with Docker
 
Docker and kubernetes
Docker and kubernetesDocker and kubernetes
Docker and kubernetes
 
A Gentle Introduction to Docker and Containers
A Gentle Introduction to Docker and ContainersA Gentle Introduction to Docker and Containers
A Gentle Introduction to Docker and Containers
 
Docker from A to Z, including Swarm and OCCS
Docker from A to Z, including Swarm and OCCSDocker from A to Z, including Swarm and OCCS
Docker from A to Z, including Swarm and OCCS
 
Cassandra and docker
Cassandra and dockerCassandra and docker
Cassandra and docker
 
Introduction to Docker and Monitoring with InfluxData
Introduction to Docker and Monitoring with InfluxDataIntroduction to Docker and Monitoring with InfluxData
Introduction to Docker and Monitoring with InfluxData
 
The internals and the latest trends of container runtimes
The internals and the latest trends of container runtimesThe internals and the latest trends of container runtimes
The internals and the latest trends of container runtimes
 
codemotion-docker-2014
codemotion-docker-2014codemotion-docker-2014
codemotion-docker-2014
 
Why everyone is excited about Docker (and you should too...) - Carlo Bonamic...
Why everyone is excited about Docker (and you should too...) -  Carlo Bonamic...Why everyone is excited about Docker (and you should too...) -  Carlo Bonamic...
Why everyone is excited about Docker (and you should too...) - Carlo Bonamic...
 
Docker 101
Docker 101 Docker 101
Docker 101
 
Docker handons-workshop-for-charity
Docker handons-workshop-for-charityDocker handons-workshop-for-charity
Docker handons-workshop-for-charity
 
Improving Development and Deployment with Docker
Improving Development and Deployment with DockerImproving Development and Deployment with Docker
Improving Development and Deployment with Docker
 
Containerize! Between Docker and Jube.
Containerize! Between Docker and Jube.Containerize! Between Docker and Jube.
Containerize! Between Docker and Jube.
 

More from Guangya Liu

Istio简介
Istio简介Istio简介
Istio简介
Guangya Liu
 
IBM Cloud Private Introduction
IBM Cloud Private IntroductionIBM Cloud Private Introduction
IBM Cloud Private Introduction
Guangya Liu
 
Open source-summit-k8 s-mesos-v7
Open source-summit-k8 s-mesos-v7Open source-summit-k8 s-mesos-v7
Open source-summit-k8 s-mesos-v7
Guangya Liu
 
Mesos vs Kubernetes: What We Learned Working With Both For Chinese Customers
Mesos vs Kubernetes: What We Learned Working With Both For Chinese CustomersMesos vs Kubernetes: What We Learned Working With Both For Chinese Customers
Mesos vs Kubernetes: What We Learned Working With Both For Chinese Customers
Guangya Liu
 
Ibm dnt-dcos-v9-3
Ibm dnt-dcos-v9-3Ibm dnt-dcos-v9-3
Ibm dnt-dcos-v9-3
Guangya Liu
 
Mesos con 2015 recap meetup-v2
Mesos con 2015 recap meetup-v2Mesos con 2015 recap meetup-v2
Mesos con 2015 recap meetup-v2
Guangya Liu
 
OpenStack and Docke Integration V6
OpenStack and Docke Integration V6OpenStack and Docke Integration V6
OpenStack and Docke Integration V6
Guangya Liu
 
Pairs OpenStack Summit Summary
Pairs OpenStack Summit SummaryPairs OpenStack Summit Summary
Pairs OpenStack Summit Summary
Guangya Liu
 
Docker集群管理 工具篇
Docker集群管理 工具篇Docker集群管理 工具篇
Docker集群管理 工具篇
Guangya Liu
 
OpenStack Resource Scheduling
OpenStack Resource SchedulingOpenStack Resource Scheduling
OpenStack Resource Scheduling
Guangya Liu
 
Organizing open stack-meetup-in-china
Organizing open stack-meetup-in-chinaOrganizing open stack-meetup-in-china
Organizing open stack-meetup-in-china
Guangya Liu
 
Docker open stack
Docker open stackDocker open stack
Docker open stack
Guangya Liu
 

More from Guangya Liu (12)

Istio简介
Istio简介Istio简介
Istio简介
 
IBM Cloud Private Introduction
IBM Cloud Private IntroductionIBM Cloud Private Introduction
IBM Cloud Private Introduction
 
Open source-summit-k8 s-mesos-v7
Open source-summit-k8 s-mesos-v7Open source-summit-k8 s-mesos-v7
Open source-summit-k8 s-mesos-v7
 
Mesos vs Kubernetes: What We Learned Working With Both For Chinese Customers
Mesos vs Kubernetes: What We Learned Working With Both For Chinese CustomersMesos vs Kubernetes: What We Learned Working With Both For Chinese Customers
Mesos vs Kubernetes: What We Learned Working With Both For Chinese Customers
 
Ibm dnt-dcos-v9-3
Ibm dnt-dcos-v9-3Ibm dnt-dcos-v9-3
Ibm dnt-dcos-v9-3
 
Mesos con 2015 recap meetup-v2
Mesos con 2015 recap meetup-v2Mesos con 2015 recap meetup-v2
Mesos con 2015 recap meetup-v2
 
OpenStack and Docke Integration V6
OpenStack and Docke Integration V6OpenStack and Docke Integration V6
OpenStack and Docke Integration V6
 
Pairs OpenStack Summit Summary
Pairs OpenStack Summit SummaryPairs OpenStack Summit Summary
Pairs OpenStack Summit Summary
 
Docker集群管理 工具篇
Docker集群管理 工具篇Docker集群管理 工具篇
Docker集群管理 工具篇
 
OpenStack Resource Scheduling
OpenStack Resource SchedulingOpenStack Resource Scheduling
OpenStack Resource Scheduling
 
Organizing open stack-meetup-in-china
Organizing open stack-meetup-in-chinaOrganizing open stack-meetup-in-china
Organizing open stack-meetup-in-china
 
Docker open stack
Docker open stackDocker open stack
Docker open stack
 

Recently uploaded

June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
Ivanti
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
Zilliz
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
David Brossard
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Pixlogix Infotech
 
Mariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceXMariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceX
Mariano Tinti
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
Things to Consider When Choosing a Website Developer for your Website | FODUU
Things to Consider When Choosing a Website Developer for your Website | FODUUThings to Consider When Choosing a Website Developer for your Website | FODUU
Things to Consider When Choosing a Website Developer for your Website | FODUU
FODUU
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
tolgahangng
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
Uni Systems S.M.S.A.
 

Recently uploaded (20)

June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
 
Mariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceXMariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceX
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
Things to Consider When Choosing a Website Developer for your Website | FODUU
Things to Consider When Choosing a Website Developer for your Website | FODUUThings to Consider When Choosing a Website Developer for your Website | FODUU
Things to Consider When Choosing a Website Developer for your Website | FODUU
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
 

Docker volume-isolator-in-mesos

  • 1. 1 Docker Volume Isolator In Mesos Jie Yu, Gilbert Song, Guangya Liu
  • 2. 2 Gilbert Song Distributed Systems Engineer Mesosphere Guangya LiuJie Yu Distributed Systems Engineer Apache Mesos PMC & Committer Mesosphere Senior Software Engineer IBM
  • 3. 3 If your containers are “cattle”, you have nothing to lose... The test: I don’t need persistence beyond the life of the container instance or host. But what to do if your containers are pet, if you containers need to persist data? Motivation
  • 4. 4 Stateful Service: A service that needs to persist data/state A typical stateful service ● Use external volume services ● Stateless to Mesos Stateful services (like databases) need durable and scalable state Motivation
  • 5. 5 There are many docker images on DockerHub are databases... Motivation
  • 6. 6 Current state of storage solution in `MesosContainerizer` Local ● Persistent Volume Master Mesos Agent Aurora Executor Tasks mysqld datanode Hadoop Executor Tasks Operating System
  • 7. 7 But local is not enough!! Local ● Availability Risk ○ Migrate container to another host – your storage is gone ○ Host goes down – your service goes down ● Scale Limitation ○ Need more storage than the host has? Sorry…
  • 8. 8 What we would like? External Distributed Volume for stateful service Master Mesos Agent Aurora Executor Tasks mysqld datanode Hadoop Executor Tasks Operating System
  • 9. 9 The challenges + others Give operator the freedom to choose different external storage platforms.
  • 10. 10 Docker with External Volume IBM Spectrum Scale AWS GCE OpenStack Docker Volume Driver
  • 11. 11 Mesos with External Volume IBM Spectrum Scale AWS GCE OpenStack Docker Volume Driver
  • 12. Docker Volume Isolator Docker Volume Isolator IBM Spectrum Scale AWS GCE OpenStack Docker Volume Driver ● A driver based model ● Docker Volume Isolator takes care of business intelligence: ○ Mount with driver name, volume name and some options. ○ Unmount with driver name and volume name. 12
  • 13. 13 Docker Volume Isolator Architecture HTTP/JSON UNIX/TCP SOCKETS for Docker Volume API Docker Volume Driver Plugin (Convoy, Rex-ray, Flocker) Docker Volume Driver Client Docker Volume Isolator dvdcli Docker Daemon (Optional) Docker Volumes Mount: dvdcli mount –volumedriver=convoy –volumename=vol1 --volumeopts=size=5 --volumeopts=iops=150 Unmount: dvdcli unmount --volumedriver=convoy --volumename=vol1 DeviceMapper NFS Amazon EBS GlusterFS Ceph
  • 15. How does it works (with Rex-ray)? Mesos Agent docker/volume dvdcli Rex-ray Container 1) Framework launch a task 2) Provision the container 3) Get the mount point from dvdcli 4) Mount the external volume to mesos container EBS
  • 16. 16 ● Have `dvdcli` installed on each agent ● Volume Lifecycle management (create, delete) ● Use the same Docker volume in both the DockerContainerizer and the MesosContainerizer simultaneously is strongly discouraged ○ MesosContainerizer has its own reference counting to decide when to unmount a Docker volume. ○ It would be problematic if a Docker volume is unmounted by MesosContainerizer but the DockerContainerizer is still using it. Limitations
  • 17. 17 ● Get rid of `dvdcli` Future Work
  • 18. 18 ● Part of DC/OS ● Create Volumes with your normal Marathon App Definition ● Track via Marathon UI ● Reuse the same volume after app restart External Volumes in Marathon and DC/OS