Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware


Published on

Containers are showing rising interest in the enterprise world for cloud-native applications, but there remain significant roadblocks to adoption. For example, containers have had significant issues in using existing storage for interacting with persistent, stateful applications and data. VMware is developing new features that leverage vSphere to help provide persistent storage and data lifecycle management for Docker containers. In this technical session, we will go over use cases for container-based storage, describe how VMware is providing solutions for storage pain points, and showcase a demo so you can see some solutions in action. Come see how VMware provides Containers without Compromise.

Published in: Technology
  • Be the first to comment

VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware

  1. 1. Containers without Compromise: Persistent Storage for Docker Containers with VMware Vivek Saraswat, VMware, Inc CNA5860 #CNA5860
  2. 2. • This presentation may contain product features that are currently under development. • This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. • Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. • Technical feasibility and market demand will affect final delivery. • Pricing and packaging for any new technologies or features discussed or presented have not been determined. Disclaimer CONFIDENTIAL 2
  3. 3. Agenda CONFIDENTIAL 3 1 Intro to Cloud-Native Applications 2 Containers 101 3 Persistent Data in Containers 4 VMware Solutions for Cloud-Native Storage 5 Looking Towards the Future
  4. 4. What are Cloud-Native Applications? CONFIDENTIAL 4 Developer access via APIs Microservices, not monolithic stacks Continuous integration and deployment App-defined Availability Built for scale Decoupled from infrastructure Application
  5. 5. Hardware OS Kernel OS File system Userspace Container Appprocess Appprocess Appprocess Appprocess Appprocess Container Appprocess Appprocess Linux Containers CONFIDENTIAL 55 • App-level Isolation – Isolation at individual kernel subsystem level (e.g. filesystem, CPU resources, etc.) – User-level process (LXC, libcontainer) orchestrates these subsystems to create a container • Existed for Many Years – Solaris Zones, FreeBSD Jails, etc. • Why? – Process isolation – Reproducible environment – Enables management at scale
  6. 6. is a “Shipping Container” for Code Frictionless deployment and maximum portability On developer laptop: Then on server: That’s it!! A natural fit for 3rd Platform, 12 factor, microservices It makes DevOps much much easier CONFIDENTIAL 6 Developers because … ~# docker build my_app ~# docker push my_app ~# ~# docker pull my_app ~# docker run my_app ~#
  7. 7. Containers Are Stateless “Cattle” CONFIDENTIAL 7 Source: “CERN Data Centre Evolution”
  8. 8. But…What about Your Data? • If you start a new container, you might lose all the data from the old one! • “Stateful” data needs to be accessed and protected separately • Original model: Persist stateful data to noncontainerized managed storage • But, problems arise: – No local control over storage management – Latency/access issues – Issues at scale CONFIDENTIAL 8
  9. 9. Containerized Storage Apps are Rapidly Increasing CONFIDENTIAL 9 Thousands of DB apps Millions of downloads
  10. 10. Bring in the Container Data Volumes! • Usage – Contains persistent data for local containers – Appears as directory within host file system (e.g. “docker run –v /mount/mydata/”) – Can store on external storage and mount/unmount from a host • Benefits – Manage and preserve your stateful data – Utilize storage platform data services Volumes open up new possibilities for containerized applications! CONFIDENTIAL 10 Host Container Storage Platform Container Data Volumes
  11. 11. Container Storage Use Cases CONFIDENTIAL 11 Unshared Volumes Shared Volumes Persist to External Storage Use Case: Running container- based SQL or noSQL DB Use Case: Sharing a set of tools or content across app instances Use Case: Object store for retention/archival, DBaaS for config/transactions Host C C Storage Platform Host C C Host C C Host C C Storage Platform Host C C Host C C Cloud Storage API API
  12. 12. Containerized Storage in Practice • Application composed of series of loosely coupled microservices – Scheduled by an application orchestrator (e.g. Kubernetes, Mesos) – Microservices coordinate via REST API • Each microservice consists of multiple stateless and stateful containers – API frontend container – Database engine container – Actual data is persisted to a container data volume (supported by external storage) CONFIDENTIAL 12 Application Orchestrator Microservice 1 API Front-End Database Engine Data Volume Microservice 2 API Front-End Database Engine Data Volume Microservice 2 API Front-End Database Engine Data Volume
  13. 13. Container Volumes and Virtual Disks: Look Familiar? CONFIDENTIAL 13
  14. 14. Announcing vSphere Driver for Flocker 1.0! CONFIDENTIAL 14
  15. 15. Luke Marsden, CTO
  16. 16. The Problem ClusterHQ’s Flocker Solves CONFIDENTIAL 16
  17. 17. ClusterHQ Flocker Data Volume Migration (before) CONFIDENTIAL 17
  18. 18. ClusterHQ Flocker Data Volume Migration (after) CONFIDENTIAL 18
  19. 19. • Integrate “upwards” to Orchestration Frameworks – Docker Swarm and Compose via Flocker Docker Plugin – Mesosphere/Marathon – Kubernetes (coming later), CoreOS • Integrate “downwards” to storage vendors – vSphere driver enables awesome coverage – Growing ClusterHQ partner network – Also support OpenStack Cinder – Integrate with public cloud storage backends • Add depth of storage capabilities – Make data portable: enable migration of data volumes between clouds, different stages of software development lifecycle – Enable snapshotting, cloning, backup/restore, HA, DR… Get in touch! @clusterhq CONFIDENTIAL 19 ClusterHQ Flocker Roadmap ClusterHQ
  20. 20. vSphere Driver for Flocker Details • Run containerized stateful apps on your current vSphere deployment using open-source Flocker software • Move containers + data volumes from one ESX VM to another • Compatible with ALL vSphere storage (VSAN, VVOL, VMFS, NFS) • Straightforward install/configure/deploy process • Free! • Available at CONFIDENTIAL 20
  21. 21. Native Docker on vSphere CONFIDENTIAL 21 ESX VM2 Container DB App ESX VM1 Container DB App VMDK1 + Container Volume VMDK2 ESX VM2 Container DB App ESX VM1 Container DB App vSphere + Flocker Move Container Move Container VMDK1 VMDK2Container Volume VMDK When container moves, data volume stays on host VMDK. Database starts on new VM without any of its data. Data Volume stored on separate VMDK. When container moves, VMDK moves with it. Database keeps its data!
  22. 22. vSphere Flocker Demo 22
  23. 23. VMware Virtual SAN 6.1 CONFIDENTIAL 23 Radically Simple Hyperconverged Storage for VMs vSphere + Virtual SAN … • Software-defined storage optimized for VMs • Hypervisor-converged architecture • Runs on any standard x86 server • Pools HDD/SSD into a shared datastore • Delivers enterprise-level scalability and performance • Managed through per-VM storage policies • Deeply integrated with the VMware stack Overview Hard disksSSD Hard disks SSD Hard disks SSD Virtual SAN Datastore
  24. 24. But Don’t Take Our Word for It… CONFIDENTIAL 24 “ClusterHQ and VMware supported our evaluation of Flocker with vSphere and Virtual SAN, providing us building blocks for container persistence in our Docker- based PaaS.”
  25. 25. CONFIDENTIAL 25
  26. 26. Unified Hybrid Platform vSphere Integrated Containers • Give developers the flexibility, portability and speed containers deliver • Leverage existing investments in VMware infrastructure, people, processes and management tools • DRS, vMotion, HA/DR • Storage and Network Integration (VSAN and NSX) • Apply same isolation, data persistence, networking, management and robust service levels you have today • No rebuilding or re-architecture required • Full compatibility with broad ecosystem of existing tools CONFIDENTIAL 26 NSX VSAN vRealize vSphere vSphere Integrated Containers (Instant Clone, Project Bonneville, Photon OS)
  27. 27. Cloud-Native Platform VMware Photon Platform CONFIDENTIAL 27 Photon Controller Photon Machine Photon Platform Host controller & scheduler: High scale distributed control plane, includes Lightwave Compute host: Lightweight hypervisor, based on ESX Includes Photon OS Deep integration with modern, open source frameworks & app platforms
  28. 28. Looking Towards the Future of Storage • Distributed infrastructure – Global management – Automation friendly • Scalable troubleshooting – Decentralized analytics – Information gathering, processing, prediction • IT-friendly GUI, scripting – Infrastructure operations • Dev-friendly APIs, CLI – Application integration CONFIDENTIAL 28 Physical Servers Infrastructures Pools of Resources Virtual SAN Datastore Magnetic Devices Flash Devices Magnetic Devices Flash Devices Storage Abstraction Distributed storage platform storageinfrastructure management distributedmonitoringUI/APIs
  29. 29. VDI files A Distributed File System for Cloud-Native Apps • Hyper-converged scale-out file system • Relies on Object Storage backend – Hardware management – Resource provisioning, discovery – Distributed parallel data path – High availability • Backend: VSAN, others… – Can span multiple backend “clusters” • Design Requirements – POSIX file system – Cloud scale: files, clients, clones – Per file / directory access control – O(1) snapshot / clone creation CONFIDENTIAL 29 Distributed storage management service VSAN (Object) store VSAN (Object) store Distributed File system NoSQL
  30. 30. Distributed File System Demo 30
  31. 31. What’s Now? • vSphere driver for Flocker 1.0 available now for free at: • Extending functionality for VMware-based container data volumes • Storage integration with vSphere Integrated Containers • Storage integration with Photon Platform Follow along on Twitter: @theVSaraswat CONFIDENTIAL 31 What’s Next?
  32. 32. Containers without Compromise: Persistent Storage for Docker Containers with VMware Vivek Saraswat, VMware, Inc CNA5860 #CNA5860