Deep Dive on Container Storage Architectures - Clinton Kitson and Chris Duchesne - Dell EMC World 2017
•Download as PPTX, PDF•
1 like•2,748 views
This document discusses container storage architectures and provides a deep dive on using REX-Ray and ScaleIO with Docker, Mesos, and Kubernetes. It covers topics like storage in cloud native environments, container storage orchestration, the container storage interface, and using operators to automate storage operations. The document is authored by Dell EMC open source engineers and advocates building community around software-based infrastructure solutions.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Deep Dive on Container Storage Architectures - Clinton Kitson and Chris Duchesne - Dell EMC World 2017
Similar to Deep Dive on Container Storage Architectures - Clinton Kitson and Chris Duchesne - Dell EMC World 2017 (20)
Recently uploaded
Recently uploaded (20)
Deep Dive on Container Storage Architectures - Clinton Kitson and Chris Duchesne - Dell EMC World 2017
- 1. Deep Dive on Container Storage Architectures Clint Kitson Technical Director @clintkitson {code} by Dell EMC Chris Duchesne Open Source Engineer @cduchesne {code} by Dell EMC
- 2. © Copyright 2017 Dell Inc.2 Open Source at Dell EMC – Contributing to and maintaining open source projects – Community engagement – Technical solution leadership {code} by Dell EMC is a group of passionate open source engineers and advocates building community around software-based infrastructure. Platinum Sponsor
- 3. There is no such thing as a stateless architecture. Applications need data.
- 4. © Copyright 2017 Dell Inc.4 Yes, applications need data Lots of different types of persistent services to consider Files Blocks Documents Logstreams Time Series Media and Streaming Modern or Traditional Applications Storage Services Other Persistence Services Objects
- 5. © Copyright 2017 Dell Inc.5 Use services from any cloud True portability of applications requires interoperability My Cloud Public Cloud Universal Network and Storage Interfaces Container Orchestrators ..and interoperating with cloud services ..with your preferred CO Running Docker Container Images ..with or without virtualization DEW2017 Sessions No teleportation here.
- 6. © Copyright 2017 Dell Inc.6 Storage with persistent applications Cloud Native thinking… Use available storage features Plan for failures and transience Interoperate and orchestrate A cloud operating model supporting your applications.
- 7. © Copyright 2017 Dell Inc.7 Storage in Cloud Native Supporting persistent applications and making smarter applications • Interoperate with cloud resources • Define storage requirements with applications • Automate the operations of complex persistent applications
- 8. © Copyright 2017 Dell Inc.8 Storage is easy when abstracted by a hypervisor Everything is software defined with virtualized infrastructure Physical Servers Hypervisor/IaaS You get software defined storage from any storage platformCompute Network Storage Shared storage (FC/iSCSINFS) Provided through hyper-converged storage Data-plane abstraction makes it easy to connect storage to VMs. OR/AND VM Abstracted by a hypervisor for VMs With many heterogeneous servers and storage Maybe this is the right answer for you. DATA FLOW
- 9. © Copyright 2017 Dell Inc.9 Cloud Native thinking applied.. Be portable, focus on software and interoperability instead of data-plane abstraction Container Orchestrators Container OSs Storage resources are interoperable Private Cloud Storage Service Cloud Native Storage Compute Network Storage DATA FLOW
- 10. © Copyright 2017 Dell Inc.10 I believe we can get here Enable a cloud operating model supporting all your applications deploy: application: cassandra cpu: 2000ghz availability: n+2 size: 1048576GB iops: 1048576 Container Orchestrators Operator Automation Any Cloud
- 11. Let’s go for a swim.
- 12. © Copyright 2017 Dell Inc.12 Deploying applications with storage How do the basics work? deploy: container: postgres volume: name: myvolume1 Storage Orchestration • Orchestrator ensures application and container are running at all times. • A new container will be created with the existing data if necessary. • The application process and container maintain ephemeral. • Storage is orchestrated to the host and made available to container.
- 13. © Copyright 2017 Dell Inc.13 Storage orchestration with containers How do the basics work? Storage is attached and detached from host instances where containers are targeted to run. Container Host Container Runtime/Orchestrator REX-Ray, the container storage orchestration engine
- 14. © Copyright 2017 Dell Inc.14 Interoperability for storage services Digging a little deeper • Orchestrators communicate with external storage platforms to perform storage lifecycle and orchestration features. • These integrations take place internal and external to container runtimes and orchestrators and their code base. Container Orchestrator Cloud and Storage Platform Container Hosts Cloud and Storage Platform Storage Plugins In-Tree Out-of-Tree Host Process
- 15. © Copyright 2017 Dell Inc.15 $ docker plugin install rexray/scaleio Installed plugin rexray/scaleio:latest $ docker plugin ls ID NAME DESCRIPTION ENABLED 450420731dc3 rexray/scaleio:latest REX-Ray for ScaleIO true $ docker run -ti --volume-driver=rexray/scaleio -v test:/test busybox $ df -h /test REX-Ray with ScaleIO Docker Managed Plugin
- 16. © Copyright 2017 Dell Inc.16 Storage Plugins Interoperability today Docker Volume Driver Interface DVDI DVDCLI Flex Interface DVDI JSON over RPC JSON over Proc JSON/RPC over HTTP Storage Platform JSON/RPC over HTTP In-Tree
- 17. © Copyright 2017 Dell Inc.17 • When out-of-tree plugins are being used, REX-Ray is the glue between a storage and container platform. • It is a trusted tool that provides integration to heterogeneous storage platforms. • Currently 6/16 store.docker.com plugins are REX-Ray A container storage orchestrator storage plugin Digging a little deeper • Any block/file/object platform • Amazon AWS/EFS/S3 • Dell EMC ScaleIO/Isilon/ECS • Digital Ocean Block Storage • FittedCloud EBS • Google Compute Engine PD/CSB • OpenStack • Red hat Ceph RBD/RGW • S3-Compatible Object Storage • .. Contribute!!
- 18. © Copyright 2017 Dell Inc.18 Interfacing for storage services Digging a little deeper Container orchestrators and runtimes are able to make specific requests for storage services. Container Orchestrator Cloud and Storage Platform Container Hosts Interface achieves these things.. • Create/Remove volumes • Inspect/List volumes • Attach/Detach volumes • Mount/Unmount volumes
- 19. © Copyright 2017 Dell Inc.19 Storage resource model Digging a little deeper Volumes Devices Storage Pools Storage Platform Snapshots A full storage model would include the volume resources which store data, but also the logical parents and related resources.
- 20. © Copyright 2017 Dell Inc.20 Volume parameters Digging a little deeper volume: name: unique name size: in bytes iops: limit type: pool or type encryption: true encryptionKey: string id: unique id shared: multiple hosts readWrite: true fields: any info • Storage resources can include parameters that are common and useful to application consumers and automation. Additional parameters are available but not exposed for common consumption. volume: name: unique name fields: any info
- 21. © Copyright 2017 Dell Inc.21 Storage roles Digging a little deeper Roles should allow the separation of concerns where the provisioning and consumption of storage can be controlled. • Operations Team – Creates policies that limits storage consumption – Identifies available volumes and available resources • Application Owners – Provisions from available volumes – Deploys applications with storage
- 22. © Copyright 2017 Dell Inc.22 Software managing software Digging a little deeper Automating the operations of software inclusive of storage resources is important for more complex applications. Container Orchestrators Operator Automation Etcd for Dummies Container Orchestrators
- 24. © Copyright 2017 Dell Inc.24 Container Orchestrators Yesterday (2017/04/10) Container Orchestrator Storage Interface Centraliz ation Storage Roles Storage Resource Model Resource Details Storage Services Operator Automation Docker Swarm DVD (RPC/JSON) Plugin responsibi lity AppOwners Volumes N/A Plugin in container N/A Mesos DVDI proc (RPC/JSON) Plugin responsibi lity AppOwners N/A N/A Plugin on host Frameworks Kubernetes Flex proc, In- tree kubectl and kubelet plugin Ops, AppOwners Volumes Some Kube node plugin, Host process Custom Controllers Cloud Foundry N/A, In-tree N/A AppOwners Volumes N/A N/A N/A Container Storage Interface** gRPC Native Ops, AppOwners Volumes, Snapshots, Pool, Platform Material parameters Plugin in container or on Storage platform CO Choice
- 26. © Copyright 2017 Dell Inc.26 Storage for containers in a cloud operating model Any Cloud Positives Negatives Deployed and managed as software Has hardware-defined qualities Interoperates with container orchestrators and runtimes Complex availability Exposes common parameters, ie. Type, Size, IOPS, Encryption Complex deployment RBAC Slow orchestration Snapshots/Replication Only scale-up
- 27. © Copyright 2017 Dell Inc.27 Cloud Native Storage ScaleIO provides storage built to support Cloud Native workloads in Private Clouds • Freedom and flexibility from deploying as a software-based platform • Efficient interoperability with container orchestrators • Docker and Mesos with REX-Ray Volume Plugin • Kubernetes ScaleIO Volume Plugin • Simple availability configuration (inherent multi-path) • Scales multi-dimensionally (up and out) • Supports data services (snapshots, replication) • Common abstractions of core capabilities (size, volume type)
- 29. What’s next?
- 30. © Copyright 2017 Dell Inc.30 Kubernetes operators with storage Orchestrator Storage Interface Centralizati on Storage Roles Storage Resource Model Resource Details Storage Services Operator Automation Kubernetes Flex proc, In-tree kubectl and kubelet plugin Ops, AppOwners Volumes Some Kube node plugin, Host process Custom Controllers SSDKubelet Nodes operator Storage Classes In-Tree ScaleIO Plugin
- 31. © Copyright 2017 Dell Inc.31 Summary • Cloud native environments are optimized for traditional and modern applications in any cloud – think differently about your infrastructure • Enable smarter applications through automating their operations • Storage interoperability with cloud native further enables portability of your applications • Container orchestrators and runtimes have different levels of storage support • REX-Ray provides container storage orchestration for cloud native workloads
- 32. © Copyright 2017 Dell Inc.32 codedellemc.com community.codedellemc.com @codeDellEMC blog.codedellemc.com {code} by Dell EMC is a group of passionate open source engineers and advocates building community around software-based infrastructure. rexray.codedellemc.com github.com/codedellemc/labs Clint Kitson @clintkitson github.com/clintkitson HOL01 Use REX-Ray & ScaleIO w/ Docker, Mesos and Kubernetes “Deep Dive on REX-Ray, libStorage, and the Container Storage Interface” Tuesday 12p and Wednesday 830a