Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

{code} and containers


Published on

An introduction to {code} by Dell EMC, our mission on containers, and our core project REX-Ray. This will give the audience an understanding of why REX-Ray is important and where you can go to learn more.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

{code} and containers

  1. 1. {code} and containers Open Source Software, Dell EMC’s Contributions, and Highly Available Applications July 2017
  3. 3. Intangible Benefits of Open Source Not reinventing the wheel Customization with benefits Motivated workforce Attracting top talent Standardized practices Business acceleration Cleaner software Cheaper Customer goodwill Community support Innovation Flexibility Freedom Integration
  4. 4. True Benefits of Open Source Innovation Flexibility Freedom Integration
  5. 5. FIFI and Integration • Freedom – to run software for any purpose and to redistribute it • Innovation – the ability to leverage a collaborative community of developers • Flexibility – to deploy the software in a manner that best meets the organization’s requirements • Integration – Ability to easily integrate open source software with existing infrastructure It’s not about cost, it’s about integration
  6. 6. Open Source at Dell EMC – Contributing to and maintaining open source projects – Community engagement – Technical solution leadership {code} by Dell EMC is a group of passionate open source engineers and advocates building community around software-based infrastructure. Platinum Sponsor
  7. 7.
  8. 8. Digital Business is Driving Software Innovation Microservices APIs Open source Containers Cloud Native Frameworks Analytics Insights Drive New Functionality, Which Drives New Data Applications/IoT Transforms Business Data Generated By New Applications
  9. 9. Open source Containers Digital Business is Driving Software Innovation Microservices APIs Cloud Native Frameworks Analytics Insights Drive New Functionality, Which Drives New Data Applications/IoT Transforms Business Data Generated By New Applications
  10. 10. Keys to Freedom and Flexibility • Embrace DevOps and open source • Remove unnecessary complexity • Run all applications from containers • Operate everything as software, including storage • Run applications as services • Use platform that orchestrates and consumes all infrastructure natively for applications
  11. 11. Applications are Changing Loosely Coupled Services Many Servers ~2000 Today Monolithic Big Servers Slow Changing Rapidly updated
  12. 12. Containers are Key • Lightweight • Becoming defacto application packaging standard • Package of software binaries and dependencies • Easily portable across environments (on-prem and cloud) • Allows ecosystem to develop around its standard • Docker, Mesos, and Kubernetes are currently the most popular container technologies • A persistent container stores data in an external volume • Additionally an opportunity to run old applications in new ways Code and Dependencies Container
  13. 13. Optimizing and Enabling Rapid Innovation Virtual machines Server Public Cloud Disaster Recovery Developer Laptop Server Cluster Data Center Generic Persistence Web Front EndBackground Workers SQL Database NoSQL Scale-Out Database Queue API Endpoint Development Test & QA Production Scale Out
  14. 14. Virtual Machines and Containers Platform 2/Mode 1 Platform 3/Mode 2
  15. 15. Challenges of deploying container platforms Visibility & Ease of Use Elasticity Data Storage & Protection Platform Sprawl & Siloed Organizations Financial Viability & Existing Investments Many Applications, Different Needs Business Challenges Infrastructure Challenges
  17. 17. Do you have any of these in your data center? • Databases – Postgres, MongoDB, MySQL, MariaDB, Redis, Cassandra • Search, Analytics, Messaging – ElasticSearch, LogStash, Kafka, RabbitMQ • Content Management – Wordpress, Joomla, Drupal, SugarCRM • Service Discovery – Consul, Zookeeper, etcd • Continuous Integration and Delivery – Jenkins, GitLab, SonarQube, Selenium, Nexus • Custom Applications – That Java app your company built Stateful and persistent applications
  18. 18. Applications need data Lots of different types of persistent services to consider Files Blocks Documents Logstreams Time Series Media and Streaming Modern or Traditional Applications Storage Services Objects Your use case here…
  19. 19. 6 of the top 10 require persistence 7/20/17
  20. 20. What's the problem? • When I run a persistent application in a container, where does my data get stored? – The container holds the data directory and structure of the entire application – Optionally use local volumes • Stateless applications work well – nginx, httpd, kibana, haproxy, memcached, solr, celery $ docker run -v redisData:/data redis redisData /etc /var /bin /opt /data
  21. 21. What's the problem? • Lose a container – Lose the data • Lose a server – Lose the data • Local data storage – Failed hard drives or failed RAID – Can not scale beyond the physical limit of the server /etc /var /bin /opt /data
  22. 22. Storage is easy when abstracted by a hypervisor Everything is software defined with virtualized infrastructure Physical Servers Hypervisor/IaaS You get software defined storage from any storage platformCompute Network Storage Shared storage (FC/iSCSINFS) Provided through hyper-converged storage Data-plane abstraction makes it easy to connect storage to VMs. OR/AND VM Abstracted by a hypervisor for VMs With many heterogeneous servers and storage Maybe this is the right answer for you. DATA FLOW
  23. 23. Cloud Native Thinking Applied.. Be portable, focus on software and interoperability instead of data-plane abstraction Container Orchestrators Container OSs Storage resources are interoperable Cloud Storage Service Integration with Storage Orchestrators Cloud Native Storage Compute Network Storage DATA FLOW
  24. 24. Interoperability for Storage Services • Orchestrators communicate with external storage platforms to perform storage lifecycle and orchestration features. • These integrations take place internal and external to container runtimes and orchestrators and their code base. Container Orchestrator Cloud and Storage Platform Container Hosts Cloud and Storage Platform Storage Plugins In-Tree Out-of-Tree Managed Plugins or Host Process
  25. 25. Deploying Applications with Storage • Orchestrator ensures application and container are running at all times. • A new container will be created with the existing data if necessary. • The application process and container maintain ephemeral. • Storage is orchestrated to the host and made available to container. Storage is attached and detached from host instances where containers are targeted to run. deploy: container: redis volume: name: redisData /redisData /etc /var /bin /opt /data
  26. 26. Interfacing for Storage Services Container orchestrators and runtimes are able to make specific requests for storage services. Container Orchestrator Cloud and Storage Platform Container Hosts Interface achieves these things.. • Create/Remove volumes • Inspect/List volumes • Attach/Detach volumes • Mount/Unmount volumes
  27. 27. Storage Plugins Interoperability Today Docker Volume Driver Interface DVDI DVDCLI Flex Interface DVDI JSON over RPC JSON over Proc JSON/RPC over HTTP Storage Platform JSON/RPC over HTTP In-Tree
  28. 28. Introducing REX-Ray REX-Ray The leading container storage orchestration engine enabling persistence for cloud native workloads • Out of Tree Plugin
  29. 29. Cloud Native Interoperability DOCKER APACHE MESOSKUBERNETES AND OPENSHIFT Use the stand alone Docker Engine to run a stateful application or combine it with Docker Swarm Mode to turn your application into a robust service. Use any framework that orchestrates containers such as Marathon or Aurora to provide persistent storage for stateful services. Run stateful applications in pods and stateful sets through FlexREX, and benefit from a broader set of storage platforms and CLI management capabilities.
  30. 30. REX-Ray Features PERSISTENT STORAGE ORCHESTRATION FOR CONTAINERS ENTERPRISE READY OPEN SOURCE TRUSTED INTEROPERABILITY • Run any application using multiple storage platforms. • Resume state and save data beyond the lifecycle of a container. • Containers aren’t just for stateless applications anymore. • High-availability features for container restarts across host • CLI intuitiveness • Contributions from the cloud native community and the leading storage vendor in the world. • A completely open and community driven project • Constantly innovating and providing new integration points. • Community contributed drivers, features, and additional functionality • Built using an interface library called libStorage. • Includes reusable orchestration and lifecycle operations that satisfy the integration of cloud native and storage platforms today and tomorrow.
  31. 31. REX-Ray Features • A single interface to all of your storage platforms. • A common orchestration and volume lifecycle operations no matter what type of platform you choose. • Includes support for storage platforms types that cover block, file, and object. • Run any application that has any type of storage requirement • Installed as a single binary or deployed as a container. • Can be configured to include one or multiple storage platforms from a single stateless service. • Multiple architectural choices allow flexibility for deployments. • Configure as standalone to serve in a decentralized architecture. • Leverage the client/agent and controller for a centralized architecture MULTIPLE STORAGE PLATFORM SUPPORT STORAGE AGNOSTIC EFFORTLESS DEPLOYMENT STANDALONE OR CENTRALIZED
  32. 32. REX-Ray Features • All traffic is encrypted with TLS using pre- configured keys and certificates or auto- generated self-signed certificates. • A fingerprint feature using Token Based Authentication prompts an Agent to trust a Controller. SECURE BY DEFAULT
  33. 33. Storage Platform Integration • Elastic Block Storage (EBS) • Elastic File Storage (EFS) • Simple Storage Service (S3) • Use thin- provisioned EBS volumes as an alternative to reduce costs in AWS. • ScaleIO (block) • Isilon (NFS) • ECS (object) • Persistent Disks (PD) • Cloud Storage Buckets (CSB) • Digital Ocean Block Storage attaches persistent disks your droplets. • Blob Storage Unmanaged Disks (block) • S3 compatible object storage • Local Disk • RADOS Block Devices (RBD) • RADOS Gateway (RGW) • Mount a S3 bucket as a directory using Fuse • Cinder volumes on any hardware (virtualized or bare-metal) can be used.
  34. 34. Development to Production Lifecycle Deploying containers using REX-Ray stays consistent from local development, testing in the cloud, to production in on-premise datacenters ScaleIO Isilon all supported storage platforms are not shown*
  35. 35. Who’s Using REX-Ray? …and many more
  36. 36. Docker Integration • Use the stand alone Docker Engine to run a stateful application or combine it with Docker Swarm Mode to turn your application into a robust service. • Available in the Docker Store as a certified and trusted plugin • REX-Ray development started when the Docker Volume Driver interface was introduced in 1.7 Experimental • One of 6 available drivers during DockerCon 2014 debut of the Docker Volume Driver • {code} team has submitted 6 patches and over 100 lines of code to the Docker Engine for Storage • Provides full volume lifecycle capability
  37. 37. Docker Integration • Deploy as a host based service or as a Docker Plugin Available in the Docker Store
  38. 38. $ docker service create --name redis --replicas 1 --mount type=volume,src=redisData, dst=/data,volume-driver=rexray redis redisData /etc /var /bin /opt /data Storage Platform Persistent Volume Docker Swarm Integration
  39. 39. Mesos Integration • Docker Integration • UseDocker asthe underlyingcontainertechnology • SpecifyREX-Ray astheVolume Driver foranypersistentstorageusecase REX-Ray
  40. 40. Mesos Integration • Docker Volume Driver Isolator Module • Noremote storagecapabilityusingMesosbeforeSeptember2015 • {code}was firstto bring an abstractedpersistentstoragemodel • Freedomto usethe MesosContainerizerwith ANY dockervolume driver (thatmeansyoudon’tneedDocker) • Mergedmesos-module-dvdi upstream into Mesos 1.0 to providestorageplatformsupportforanyvendorwho has writtena dockervolume driver, includingREX-Ray • Every companythat boastsstorageintegrationwith Mesosusesthe{code} contributedmodule.
  41. 41. DC/OS Integration • DC/OS is a distributed operating system based on the Apache Mesos distributed systems kernel. • DC/OS 1.7 ReleaseshipswithREX-Rayembedded • DC/OS 1.8 ReleaseshipswithDVDI embeddedforMesos 1.0 • DC/OS 1.10 ReleaseshipswithREX-Rayv0.9 andDVDCLI 0.2 via Community(#1430) • Featuredin the documentation-
  42. 42. End-to-End Container Persistence Leadership for Mesos What does it do Enables external storage to be created / mounted / unmounted with each agent task Abstracts Docker Volume Drivers for Mesos Storage orchestration engine Cool factor • First Mesos Agent external storage module • Merged into Mesos 1.0 (2016) • Allows the use of any Docker volume driver with Mesos! • Vendor agnostic • DC/OS 1.7+ framework (2016) mesos-module-dvdi Docker Volume Driver Client CLI DVDCLI REX-Ray
  43. 43. Mesos Container Framework Integration • Marathon from Mesosphere • Apache Aurora (originally used by Twitter) • ElasticSeach
  44. 44. ScaleIO Framework for Mesos • A Software-based Storage Framework using ScaleIO to turn commidy hardware and direct attached storage (DAS) into a globally accessible and scalable storage platform that can be deployed anywhere • Framework installs and configures ScaleIO on all Mesos Agent (compute) nodes. As more Agents are added to the cluster, ScaleIO is installed and adds storage resources to the cluster. • Deploy and Configure ScaleIO without knowing operational details • Centralized Monitoring of the storage platform • Persistent storage native to the container scheduling platform using REX-Ray •
  45. 45. Kubernetes Integration • Kubernetes offers two approaches for storage integration. • An “in-tree” volume plugin for a platform. The storage interface code is directly embedded into Kubernetes. The downside is that plugin velocity (the speed at which a plugin can be added, enhanced, or patched) is gated by the Kubernetes release cycle. • The second approach is to leverage the FlexVolume volume plugin. This plugin uses externally installed and managed software to handle basic Attach/Detach/Mount/Unmount storage operations.
  46. 46. Kubernetes Integration • REX-Ray provides an adapter script called FlexREX which integrates with the FlexVolume plug-in to interact with the backing storage system • Allowspodsto consumedatastoredonvolumes thatare orchestratedby REX-Ray.Using Kubernetes' FlexVolumeplug-in,REX-Raycan provideuniform accessto storage operationssuchasattach, mount, detach,and unmountforanyconfiguredstorageprovider. • Runstatefulapplicationsinpodsand statefulsetsthroughFlexREX, andbenefitfrom CLImanagement capabilities. • UseanyREX-Raysupportedstorageplatform • Dynamic Provisioningandstorageclassesarenotavailableas a limitationof FlexVolume andrequires volume create/deleteto be managedoutsideof Kubernetes $ rexray flexrex install
  47. 47. Kubernetes Integration • ScaleIO is part of the core Kubernetes code and a first class native storage provider • ScaleIO can take full advantage of the Kubernetes volume lifecycle features including dynamic provisioning and storage classes • ScaleIO driver is embedded in the standard distribution of Kubernetes • Contributed code from the {code} by Dell EMC team passes “Google” standard of quality • Opens a new opportunity for those running Kubernetes in on-premise data centers. It allows utilization of your commodity x86 server hardware for very high performance and highly available storage for running stateful apps in containers.
  48. 48. OpenShift Integration • Kubernetes is used under the covers of OpenShift • Get all the benefits of FlexREX and native use of ScaleIO without any additional configuration • Support persistence requirements for applications
  49. 49. Storage Plugins Interoperability Today Docker Volume Driver Interface DVDI DVDCLI Flex Interface DVDI JSON over RPC JSON over Proc JSON/RPC over HTTP Storage Platform JSON/RPC over HTTP In-Tree
  50. 50. Solving the problem • REX-Ray or Kubernetes is installed and configured on all hosts in the cluster as a stateless service • Container engines redirect volume operations to the storage driver – Create/Mount/Unmount/Delete $ docker run --volume-driver=rexray -v redisData:/data redis /redisData /etc /var /bin /opt /data
  51. 51. Solving the problem • Lose the container or lose the server – Data persists and remains in tact on the remote storage platform /etc /var /bin /opt /redisData
  52. 52. Solving the problem • Attach the volume to a new container on a different host – Equivalent of a hard reset. Application starts and resumes from the last write to disk – Container scheduler is responsible for creating a new container • Scalability – Application data can scale to the maximum supported by the storage platform /etc /var /bin /opt /data /redisData
  53. 53. Dell EMC ScaleIO Software that creates an enterprise class virtual SAN on top of commodity server hardware OS, Hypervisor, and Media Agnostic • Utilizes commodity hardware • “Pay as you grow”— linear predictable costs • No dedicated storage components (FC network, HBAs) • Add servers of any type or configuration to the pool • Retire servers on your own schedule • Eliminate migration • Scale performance linearly
  54. 54. Scale Out Storage for Scale Out Apps
  55. 55. #CodeOpen Demo #CodeOpen Demo
  56. 56. Take our projects for a spin at the {code} Labs
  57. 57. @codeDellEMC {code} by Dell EMC is a group of passionate open source engineers and advocates working to build a community around software-based infrastructure. vLab (For Partners) Use REX-Ray & ScaleIO w/ Docker, Mesos and Kubernetes