EMC World 2016 - code.13 State of the Container Ecosystem with Persistent Applications

1© Copyright 2016 EMC Corporation. All rights reserved. 1© Copyright 2016 EMC Corporation. All rights reserved.
STATE OF THE CONTAINER ECOSYSTEM
WITH PERSISTENT APPLICATIONS
STEVE WONG
DEVELOPER ADVOCATE –EMC{code}

2© Copyright 2016 EMC Corporation. All rights reserved.
Old: monolithic, big,
slow to change,
long lived
APPLICATIONS ARE CHANGING
New: many instances,
rapidly implemented,
updated, deployed

SOLUTION: ENTER THE CONTAINER

TODAY’S AGENDA
Two rapidly
developing areas
fields within the
container space:
1. Persistent
Storage
2. Schedulers

WHERE DOES STORAGE FIT INTO THIS?
… THE SHELL GAME
Containers and this new
“cattle” thing magically made
my storage management
headaches disappear, right?

IF YOUR CONTAINER ARE CATTLE,
YOU HAVE NOTHING TO LOSE …
Go ahead, use local storage.
• The test: I don’t need
persistence beyond the life
of the container instance or
host

I HAVE SERVICES THAT AREN’T CATTLE
WHAT IF I RUN THESE THE SAME WAY?
A portion of the container host’s
filesystem can be exposed to a
container.
• Availability Risk
– Migrate container to another host – your
storage is gone
– Host goes down – your service goes down
• Scale Limitation
– Need more storage than the host has?
Sorry…

SOLUTION: MOUNT OF EXTERNAL STORAGE

HOW DO I MOUNT EXTERNAL STORAGE?
THE REX-RAY PROJECT
• Open Source - extensible
• Supports many container
platforms, including cloud hosted
• Supports many storage platforms
• Managed and configurable by
container host,
not inside-the-container
configuration

TODAY’S AGENDA
Two rapidly developing
areas fields within the
container space:
1. Persistent Storage
2. Schedulers
..we’ll come back to
storage in a minute

GOT CONTAINERS AT SCALE?
YOU NEED A SCHEDULER…

ASSUME 1000’S OF CONTAINERS
• How do you place an
assortment of
containers of
different sizes and
shapes onto your
servers efficiently?

ASSUME 1000’S OF SERVICES
• How do you
find them?
• How do they
find each
other?

WHAT’S A SCHEDULER?
• Manages compute nodes
– Takes reports from each backend server, as to
whether they are still available, and what their
capabilities and capacities are.
• Manages incoming work request queue
– Takes user’s, batch, and API job requests
– Based on requirements, decides which
backend server(s) can fulfill each request
– If multiple backend servers are suitable,
engages optimization algorithms in favor of
random placement
on demand
scheduled
batch

MANY SCHEDULERS ARE AVAILABLE
• Differences
– Management “philosophy”
• “Tell me How” vs “Tell me What”
– aka Imperative vs Declarative, Explicit vs Implicit, declare procedure vs declare outcome
– Container technology(s) used
– Maturity
– Supported scale
– Support for external storage volume mounts
– Degree of “convergence” with a specific application stack
• Aka degree to which scheduler is “opinionated”

SCHEDULERS COMPARED
Swarm Kubernetes Mesos Cloud
Foundry
Management
style
procedure outcome outcome outcome
Maturity new new years years
Scale 1,000? 1000 10,000+ 5,000
Extensibility limited limited Well defined limited
Container
Engine
Docker Docker
Roadmap:
CoreOS
Linux+Docker
Roadmap:CoreO
S
Garden
External
storage
support
DVD RexRay in host RexRay in host RexRay
Application
specific
framework
availability
Roadmap:Auror
a for generic
proposed 20+
applications,
Marathon and
Aurora for
generic
Multiple
language
platforms

KUBERNETES
• Started by Google
• Primary container integration is with Docker, but
enhancing support for CoreOS in upcoming release
• Implicit aka Declarative configuration philosophy
– Declare desired outcome
– User defined Key/value labels on objects can be used to
support this

KUBERNETES
• pods group containers together’
– pods are the smallest management unit
– provides “plumbing” to allow containers to interoperate
(micro-service per container) inside the pods
• replication controllers handle container lifecycle
• labels provide a way to find and query containers
and other objects
• services assemble a set of containers performing a
function

KUBERNETES
• monitors health of services, with auto-healing
• Load balancing built in
• External volume mounts supported in POC / release
candidate with additional external storage features in
active planning stage

SWARM
• Started by Docker Inc.
• Stays with Docker CLI/API originally developed for
single host, and allows using it across a multiple
node cluster
• Explicit aka Imperative configuration philosophy
– Declare desired how to achieve outcome
• Compose uses a yaml file to group containers

SWARM
• Opportunities for enhancement:
– Container communication across host boundaries
– External persistent volume management

MESOS
• Mesos is the scheduler that provides maximal
flexibility
• It allows you to run sub-schedulers (aka
Frameworks), many of which are available off-the
shelf – or you can implement your own
• It can support Docker containers, but also supports
non Docker workloads.

MESOS FRAMEWORKS
• Marathon
– Supports generic workloads that run in a linux container
– Supports Docker containers
• Hadoop
• Spark
• Kafka
• Many more

MESOS STORAGE SUPPORT
• External volume mounts supported now
• Full global external storage resource management,
using an offer/accept model is in active planning
stage now

CLOUD FOUNDRY
• Cloud Foundry delivers management at the
application level
• Containers are used “under the covers” but you
aren’t exposed to the details and don’t have to
manage them

CONTAINER SCHEDULERS AS PAAS STACKS
• A variety of PAAS stacks exist that attempt to give
you “everything you need” to create an app on
Amazon, or your own hardware, provided you accept
the constraints imposed by the platform.
• But marketing people hate to call these constraints.
Instead, the PaaS stack is described as opinionated
– Some container schedulers are more opinionated than
others. Basically the platform makes technology choices for
you, and you don’t get to change these.

CLOUD FOUNDRY
• Cloud Foundry is far more opinionated than other
container schedulers – and depending on your
objective, and staffing capacity,
this could be a good thing

TODAY’S AGENDA
Two rapidly developing
areas fields within the
container space:
1. Persistent Storage
2. Schedulers
..we’ll come back to
storage in a minute

A SCHEDULER SHOULD INTERFACE WITH
STORAGE AT MULTIPLE LEVELS
Create
Utilize
Retire
Lifecycle management with
governance
Mount managementAssignment to applications
(tenancy)

WHAT POLLY STORAGE SCHEDULER DOES…
• Manages interfaces to storage providers
– Multiple concurrent providers anticipated
• Mixed vendors
– Including public cloud
– Takes reports from each provider, and what their
capabilities and capacities are
– Uses admin defined key/value labels to enable
• Outcome based placement
• Concurrent use of multiple schedulers, with quotas
• Pool advertisement, with suballocation

STORAGE SCHEDULER = DELIVER SCALE OUT
Scale out is not just about adding capacity.
A Storage Scheduler will let you:
• Add, remove, replace storage provider platforms
– in or out of public cloud
• Add, remove, replace schedulers
• This is about giving you flexibility and
shifting the burden of keeping up to vendors

DEMO
• Demo attach of a persistent volume under
Kubernetes
• Shutdown container
• Re-attach same volume to a Mesos hosted container,
under Linux containerizer instead of Docker

Data Persistence in the New Container World
Wednesday 3PM
Joshua Bernstein
VP of Technology for ETD
Tobi Knaup
CEO & Co-Founder of Mesosphere
Guru Session

@EMCcode
@cantbewong
emccode.com
community.emccode.com
cantbewong.com
Come visit us at
Booth #1044
or in the vLab
Questions?

EMC World 2016 - code.13 State of the Container Ecosystem with Persistent Applications

EMC World 2016 - code.13 State of the Container Ecosystem with Persistent Applications

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (16)

Similar to EMC World 2016 - code.13 State of the Container Ecosystem with Persistent Applications

Similar to EMC World 2016 - code.13 State of the Container Ecosystem with Persistent Applications (20)

Recently uploaded

Recently uploaded (20)

EMC World 2016 - code.13 State of the Container Ecosystem with Persistent Applications

Editor's Notes