Philip Norman Engineer with Mesosphere All Things Open October 26-27, 2016 Raleigh, North Carolina

1. © 2016 Mesosphere, Inc. All Rights Reserved. 1
MODERN
CONTAINER
ORCHESTRATION
(WITHOUT BREAKING THE BANK)

2. © 2016 Mesosphere, Inc. All Rights Reserved. 2
WAIT, WHO
ARE YOU?
Engineer at Mesosphere
DC/OS Contributor
@philipnrmn

3. © 2016 Mesosphere, Inc. All Rights Reserved. 3
A non-trivial app might
well need all the
following:
● multiple app servers
● load balancing
● message queues
● HA datastores
● analytics
RUNNING
NON-TRIVIAL
APPLICATIONS
AT SCALE IS
*REALLY*
COMPLICATED
Bring cluster computing to
non-experts: One of the most exciting
things about datacenter technology is
that it is increasingly being applied to
“big data” problems in the sciences. With
cloud computing, scientists can readily
acquire the hardware to run large
parallel computations; the main thing
missing is the right software.
These non-expert cluster users have
very different needs from those in large
corporations: they are not backed by an
operations team that will configure their
systems and tune their programs.
Instead, they need cluster software that
configures itself correctly out of the box,
rarely fails, and can be debugged
without intimate knowledge of several
interacting distributed systems. These
are difficult but worthwhile challenges
for the community to pursue.
-The Datacenter Needs an Operating
System

4. © 2016 Mesosphere, Inc. All Rights Reserved. 4
APACHE MESOS
THE KERNEL
OF THE DC/OS

5. © 2016 Mesosphere, Inc. All Rights Reserved. 5
THE BIRTH OF MESOS
APACHE MESOS
TWITTER TECH TALK
The grad students working on Mesos
give a tech talk at Twitter.
March 2010
APACHE INCUBATION
Mesos enters the Apache Incubator.
Spring 2009
CS262B
Ben Hindman, Andy Konwinski and
Matei Zaharia create “Nexus” as their
CS262B class project.
MESOS PUBLISHED
Mesos: A Platform for Fine-Grained
Resource Sharing in the Data Center is
published as a technical report.
September 2010
December 2010

6. © 2016 Mesosphere, Inc. All Rights Reserved. 6
What does an operating system provide?
● Resource management
● Programming abstractions
● Security
● Monitoring, debugging, logging
GRAD STUDENTS LEARNED HOW TO SHARE
APACHE MESOS
Sharing resources between batch
processing frameworks:
● Hadoop
● MPI
● Spark

7. © 2016 Mesosphere, Inc. All Rights Reserved. 7
CLUSTERING YOUR RESOURCES FOR YOU
APACHE MESOS
Apache Mesos is a cluster resource manager.
It handles:
● Aggregating resources and offering them to schedulers
● Launching tasks (i.e. processes) on those resources
● Communicating the state of those tasks back to schedulers
● Tasks can be:
● Long running services
● Ephemeral / batch jobs

8. © 2016 Mesosphere, Inc. All Rights Reserved. 8
SCHEDULERS AND TASKS
APACHE MESOS

9. © 2016 Mesosphere, Inc. All Rights Reserved. 9
SHARING RESOURCES FOR EFFICIENCY
APACHE MESOS
A naive approach to handling varied app
requirements: static partitioning.
This can cope with heterogeneity, but is
very expensive.

10. © 2016 Mesosphere, Inc. All Rights Reserved. 10
SHARING RESOURCES FOR EFFICIENCY
APACHE MESOS
Multiple frameworks can use the same
cluster resources, with their share
adjusting dynamically.

11. © 2016 Mesosphere, Inc. All Rights Reserved. 11
ORCHESTRATION
DC/OS IN
ACTION

12. © 2016 Mesosphere, Inc. All Rights Reserved. 12
THE DC/OS
“STACK”
ORCHESTRATION

13. © 2016 Mesosphere, Inc. All Rights Reserved. 13
THE DC/OS
“STACK”
ORCHESTRATION

14. © 2016 Mesosphere, Inc. All Rights Reserved. 14
Marathon is:
● a scheduler for long-running services
● a core component of DC/OS
● a meta-scheduler
DC/OS, MARATHON, AND LONG-RUNNING SERVICES
ORCHESTRATION

15. © 2016 Mesosphere, Inc. All Rights Reserved. 15
Marathon provides:
● monitoring and health-checks
● resilience for your services
● upgrades and deployments
DC/OS, MARATHON, AND LONG-RUNNING SERVICES
ORCHESTRATION

16. © 2016 Mesosphere, Inc. All Rights Reserved. 16
RESILIENCE IN ACTION
ORCHESTRATION

17. © 2016 Mesosphere, Inc. All Rights Reserved. 17
RESILIENCE IN ACTION
ORCHESTRATION

18. © 2016 Mesosphere, Inc. All Rights Reserved. 18
RESILIENCE IN ACTION
ORCHESTRATION

19. © 2016 Mesosphere, Inc. All Rights Reserved. 19
THE ANATOMY OF A DEPLOYMENT
ORCHESTRATION
A deployment:
● EITHER deploys a new application
OR replaces an existing application
OR changes the number of running instances
● can be rolled back if unsuccesful
● can trigger more deployments

20. © 2016 Mesosphere, Inc. All Rights Reserved. 20
THE DC/OS
“STACK”
ORCHESTRATION

21. © 2016 Mesosphere, Inc. All Rights Reserved. 21
THE OLD
WORLD
ORCHESTRATION
Jenkins master
Life is good.
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins master
Oh, is that Jenkins
cluster still up? I do
all my builds on a
Raspberry Pi now.
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins master
My builds take
hours to run!
Jenkins agent

22. © 2016 Mesosphere, Inc. All Rights Reserved. 22
JUST USE WHAT YOU NEED, WHEN YOU NEED IT AND SHARE THE LOVE RESOURCES
ORCHESTRATION
Mesos master
Mesos agentMesos agentMesos agent Mesos agent Mesos agent
Jenkins agent (master
1)
Jenkins agent (master
1)
Jenkins agent (master
3)
Jenkins agent (master
3)
Jenkins agent (master
3)
Jenkins master 1
Life is good.
Jenkins master 2 Jenkins master 3
Life is good.
Life is good.

23. © 2016 Mesosphere, Inc. All Rights Reserved. 23
JUST USE WHAT YOU NEED, WHEN YOU NEED IT AND SHARE THE LOVE RESOURCES
ORCHESTRATION
Mesos master
Mesos agentMesos agentMesos agent Mesos agent Mesos agent
Jenkins agent (master
3)
Jenkins agent (master
3)
Jenkins agent (master
3)
Jenkins agent (master
3)
Jenkins agent (master
3)
Jenkins master 1
Team dinner!
Let’s bounce.
Jenkins master 2 Jenkins master 3
Life is good.
Life is good.

24. © 2016 Mesosphere, Inc. All Rights Reserved. 24
ORCHESTRATION
LIVE DEMO

25. © 2016 Mesosphere, Inc. All Rights Reserved. 25
A MESOS
CLUSTER IN
THE WILD
ORCHESTRATION
A general diagram deployment workflow for internal SAMI developers. Credit: Samsung

26. © 2016 Mesosphere, Inc. All Rights Reserved. 26
CI/CD WITH
JENKINS AND
MARATHON
IN DC/OS
ORCHESTRATION

27. © 2016 Mesosphere, Inc. All Rights Reserved. 27
QUESTIONS?
DC/OS docs:
http://dcos.io

28. © 2016 Mesosphere, Inc. All Rights Reserved. 28
THANK YOU!
philip@mesosphere.io
@philipnrmn