The document is a presentation on DC/OS, an open source tool for managing containerized applications in a clustered environment. It discusses how DC/OS builds on Apache Mesos to provide service discovery, load balancing, security and management of applications. It also summarizes some key capabilities of DC/OS like running various frameworks including Marathon, Docker and Kubernetes, integrated storage options, and providing a centralized view of operations across all frameworks and clusters.

1. © 2017 Mesosphere, Inc. All Rights Reserved. 1
Leeds DevOps
Introduction to DC/OS
Matt Jarvis
@mattj_io

2. © 2017 Mesosphere, Inc. All Rights Reserved.
❏ Head of Community and Evangelism at Mesosphere
❏ Building stuff with open source software for 15+ years
❏ Ops, Dev and Dev/Ops
Matt Jarvis, Head of Community and Evangelism
@mattj_io

3. © 2017 Mesosphere, Inc. All Rights Reserved. 3
In the beginning there was ….

4. © 2017 Mesosphere, Inc. All Rights Reserved. 4
Things get smaller

5. © 2017 Mesosphere, Inc. All Rights Reserved. 5
Connect some smaller computers to the big one ...

6. © 2017 Mesosphere, Inc. All Rights Reserved. 6
Scaling ...

7. © 2017 Mesosphere, Inc. All Rights Reserved. 7
MapReduce is crunching data ..

8. © 2017 Mesosphere, Inc. All Rights Reserved. 8
First generation creates processing silos

9. © 2017 Mesosphere, Inc. All Rights Reserved. 9
Subdivision of infrastructure

10. © 2017 Mesosphere, Inc. All Rights Reserved. 10
Splitting applications up

11. © 2017 Mesosphere, Inc. All Rights Reserved. 11
We need to turn faster ..

12. © 2017 Mesosphere, Inc. All Rights Reserved.
MODERN APPLICATION -> FAST DATA BUILT-IN
Data Ingestion
Request/Response
Devices
Client
Sensors
Message
Queue/Bus
Microservices Distributed Storage
Analytics
(Streaming) Use Cases:
● Anomaly detection
● Personalization
● IoT Applications
● Predictive Analytics
● Machine Learning

13. © 2017 Mesosphere, Inc. All Rights Reserved.
The SMACK Stack
Data Ingestion
Request/Response
Devices
Client
Sensors
Use Cases:
● Anomaly detection
● Personalization
● IoT Applications
● Predictive Analytics
● Machine Learning
Message
Queue/Bus
Microservices Distributed Storage
Analytics
(Streaming)

14. © 2017 Mesosphere, Inc. All Rights Reserved.
Complexity increases

15. © 2017 Mesosphere, Inc. All Rights Reserved. 15
Apache Mesos:
The datacenter kernel
http://mesos.apache.org/

16. © 2017 Mesosphere, Inc. All Rights Reserved.
• A cluster resource
negotiator
• A top-level Apache project
• Scalable to 10,000s of
nodes
• Fault-tolerant, battle-tested
• An SDK for distributed apps
• Native Docker support
16
Building block of the modern internet
http://mesos.apache.org/documentation/latest/powered-by-mesos/

17. © 2017 Mesosphere, Inc. All Rights Reserved. 17
THE BIRTH OF MESOS
TWITTER TECH TALK
The grad students working on Mesos
give a tech talk at Twitter.
March 2010
APACHE INCUBATION
Mesos enters the Apache Incubator.
Spring 2009
CS262B
Ben Hindman, Andy Konwinski and
Matei Zaharia create “Nexus” as their
CS262B class project.
MESOS PUBLISHED
Mesos: A Platform for Fine-Grained
Resource Sharing in the Data Center is
published as a technical report.
September 2010
December 2010
DC/OS
April 2016

18. © 2017 Mesosphere, Inc. All Rights Reserved. 18
Solving the Fail Whale

19. © 2017 Mesosphere, Inc. All Rights Reserved. 19
THE BIRTH OF MESOS
TWITTER TECH TALK
The grad students working on Mesos
give a tech talk at Twitter.
March 2010
APACHE INCUBATION
Mesos enters the Apache Incubator.
Spring 2009
CS262B
Ben Hindman, Andy Konwinski and
Matei Zaharia create “Nexus” as their
CS262B class project.
MESOS PUBLISHED
Mesos: A Platform for Fine-Grained
Resource Sharing in the Data Center is
published as a technical report.
September 2010
December 2010
DC/OS
April 2016

20. © 2017 Mesosphere, Inc. All Rights Reserved.
Two level scheduling
Mesos Master and Agents
● Abstract resources into single pool
● Offers and tracks resources
● Guarantees isolation
● Handles workload restart on failure
Mesos Framework
● Consumes resources
● Deploys tasks
● Provides application specific logic for deployment, recovery, upgrade

21. © 2017 Mesosphere, Inc. All Rights Reserved. 21
MULTIPLEXING OF DATA, SERVICES, USERS, ENVIRONMENTS
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Apache Mesos
automated schedulers, workload multiplexing onto the
same machines
mySQL
microservice
Cassandra
Spark/Hadoop
Kafka

22. 22

23. © 2017 Mesosphere, Inc. All Rights Reserved. 23
DC/OS brings it all together
● Service Discovery
● Load Balancing
● Security
● Ease of installation
● Comprehensive tooling for operations
● Built in frameworks for long running and
scheduled jobs
● Catalog of pre-configured apps (including
Apache Spark, Apache Kafka…), browse at
http://universe.dcos.io/
● And much more https://dcos.io/

24. 24
DC/OS is … ● 100% open source (ASL2.0)
+ A big, diverse community
● An umbrella for ~30 OSS projects
+ Roadmap and designs
+ Documentation and tutorials
● Not limited in any way

25. © 2017 Mesosphere, Inc. All Rights Reserved. 25
DC/OS
Architecture
Overview
Security &
Governance
Container Orchestration Monitoring & Operations User Interface & Command Line
HDFS Jenkins Marathon Cassandra Flink
Spark Docker Kafka MongoDB +30 more...
DC/OS
Services & Containers
ANY INFRASTRUCTURE

26. © 2017 Mesosphere, Inc. All Rights Reserved. 26
Web-based GUI
https://dcos.io/docs/lates
t/usage/webinterface/
Interact with DC/OS (1/2)

27. © 2017 Mesosphere, Inc. All Rights Reserved. 27
CLI tool
https://dcos.io/docs/latest/usage/cli/
API
https://dcos.io/docs/latest/api/
Interact with DC/OS (2/2)

28. © 2017 Mesosphere, Inc. All Rights Reserved. 28
Catalog of Applications

29. © 2017 Mesosphere, Inc. All Rights Reserved. 29
Install an
Application

30. © 2017 Mesosphere, Inc. All Rights Reserved. 30
Application
JSON

31. © 2017 Mesosphere, Inc. All Rights Reserved. 31
Service Discovery
Critical to distributed systems since container can be spawned anywhere
Mesos DNS
● Each instance of a service given a DNS entry in pattern task.scheduler.mesos eg. myapp.marathon.mesos
● Basic round robin load balancing
● Requires an A and SRV lookup
Named VIPS
● Service allocated name based virtual IP eg. redis.marathon.l4lb.thisdcos.directory:6379
● Very high performance - integrates with connection tracking table in kernel for real address resolution
● Also provides low cost East/West load balancing
● Uses gossip protocol to propagate between nodes
● ~100ms update times

32. © 2017 Mesosphere, Inc. All Rights Reserved. 32
Load balancing - MarathonLB
● Based on HAProxy
● Ingests state of running applications
● Regenerates HAProxy configuration
● Can be North/South or East/West

33. © 2017 Mesosphere, Inc. All Rights Reserved. 33
Integrated storage options

34. © 2017 Mesosphere, Inc. All Rights Reserved. 34
Networking
● Integrated VXLAN based virtual networks
● CNI compatible upstream integrations - Calico, OpenContrail etc.

35. © 2017 Mesosphere, Inc. All Rights Reserved. 35
● Mesos can’t run applications on its
own.
● A Mesos framework is a distributed
system that has a scheduler.
● Schedulers like Marathon start and
keep your applications running. A bit
like a distributed init system.
● Learn more at
https://mesosphere.github.io/marat
hon/
Marathon

36. © 2017 Mesosphere, Inc. All Rights Reserved. 36
Mesos Docker containerizer

37. © 2017 Mesosphere, Inc. All Rights Reserved. 37
Mesos Universal Container Runtime - no Docker runtime

38. © 2017 Mesosphere, Inc. All Rights Reserved. 38
Kubernetes
● Beta framework for running Kubernetes clusters
● Multiple clusters within same DC/OS cluster
● Multiple versions within same DC/OS cluster

39. © 2017 Mesosphere, Inc. All Rights Reserved. 39
Centralised operations

40. © 2017 Mesosphere, Inc. All Rights Reserved. 40
Centralised operations
Framework integrations
● CLI extensions
● Integrated UI’s
Logging aggregation
● Available through API, CLI and UI
● Integration with ELK and others
Metrics aggregation
● Available through UI, CLI and API
● Integrates with Prometheus, InfluxDB etc.
Single pane of glass across all frameworks
dcos kafka topic create topic1 --partitions 1 --replication 1

41. © 2017 Mesosphere, Inc. All Rights Reserved. 41
Try it out !

42. © 2017 Mesosphere, Inc. All Rights Reserved. 42
Questions?
Matt Jarvis
Twitter: @mattj-io
Email: mjarvis@mesosphere.com
https://dcos.io
@dcos
users@dcos.io
/dcos
/dcos/examples
/dcos/demos
chat.dcos.io

43. © 2017 Mesosphere, Inc. All Rights Reserved.
The SMACK Stack

44. © 2017 Mesosphere, Inc. All Rights Reserved. 44
DEM0
Financial Transaction Processing using Apache Kafka and Apache Flink