The document discusses Apache Mesos, a top-level Apache project designed as a cluster resource negotiator for scalable data centers. It covers its architecture, functionalities, and the evolution of the Data Center Operating System (DC/OS) which supports modern distributed applications. Key features include resource management, fault tolerance, and support for multiple frameworks such as Hadoop, Spark, and Docker.

1. 1
Mesos and the
Architecture of the
New Datacenter
@joerg_schad

2. © 2016 Mesosphere, Inc. All Rights Reserved. 2

3. Flink Cassandra Rails Spark memcached

4. Flink Cassandra Rails Spark memcached

5. © 2015 Mesosphere, Inc. All Rights Reserved. 5
A naive approach to handling varied app
requirements: static partitioning.
This can cope with heterogeneity, but is
very expensive.
KEEP IT STATIC
time

6. © 2015 Mesosphere, Inc. All Rights Reserved. 6
Maintaining sufficient headroom to
handle peak workloads on all partitions
leads to poor utilization overall.
KEEP IT STATIC
time

7. © 2015 Mesosphere, Inc. All Rights Reserved. 7
Multiple frameworks can use the same
cluster resources, with their share
adjusting dynamically.
SHARED RESOURCES
time

8. © Gerard Julien/AFP
Run everything in containers!

9. © 2015 Mesosphere, Inc. All Rights Reserved.
What about container management?
9

10. © 2015 Mesosphere, Inc. All Rights Reserved.
What about container management?
10

11. © 2015 Mesosphere, Inc. All Rights Reserved. 11
MESOS:
ORIGINS

12. © 2015 Mesosphere, Inc. All Rights Reserved. 12
THE BIRTH OF MESOS
TWITTER TECH TALK
The grad students working on Mesos
give a tech talk at Twitter.
March 2010
APACHE INCUBATION
Mesos enters the Apache Incubator.
Spring 2009
CS262B
Ben Hindman, Andy Konwinski and
Matei Zaharia create “Nexus” as their
CS262B class project.
MESOS PUBLISHED
Mesos: A Platform for Fine-Grained
Resource Sharing in the Data Center is
published as a technical report.
September 2010
December 2010
DC/OS
April 2016

13. © 2015 Mesosphere, Inc. All Rights Reserved. 13
Sharing resources between batch
processing frameworks
● Hadoop
● MPI
● Spark
What does an operating system provide?
● Resource management
● Programming abstractions
● Security
● Monitoring, debugging, logging
TECHNOLOGY VISION

14. © 2015 Mesosphere, Inc. All Rights Reserved.
• A top-level Apache project
• A cluster resource
negotiator
• Scalable to 10,000s of
nodes
• Fault-tolerant, battle-tested
• An SDK for distributed apps
• Native Docker support
14
Apache Mesos

15. © 2015 Mesosphere, Inc. All Rights Reserved. 15
ARCHITECTURE
MESOS FUNDAMENTALS

16. © 2015 Mesosphere, Inc. All Rights Reserved. 16
ARCHITECTURE
MESOS FUNDAMENTALS
● Agents advertise resources to Master
● Master offers resources to Framework
● Framework rejects/uses resources
● Agents report task status to Master

17. © 2015 Mesosphere, Inc. All Rights Reserved. 17
ARCHITECTURE
MESOS FUNDAMENTALS
● Agents advertise resources to Master
● Master offers resources to Framework
● Framework rejects/uses resources
● Agents report task status to Master

18. © 2015 Mesosphere, Inc. All Rights Reserved. 18
ARCHITECTURE
MESOS FUNDAMENTALS
● Agents advertise resources to Master
● Master offers resources to Framework
● Framework rejects/uses resources
● Agents report task status to Master

19. © 2015 Mesosphere, Inc. All Rights Reserved. 19
ARCHITECTURE
MESOS FUNDAMENTALS
● Agents advertise resources to Master
● Master offers resources to Framework
● Framework rejects/uses resources
● Agents report task status to Master

20. © 2015 Mesosphere, Inc. All Rights Reserved. 20
ARCHITECTURE
MESOS FUNDAMENTALS
● Agents advertise resources to Master
● Master offers resources to Framework
● Framework rejects/uses resources
● Agents report task status to Master

21. © 2016 Mesosphere, Inc. All Rights Reserved. 21
YARN

22. © 2016 Mesosphere, Inc. All Rights Reserved. 22
YARN

23. © 2015 Mesosphere, Inc. All Rights Reserved. 23
Distributed
Systems
SDK„Developers will ask for
an API not for a VM to
run their software“

24. © 2015 Mesosphere, Inc. All Rights Reserved. 24
● Focus on application logic, not on data-center structure
● Avoid networking-related code
● Reuse of built-in fault-tolerance and high availability
● Reuse distributed (infrastructure) frameworks (e.g., storage)
Distributed System SDK
Mesos

25. © 2015 Mesosphere, Inc. All Rights Reserved. 25
Scheduler Interface
● Utilise resources
● ResourceOffer
● React on Tasks Events
● TaskStatus Updates
Executor Interface (optional)
● Task life-cycle and monitoring
● Provided executors
● Command Executor
● Docker Executor
First Framework/Service
Mesos

26. © 2015 Mesosphere, Inc. All Rights Reserved. 26

27. 27
WHERE DC/OS CAME FROM
PHYSICAL (x86) VIRTUAL HYPERSCALEMAINFRAME
SERVER VIRTUAL MACHINEPARTITION (LPAR)
UNIT OF
INTERACTION
● ERP, CRM, PRODUCTIVITY,
MAIL & WEB SERVER
● LINUX, WINDOWS
● DATA / TRANSACTION
PROCESSING
● UNIX, IBM OS/360
DEFINITIVE
APPS AND OS
● ERP, CRM, PRODUCTIVITY,
MAIL & WEB SERVER
● HYPERVISOR + GUEST OS
● BIG DATA, INTERNET OF
THINGS, MOBILE APPS
● ???
???DATACENTER
NEW FORM FACTOR FOR
DEVELOPING AND RUNNING APPS
● BIG DATA, INTERNET OF
THINGS, MOBILE APPS
● DATACENTER OPERATING
SYSTEM (DC/OS)

28. Datacenter Operating System (DC/OS)
Distributed Systems Kernel (Mesos)
DC/OS ENABLES MODERN DISTRIBUTED APPS
Big Data + Analytics EnginesMicroservices (in containers)
Streaming
Batch
Machine Learning
Analytics
Functions &
Logic
Search
Time Series
SQL / NoSQL
Databases
Modern App Components
Distributed systems kernel to
abstract resources
Ecosystem of frameworks & apps
Consistent architecture to run on
top of kernel
User Interface (GUI & CLI)
Core system services
(e.g., distributed init, cron, service
discovery, package mgt & installer,
storage)
Any Infrastructure (Physical, Virtual, Cloud)
28

29. 29
THE
BASICS
DC/OS is …
● 100% open source (ASL2.0)
+ A big, diverse community
● An umbrella for ~30 OSS projects
+ Roadmap and designs
+ The build tool chain
+ Docs and tutorials
● Not limited in any way
● Familiar, with a few new features

30. 30

31. 31
THE
UNIVERSE

32. 32

33. © 2015 Mesosphere, Inc. All Rights Reserved. 33
Questions?
Thank you!