Papers we Love @ Seattle, 08/14/2015 Abstract We present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI. Sharing improves cluster utilization and avoids per-framework data replication. Mesos shares resources in a fine-grained manner, allowing frameworks to achieve data locality by taking turns reading data stored on each machine. To support the sophisticated schedulers of today's frameworks, Mesos introduces a distributed two-level scheduling mechanism called resource offers. Mesos decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to run on them. Our results show that Mesos can achieve near-optimal data locality when sharing the cluster among diverse frameworks, can scale to 50,000 (emulated) nodes, and is resilient to failures.

1. Mesos
A Platform for Fine-Grained Resource Sharing in the Data Center
@PapersWeLoveSEA
@ankurcha - Ankur Chauhan
Benjamin Hindman, Andy konwinski, Matei Zaharia,
Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, Ion Stoica

2. Background
Rapid innovation in distributed/cluster
computing
● Hadoop, Spark, Flink, Tez, HDFS, Dryad etc
● Micro-services / web-services (long running)
● <Your custom framework> …
… There is a lot of wasted effort.

3. Problem
● Rapid innovation in cluster/dist. frameworks
● No single framework is optimal for all
workloads
● What do we want?
○ Run multiple frameworks on a single cluster (big)
■ Maximize utilisation of resources. (elastic)
■ Share data between frameworks.

4. Goal

5. Solution

6. Mesos Goals
● High utilization of resources
● Support diverse frameworks
● Scalability to 10,000s of nodes
● Reliability in face of failures
● Efficient with minimal overhead
● Highly available via leader election of
master.

7. Other benefits
● Run multiple instances of a framework
○ Isolate production and experimental tasks
○ Run multiple version of a framework
● Build special framework targeting a
particular domain
○ eg: spark

8. Mesos Architecture
● Resource Offers
○ Offer available resources to frameworks
● 2-level scheduler
○ Mesos takes the decision of which framework gets
the offer.
○ Frameworks decide whether to accept/reject offer.
● Fine grained sharing
○ Improved utilization, responsiveness, data locality
● Pluggable allocation module

9. Mesos Architecture
High level architecture
Design components
● Mesos Master
○ Runs the allocation module
● Mesos Slave
○ Offers resources
○ task/executor isolation
● Scheduler
○ Accepts/rejects resource offers
● Executor
○ Runs tasks
● Task
○ Unit of work

10. Mesos Architecture
● Making resource offers scalable and robust
○ Let schedulers define filters
○ Offered resources counted as allocations
○ Rescind offers if not accepted within timeout
● Fault tolerance
○ Zookeeper for leader election (hot-standby)
○ Mesos reports slave and executor failures to
scheduler
○ Minimal internal state in master

11. Resource offers
● Mesos decides how many
resources to offer to each
framework, based on
allocation policy (fair share),
while framework decide which
resource offers to accept and
which tasks to run on them.
● When a framework accepts an
offer, it passes Mesos a
description of the task (and
executor) to launch.

12. Analysis
● Resource offers work well when:
○ Frameworks can scale elastically.
○ Task durations are homogeneous.
○ Frameworks have many preferred nodes.
● Conditions hold true in many frameworks
(hadoop, dryad, spark … )
○ Work is divided into short tasks
○ Data is replicated across multiple nodes

13. Limitations of distributed scheduling
● Fragmentation
○ With heterogeneous resource demands, distributed
collection of frameworks may not be able to bin pack
optimally.
● Independent framework constraints
○ Esoteric dependency scenarios can only be satisfied
with a centralized scheduler
● Framework complexity
○ Need to deal with resource offers ( not onerous )

14. Implementation
● ~10,000 lines of C++
● libprocess - actor-based programming (akka
for c++)
● Cgroups / Docker for task isolation
● Zookeeper for leader election (HA)
● Ports / plugins
○ Hadoop - map reduce
○ Torque / MPI
○ Spark Framework - iterative jobs

15. Evaluation
Dynamic Resource
sharing

16. Evaluation
Data locality with resource offers
○ 16 Hadoop instances using 93 EC2 instances
○ 1.7x speedup with mesos
○ 97% data locality with 5sec delay scheduling

17. Evaluation
Mesos scalability
● 99 EC2 instances
● Scaled to 50k slaves, 200 frameworks, 100k tasks

18. Evaluation
● Failure recovery
○ Mean time to recovery was between 4-8 seconds
with 95% confidence of 3secs on either side.
● Performance isolation
○ Linux containers are not perfect at isolation
○ 30% increase in request latency vs 550% increase
during tests
■ apache server + cpu hog process

19. The end