This document provides an overview of Apache Mesos, including: - Mesos is a cluster manager and meta-scheduler that provides efficient resource isolation and sharing across distributed applications or frameworks. - It introduces core Mesos concepts like resource allocation, high availability, schedulers, executors, and fine-grained vs. coarse-grained task execution. - Mesos is compared to Hadoop YARN, and supports frameworks like HDFS, Hadoop/Spark, Hive/Spark SQL, Pig, HBase, and Storm.

1. Datacenter Computing and
Resource Management Using
Apache Mesos
Ajay Viswanathan
Software Engineer - Research

2. The Agenda
• Introduction to Apache Mesos
• Core concepts
• Resource allocation
• High Availability and Failure Handling
• Schedulers and Executors
• Fine-grained and Coarse-grained execution
• Mesos vs YARN
• Building a Distributed Framework: Hands on tutorial
• Integration with Apache Spark: Demo

3. What is a distributed system?

4. • run simultaneous tasks / processes - distributed
• handle process failures - fault tolerant
• optimize performance - elasticity
What is a distributed system?

5. The Agenda
• Introduction to Apache Mesos
• Core concepts
• Resource allocation
• High Availability and Failure Handling
• Schedulers and Executors
• Fine-grained and Coarse-grained execution
• Building a Distributed Framework: Hands on tutorial
• Integration with Apache Spark: Demo

6. Who is using Mesos?
http://www.slideshare.net/timothysc/introduction-to-mesos

7. • is based on the paper - [Mesos: A Platform for Fine-Grained
Resource Sharing in the Datacenter] by Benjamin Hindman
et all. (part of the Berkeley Data Analytics Stack)
• has been developed at Twitter and is based on Google’s Borg
• is a cluster manager
• is a meta-scheduler and a microkernel for the datacenter
• provides efficient resource isolation
• allows sharing across distributed applications, or
frameworks
Mesos …

8. Where does it fit?
Mesos
HDFS
Hadoop / Spark
Hive / Spark SQL Pig
HBase Storm…
Kernel layer
Data layer
Framework layer
Distributed resources: CPU, RAM etc. Resource layer

9. Supported Frameworks - Constantly
being updated
http://www.enterprisetech.com/2014/08/19/google-mesosphere-make-google-clone-masses/

10. • Single-tenancy vs Multi-tenancy - better utilisation of
resources
• One large pool of resources - feels like programming on
personal machine
• Scalable and fault-tolerant - fire-and-forget
• Isolation - leads to better performance and safety
What are the benefits?

11. The Agenda
• Introduction to Apache Mesos
• Core concepts
• Resource allocation
• High Availability and Failure Handling
• Schedulers and Executors
• Fine-grained and Coarse-grained execution
• Mesos vs YARN
• Building a Distributed Framework: Hands on tutorial
• Integration with Apache Spark: Demo

12. • Master determines how many resources to offer,
Framework decides which offered resources to use
• Frameworks can reject offers
• Frameworks can also set filters to boost efficient in
satisfying constraints
• Mesos may revoke (kill) buggy / greedy jobs after giving its
Framework a grace period to clean up
• Guaranteed allocation allows frameworks to ensure tasks
don’t get killed within its defined limits
Resource allocation

13. A typical architecture
http://mesos.apache.org/documentation/latest/mesos-architecture/

14. Task Execution
http://mesos.apache.org/documentation/latest/mesos-architecture/

15. • Scheduler Callbacks
• resourceOffer(offerId, offers)
• offerRescinded(offerId)
• statusUpdate(taskId, status)
• slaveLost(slaveId)
• Scheduler Actions
• replyToOffer(offerId, tasks)
• setNeedsOffers(bool)
• setFilters(filters)
• getGuaranteedShare()
• killTask(taskId)
The Framework API
• Executor Callbacks
• launchTask(taskDescriptor)
• killTask(taskId)
• Executor Actions
• sendStatus(taskId, status)

16. • Both allow sharing of resources between different frameworks
• YARN is written in Java, while Mesos is written in C++
• YARN only offers memory scheduling, while Mesos offers CPU and
memory scheduling
• YARN uses UNIX processes to run tasks, Mesos uses Linux
Container groups to run tasks
• YARN allows specific requests and locality preference for offer
requests, Mesos is more generic but more coding for writing
frameworks
• Mesos is relatively more mature and supports a wider range of
Frameworks currently
Mesos vs YARN

17. The Agenda
• Introduction to Apache Mesos
• Core concepts
• Resource allocation
• High Availability and Failure Handling
• Schedulers and Executors
• Fine-grained and Coarse-grained execution
• Mesos vs YARN
• Building a Distributed Framework: Hands on tutorial
• Integration with Apache Spark: Demo

18. The Agenda
• Introduction to Apache Mesos
• Core concepts
• Resource allocation
• High Availability and Failure Handling
• Schedulers and Executors
• Fine-grained and Coarse-grained execution
• Mesos vs YARN
• Building a Distributed Framework: Hands on tutorial
• Integration with Apache Spark: Demo

19. • The original paper on Mesos - https://www.cs.berkeley.edu/
~alig/papers/mesos.pdf
• Boilerplate Mesos Framework code - https://github.com/
mesosphere/scala-sbt-mesos-framework.g8
• Google Borg, the precursor to Mesos - http://www.wired.com/
2013/03/google-borg-twitter-mesos/all/
• Configuring a Mesos cluster - http://mesosphere.com/docs/
getting-started/datacenter/install/
Links to explore

20. Thank You
You could also mail me your queries
at ajay@sigmoidanalytics.com
Questions time