The document provides an introduction to Apache Mesos, an open-source cluster manager that efficiently handles workloads in a distributed environment. It explains Mesos's architecture, its dynamic resource sharing, and its fault-tolerant, scalable nature, along with examples of frameworks that run on it, such as Spark and Hadoop. Notable companies like Apple, Netflix, and Twitter utilize Mesos for various applications, including backend operations and real-time data processing.

1. Introduction to Mesos
Mahesh Chand
Software Consultant
KNOLDUS SOFTWARE LLP

2. Agenda
● What is Cluster Manager ?
● What is Mesos ?
● Why Mesos ?
● Mesos Architecture
● What is framework
● Example of Resource Offer
● Mesos API
● Mesos Implementation
● Who is using it ?
● Demo

3. What is Cluster Manager ?

4. What is Mesos ?
In layman’s term, Imagine a busy airport.
Airplanes are constantly taking off and landing.
There are multiple runways, and an airport dispatcher is assigning time-slots
to airplanes to land or takeoff.
So Mesos is the airport dispatcher, runways are compute nodes, airplanes
are compute tasks, and frameworks like Hadoop, Spark and Google
Kubernetes are airlines companies.

5. What is Mesos ?
In technical terms, Apache Mesos is the first open source cluster manager
that handles the workload efficiently in distributed environment through
dynamic resource sharing and isolation. This means that you can run any
distributed application i.e spark, hadoop etc., which requires clustered
resources.
It uses two level scheduler mechanism called resource offers.
Mesos decides how many resources to offer each framework, while
framework decides which resources to accept and which computations to run
on them.

6. Mesos Architecture

7. Why Mesos ?
1) Abstracts out the managing resources from processing part.
2) Fault-tolerant
3) Scalable
4) Fine grained resource sharing
5) It provides Web UI to monitor cluster state
6) Provides API’s in multiple language like c++, java, python etc

8. What is Framework ?
Framework is an application that run on mesos i.e spark, storm, kafka etc
A framework has two parts:
● the controller portion, which is called the scheduler,
● and the worker portion, which are called the executors.

9. Framework:

10. Example Of Resource Offer

11. Mesos API

12. Mesos Implementation
1) Mesos is implemented in C++.
2) To reduce the complexity of implementation, c++ library libprocess that
provides an actor based programming model using efficient
asynchronous I/O mechanism.
3) Use linux cgroups for resource isolation i.e, cpu, memory, i/o etc.
4) It uses protocol buffer for serialization.
5) It uses mostly factory and builder design patterns.

13. Who is using it ?
Apple uses Mesos to power the entire backend for its Siri personal-assistant
application and has built a custom scheduler called J.A.R.V.I.S to handle job
deployment and resource allocation.
Netflix uses it to run a mix of batch, stream processing, and variety of use
cases including real time anomaly detection, training, machine learning
orchestration and node.js based microservices.
Twitter is running several key services including analytics and ads.

14. Demo
1 ) Start the mesos master.
2) Start the mesos slave.
3) Deploy your framework on mesos
CODE: spark-on-mesos

15. References
Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center
Mesos Documentation
Building Distributed Systems from Scratch - Part 1
Getting Started With Mesos