Apache mesos - cluster manager for resource management

1. Apache Mesos

2. Apache mesos
Apache Mesos is an open-source cluster manager that was developed at
the University of California, Berkeley. It "provides efficient resource isolation and
sharing across distributed applications, or frameworks". The software enables
resource sharing in a fine-grained manner, improving cluster utilization.
Since being developed at UC Berkeley, it has been adopted by several large
software companies, including Twitter, Airbnb and Apple. At least 50 organizations
currently use Mesos.[

3. What is Apache
mesos ?
As said Apache mesos is scheduler capable of handling multiple
resources, provides efficient resource isolation and sharing across
distributed applications, or frameworks. The software enables resource
sharing in a fine-grained manner, improving cluster utilization. In a
distributed environment, Mesos runs on every machine.
You put Mesos in charge of a bunch of machines. Then you can run
"applications". Hadoop, Spark, graph processing, machine learning. All
applications will "ask" for resources. Mesos gives resources and keeps the
accounting.Not all applications need to allocate resources the same way.
Schedulers know what their client applications want as resources and
lifecycle and can ask Mesos.

5. Why Apache
mesos ?
Mesos offers huge benefits to both developers and operators. The ability of
Mesos to consolidate various frameworks on a common infrastructure not only
saves on infrastructure costs, but also provides operational benefits to the Ops
teams and simplifies developers' view of the infrastructure, ultimately leading
to business success. Here are some of the reasons for organizations to embrace
Mesos:
 Scalability to thousands of nodes
Dynamic resource sharing through Mesos drives higher utilization and
throughput
Fault-tolerant replicated master using ZooKeeper
Frameworks can be built on Java/Python/Scala/C/C++
Isolation between tasks with Linux containers
Web UI for viewing cluster state

6. Components
Apache mesos is a fault-tolerant cluster manager. It’s designed for distributed
computing environments to provide resource isolation and management across a cluster of slave nodes.
A Mesos cluster is made up of four major components:
1.ZooKeepers
2.Mesos masters
3.Mesos slaves
4.Frameworks

7. Components
1.ZooKeeper
Apache ZooKeeper is a centralized configuration manager, used by distributed applications such
as Mesos to coordinate activity across a cluster. Mesos uses ZooKeeper to elect a leading master
and for slaves to join the cluster.
2.Mesos masters
A Mesos master is a Mesos instance in control of the cluster.
A cluster will typically have multiple Mesos masters to provide fault-tolerance, with one instance
elected the leading master.

8. Components
3.Mesos slaves
A Mesos slave is a Mesos instance which offers resources to the cluster. They are the ‘worker’ instances - tasks are
allocated to the slaves by the Mesos master.
4.Frameworks
On its own, Mesos only provides the basic “kernel” layer of your cluster. It lets other applications request
resources in the cluster to perform tasks, but does nothing itself. Frameworks bridge the gap between the Mesos
layer and your applications. They are higher level abstractions which simplify the process of launching tasks on
the cluster.
Components:
Scheduler: Receives resource offers and Launch tasks
Executor: Executor is launched by the slave to execute tasks on the slave

9. Mesos Architecture
Mesos architecture consists of a Mesos Master that manages Mesos Slaves and frameworks that run tasks on
Mesos Slaves. Master is in soft state with one or more stand-by managed by ZooKeeper to implement the
failover mechanism.
A framework is an application running on top of Mesos. A Mesos framework normally uses Mesos APIs (C++,
Python, Ruby, JVM, Go) exposed by Mesos kernel. Using these APIs a framework interacts with Mesos Master
to accept and register the resources allocation on Mesos Slaves.
A framework consists of two key components,
a scheduler accepts and registers resources from master, and
an executor is a process launched on slave nodes to run the framework’s tasks.
Some frameworks distributed in nature such as Hadoop, Spark, etc. Distributed frameworks normally employ
distributed executors.

11. Running the task
Step 1:- Slave 1 reports to the master that it has 4 CPUs and 4 GB of memory free. The master
then invokes the allocation module, which tells it that framework 1 should be offered all
available resources.
Step 2:- The master sends a resource offer describing these resources to framework 1.
Step 3:- The framework's scheduler replies to the master with information about two tasks to
run on the slave, using 2 CPUs; 1 GB RAM for the first task, and 1 CPUs; 2 GB RAM for the second
task.
Step 4:- Finally, the master sends the tasks to the slave, which allocates appropriate resources to
the framework's executor, which in turn launches the two tasks (depicted with dotted borders).
Because 1 CPU and 1 GB of RAM are still free, the allocation module may now offer them to
framework 2. In addition, this resource offer process repeats when tasks finish and new
resources become free.

13. Excecution flow
slave notifies master about its available resources
tasks is scheduled by the scheduler. So scheduler have info about available tasks
Scheduler sends the tasks to the right slave based on the available resources of the slave
Slave check of executor that is already running,if not it launches a new one and execute the
task on the executor

14. Mesos allows you to share a pool of servers that can all run different
parts of your application without them interfering with each other and
with the ability to dynamically allocate resources across the cluster as
needed.

15. Containerizer
Containerizers are intended to run tasks in ‘containers’ which in turn are
used to:
Isolate a task from other running tasks.
‘Contain’ tasks to run in limited resource runtime environment.
Control task’s individual resources (e.g, CPU, memory) programatically.
Run software in a pre-packaged file system image, allowing it to run in
different environments.
Composing containerizer
Allows multiple container technologies to play together. It is
enabled when you configure the --containerizers=cont1,cont2,.. The order
of the comma separated list is important as the first containerizer that
supports the task’s container configuration will be used to launch the task.
Eg:- --containerizers=mesos,docker
Use cases:
For testing tasks with different types of resource isolations.

16. Docker containerizer
Docker containerizer allows tasks to be run inside docker container. This containerizer is enabled when you configure the
agent flag as --containerizers=docker
Use cases:
•If task needs to be run with the tooling that comes with the docker package.
•If Mesos agent is running inside a docker container.
Mesos containerizer
This containerizer allows tasks to be run with an array of pluggable isolators provided by Mesos. This is the native Mesos
containerizer
solution and is enabled when you configure the agent flag as --containerizers=mesos,docker
Use cases:
•Allow Mesos to control the task’s runtime environment without depending on other container technologies (e.g., docker).
•Want fine grained operating system controls (e.g., cgroups/namespaces provided by linux).
•Want Mesos’s latest container technology features.
•Need additional resource controls like disk usage limits, which might not be provided by other container technologies.
•Want to add custom isolation for tasks.

17. In a simpler way…..
Imagine a busy airport. Airplanes are constantly taking off and landing.
There are multiple runways, and an airport dispatcher is assigning time-slots to airplanes to land
or takeoff. So Mesos is the airport dispatcher, runways are compute nodes, airplanes are
compute tasks, and frameworks like Hadoop, Spark and Google Kubernetes are airlines
companies.