Apache Mesos: Architecture, Design and Code Review

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Apache Mesos:
Architecture, Design and Code Review
By: Morteza Zakeri
Software Architectures Course
Instructor: Dr. Mehrda...

25
Outline
• Introduction
• Architectural review
• Implementation
• Design pros and cons
• Reverse engineering Mesos
• Con...

25
What is Apache Mesos?
• A platform for sharing clusters between multiple diverse
cluster computing frameworks.
• A dist...

25
Motivation to sharing a cluster
• We live in the high performance computing (HPC) era!
• There is no single framework t...

25
Common solutions for sharing a cluster
1. Statically partition the cluster and run one framework per
partition.
2. Allo...

25
Fine-grained sharing
Framework 1
Framework 2
Framework 3
321
654
987
Storage System (e.g. HDFS)
321
654
987
Storage Sys...

25
Mesos design decisions
• Small microkernel-like core
• Implements fine-grained sharing
• Allocation at the level of tas...

25
Mesos design choices
• Centralized scheduler vs. Resource Offer
• Resource Offer:
• Delegating control over scheduling ...

25
Apache Mesos architecture
Mesos architecture diagram, showing two running frameworks (Hadoop and MPI).
Introduction Arc...

25
Apache Mesos architecture: Resource Offer
• The master implements ﬁne-grained sharing across
frameworks using resource ...

25
Apache Mesos architecture: Resource Offer
• Each framework running on Mesos consists of two
components:
• A scheduler t...

25
Resource Offer: an example
Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion
October ...

25
Apache Mesos API (early versions)
Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion
O...

25
Implementation
• About 10,000 lines of C++ code (early versions).
• Today version (full): > 250,000
• Runs on Linux, So...

25
Implementation: why C++?
I. Performance
II. Maintainability (static typing)
III. Object oriented (and its benefits)
IV....

25
Design good points
• Uses object oriented design elements:
• Abstraction, modularization, and encapsulation.
• Uses som...

25
Design weak points
• Backward incompatible
• E.g. changed function signature for create().
• Unutilized abstractions
• ...

25
Dive into source code
• Code breakdown
• About 2% inactive code
• 9806 lines
• Smell: unutilized abstractions
Remove t...

25
Dive into source code
• Method breakdown
• 86% public methods
• Weak (leaky) encapsulation
• Public methods in the same...

25
Dive into source code
• Largest methods
• _accept()
• 1147 lines
Break them into sub
methods!
• Improve some aspects o...

25
More on abstraction mesos::internal::master::Master::Http
mesos::internal::checks::check::Http
Smell: duplicate abstrac...

25
Conclusion
• Mesos shares clusters efficiently among diverse frameworks
• A framework ensemble / orchestration builder!...

25
Conclusion
• What do you think about this?

Hardware
OS
MESOS
Introduction Architecture Implementation Design pros & …...

25
References
• [1] Hindman, B., Konwinski, A., Matei, Z., Ghodsi, A., D.Joseph, A., Katz, R., …
Stoica, I. (2011). Mesos:...

25
Fine-grained outline!
• Introduction
• What is Mesos?
• Motivation to sharing a cluster
• Problems and solutions
• Arch...

25
Cluster computing main applications
• High Performance Computing (HPC)
Large internet services
• E.g. social networks,...

25
Terminology
• Cluster
• A collection of distributed machines that collaborate to present a
single integrated computing ...

25
Terminology
• Job
• A unit of work run on the cluster nodes.
• Slot
• Nodes are subdivided into “slots”.
• (In some fra...

25
Terminology
• Elastic framework
• Can scale its resources up and down during execution tasks
(dynamically).
• Such as H...

25
Goals to Design Mesos
• High utilization
• Efficient data sharing
• Support diverse frameworks (current and future)
• S...

25
Background
• Diverse array of cluster computing frameworks:
• Hadoop/ Map-Reduce (multi-terabyte data-sets)
• Dryad (ma...

25
Problem
• Rapid innovation in cluster computing frameworks.
There is no single framework that optimal for all
applicat...

25
Problem
• Hadoop data warehouse at Facebook:
• Include 2000-node Hadoop cluster,
• Uses a fair scheduler for Hadoop,
• ...

25
Better solution: Mesos!
Hadoop
Pregel
MPI
Shared cluster
Today: static partitioning Mesos: dynamic sharing
Backup slide...

25
Resource Offer: Intuition
Backup slides
October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +11

25
Isolation
• Isolate resources using OS container technologies
• Linux Containers (LXC)
• Solaris Projects
• Limit the C...

25
Fault Tolerance
• Master’s only state is the list of active slaves, active
frameworks, and running tasks.
• Run multipl...

25
More on abstraction routing::filter::icmp::Classifier
routing::filter::ip::Classifier
Smell: duplicate abstraction
Back...

25
The End
• See my previous presentation about Apache Mesos:
• Winter 2017
• Cluster, grid and cloud (CGC) computing cour...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1011447 views
Storytelling For The Web: Integrate... by Chiara Aliotta 919244 views
Artificial Intelligence, Data and C... by OECD Directorate ... 820562 views
How to Leverage AI to Boost Employe... by SocialHRCamp 342424 views
2024 State of Marketing Report – by... by Marius Sescu 195380 views
Everything You Need To Know About C... by Expeed Software 218594 views

View on Slideshare

1 of 40 Ad

View on Slideshare

1 of 40 Ad