Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Hita Barasm Mamagani
Spring 2019 – Cluster and Grid Computing Course (By Prof. M. Sharifi)
Iran University of Science and ...
Pig
BACKGROUND
Rapid innovation in cluster computing
frameworks
Dryad
Pregel
Percolator
CIEL
PROBLEM
No single framework optimal for all
applications
Want to run multiple frameworks in a single
cluster
…to maximize ...
DEDICATED SERVERS
• Low utilization rates
• Longer time to ramp up new services
VIRTUALIZATION
• Even more machines to manage
• Substantial performance decrease due to virtualization
• VM licensing costs
VIRTUALIZATION
• Even more machines to manage
• Substantial performance decrease due to virtualization
• VM licensing cost...
VIRTUALIZATION
“We wanted people to be able to program for the datacenter
just like they program for their laptop.”
Ben Hi...
OUR GOAL
Hadoop
Pregel
MPI
Shared cluster
Today: static partitioning Mesos: dynamic sharing
%100
%50
%0
%0
%0
%0
%17
%17
%...
SOLUTION IS MESOS
Mesos is a common resource sharing layer over
which diverse frameworks can run
Mesos
Node
Hadoop
Node No...
MESOS ARCHITECTURE
MESOS ARCHITECTURE
Given the use of Mesos as a Datacenter OS kernel…
• Chronos provides complex scheduling capabilities, m...
HOW IS THE SHARING
Framework 1
Framework 2
Framework 3
Coarse-Grained Sharing (HPC): Fine-Grained Sharing (Mesos):
Storage...
TWO-LEVEL SCHEDULER
MPI job
MPI
scheduler
Hadoop job
Hadoop
scheduler
Allocation
module
Mesos
master
Mesos slave
MPI
execu...
TWO-LEVEL SCHEDULER
MPI job
MPI
scheduler
Hadoop job
Hadoop
scheduler
Allocation
module
Mesos
master
Mesos slave
MPI
execu...
DOCKER ON MESOS
Marathon can launch
and monitor service
containers from one or
more Docker registries,
using the Docker
ex...
DOCKER ON MESOS
MESOS HIGH-AVAILABILITY MODE
• Uses multiple Mesos masters:
- one active master (called the leader)
- several backups in c...
FAULT TOLERANCE
• Mesos master has only soft state:
List of currently running frameworks and tasks
• Rebuild when framewor...
SCALABILITY
Mesos only performs inter-framework scheduling (e.g. fair
sharing), which is easier than intra-framework sched...
WHAT IS THE SHARING
WHAT IS THE SHARING
A MESOS USER
“Mesos is the cornerstone of our elastic compute
infrastructure – it’s how we build all our new services and ...
ANOTHER MESOS USER
“We think we might be pushing data science in the field of
travel more so than anyone has ever done bef...
WHO USES MESOS?
• Airbnb
• Apple
• Cisco
• Coursera
• eBay
• Ericsson
• Foursquare
• Netflix
• Opera
• PayPal
• Pinterest
...
RESOURCES
Apache Mesos Project Website - Link
Mesosphere Website - Link
Dharmesh Kakadia, “Apache Mesos Essentials” Pac...
THE END!
Any
more
questions?
Thanks for giving your time 
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

Share

Apache Mesos Introduction

Download to read offline

These slides were presented at the Iran University of Science and Technology. It was for Cluster and Grid Computing course in Spring 2019.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Apache Mesos Introduction

  1. 1. Hita Barasm Mamagani Spring 2019 – Cluster and Grid Computing Course (By Prof. M. Sharifi) Iran University of Science and Technology
  2. 2. Pig BACKGROUND Rapid innovation in cluster computing frameworks Dryad Pregel Percolator CIEL
  3. 3. PROBLEM No single framework optimal for all applications Want to run multiple frameworks in a single cluster …to maximize utilization …to share data between frameworks
  4. 4. DEDICATED SERVERS • Low utilization rates • Longer time to ramp up new services
  5. 5. VIRTUALIZATION • Even more machines to manage • Substantial performance decrease due to virtualization • VM licensing costs
  6. 6. VIRTUALIZATION • Even more machines to manage • Substantial performance decrease due to virtualization • VM licensing costs • Static partitioning limits elasticity
  7. 7. VIRTUALIZATION “We wanted people to be able to program for the datacenter just like they program for their laptop.” Ben Hindman
  8. 8. OUR GOAL Hadoop Pregel MPI Shared cluster Today: static partitioning Mesos: dynamic sharing %100 %50 %0 %0 %0 %0 %17 %17 %17 %33 %33 %33
  9. 9. SOLUTION IS MESOS Mesos is a common resource sharing layer over which diverse frameworks can run Mesos Node Hadoop Node Node Spark Node Node MPI Node Node Node Node Node Node Node Hadoop Spark MPI
  10. 10. MESOS ARCHITECTURE
  11. 11. MESOS ARCHITECTURE Given the use of Mesos as a Datacenter OS kernel… • Chronos provides complex scheduling capabilities, much like a distributed Unix “cron” • Marathon provides highly-available long-running services, much like a distributed Unix “init.d” • next time you need to build a distributed app, consider using these as building blocks
  12. 12. HOW IS THE SHARING Framework 1 Framework 2 Framework 3 Coarse-Grained Sharing (HPC): Fine-Grained Sharing (Mesos): Storage System (e.g. HDFS) Storage System (e.g. HDFS) Fw. 1 Fw. 1Fw. 3 Fw. 3 Fw. 2Fw. 2 Fw. 2 Fw. 1 Fw. 3 Fw. 2Fw. 3 Fw. 1 Fw. 1 Fw. 2Fw. 2 Fw. 1 Fw. 3 Fw. 3 Fw. 3 Fw. 2 Fw. 2
  13. 13. TWO-LEVEL SCHEDULER MPI job MPI scheduler Hadoop job Hadoop scheduler Allocation module Mesos master Mesos slave MPI executor Mesos slave MPI executor tasktask Resource offer Pick framework to offer resources to Resource offer = list of (node, availableResources) E.g. { (node1, <2 CPUs, 4 GB>), (node2, <3 CPUs, 2 GB>) }
  14. 14. TWO-LEVEL SCHEDULER MPI job MPI scheduler Hadoop job Hadoop scheduler Allocation module Mesos master Mesos slave MPI executor Mesos slave MPI executor tasktask Resource offer Pick framework to offer resources to Framework-specific scheduling Launches and isolates executorsHadoop executor task
  15. 15. DOCKER ON MESOS Marathon can launch and monitor service containers from one or more Docker registries, using the Docker executor for Mesos
  16. 16. DOCKER ON MESOS
  17. 17. MESOS HIGH-AVAILABILITY MODE • Uses multiple Mesos masters: - one active master (called the leader) - several backups in case leader fails • The masters elect the leader, with Apache ZooKeeper both coordinating the election and handling leader detection by masters, nodes, and scheduler drivers.
  18. 18. FAULT TOLERANCE • Mesos master has only soft state: List of currently running frameworks and tasks • Rebuild when frameworks and slaves re-register with new master after a failure • Fault detection and recovery in ~10 sec
  19. 19. SCALABILITY Mesos only performs inter-framework scheduling (e.g. fair sharing), which is easier than intra-framework scheduling 0 0.2 0.4 0.6 0.8 1 -10000 10000 30000 50000 TaskStartOverhead(s) Number of Slaves Result: Scaled to 50,000 emulated slaves, 200 frameworks, 100K tasks (30s Len.)
  20. 20. WHAT IS THE SHARING
  21. 21. WHAT IS THE SHARING
  22. 22. A MESOS USER “Mesos is the cornerstone of our elastic compute infrastructure – it’s how we build all our new services and is critical for Twitter’s continued success at scale. It's one of the primary keys to our data center efficiency.” Chris Fry, SVP Engineering • Instead of thinking about static machines, engineers think about resources like CPU, memory and disk • Allows services to scale and leverage a shared pool of servers across datacenters efficiently • Reduces the time between prototyping and launching
  23. 23. ANOTHER MESOS USER “We think we might be pushing data science in the field of travel more so than anyone has ever done before… a smaller number of engineers can have higher impact through automation on Mesos.” Mike Curtis, VP Engineering • Helps advance engineering strategy of building small teams that can move fast • Enables use of Hadoop along with Chronos, Spark, Storm, etc.
  24. 24. WHO USES MESOS? • Airbnb • Apple • Cisco • Coursera • eBay • Ericsson • Foursquare • Netflix • Opera • PayPal • Pinterest • Twitter • Uber • Udacity • Verizon Labs • Xiaomi • etc. Link to source (Apache Mesos Website)
  25. 25. RESOURCES Apache Mesos Project Website - Link Mesosphere Website - Link Dharmesh Kakadia, “Apache Mesos Essentials” Packt Publishing Ltd, 2015. (Book) - Link Paco Nathan, “CHUG: Datacenter Computing with Apache Mesos”, 2013. (Conference) – Link Ben Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony Joseph, Randy Katz, Scott Shenker, Ion Stoica, “Mesos, A Platform for Fine-Grained Resource Sharing in the Data Center”, Berkeley University, 2011. (Conference) - Link
  26. 26. THE END! Any more questions? Thanks for giving your time 

These slides were presented at the Iran University of Science and Technology. It was for Cluster and Grid Computing course in Spring 2019.

Views

Total views

23

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

0

Shares

0

Comments

0

Likes

0

×