SlideShare a Scribd company logo
Centralized Scheduling made Simple
Scott Weiss
Once upon a time, there was a
meeting about task scheduling…
Mesos
• Popular, mature cluster manager
• Widely used at hyperscale
Mesos Architecture
Mesos Architecture
Applications do their
own scheduling!
Mesos Architecture
Your
application
Frameworks
Mesos Master
(not a scheduler)
Mesos Slaves
How It Works
“Hey Jenkins, Do you
want to run something
on this node?”
“Sure, that node looks
good for this task”
“OK, let me run
that for you.”
Distributed Scheduling Model
• Pros:
– Applications have more control
• Cons:
– Schedulers have only a localized view of the
cluster
Distributed Scheduling Model
• Pros:
– Applications have more control
• Cons:
– Schedulers have only a localized view of the
cluster
• What if scheduling decisions need to be
made with a global view of the cluster?
Fenzo
Problems
“Hey Jenkins, Do you
want to run something
on this node?”
“I don’t like this node”
“It’s up to you to
wait for a better
one”
Schedulers only see the nodes Mesos shows them
Problems
“Hey Jenkins, Do you
want to run something
on this node?”
“I want on the same node
as Marathon”
Schedulers only know where their own tasks are running
Problems
“Hey Jenkins, Do you
want to run something
on this node?”
“My tasks are more
important than Jenkins’, I
need that node more”
Schedulers cannot overrule each other
“Then you should have
reserved it ahead of
time”
Problems
“Hey Jenkins, Do you
want to run something
on these nodes?”
Continuous placement is not possible
“Only if it belongs to
you.”
“Those nodes are too
small for me. Can I cancel
one of the tasks and move
it to another node?”
Adding Centralized Scheduling to Mesos
Your
application
Adding Centralized Scheduling to Mesos
Your
application
Layer of Abstraction
Layer-X Was born
Your
application
How it works
• Layer-X represents itself to Mesos Applications (called
“Frameworks”) as an actual Mesos cluster
• Mesos Frameworks register to Layer-X the same way they
register to Mesos itself
marathon/bin/start --master=LAYERX_URL
How it works
• Layer-X registers to Mesos like a normal
framework
config := scheduler.DriverConfig{
Scheduler: layerxScheduler,
Framework: layerxFramework,
Master: *master,
Credential: (*mesosproto.Credential)(nil),
}
driver, _ := scheduler.NewMesosSchedulerDriver(config)
status, _ := driver.Run()
How it works
• Layer-X collects tasks from frameworks with
fictitious resource offers
• Layer-X then collects resources from Mesos
and sits on them
How it works
• By collecting all tasks and all resources,
Layer-X has a global view of the cluster
• Layer-X can then be controlled through a REST
API by an external application to make better
scheduling decisions
How it works
• External Schedulers read the state of the
cluster through Layer-X API
• Using this information, External Schedulers
make scheduling decisions with Layer-X API
What’s possible with Layer-X?
Your
application
What’s possible with Layer-X?
Your
application
Plug in any scheduler
Fenzo
What’s possible with Layer-X?
Your
application
Plug in any scheduler
Fenzo
Intercept, Resize Tasks
What’s possible with Layer-X?
Your
application
Plug in any scheduler
Fenzo
Intercept, Resize Tasks
Live migration,
continuous placement
What’s possible with Layer-X?
Your
application
Plug in any scheduler
Fenzo
Intercept, Resize Tasks
Live migration,
continuous placementMulti-cluser management
us-east-1us-west-1
Demo!
What else is possible with Layer-X?
Your
application
What else is possible with Layer-X?
Your
application
We all speak
Docker!
What else is possible with Layer-X?
Your
application
Next Steps for Layer-X
• Open Source
• Plugins for more cluster managers
– Kubernetes
– Docker Swarm
Follow us on Twitter & Github
• For news about the release of Layer-X and other
interesting OSS projects at EMC, follow
• Our Github page:
https://github.com/emc-advanced-dev
• Twitter
@ilackarms
@Idit_Levine
• And please check out my Unikernels workshop,
tomorrow at 1:00 PM
• Questions?

More Related Content

Viewers also liked

Birkman Review - Joseph_Hyslop-sm
Birkman Review - Joseph_Hyslop-smBirkman Review - Joseph_Hyslop-sm
Birkman Review - Joseph_Hyslop-sm
Joseph Hyslop
 
Financing for Development
Financing for DevelopmentFinancing for Development
Financing for Development
snowedunder
 
Two page spread pdf
Two page spread pdfTwo page spread pdf
Two page spread pdf
stoliros
 
The last days part 4
The last days part 4The last days part 4
The last days part 4
EdJacquin
 

Viewers also liked (20)

strengths-guide
strengths-guidestrengths-guide
strengths-guide
 
Birkman Review - Joseph_Hyslop-sm
Birkman Review - Joseph_Hyslop-smBirkman Review - Joseph_Hyslop-sm
Birkman Review - Joseph_Hyslop-sm
 
Financing for Development
Financing for DevelopmentFinancing for Development
Financing for Development
 
UniK: Deploy Unikernels with Ease
UniK: Deploy Unikernels with EaseUniK: Deploy Unikernels with Ease
UniK: Deploy Unikernels with Ease
 
Khanna_Embedded
Khanna_EmbeddedKhanna_Embedded
Khanna_Embedded
 
Proyecto n. 6
Proyecto n. 6Proyecto n. 6
Proyecto n. 6
 
Startup Secrets - Getting Behind the Perfect Investor Pitch
Startup Secrets - Getting Behind the Perfect Investor PitchStartup Secrets - Getting Behind the Perfect Investor Pitch
Startup Secrets - Getting Behind the Perfect Investor Pitch
 
Tesla
TeslaTesla
Tesla
 
Up, up & away
Up, up & awayUp, up & away
Up, up & away
 
Refugeelaw
RefugeelawRefugeelaw
Refugeelaw
 
Family tree
Family tree Family tree
Family tree
 
12 dom ordinario, ciclo 'c' (5)
12  dom  ordinario,  ciclo 'c' (5)12  dom  ordinario,  ciclo 'c' (5)
12 dom ordinario, ciclo 'c' (5)
 
Acland Burghley Healthy Eating Champions: Healthy London Hackerthon
Acland Burghley Healthy Eating Champions:  Healthy London HackerthonAcland Burghley Healthy Eating Champions:  Healthy London Hackerthon
Acland Burghley Healthy Eating Champions: Healthy London Hackerthon
 
Wizards in winter
Wizards in winterWizards in winter
Wizards in winter
 
Two page spread pdf
Two page spread pdfTwo page spread pdf
Two page spread pdf
 
The last days part 4
The last days part 4The last days part 4
The last days part 4
 
Treasured space, new place
Treasured space, new placeTreasured space, new place
Treasured space, new place
 
Kuliner Makassar
Kuliner MakassarKuliner Makassar
Kuliner Makassar
 
Q.1
Q.1Q.1
Q.1
 
Thriller powerpoint
Thriller powerpointThriller powerpoint
Thriller powerpoint
 

Similar to Layer-X ContainerDays Slides May 24 2016

OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating System
OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating SystemOSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating System
OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating System
NETWAYS
 
Containerization - The DevOps Revolution
Containerization - The DevOps RevolutionContainerization - The DevOps Revolution
Containerization - The DevOps Revolution
Yulian Slobodyan
 
Thinking in Terms of a Data Center Operating System
Thinking in Terms of a Data Center Operating SystemThinking in Terms of a Data Center Operating System
Thinking in Terms of a Data Center Operating System
Weston Bassler
 

Similar to Layer-X ContainerDays Slides May 24 2016 (20)

Mesos: The Operating System for your Datacenter
Mesos: The Operating System for your DatacenterMesos: The Operating System for your Datacenter
Mesos: The Operating System for your Datacenter
 
Mesos study report 03v1.2
Mesos study report  03v1.2Mesos study report  03v1.2
Mesos study report 03v1.2
 
Mesos: Cluster Management System
Mesos: Cluster Management SystemMesos: Cluster Management System
Mesos: Cluster Management System
 
Mesos Introduction
Mesos IntroductionMesos Introduction
Mesos Introduction
 
Mesos introduction
Mesos introductionMesos introduction
Mesos introduction
 
Apache Mesos
Apache MesosApache Mesos
Apache Mesos
 
Cloud Infrastructures Slide Set 8 - More Cloud Technologies - Mesos, Spark | ...
Cloud Infrastructures Slide Set 8 - More Cloud Technologies - Mesos, Spark | ...Cloud Infrastructures Slide Set 8 - More Cloud Technologies - Mesos, Spark | ...
Cloud Infrastructures Slide Set 8 - More Cloud Technologies - Mesos, Spark | ...
 
Mesos sys adminday
Mesos sys admindayMesos sys adminday
Mesos sys adminday
 
Apache mesos - overview
Apache mesos - overviewApache mesos - overview
Apache mesos - overview
 
Cloud infrastructure on Apache Mesos
Cloud infrastructure on Apache MesosCloud infrastructure on Apache Mesos
Cloud infrastructure on Apache Mesos
 
OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating System
OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating SystemOSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating System
OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating System
 
DEPLOYING A DOCKERIZED DISTRIBUTED APPLICATION IN MESOS
DEPLOYING A DOCKERIZED DISTRIBUTED APPLICATION IN MESOSDEPLOYING A DOCKERIZED DISTRIBUTED APPLICATION IN MESOS
DEPLOYING A DOCKERIZED DISTRIBUTED APPLICATION IN MESOS
 
Containerization - The DevOps Revolution
Containerization - The DevOps RevolutionContainerization - The DevOps Revolution
Containerization - The DevOps Revolution
 
Introduction to Apache Mesos
Introduction to Apache MesosIntroduction to Apache Mesos
Introduction to Apache Mesos
 
Modern distributed-systems
Modern distributed-systemsModern distributed-systems
Modern distributed-systems
 
Architecting for Failure in a Containerized World
Architecting for Failure in a Containerized WorldArchitecting for Failure in a Containerized World
Architecting for Failure in a Containerized World
 
Thinking in Terms of a Data Center Operating System
Thinking in Terms of a Data Center Operating SystemThinking in Terms of a Data Center Operating System
Thinking in Terms of a Data Center Operating System
 
ContainerCon EU 2016 - Software-Defined Storage and Container Schedulers
ContainerCon EU 2016 - Software-Defined Storage and Container SchedulersContainerCon EU 2016 - Software-Defined Storage and Container Schedulers
ContainerCon EU 2016 - Software-Defined Storage and Container Schedulers
 
DSD-INT 2020 Scripting a Delft-FEWS configuration - Verkade
DSD-INT 2020 Scripting a Delft-FEWS configuration - VerkadeDSD-INT 2020 Scripting a Delft-FEWS configuration - Verkade
DSD-INT 2020 Scripting a Delft-FEWS configuration - Verkade
 
TechBeats #2
TechBeats #2TechBeats #2
TechBeats #2
 

Recently uploaded

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
Alluxio, Inc.
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
Max Lee
 

Recently uploaded (20)

The Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionThe Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion Production
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
 
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
 
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesGraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Kraków
 
A Guideline to Gorgias to to Re:amaze Data Migration
A Guideline to Gorgias to to Re:amaze Data MigrationA Guideline to Gorgias to to Re:amaze Data Migration
A Guideline to Gorgias to to Re:amaze Data Migration
 
Designing for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web ServicesDesigning for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web Services
 
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfA Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
 
CompTIA Security+ (Study Notes) for cs.pdf
CompTIA Security+ (Study Notes) for cs.pdfCompTIA Security+ (Study Notes) for cs.pdf
CompTIA Security+ (Study Notes) for cs.pdf
 
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
 
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
 
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
 
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning Framework
 
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
 

Layer-X ContainerDays Slides May 24 2016

  • 1. Centralized Scheduling made Simple Scott Weiss
  • 2. Once upon a time, there was a meeting about task scheduling…
  • 3. Mesos • Popular, mature cluster manager • Widely used at hyperscale
  • 5. Mesos Architecture Applications do their own scheduling!
  • 7. How It Works “Hey Jenkins, Do you want to run something on this node?” “Sure, that node looks good for this task” “OK, let me run that for you.”
  • 8. Distributed Scheduling Model • Pros: – Applications have more control • Cons: – Schedulers have only a localized view of the cluster
  • 9. Distributed Scheduling Model • Pros: – Applications have more control • Cons: – Schedulers have only a localized view of the cluster • What if scheduling decisions need to be made with a global view of the cluster? Fenzo
  • 10. Problems “Hey Jenkins, Do you want to run something on this node?” “I don’t like this node” “It’s up to you to wait for a better one” Schedulers only see the nodes Mesos shows them
  • 11. Problems “Hey Jenkins, Do you want to run something on this node?” “I want on the same node as Marathon” Schedulers only know where their own tasks are running
  • 12. Problems “Hey Jenkins, Do you want to run something on this node?” “My tasks are more important than Jenkins’, I need that node more” Schedulers cannot overrule each other “Then you should have reserved it ahead of time”
  • 13. Problems “Hey Jenkins, Do you want to run something on these nodes?” Continuous placement is not possible “Only if it belongs to you.” “Those nodes are too small for me. Can I cancel one of the tasks and move it to another node?”
  • 14. Adding Centralized Scheduling to Mesos Your application
  • 15. Adding Centralized Scheduling to Mesos Your application Layer of Abstraction
  • 17. How it works • Layer-X represents itself to Mesos Applications (called “Frameworks”) as an actual Mesos cluster • Mesos Frameworks register to Layer-X the same way they register to Mesos itself marathon/bin/start --master=LAYERX_URL
  • 18. How it works • Layer-X registers to Mesos like a normal framework config := scheduler.DriverConfig{ Scheduler: layerxScheduler, Framework: layerxFramework, Master: *master, Credential: (*mesosproto.Credential)(nil), } driver, _ := scheduler.NewMesosSchedulerDriver(config) status, _ := driver.Run()
  • 19. How it works • Layer-X collects tasks from frameworks with fictitious resource offers • Layer-X then collects resources from Mesos and sits on them
  • 20. How it works • By collecting all tasks and all resources, Layer-X has a global view of the cluster • Layer-X can then be controlled through a REST API by an external application to make better scheduling decisions
  • 21. How it works • External Schedulers read the state of the cluster through Layer-X API • Using this information, External Schedulers make scheduling decisions with Layer-X API
  • 22. What’s possible with Layer-X? Your application
  • 23. What’s possible with Layer-X? Your application Plug in any scheduler Fenzo
  • 24. What’s possible with Layer-X? Your application Plug in any scheduler Fenzo Intercept, Resize Tasks
  • 25. What’s possible with Layer-X? Your application Plug in any scheduler Fenzo Intercept, Resize Tasks Live migration, continuous placement
  • 26. What’s possible with Layer-X? Your application Plug in any scheduler Fenzo Intercept, Resize Tasks Live migration, continuous placementMulti-cluser management us-east-1us-west-1
  • 27. Demo!
  • 28. What else is possible with Layer-X? Your application
  • 29. What else is possible with Layer-X? Your application We all speak Docker!
  • 30. What else is possible with Layer-X? Your application
  • 31. Next Steps for Layer-X • Open Source • Plugins for more cluster managers – Kubernetes – Docker Swarm
  • 32. Follow us on Twitter & Github • For news about the release of Layer-X and other interesting OSS projects at EMC, follow • Our Github page: https://github.com/emc-advanced-dev • Twitter @ilackarms @Idit_Levine • And please check out my Unikernels workshop, tomorrow at 1:00 PM • Questions?