Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Layer-X ContainerDays Slides May 24 2016

388 views

Published on

Scott Weiss presentation about Layer-X, centralized scheduling for Mesos Frameworks

Published in: Software
  • Be the first to comment

Layer-X ContainerDays Slides May 24 2016

  1. 1. Centralized Scheduling made Simple Scott Weiss
  2. 2. Once upon a time, there was a meeting about task scheduling…
  3. 3. Mesos • Popular, mature cluster manager • Widely used at hyperscale
  4. 4. Mesos Architecture
  5. 5. Mesos Architecture Applications do their own scheduling!
  6. 6. Mesos Architecture Your application Frameworks Mesos Master (not a scheduler) Mesos Slaves
  7. 7. How It Works “Hey Jenkins, Do you want to run something on this node?” “Sure, that node looks good for this task” “OK, let me run that for you.”
  8. 8. Distributed Scheduling Model • Pros: – Applications have more control • Cons: – Schedulers have only a localized view of the cluster
  9. 9. Distributed Scheduling Model • Pros: – Applications have more control • Cons: – Schedulers have only a localized view of the cluster • What if scheduling decisions need to be made with a global view of the cluster? Fenzo
  10. 10. Problems “Hey Jenkins, Do you want to run something on this node?” “I don’t like this node” “It’s up to you to wait for a better one” Schedulers only see the nodes Mesos shows them
  11. 11. Problems “Hey Jenkins, Do you want to run something on this node?” “I want on the same node as Marathon” Schedulers only know where their own tasks are running
  12. 12. Problems “Hey Jenkins, Do you want to run something on this node?” “My tasks are more important than Jenkins’, I need that node more” Schedulers cannot overrule each other “Then you should have reserved it ahead of time”
  13. 13. Problems “Hey Jenkins, Do you want to run something on these nodes?” Continuous placement is not possible “Only if it belongs to you.” “Those nodes are too small for me. Can I cancel one of the tasks and move it to another node?”
  14. 14. Adding Centralized Scheduling to Mesos Your application
  15. 15. Adding Centralized Scheduling to Mesos Your application Layer of Abstraction
  16. 16. Layer-X Was born Your application
  17. 17. How it works • Layer-X represents itself to Mesos Applications (called “Frameworks”) as an actual Mesos cluster • Mesos Frameworks register to Layer-X the same way they register to Mesos itself marathon/bin/start --master=LAYERX_URL
  18. 18. How it works • Layer-X registers to Mesos like a normal framework config := scheduler.DriverConfig{ Scheduler: layerxScheduler, Framework: layerxFramework, Master: *master, Credential: (*mesosproto.Credential)(nil), } driver, _ := scheduler.NewMesosSchedulerDriver(config) status, _ := driver.Run()
  19. 19. How it works • Layer-X collects tasks from frameworks with fictitious resource offers • Layer-X then collects resources from Mesos and sits on them
  20. 20. How it works • By collecting all tasks and all resources, Layer-X has a global view of the cluster • Layer-X can then be controlled through a REST API by an external application to make better scheduling decisions
  21. 21. How it works • External Schedulers read the state of the cluster through Layer-X API • Using this information, External Schedulers make scheduling decisions with Layer-X API
  22. 22. What’s possible with Layer-X? Your application
  23. 23. What’s possible with Layer-X? Your application Plug in any scheduler Fenzo
  24. 24. What’s possible with Layer-X? Your application Plug in any scheduler Fenzo Intercept, Resize Tasks
  25. 25. What’s possible with Layer-X? Your application Plug in any scheduler Fenzo Intercept, Resize Tasks Live migration, continuous placement
  26. 26. What’s possible with Layer-X? Your application Plug in any scheduler Fenzo Intercept, Resize Tasks Live migration, continuous placementMulti-cluser management us-east-1us-west-1
  27. 27. Demo!
  28. 28. What else is possible with Layer-X? Your application
  29. 29. What else is possible with Layer-X? Your application We all speak Docker!
  30. 30. What else is possible with Layer-X? Your application
  31. 31. Next Steps for Layer-X • Open Source • Plugins for more cluster managers – Kubernetes – Docker Swarm
  32. 32. Follow us on Twitter & Github • For news about the release of Layer-X and other interesting OSS projects at EMC, follow • Our Github page: https://github.com/emc-advanced-dev • Twitter @ilackarms @Idit_Levine • And please check out my Unikernels workshop, tomorrow at 1:00 PM • Questions?

×