Scheduling and sharing resources in Data Clusters
•
3 likes•1,624 views
This presentation is part of my work for the course 'Big Data Seminar' at TU Berlin within the IT4BI (Information Technology for Business Intelligence) master programme.
Recommended
More Related Content
Similar to Scheduling and sharing resources in Data Clusters
Similar to Scheduling and sharing resources in Data Clusters (20)
More from Jose Luis Lopez Pino
More from Jose Luis Lopez Pino (20)
Recently uploaded
Recently uploaded (20)
Scheduling and sharing resources in Data Clusters
- 1. Introduction YARN Mesos Omega Related work Conclusions Scheduling and sharing resources in Data Clusters Jose Luis Lopez Pino December 12, 2013 Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 2. Introduction YARN Mesos Omega Related work Conclusions Table of contents 1 2 3 Introduction The problem Solutions YARN Architecture Advantages Drawbacks Performance Mesos Architecture Advantages 4 5 6 Jose Luis Lopez Pino Drawbacks Performance Omega Architecture Advantages Drawbacks Performance Related work Resource managers Scheduling techniques Conclusions Scheduling and sharing resources in Data Clusters
- 3. Introduction YARN Mesos Omega Related work Conclusions The problem Solutions The problem Data lake Multiple frameworks[6] Duplicate de data Cluster utilization Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 4. Introduction YARN Mesos Omega Related work Conclusions The problem Solutions Solutions[8] 1 Static partitioning 2 Monolithic schedulers 3 Two-level scheduler 4 Shared state approach Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 5. Introduction YARN Mesos Omega Related work Conclusions Architecture Advantages Drawbacks Performance Architecture[10] Figure: YARN architecture Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 6. Introduction YARN Mesos Omega Related work Conclusions Architecture Advantages Drawbacks Performance Advantages Scale Data locality Easy to port a new framework Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 7. Introduction YARN Mesos Omega Related work Conclusions Architecture Advantages Drawbacks Performance Drawbacks Failure recovery High latency? Network overload? Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 8. Introduction YARN Mesos Omega Related work Conclusions Architecture Advantages Drawbacks Performance Performance Job throughput Application Master flooding Preemption Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 9. Introduction YARN Mesos Omega Related work Conclusions Architecture Advantages Drawbacks Performance Architecture[9] Figure: Mesos architecture Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 10. Introduction YARN Mesos Omega Related work Conclusions Architecture Advantages Drawbacks Performance Advantages Flexible Extensible Fault tolerance Backup master node Recreate master using communication Use checkpoints for the slaves Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 11. Introduction YARN Mesos Omega Related work Conclusions Architecture Advantages Drawbacks Performance Drawbacks Complex to port a framework Intensive communication Revocation might be dangerous Penalizes long jobs Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 12. Introduction YARN Mesos Omega Related work Conclusions Architecture Advantages Drawbacks Performance Performance Missing: comparison of different policies and modules Scalable + 18% memory + 10% CPU utilization less than 1s launching tasks with 50k nodes Small tasks Data locality with delay scheduling[12] MPITorque and gang scheduling Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 13. Introduction YARN Mesos Omega Related work Conclusions Architecture Advantages Drawbacks Performance Architecture[8] Figure: Omega architecture Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 14. Introduction YARN Mesos Omega Related work Conclusions Architecture Advantages Drawbacks Performance Advantages Schedulers work in parallel Very scalable Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 15. Introduction YARN Mesos Omega Related work Conclusions Architecture Advantages Drawbacks Performance Drawbacks Unfair distribution Conflicts Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 16. Introduction YARN Mesos Omega Related work Conclusions Architecture Advantages Drawbacks Performance Performance Decision time and busyness of the scheduler Real workloads Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 17. Introduction YARN Mesos Omega Related work Conclusions Resource managers Scheduling techniques Resource managers Heterogeneous environments: Corona and Cosmos [1] Homogeneous environments: Quincy[4] Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 18. Introduction YARN Mesos Omega Related work Conclusions Resource managers Scheduling techniques Scheduling techniques Lottery scheduling[11] Dynamic Proportional Share Scheduling[7] Calibration: how does a particular task perform in a particular node?[5] Stragglers and speculative relaunch[13] Delay scheduling: achieve locality, relax fairness[12] Rich resource-requests[2] Optimize short jobs[3] Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 19. Introduction YARN Mesos Omega Related work Conclusions Conclusions Different models YARN: Easier to port a new framework Data locality Mesos Flexible and modular Fault tolerance More scalable Omega: Flexible Highly scalable Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 20. Introduction YARN Mesos Omega Related work Conclusions References I [1] Ronnie Chaiken, Bob Jenkins, Per-˚ke Larson, Bill Ramsey, A Darren Shakib, Simon Weaver, and Jingren Zhou. Scope: easy and efficient parallel processing of massive data sets. Proceedings of the VLDB Endowment, 1(2):1265–1276, 2008. [2] Carlo Curino, Djellel Difallah, Chris Douglas, Raghu Ramakrishnan, and Sriram Rao. Reservation-based scheduling: If youre late dont blame us! [3] Khaled Elmeleegy. Piranha: Optimizing short jobs in hadoop. Proceedings of the VLDB Endowment, 6(11):985–996, 2013. Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 21. Introduction YARN Mesos Omega Related work Conclusions References II [4] Michael Isard, Vijayan Prabhakaran, Jon Currey, Udi Wieder, Kunal Talwar, and Andrew Goldberg. Quincy: fair scheduling for distributed computing clusters. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles, pages 261–276. ACM, 2009. [5] Gunho Lee, Byung-Gon Chun, and Randy H Katz. Heterogeneity-aware resource allocation and scheduling in the cloud. In Proceedings of the 3rd USENIX Workshop on Hot Topics in Cloud Computing, HotCloud, volume 11, 2011. Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 22. Introduction YARN Mesos Omega Related work Conclusions References III [6] Kyong-Ha Lee, Yoon-Joon Lee, Hyunsik Choi, Yon Dohn Chung, and Bongki Moon. Parallel data processing with mapreduce: a survey. ACM SIGMOD Record, 40(4):11–20, 2012. [7] Thomas Sandholm and Kevin Lai. Dynamic proportional share scheduling in hadoop. In Job scheduling strategies for parallel processing, pages 110–131. Springer, 2010. Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 23. Introduction YARN Mesos Omega Related work Conclusions References IV [8] Malte Schwarzkopf, Andy Konwinski, Michael Abd-El-Malek, and John Wilkes. Omega: Flexible, scalable schedulers for large compute clusters. In Proceedings of the 8th ACM European Conference on Computer Systems, EuroSys ’13, pages 351–364, New York, NY, USA, 2013. ACM. [9] Facebook Engineering Team. Under the hood: Scheduling mapreduce jobs more efficiently with corona. Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 24. Introduction YARN Mesos Omega Related work Conclusions References V [10] Vinod K. Vavilapalli. Apache Hadoop YARN: Yet Another Resource Negotiator. In Proc. SOCC, 2013. [11] Carl A Waldspurger and William E Weihl. Lottery scheduling: Flexible proportional-share resource management. In Proceedings of the 1st USENIX conference on Operating Systems Design and Implementation, page 1. USENIX Association, 1994. Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters
- 25. Introduction YARN Mesos Omega Related work Conclusions References VI [12] Matei Zaharia, Dhruba Borthakur, Joydeep Sen Sarma, Khaled Elmeleegy, Scott Shenker, and Ion Stoica. Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In Proceedings of the 5th European conference on Computer systems, pages 265–278. ACM, 2010. [13] Matei Zaharia, Andy Konwinski, Anthony D Joseph, Randy H Katz, and Ion Stoica. Improving mapreduce performance in heterogeneous environments. In OSDI, volume 8, page 7, 2008. Jose Luis Lopez Pino Scheduling and sharing resources in Data Clusters