Uploaded on

Mesos, Fine-grained resource sharing

Mesos, Fine-grained resource sharing

More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
1,885
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
93
Comments
1
Likes
6

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Motivation Mesos Implementation Evaluation ConclusionMesos: A Platform for Fine-Grained Resource Sharing in the Data Center Muhammad Anis uddin Nasir KTH Royal Institute of Technology November 20, 2012 Muhammad Anis uddin Nasir Mesos 1/32
  • 2. Motivation Mesos Implementation Evaluation Conclusion1 Motivation2 Mesos Overview Architecture3 Implementation4 Evaluation5 Conclusion Muhammad Anis uddin Nasir Mesos 2/32
  • 3. Motivation Mesos Implementation Evaluation ConclusionMotivation Diverse cluster computer framework Muhammad Anis uddin Nasir Mesos 3/32
  • 4. Motivation Mesos Implementation Evaluation ConclusionMotivation No optimal framework Muhammad Anis uddin Nasir Mesos 4/32
  • 5. Motivation Mesos Implementation Evaluation ConclusionMotivation No optimal framework Run multiple frameworks Muhammad Anis uddin Nasir Mesos 4/32
  • 6. Motivation Mesos Implementation Evaluation ConclusionMotivation No optimal framework Run multiple frameworks Higher Utilization Muhammad Anis uddin Nasir Mesos 4/32
  • 7. Motivation Mesos Implementation Evaluation ConclusionMotivation No optimal framework Run multiple frameworks Higher Utilization Data sharing between clusters Muhammad Anis uddin Nasir Mesos 4/32
  • 8. Motivation Mesos Implementation Evaluation ConclusionMotivation No optimal framework Run multiple frameworks Higher Utilization Data sharing between clusters Reduce Cost Muhammad Anis uddin Nasir Mesos 4/32
  • 9. Motivation Mesos Implementation Evaluation ConclusionExisting Solutions Static Partitioning Muhammad Anis uddin Nasir Mesos 5/32
  • 10. Motivation Mesos Implementation Evaluation ConclusionExisting Solutions Static Partitioning Virtual Machines Muhammad Anis uddin Nasir Mesos 5/32
  • 11. Motivation Mesos Implementation Evaluation ConclusionFine-grained Hadoop and Dryad Muhammad Anis uddin Nasir Mesos 6/32
  • 12. Motivation Mesos Implementation Evaluation ConclusionFine-grained Hadoop and Dryad Slots Muhammad Anis uddin Nasir Mesos 6/32
  • 13. Motivation Mesos Implementation Evaluation ConclusionFine-grained Hadoop and Dryad Slots Tasks Muhammad Anis uddin Nasir Mesos 6/32
  • 14. Motivation Mesos Implementation Evaluation ConclusionFine-grained Hadoop and Dryad Slots Tasks Benefits Muhammad Anis uddin Nasir Mesos 6/32
  • 15. Motivation Mesos Implementation Evaluation ConclusionFine-grained Hadoop and Dryad Slots Tasks Benefits Data Locality Muhammad Anis uddin Nasir Mesos 6/32
  • 16. Motivation Mesos Implementation Evaluation ConclusionFine-grained Hadoop and Dryad Slots Tasks Benefits Data Locality Utilization Muhammad Anis uddin Nasir Mesos 6/32
  • 17. Motivation Mesos Overview Implementation Architecture Evaluation Conclusion1 Motivation2 Mesos Overview Architecture3 Implementation4 Evaluation5 Conclusion Muhammad Anis uddin Nasir Mesos 7/32
  • 18. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMesos Common resource sharing layer Fine grained sharing Across diverse cluster computing frameworks Muhammad Anis uddin Nasir Mesos 8/32
  • 19. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionGoals High Utilization Muhammad Anis uddin Nasir Mesos 9/32
  • 20. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionGoals High Utilization Data sharing among frameworks Muhammad Anis uddin Nasir Mesos 9/32
  • 21. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionChallenges Scalable Muhammad Anis uddin Nasir Mesos 10/32
  • 22. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionChallenges Scalable Support diverse frameworks Muhammad Anis uddin Nasir Mesos 10/32
  • 23. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionChallenges Scalable Support diverse frameworks Efficient Muhammad Anis uddin Nasir Mesos 10/32
  • 24. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionChallenges Scalable Support diverse frameworks Efficient Fault tolerant Muhammad Anis uddin Nasir Mesos 10/32
  • 25. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionChallenges Scalable Support diverse frameworks Efficient Fault tolerant Highly available Muhammad Anis uddin Nasir Mesos 10/32
  • 26. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionOther Benefits Run Multiple instances of same framework Muhammad Anis uddin Nasir Mesos 11/32
  • 27. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionOther Benefits Run Multiple instances of same framework Isolate production and experimental jobs Muhammad Anis uddin Nasir Mesos 11/32
  • 28. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionOther Benefits Run Multiple instances of same framework Isolate production and experimental jobs Run multiple versions of a framework Muhammad Anis uddin Nasir Mesos 11/32
  • 29. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionOther Benefits Run Multiple instances of same framework Isolate production and experimental jobs Run multiple versions of a framework Build special framework targeting particular problem domain Muhammad Anis uddin Nasir Mesos 11/32
  • 30. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionDesign Elements Fine-grained sharing Muhammad Anis uddin Nasir Mesos 12/32
  • 31. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionDesign Elements Fine-grained sharing Allocation at the level of tasks within a job Muhammad Anis uddin Nasir Mesos 12/32
  • 32. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionDesign Elements Fine-grained sharing Allocation at the level of tasks within a job Improve utilization, latency and data locality Muhammad Anis uddin Nasir Mesos 12/32
  • 33. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionDesign Elements Fine-grained sharing Allocation at the level of tasks within a job Improve utilization, latency and data locality Resource offers Muhammad Anis uddin Nasir Mesos 12/32
  • 34. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionDesign Elements Fine-grained sharing Allocation at the level of tasks within a job Improve utilization, latency and data locality Resource offers Simple and Scalable Muhammad Anis uddin Nasir Mesos 12/32
  • 35. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionDesign Elements Fine-grained sharing Allocation at the level of tasks within a job Improve utilization, latency and data locality Resource offers Simple and Scalable Application-controlled scheduling mechanism Muhammad Anis uddin Nasir Mesos 12/32
  • 36. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionFine-Grained Sharing Muhammad Anis uddin Nasir Mesos 13/32
  • 37. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionResource Offers Offers resources to framework Muhammad Anis uddin Nasir Mesos 14/32
  • 38. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionResource Offers Offers resources to framework Framework choose resources according to needs Muhammad Anis uddin Nasir Mesos 14/32
  • 39. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionResource Offers Offers resources to framework Framework choose resources according to needs Benefits Muhammad Anis uddin Nasir Mesos 14/32
  • 40. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionResource Offers Offers resources to framework Framework choose resources according to needs Benefits gives freedom to framework for implementation Muhammad Anis uddin Nasir Mesos 14/32
  • 41. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionResource Offers Offers resources to framework Framework choose resources according to needs Benefits gives freedom to framework for implementation keep Mesos simple and scalable Muhammad Anis uddin Nasir Mesos 14/32
  • 42. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionResource Offers Offers resources to framework Framework choose resources according to needs Benefits gives freedom to framework for implementation keep Mesos simple and scalable Drawback Muhammad Anis uddin Nasir Mesos 14/32
  • 43. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionResource Offers Offers resources to framework Framework choose resources according to needs Benefits gives freedom to framework for implementation keep Mesos simple and scalable Drawback decentralized decision might not be optimal Muhammad Anis uddin Nasir Mesos 14/32
  • 44. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionArchitecture Muhammad Anis uddin Nasir Mesos 15/32
  • 45. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionArchitecture Muhammad Anis uddin Nasir Mesos 16/32
  • 46. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Resource Allocation Muhammad Anis uddin Nasir Mesos 17/32
  • 47. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Resource Allocation Fair sharing Muhammad Anis uddin Nasir Mesos 17/32
  • 48. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Resource Allocation Fair sharing Strict priorities Muhammad Anis uddin Nasir Mesos 17/32
  • 49. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Resource Allocation Fair sharing Strict priorities Delay sharing Muhammad Anis uddin Nasir Mesos 17/32
  • 50. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Resource Allocation Fair sharing Strict priorities Delay sharing Isolation Muhammad Anis uddin Nasir Mesos 17/32
  • 51. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Resource Allocation Fair sharing Strict priorities Delay sharing Isolation Linux Containers Muhammad Anis uddin Nasir Mesos 17/32
  • 52. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Resource Allocation Fair sharing Strict priorities Delay sharing Isolation Linux Containers Solaris Project Muhammad Anis uddin Nasir Mesos 17/32
  • 53. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Scalability and Resource Allocation Robustness Fair sharing Strict priorities Delay sharing Isolation Linux Containers Solaris Project Muhammad Anis uddin Nasir Mesos 17/32
  • 54. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Scalability and Resource Allocation Robustness Fair sharing Filters Strict priorities Delay sharing Isolation Linux Containers Solaris Project Muhammad Anis uddin Nasir Mesos 17/32
  • 55. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Scalability and Resource Allocation Robustness Fair sharing Filters Strict priorities Re-offering resources Delay sharing Isolation Linux Containers Solaris Project Muhammad Anis uddin Nasir Mesos 17/32
  • 56. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Scalability and Resource Allocation Robustness Fair sharing Filters Strict priorities Re-offering resources Delay sharing Incentives Isolation Linux Containers Solaris Project Muhammad Anis uddin Nasir Mesos 17/32
  • 57. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Scalability and Resource Allocation Robustness Fair sharing Filters Strict priorities Re-offering resources Delay sharing Incentives Fault Tolerance Isolation Linux Containers Solaris Project Muhammad Anis uddin Nasir Mesos 17/32
  • 58. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Scalability and Resource Allocation Robustness Fair sharing Filters Strict priorities Re-offering resources Delay sharing Incentives Fault Tolerance Isolation Soft state master Linux Containers Solaris Project Muhammad Anis uddin Nasir Mesos 17/32
  • 59. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Scalability and Resource Allocation Robustness Fair sharing Filters Strict priorities Re-offering resources Delay sharing Incentives Fault Tolerance Isolation Soft state master Linux Containers Report node failure to Solaris Project framework Muhammad Anis uddin Nasir Mesos 17/32
  • 60. Motivation Mesos Overview Implementation Architecture Evaluation ConclusionMore Features Scalability and Resource Allocation Robustness Fair sharing Filters Strict priorities Re-offering resources Delay sharing Incentives Fault Tolerance Isolation Soft state master Linux Containers Report node failure to Solaris Project framework Multiple schedulers Muhammad Anis uddin Nasir Mesos 17/32
  • 61. Motivation Mesos Implementation Evaluation Conclusion1 Motivation2 Mesos Overview Architecture3 Implementation4 Evaluation5 Conclusion Muhammad Anis uddin Nasir Mesos 18/32
  • 62. Motivation Mesos Implementation Evaluation ConclusionImplementation 10,000 lines of code in C++ Muhammad Anis uddin Nasir Mesos 19/32
  • 63. Motivation Mesos Implementation Evaluation ConclusionImplementation 10,000 lines of code in C++ Linux, Solaris, OS X Muhammad Anis uddin Nasir Mesos 19/32
  • 64. Motivation Mesos Implementation Evaluation ConclusionImplementation 10,000 lines of code in C++ Linux, Solaris, OS X support framework written in Java, C++, Python Muhammad Anis uddin Nasir Mesos 19/32
  • 65. Motivation Mesos Implementation Evaluation ConclusionImplementation 10,000 lines of code in C++ Linux, Solaris, OS X support framework written in Java, C++, Python ZooKeeper Muhammad Anis uddin Nasir Mesos 19/32
  • 66. Motivation Mesos Implementation Evaluation ConclusionImplementation 10,000 lines of code in C++ Linux, Solaris, OS X support framework written in Java, C++, Python ZooKeeper Frameworks Muhammad Anis uddin Nasir Mesos 19/32
  • 67. Motivation Mesos Implementation Evaluation ConclusionImplementation 10,000 lines of code in C++ Linux, Solaris, OS X support framework written in Java, C++, Python ZooKeeper Frameworks Hadoop Muhammad Anis uddin Nasir Mesos 19/32
  • 68. Motivation Mesos Implementation Evaluation ConclusionImplementation 10,000 lines of code in C++ Linux, Solaris, OS X support framework written in Java, C++, Python ZooKeeper Frameworks Hadoop Torque and MPI Muhammad Anis uddin Nasir Mesos 19/32
  • 69. Motivation Mesos Implementation Evaluation ConclusionImplementation 10,000 lines of code in C++ Linux, Solaris, OS X support framework written in Java, C++, Python ZooKeeper Frameworks Hadoop Torque and MPI Spark Muhammad Anis uddin Nasir Mesos 19/32
  • 70. Motivation Mesos Implementation Evaluation ConclusionSpark Data flow for logistic regression Muhammad Anis uddin Nasir Mesos 20/32
  • 71. Motivation Mesos Implementation Evaluation Conclusion1 Motivation2 Mesos Overview Architecture3 Implementation4 Evaluation5 Conclusion Muhammad Anis uddin Nasir Mesos 21/32
  • 72. Motivation Mesos Implementation Evaluation ConclusionDynamic Resource Sharing 96 Node Mesos Cluster Muhammad Anis uddin Nasir Mesos 22/32
  • 73. Motivation Mesos Implementation Evaluation ConclusionDynamic Resource Sharing 96 Node Mesos Cluster 4 CPU Cores and 15GB Ram Muhammad Anis uddin Nasir Mesos 22/32
  • 74. Motivation Mesos Implementation Evaluation ConclusionDynamic Resource Sharing Muhammad Anis uddin Nasir Mesos 23/32
  • 75. Motivation Mesos Implementation Evaluation ConclusionData Locality with Resource Offers 16 instance of Hadoop using 93 EC2 nodes Muhammad Anis uddin Nasir Mesos 24/32
  • 76. Motivation Mesos Implementation Evaluation ConclusionData Locality with Resource Offers 16 instance of Hadoop using 93 EC2 nodes 1.7x speed up with Mesos Muhammad Anis uddin Nasir Mesos 24/32
  • 77. Motivation Mesos Implementation Evaluation ConclusionData Locality with Resource Offers 16 instance of Hadoop using 93 EC2 nodes 1.7x speed up with Mesos 97% data locality with 5sec data scheduling Muhammad Anis uddin Nasir Mesos 24/32
  • 78. Motivation Mesos Implementation Evaluation ConclusionScalability 99 Amazon EC2 nodes Muhammad Anis uddin Nasir Mesos 25/32
  • 79. Motivation Mesos Implementation Evaluation ConclusionScalability 99 Amazon EC2 nodes Scaled to 50,000 emulated slaves, 200 frameworks, 100K tasks Muhammad Anis uddin Nasir Mesos 25/32
  • 80. Motivation Mesos Implementation Evaluation ConclusionScalability 99 Amazon EC2 nodes Scaled to 50,000 emulated slaves, 200 frameworks, 100K tasks unable to scale beyond 50,000 slaves as Amazon EC2 cluster was the bottleneck Muhammad Anis uddin Nasir Mesos 25/32
  • 81. Motivation Mesos Implementation Evaluation ConclusionFurther Experiments Fault Tolerance Muhammad Anis uddin Nasir Mesos 26/32
  • 82. Motivation Mesos Implementation Evaluation ConclusionFurther Experiments Fault Tolerance Mesos masters connected to a 5-node ZooKeeper quorum Muhammad Anis uddin Nasir Mesos 26/32
  • 83. Motivation Mesos Implementation Evaluation ConclusionFurther Experiments Fault Tolerance Mesos masters connected to a 5-node ZooKeeper quorum fault detection and recovery in 10 sec Muhammad Anis uddin Nasir Mesos 26/32
  • 84. Motivation Mesos Implementation Evaluation ConclusionFurther Experiments Fault Tolerance Mesos masters connected to a 5-node ZooKeeper quorum fault detection and recovery in 10 sec Overhead Muhammad Anis uddin Nasir Mesos 26/32
  • 85. Motivation Mesos Implementation Evaluation ConclusionFurther Experiments Fault Tolerance Mesos masters connected to a 5-node ZooKeeper quorum fault detection and recovery in 10 sec Overhead LINPACK for MPI and Wordcount Hadoop Becnhmark Muhammad Anis uddin Nasir Mesos 26/32
  • 86. Motivation Mesos Implementation Evaluation ConclusionFurther Experiments Fault Tolerance Mesos masters connected to a 5-node ZooKeeper quorum fault detection and recovery in 10 sec Overhead LINPACK for MPI and Wordcount Hadoop Becnhmark Overhead of Mesos was less than 4% Muhammad Anis uddin Nasir Mesos 26/32
  • 87. Motivation Mesos Implementation Evaluation Conclusion1 Motivation2 Mesos Overview Architecture3 Implementation4 Evaluation5 Conclusion Muhammad Anis uddin Nasir Mesos 27/32
  • 88. Motivation Mesos Implementation Evaluation ConclusionConclusion Mesos shares clusters efficiently among diverse frameworks Muhammad Anis uddin Nasir Mesos 28/32
  • 89. Motivation Mesos Implementation Evaluation ConclusionConclusion Mesos shares clusters efficiently among diverse frameworks Fine-grained sharing at the level of tasks Muhammad Anis uddin Nasir Mesos 28/32
  • 90. Motivation Mesos Implementation Evaluation ConclusionConclusion Mesos shares clusters efficiently among diverse frameworks Fine-grained sharing at the level of tasks Resource Sharing, a scalable mechanism for application-controlled scheduling Muhammad Anis uddin Nasir Mesos 28/32
  • 91. Motivation Mesos Implementation Evaluation ConclusionConclusion Mesos shares clusters efficiently among diverse frameworks Fine-grained sharing at the level of tasks Resource Sharing, a scalable mechanism for application-controlled scheduling Enabales co-existence of current frameworks and development of new specialized frameworks Muhammad Anis uddin Nasir Mesos 28/32
  • 92. Motivation Mesos Implementation Evaluation ConclusionReferences Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph, A. D., Katz, R., Shenker, S., et al. (n.d.). Mesos : A Platform for Fine-Grained Resource Sharing in the Data Center. http://incubator.apache.org/mesos/ http://datainthecloud.blogspot.se/2011/10/ mesos-platform-for-fine-grained.html https://www.usenix.org/conference/nsdi11/ mesos-platform-fine-grained-resource-sharing-data-cent http://static.usenix.org/event/nsdi11/tech/ slides/hindman.pdf Muhammad Anis uddin Nasir Mesos 29/32
  • 93. Motivation Mesos Implementation Evaluation ConclusionMesos: A Platform for Fine-Grained Resource Sharing in the Data Center Muhammad Anis uddin Nasir KTH Royal Institute of Technology November 20, 2012 Muhammad Anis uddin Nasir Mesos 30/32
  • 94. Motivation Mesos Implementation Evaluation ConclusionDifferent workloads Homogeneous Tasks Muhammad Anis uddin Nasir Mesos 31/32
  • 95. Motivation Mesos Implementation Evaluation ConclusionDifferent workloads Homogeneous Tasks elastic frameworks with constant task duration Muhammad Anis uddin Nasir Mesos 31/32
  • 96. Motivation Mesos Implementation Evaluation ConclusionDifferent workloads Homogeneous Tasks elastic frameworks with constant task duration rigid frameworks with exponential task duration Muhammad Anis uddin Nasir Mesos 31/32
  • 97. Motivation Mesos Implementation Evaluation ConclusionDifferent workloads Homogeneous Tasks elastic frameworks with constant task duration rigid frameworks with exponential task duration Placement Preferences Muhammad Anis uddin Nasir Mesos 31/32
  • 98. Motivation Mesos Implementation Evaluation ConclusionDifferent workloads Homogeneous Tasks elastic frameworks with constant task duration rigid frameworks with exponential task duration Placement Preferences weighted fair allocation policy Muhammad Anis uddin Nasir Mesos 31/32
  • 99. Motivation Mesos Implementation Evaluation ConclusionDifferent workloads Homogeneous Tasks elastic frameworks with constant task duration rigid frameworks with exponential task duration Placement Preferences weighted fair allocation policy lottery scheduling Muhammad Anis uddin Nasir Mesos 31/32
  • 100. Motivation Mesos Implementation Evaluation ConclusionDifferent workloads Homogeneous Tasks elastic frameworks with constant task duration rigid frameworks with exponential task duration Placement Preferences weighted fair allocation policy lottery scheduling Heterogeneous Tasks Muhammad Anis uddin Nasir Mesos 31/32
  • 101. Motivation Mesos Implementation Evaluation ConclusionDifferent workloads Homogeneous Tasks elastic frameworks with constant task duration rigid frameworks with exponential task duration Placement Preferences weighted fair allocation policy lottery scheduling Heterogeneous Tasks random task assignment Muhammad Anis uddin Nasir Mesos 31/32
  • 102. Motivation Mesos Implementation Evaluation ConclusionDifferent workloads Homogeneous Tasks elastic frameworks with constant task duration rigid frameworks with exponential task duration Placement Preferences weighted fair allocation policy lottery scheduling Heterogeneous Tasks random task assignment reserve some resources on each node for small tasks Muhammad Anis uddin Nasir Mesos 31/32
  • 103. Motivation Mesos Implementation Evaluation ConclusionDifferent workloads Homogeneous Tasks elastic frameworks with constant task duration rigid frameworks with exponential task duration Placement Preferences weighted fair allocation policy lottery scheduling Heterogeneous Tasks random task assignment reserve some resources on each node for small tasks maximum task duration Muhammad Anis uddin Nasir Mesos 31/32
  • 104. Motivation Mesos Implementation Evaluation ConclusionDifferent workloads Homogeneous Tasks elastic frameworks with constant task duration rigid frameworks with exponential task duration Placement Preferences weighted fair allocation policy lottery scheduling Heterogeneous Tasks random task assignment reserve some resources on each node for small tasks maximum task duration Framework incentives Muhammad Anis uddin Nasir Mesos 31/32
  • 105. Motivation Mesos Implementation Evaluation ConclusionDifferent workloads Homogeneous Tasks elastic frameworks with constant task duration rigid frameworks with exponential task duration Placement Preferences weighted fair allocation policy lottery scheduling Heterogeneous Tasks random task assignment reserve some resources on each node for small tasks maximum task duration Framework incentives short tasks Muhammad Anis uddin Nasir Mesos 31/32
  • 106. Motivation Mesos Implementation Evaluation ConclusionDifferent workloads Homogeneous Tasks elastic frameworks with constant task duration rigid frameworks with exponential task duration Placement Preferences weighted fair allocation policy lottery scheduling Heterogeneous Tasks random task assignment reserve some resources on each node for small tasks maximum task duration Framework incentives short tasks elastic tasks Muhammad Anis uddin Nasir Mesos 31/32
  • 107. Motivation Mesos Implementation Evaluation ConclusionDifferent workloads Homogeneous Tasks elastic frameworks with constant task duration rigid frameworks with exponential task duration Placement Preferences weighted fair allocation policy lottery scheduling Heterogeneous Tasks random task assignment reserve some resources on each node for small tasks maximum task duration Framework incentives short tasks elastic tasks do not accept unknown resources Muhammad Anis uddin Nasir Mesos 31/32
  • 108. Motivation Mesos Implementation Evaluation ConclusionLimitations Fragmentation Muhammad Anis uddin Nasir Mesos 32/32
  • 109. Motivation Mesos Implementation Evaluation ConclusionLimitations Fragmentation not optimal bin packing Muhammad Anis uddin Nasir Mesos 32/32
  • 110. Motivation Mesos Implementation Evaluation ConclusionLimitations Fragmentation not optimal bin packing large jobs may starve Muhammad Anis uddin Nasir Mesos 32/32
  • 111. Motivation Mesos Implementation Evaluation ConclusionLimitations Fragmentation not optimal bin packing large jobs may starve minimum offer size on each slave Muhammad Anis uddin Nasir Mesos 32/32
  • 112. Motivation Mesos Implementation Evaluation ConclusionLimitations Fragmentation not optimal bin packing large jobs may starve minimum offer size on each slave Interdependent framework constraints Muhammad Anis uddin Nasir Mesos 32/32
  • 113. Motivation Mesos Implementation Evaluation ConclusionLimitations Fragmentation not optimal bin packing large jobs may starve minimum offer size on each slave Interdependent framework constraints scenarios where only one task can be accommodated Muhammad Anis uddin Nasir Mesos 32/32
  • 114. Motivation Mesos Implementation Evaluation ConclusionLimitations Fragmentation not optimal bin packing large jobs may starve minimum offer size on each slave Interdependent framework constraints scenarios where only one task can be accommodated Framework Complexity Muhammad Anis uddin Nasir Mesos 32/32
  • 115. Motivation Mesos Implementation Evaluation ConclusionLimitations Fragmentation not optimal bin packing large jobs may starve minimum offer size on each slave Interdependent framework constraints scenarios where only one task can be accommodated Framework Complexity framework scheduling is complex Muhammad Anis uddin Nasir Mesos 32/32
  • 116. Motivation Mesos Implementation Evaluation ConclusionLimitations Fragmentation not optimal bin packing large jobs may starve minimum offer size on each slave Interdependent framework constraints scenarios where only one task can be accommodated Framework Complexity framework scheduling is complex framework has a choice Muhammad Anis uddin Nasir Mesos 32/32
  • 117. Motivation Mesos Implementation Evaluation ConclusionLimitations Fragmentation not optimal bin packing large jobs may starve minimum offer size on each slave Interdependent framework constraints scenarios where only one task can be accommodated Framework Complexity framework scheduling is complex framework has a choice failures are easy to handle with resource offers Muhammad Anis uddin Nasir Mesos 32/32