Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Jun 2017 HUG: YARN Scheduling – A Step Beyond

589 views

Published on

In recent times, YARN Capacity Scheduler has improved a lot in terms of some critical features and refactoring.  Here is a quick look into some of the recent changes in scheduler:

Global Scheduling Support

General placement support

Better preemption model to handle resource anomalies across and within queue.

Absolute resources’ configuration support

Priority support between Queues and Applications

In this talk, we will deep dive into each of these new features to give a better picture of their usage and performance comparison. We will also provide some more brief overview about the ongoing efforts and how they can help to solve some of the core issues we face today.

Speakers:

Sunil Govind (Hortonworks), Jian He (Hortonworks)

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Jun 2017 HUG: YARN Scheduling – A Step Beyond

  1. 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Hadoop YARN: Capacity Scheduler Improvements June 2017 Sunil Govindan and Junping Du
  2. 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved About us Sunil G Hortonworks Apache Hadoop Committer Junping Du Hortonworks Apache Hadoop PMC and Committer junping_du@apache.org sunilg@apache.org
  3. 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda ⬢ Overview: Capacity Scheduler ⬢ Current features in Capacity Scheduler ⬢ Ongoing work in Capacity Scheduler ⬢ Q & A
  4. 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Hadoop YARN Scheduler Inter/Intra queue pre-emption Application Queue B – 25% Queue C – 25% Label: SAS (exclusive) Queue A – 50% FIFO ResourceManager (active) Application, Queue A, 4G, 1 vcore Reservation for application User
  5. 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Capacity Scheduler : Feature Overview (Available) ⬢ Priority support for Queue and Application – Application Priority – Queue Priority ⬢ Preemption Support – Improvement in Inter Queue preemption model – Intra Queue preemption support based on application priority and user-limit
  6. 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Capacity Scheduler : Application/Queue Priority ⬢ Overview – Available as part of Hadoop 2.8 release – Done in YARN-1963 and YARN-5864 ⬢ Application Priority – Execute some YARN applications at higher priority, regardless of other applications running – Helps to avoid creating multiple queues for priority – Enable support to set priority at each application level and dynamically change at runtime – Control the users who are abusing higher priorities via ACLs configurable at queue level
  7. 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Capacity Scheduler : Application/Queue Priority ⬢ Queue Priority – Currently queues are ordered according to relative used-capacities – Possibility of scarce resources allocation to less-important apps first • Latency sensitivity • Resource fragmentation for large-container apps – Queue priority helps to configure a higher integer value to critical queue such as long running service queue.
  8. 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Capacity Scheduler : Preemption Support ⬢ Resource Preemption – Resource Preemption in YARN allows businesses to • maximize the use of their cluster compute power • decrease compute time for most applications • ensure that resources are available in a timely manner for critical applications. ⬢ Inter-Queue Preemption – helps in scenarios such as  Over-committed cluster where queue elasticity is used extensively  Starving applications present in under-served queues for resources – addresses queue starvation by finding the best suited resources for under served queue – Improved support for reservations and queue-priority (Available as part of Hadoop 2.8 release)
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Capacity Scheduler : Preemption Support ⬢ Intra-Queue Preemption – Available as part of Hadoop 2.8 and done in YARN-2009 and YARN-2113 – helps in scenarios such as when  Lower priority applications consumed Queue’s entire quota starving higher priority apps  Few users could consume entire user quota to starve other user’s applications –“normalize resources based on application’s priority and user-limit within the queue”
  10. 10. 1 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Capacity Scheduler : Preemption Support ⬢ Intra-Queue Preemption based on Application Priority Before Input: app1 , p1, u1 <pending=20 , used=50 > app2 , p1, u1 <pending=20 , used= 20> app3 , p3, u1 <pending=50 , used= 0> Configuration: intra-queue-preemption.enabled = true root.qA.capacity = 70% root.qB.capacity = 30% Cluster resource = 100 (qA.used=70, qB.used=30) After Preempted: app1 , p1, u1 <preempted=31, used=19> app2 , p1, u1 <preempted=19, used=1> app3 , p3, u1 <pending=0 , used=50> Analysis: ⬢ 30 resources were preempted from app1 and app2 ⬢ App2’s AM container got spared. Pending Used app1 app2 app3 app1 app2 app3 p3 Preempted
  11. 11. 1 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Capacity Scheduler : Preemption Support ⬢ Intra-Queue Preemption based on User Limit Before Input: app1 , p1, u1 <pending=20 , used=25 > app2 , p1, u2 <pending=20 , used= 25> app3 , p1, u3 <pending=30 , used= 50> Configuration: intra-queue-preemption.enabled = true root.qA.capacity = 10% User-limit = 33% Cluster resource = 100 (qA.used=100) After Preempted: app1 , p1, u1 <preempted=0, used=33> app2 , p1, u2 <preempted=0, used=33> app3 , p1, u3 < preempted=16, used=34> Analysis: ⬢ 16 resources were preempted from app3 ⬢ App1 and app2 shared these preempted resources. Pending Used app1 app2 app3 app1 app2 app3 u3 Preempted u2u1
  12. 12. 1 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Capacity Scheduler : Feature Overview (Ongoing) ⬢ Global Scheduling Support – Scheduling placement support – 8x performance improvement ⬢ Absolute Resource Configuration support – Capacity scheduler queue planning was based on percentage – Introduce Absolute resource configuration per queue level – Interaction with Resource profile feature
  13. 13. 1 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Capacity Scheduler : Global Scheduling Support ⬢ Global Scheduling Support – Available as part of Hadoop 3.0 Alpha-3 release and done in YARN-5139 – Better support for placing resource requests for applications – 8x performance improvement ⬢ Overview – Current design of one-node-at-a-time allocation cycle can lead to suboptimal decisions. – Considering future complex resource placement requirements, such as node constraints (give me "a && b || c") or anti-affinity (“do not allocate HBase regionsevers and Storm workers on the same host”), YARN scheduler is moving towards Global Scheduling. – With global scheduling, YARN scheduler should be able to look at more nodes and select the best nodes based on application requirements unlike existing schedulers.
  14. 14. 1 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Capacity Scheduler : Global Scheduling Support ⬢ Design Overview – Allocation decision on multiple nodes instead of single node. – Improved locking mechanism to allow multiple allocation threads looking at cluster states to create allocation proposals. – Each of these allocation-proposal will be sent to scheduler to commit or reject. ⬢ This design is not finalized to consider few more options. YARN-6592
  15. 15. 1 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Capacity Scheduler : Absolute Resource Configuration ⬢ Current Model – Use Queues to manage resource-usage by different team/department/BU and different workloads – Using percentages – Hierarchical queues helps for better queue management – Work is getting done as part of YARN-5881 root (100%) Sales (30%) Engineering (65%) Default (5%) Dev (50%) QE (50%)
  16. 16. 1 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Capacity Scheduler : Absolute Resource Configuration ⬢ Issues in current model – Resource management through percentages is not easy  works perfectly fine when the ratio between queues are fixed  not easy for admins who want fine control of resources of queues  With nodes getting added or removed, it will be tougher to set a specific resource limit for specific queue – One percentage value for all resource-types – Sum of the min-resources of all the queues must be 100%
  17. 17. 1 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Capacity Scheduler : Absolute Resource Configuration ⬢ New Approach – Can specify absolute resource values as min-resource to queues. – For better elasticity, also can support specify absolute resource values as max-resource to queues. – parent.min-resource >= Σ(child.min-resource). – “relax the exactly-sum-to-100% requirement of today” ⬢ Challenges of Absolute Configuration – Ensuring SLAs when cluster scales down – Handling min-resources when cluster scales up root [memory=100Gi,vcores=100] Sales [memory=30Gi,v cores=30] Engineering [memory=65Gi, vcores=60] Default [memory=1Gi, vcores=1] Dev [memory=30Gi, vcores=50] QE [memory=35Gi, vcores=10]
  18. 18. 1 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved1 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ongoing Features in YARN and CS
  19. 19. 1 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved YARN features : Capacity scheduler ⬢ New YARN UI (YARN-3368) – Available as part of Hadoop 3.0 Alpha-3 release – Support for native-service to launch service – Makes Hadoop YARN much easier to manage ⬢ Resource profiles (YARN-3926) – Ongoing effort to support different resources – Performance improvement ⬢ Application Timeout (YARN-3813) – Control lifetime of an application by YARN – Helps to kill applications which runs over the limited time allotted ⬢ Opportunistic Containers – Two approaches: Distributed Scheduling (YARN-2877) and Centralized (YARN-5220)
  20. 20. 2 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ongoing work ⬢ GPU Scheduling (YARN-3926) – Ongoing effort to support GPU while scheduling – GPU isolation per core ⬢ Distributed scheduling –YARN-2877, YARN-4742 –NMs run as a local scheduler –Allows faster scheduling turnaround ⬢ Better support for disk and network isolation (YARN-2619, YARN-2140) –Tied to supporting arbitrary resource types
  21. 21. 2 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved YARN features : New UI (Dashboard)
  22. 22. 2 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved YARN features : New UI (Queues)
  23. 23. 2 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved YARN features : New UI (Applications)
  24. 24. 2 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved2 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank you!

×