Lessons learned from scaling YARN to 40K machines in a multi tenancy environment
•Download as PPTX, PDF•
3 likes•1,158 views
This document summarizes Microsoft's experience running YARN at a massive scale of over 40,000 machines to support its Cosmos big data platform. Some key points: - Cosmos processes over 500,000 jobs and 2 million containers per hour with high reliability and CPU utilization. - Scaling YARN to this level required optimizations like scheduler key pruning, time-based locality decay, and asynchronous logging to achieve sub-5 second allocation latencies. - A federated approach was used to partition the large cluster into independent YARN sub-clusters for improved scalability and maintenance. - Ongoing work involves tuning multi-cast policies, opportunistic container utilization, and log management to maximize scal
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Lessons learned from scaling YARN to 40K machines in a multi tenancy environment
Similar to Lessons learned from scaling YARN to 40K machines in a multi tenancy environment (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
Lessons learned from scaling YARN to 40K machines in a multi tenancy environment
- 1. LESSONS LEARNED FROM SCALING YARN TO 40K MACHINES IN A MULTI TENANCY ENVIRONMENT Roni Burd : Principal Software Eng Mgr Hitesh Sharma: Principal Software Eng Sarvesh Sakalanaga: Senior Software Eng Mgr
- 2. Cosmos: World’s Biggest YARN Cluster! Cosmos in numbers • Single DC >40,000 machines • Multiple DCs (do the math ) • >500,000 jobs / day • >2,000,000 containers / hour • Up to 90% CPU utilization • 99.9% reliability/availability • Several Exabytes in storage • Hundreds of petabytes processed per day Journey Migrate Cosmos to YARN Run OSS workloads Multiplier effect from the community
- 3. • Running different YARN applications with different characteristics • SCOPE (AKA U-SQL) is the most common application today • Spark, custom REEF code and custom YARN AM • As we migrate existing workloads to YARN, we have to remain compatible with legacy apps • Cosmos supports running SLA and non-SLA jobs in the same datacenter • Customer are allocated Virtual cluster (VC) with a max number of containers • Machines allocated to each VC are shared across all tenants • Virtual clusters have guaranteed capacity and can use idle capacity from other tenants • Maximize COGS and reduce latency • Maximize data locality and prevent data movement/copy • Don’t leave any resource unused – CPU, Memory, IOs and Bandwidth Cosmos environment
- 4. NM1 NM 2 NM 3 NM 4 NM 5 NM 6 NM 7 NM 8 NM 9 NM 10 NM 11 NM 12 NM 14 NM 15 NM 13 NM 3900 NM 3901 NM 3903 NM 3904 NM 3902 Scope AM Time Parallel containers Max containers allowed • Scope submits containers in ‘waves’ • Waves are capped by max allocation • Waves can be very big (2000 or more) • Jobs achieve SLA by avoiding contention • Allocated capacity < Total Cluster Capacity • This leads to natural underutilization Scope AM Scope request characteristics … Rack Rack Rack
- 5. Achieving 90% CPU utilization: Mercury (YARN-2877) Opportunistic containers • Scope allocates OPPORTUNISTIC containers • OPP are queued in NM based on NMs capacity • Scheduled after exhausting max GUARANTEED • Once GUARANEED finish, SCOPE can either: • Promote running OPPORTUNISTIC • Scheduled new GUARANEED wave instead • OPPORTUNISTIC get paused (YARN-5972) • This improved latencies, reduced COGS and preserves SLAs DC1 DC2
- 6. In aggregate SCOPE generate 4000 QPS (avg) /10,000 QPS (max) YARN RM Scope AM Pending containers Allocated Guaranteed NM Scheduler SCOPE Aux Service Allocate(G) Allocate(OPP) Immediate response “Promote”* Response only when allocated *:based on 2.7 patch Working with OSS for 2.9 Opportunistic queue Paused queue O O O G G G Running StartContainer(G) G OLoad information O O Response only when allocated
- 7. Latency sensitive workloads • Duration of scope containers vary widely, but most are small – delays are expensive! • Any delays in Allocate() or StartContainer() can lead to job slowdown and failures • A 20sec delay can mean millions of U$S • Locality is extremely important for latency and COGS • Scope has visibility to all 40,000 machines through a Aux Service • Scope AM is smart and makes several optimizations (e.g. future JOINs, RACK aggregations) • RACK locality is “almost” as good as NODE locality – OFF_SWITCH is bad Container allocation need to be <5sec @ 95%tile Node Locality on every request across the DC is important for latency
- 8. How did we scale YARN to deal with our QPS, latency and locality requirements? How did we scale YARN to 40K machines? Hard lessons (that we have time to cover today)
- 9. YARN RM Scalability Hitesh Sharma Principal Software Engineer
- 10. Scope – main workload In aggregate SCOPE generate 4000 QPS (avg) /10,000 QPS (max) Container allocation need to be <5sec @ 95%tile Node Locality on every request across the DC is important for latency
- 11. • A few Scope jobs would overwhelm YARN RM in a few minutes • Long delays in allocating containers would result in job failures • Scale testing in big clusters is a very expensive process • Lack of metrics and telemetry to understand what’s happening in YARN RM Legend • Red line : pending containers • Blue line :allocated containers Allocated vs Pending containers (Note: Cluster had resources to satisfy all the requests being made) Setup • Test cluster with 3000 nodes • Running Hadoop 2.7.1 • CapacityScheduler with single queue Challenges
- 12. Scheduling in capacity-scheduler YARN RM NM NM NM NM HB HB HB HB Thread Scheduler loop (thread driven by HB) Thread Thread Allocate Allocate Allocate Pending request YARN RM takes a lock and looks at all the outstanding container requests in the queue on each HB If the node can satisfy a pending request then YARN RM allocates otherwise it just looks at the next one AM AM AM Look at _every_ pending requests (potentially thousands) - algorithm uses this to downgrade locality
- 13. • Scheduler loop is very expensive as it looks over all the outstanding requests • Counting missed opportunities to relax locality impacts scale and hard to tune • Creating immutable “Resource” objects during heartbeat processing is very expensive • Log4j is synchronous by default • Only one allocation per node heartbeat Bottlenecks
- 14. 1. Scheduler key pruning • Each node heartbeat looks at the outstanding allocation requests for that node only 2. Time based decay for locality • Use time to decide if we should downgrade to rack or any where in the cluster • Reduces the work done in each node heartbeat 3. Switched to async log4j appender • Reduces expensive lock and IO contention 4. Metrics to track allocation latencies and QPS • TTD and TTM are critical to achieve 99.9% availability/reliability Improvements
- 15. • Test cluster with 4000 nodes • RM-NM and RM-AM heartbeat set to 1sec sec • Using log4j async logger Stage Impact Before improvements + relaxLocality ON • Allocation latency @ 95th%tile< 10s • Promotion latency @ 95th%tile< 10s • Node locality – <10% • ANY> locality – >80% • Sustained load < 500 QPS Scheduler key pruning • Allocation latency @ 95th%tile< 4s • Promotion latency @ 95th%tile< 4s • Node locality – 99.51% • Rack locality– 0.23% • Sustained load < 2000 QPS Time based decay for locality • Allocation latency @ 95th%tile< 3s • Promotion latency @ 95th%tile< 3s • Node locality – 99.84% • Rack locality– 0.11% • Sustained load < 3000 QPS Test results
- 16. Allocated vs Pending containers Before After Test results
- 17. Pending/Allocated Paused containers Queued container Percentile latency and QPS for Allocate, Promote, Locality metrics Containers running and type Containers Queued Containers Paused Example dashboard
- 18. • Single machine setup to stress test YARN scheduler (part of OSS since 2.6) • Allows us to try out cluster configurations with any number of nodes and different configurations (NM-RM heartbeat frequency, scheduler configurations, etc.) • Updated SLS to generate load similar to our workloads We can now try out different settings and fixes in a matter of minutes! Simulator (mimic Scope AM) Real YARN RM Mock NM Mock NM Mock NM HB HB HB • Container requests wave • Container duration • Multiple apps • Allocation ID • Relax locality • Opportunistic allocations • Container promotions thread thread threads Scheduler Load Simulator (aka SLS)
- 19. Scaling YARN to 40K+ nodes and beyond Sarvesh Sakalanaga Senior Engineering Manager
- 20. Yarn Sub-Cluster #1 Yarn Sub-Cluster #3Yarn Sub-Cluster #2 RM Task RM Task RM Task AM RM Proxy Service (Per NodeManager) Policy StateRouter Service YARN Client Federation Services YARN Sub Clusters Servers in Datacenter AM AM • Implements Client-RM Protocol • Stateless, Scalable Service • Multiple Instances with Load Balancer • Implements AM-RM Protocol • Hosted in NM • Intercepts all AM-RM communications • Sub-clusters are unmodified standalone YARN clusters with about 4K nodes. Start Containers Submit App • Voila! Applications can transparently span across multiple YARN sub clusters and scale to Datacenter level • No code change in any application• Centralized, highly-available repository Federation Interceptor UAM POOL Smart Policy Federation – High level architecture
- 21. Scope – main workload In aggregate SCOPE generate 4000 QPS (avg) /10,000 QPS (max) Container allocation need to be <5sec @ 95%tile Node Locality on every request across the DC is important for latency
- 22. 1. Load shaping 2. Cluster maintenance 3. Log management Production challenges
- 23. Limitations of BroadcastAMRMProxyPolicy •RM Scalability: increases QPS in all the sub-clusters •Cluster Utilization: duplicate allocations for each sub- cluster Solution: LocalityMulticastAMRMProxyPolicy 1. Create UAMs on demand 2. Allocate on sub-clusters on a rack 1: Load shaping
- 24. 2: Cluster maintenance Each DC will have constants machine movements • 13 sub-clusters split our 40K+ DC • About ~800 machines/day need some form of maintenance • Clusters keeps growing and changing (e.g. decommissioning RACKs) Solution: Use a Sub-Cluster Manger for balancing 1. Node to sub-cluster resolver service 2. Dynamic balancing of sub-cluster capacity 3. NM Maintenance mode: Container draining
- 25. Yarn Sub-Cluster #1 2: Cluster maintenance scenario: adding machines O O G G HEALTHY HEALTHY O O G G O O G G HEALTHY HEALTHY O O G G O O G G HEALTHY HEALTHY O O G G Yarn Sub-Cluster #2 O O G G HEALTHY HEALTHY O O G G O O G G HEALTHY HEALTHY O O G G O O G G HEALTHY HEALTHY O O G G Yarn Sub-Cluster #3 HEALTHY HEALTHY HEALTHY HEALTHY MAINTENANCE MAINTENANCE Sub-cluster Manager
- 26. Log volume per DC: 2 PB/day ; 2.5 GB/hour per machine 1. Custom log aggregator that works with our Scope AM to keep logs only for critical path containers and failed containers 2. Custom tools to aggregates container logs outside of YARN on need basis 3. Custom log search tool (Helios – Internal code name) that indexes YARN logs bases on keywords 4. Scope NM AUX service to keep track of key information on container stats 3: Log handing
- 27. Scalability Multi-cast policy tuning for better placements, Container reuse, Have light weight “Resource” objects, Multiple node allocations in the same heartbeat Utilization Container resize, Improve opportunistic containers utilization with system priorities, Federation Global policy generator enhancements and Relax locality for Opportunistic containers Operability ATS V2, more metrics and logs and better log management And many more…! Next Steps
- 28. Fully committed to YARN and Open Source Committed • Federation: YARN-2915 • AllocationID: YARN-4879 • Support distributed scheduling (AKA mercury): YARN-2877 • AMRMProxy: YARN-2884 • UnmanagedAM pool manager: YARN-5531 • Federation Interceptor: YARN-6511 • Locality – Multicast Policy: YARN-5325 In progress/Open • Federation phase 2: YARN-5597 • GPG: YARN-3660 • Router: YARN-3659 • Pausing of Opportunistic containers: YARN-5972 • Container Promotion: YARN-5085 • Scheduling of Opp through YARN RM: YARN-5220
- 29. Conclusion Largest YARN deployment in the world! ...and we are growing 5X next year …with more OSS workloads Our Journey has just started!
Editor's Notes
- 1) When we started to run YARN in a cluster with over 3000 nodes then a few Scope jobs would easily overwhelm YARN RM. For instance, just 10 terasorts could actually cause YARN RM to fallover. 2) We would see long delays in allocation latencies and in most cases never got a container back from YARN RM. 3) The other challenge that we have had is that telemetry and logging were some what missing and it was incredibly hard to diagnose issues in production. 4) It is also quite expensive to test in big clusters as the cycle of deployment and testing can be a few hours in the best case. 5) This graph here shows the number of allocated and pending containers in YARN RM. We use these metrics as a proxy to know whether YARN RM can keep up with the rate at which AMs are requesting for containers. 6) The Y axis is the # of containers and X axis shows time. The blue line shows the # of allocated containers while the red line shows the # of pending containers. If the number of outstanding containers becomes too high then YARN RM won’t be able to do any allocations.
- At a very high level, here is a quick recap of how capacity scheduler does scheduling The application masters are sending container requests to YARN RM. These get added in a “pending queue” inside YARN RM. YARN RM also receives heartbeats from all the nodes in the cluster. Upon receiving a node heartbeat YARN RM takes a lock and looks at the pending queue to see what can be satisfied by that node. If the node can satisfy a request then an allocation is done. Otherwise YARN RM would look through the other pending requests in the queue. One major issue here is that upon each node heartbeat YARN RM is looking at all the outstanding requests, which may not be even satisfiable by that node. Further the algorithm used to decay locality from node to rack and any is based on heartbeat counting. The summary is that we spend a lot of time in each heartbeat processing and if you have high allocation request rate with 1 sec node heartbeats, it becomes hard for YARN RM to do allocations fast enough.
- Testing on real clusters is quite expensive as it involves deploying the bits, generating the load, and then analyze GBs of logs. This is a long cycle and to improve on it we started to leverage SLS. To recap, SLS is Scheduler Load Simulator and is a tool in Hadoop since 2.6. It allows you to start YARN RM with the given config and simulates multiple NMs heartbeating into it. It can then simulate submitting different apps with containers starting and exiting. We have extended SLS to generate load that is similar to the workload we run. This included sending waves of container requests, containers can live for different durations, multiple apps, do allocations with alloc ID, relaxed locality, and O allocations. A big shout out to everyone who worked and contributed to SLS. This tool has been a life saver! Today SLS is one of the key tools we use for testing any changes being made to YARN RM.
- Add callout http://hadoopsummit.org/san-jose/agenda/
- Talk about CISL Router add multiple instances Talk about policies
- Talk why we did not use YARN log aggregation Talk about our custom AM using talking to PN to handle logs Talk about sub-cluster to node mapping (CWS app id) Timeline server
- Remove links