Dawn of YARN @ Rocket Fuel
•
2 likes•1,114 views
This document discusses Rocket Fuel's use of YARN (Yet Another Resource Negotiator) for managing big data workloads. It summarizes Rocket Fuel's motivation for adopting YARN due to limitations in Hadoop's original resource management system. It describes how Rocket Fuel set up YARN and the various applications now running on YARN including Hive, Spark, Storm and HBase. It also discusses challenges faced including HDFS issues, resource manager failures and fair scheduling. It concludes by looking at next steps such as HDFS federation, RM HA and using YARN for additional applications and services.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Dawn of YARN @ Rocket Fuel
Similar to Dawn of YARN @ Rocket Fuel (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
Dawn of YARN @ Rocket Fuel
- 1. Proprietary & Confidential. Copyright © 2015.Proprietary & Confidential. Copyright © 2015. DAWN of YARN Abhijit Pol Nikhil Mulley
- 2. Proprietary & Confidential. Copyright © 2015.Proprietary & Confidential. Copyright © 2015. ADVERTISERS ROCKET FUEL 200+ Ad Exchanges 50+ Mn Websites 90+ Bn Daily Opportunities 3B CONSUMERS
- 3. Proprietary & Confidential. Copyright © 2015. 5 B 35 B 90 B Searches on Google Facebook Events Bid Requests Considered by Rocketfuel Requests per day Throughput
- 4. Proprietary & Confidential. Copyright © 2015. 200 Servers 1500 Servers 5 PB 65 PB13x Data Warehouse Growth
- 5. Big Data Fuels AI at Rocket Fuel
- 6. Proprietary & Confidential. Copyright © 2015. Ad Serving ETL-Reporting-Analytics Data Mining-Machine Learning Artificial Intelligence 2009: Hadoop 2010: HBase for Real-Time Workload 2012: Kafka-Storm for Stream Processing 2014: YARN, Spark for ML Early Adopters
- 7. Proprietary & Confidential. Copyright © 2015. Rocket Fuel Intro Motivation for YARN YARN Setup YARN Today Challenges and Learnings
- 11. Multi-Tenancy
- 12. Scalability
- 13. Proprietary & Confidential. Copyright © 2015. Rocket Fuel Intro Motivation for YARN YARN Setup YARN Today Challenges and Learnings
- 14. Proprietary & Confidential. Copyright © 2015. YARN Setup
- 15. Proprietary & Confidential. Copyright © 2015. Automation Is The Key! • Fast Resource Provisioning • Template Driven Configuration • Service Discovery + Monitoring • Change Management
- 16. Proprietary & Confidential. Copyright © 2015. Deployment • Internal Fork • Jenkins & FPM • Puppet Manifests • Multiple Versions Installed • Activate The Nice One
- 17. Proprietary & Confidential. Copyright © 2015. HDFS High Availability • QJM Based Namenode HA • Automatic Failover • Dedicated Disks for ZK & JN • Appropriate Timeouts in HA Configs • Appropriate Checkpoint Periods • ConfiguredFailoverProxyProvider
- 18. Proprietary & Confidential. Copyright © 2015. YARN MRV2 Config • Use Defaults – They Work! • Few Properties: • mapreduce.task.files.preserve.failedtasks => false • mapreduce.{map|reduce}.memory.mb => ~2G • mapreduce.{map|reduce}.java.opts => 85% of above • yarn.app.mapreduce.ap.resource.am => 1.5G • yarn.nodemanager.vmem-check-enabled => false • yarn-utils.py • Log Aggregation • yarn.log-aggregation-enable => true • yarn.log-aggregation.retain-seconds => 7 * 86400
- 19. Proprietary & Confidential. Copyright © 2015. Scheduler • Fair Share Scheduler • ACLs • Preemption • Measure and Tune!
- 20. Proprietary & Confidential. Copyright © 2015. Rocket Fuel Intro Motivation for YARN YARN Setup YARN Today Challenges and Learnings
- 21. Proprietary & Confidential. Copyright © 2015. Happy Map Reduce • No Static Slots • Performance • Lean RM • Own AM! Each time! • Completed Jobs History • Log Aggregation
- 22. Proprietary & Confidential. Copyright © 2015. Habitat - Sharing is Caring • Hive and Map-Reduce • Tez and Spark • Storm on YARN • HBase on YARN DeathStar: Multi Tenant HBase on YARN By Ishan and Nitin Tomorrow 1:30pm at Hall 211
- 23. Proprietary & Confidential. Copyright © 2015. YARN Cluster Numbers • 1.5k+ Nodes • 500+ Active Users • 12k+ Applications/Day • 10+ HBase Clusters • 10+ Storm Topologies • 500+ Hive Queries on Tez • 20k+ MR Jobs/Day
- 24. Proprietary & Confidential. Copyright © 2015. YARN is Relatively New!
- 25. Proprietary & Confidential. Copyright © 2015. OpenTSDB Tidbits @ Rocket Fuel • HBase-0.96 Cluster • OpenTSDB 2.0 • 5k+ Individual Metrics • 200k+ Data Points/Sec • 2500+ Nodes monitored • Retention of 365 days
- 26. Proprietary & Confidential. Copyright © 2015. Nagios For Alerting • Simple Health Checks • OpenTSDB Metrics Via check_tsd • Summary Alerts/Cluster Checks • Do Not Get Swamped • Use hostgroup and templates • Use service/host dependency checks • Use event handlers appropriately • PagerDuty for Critical Ones
- 27. Proprietary & Confidential. Copyright © 2015. Rocket Fuel Intro Motivation for YARN YARN Setup YARN Today Challenges and Learnings
- 28. Proprietary & Confidential. Copyright © 2015. Elementary • IPV6 Not Ready (NoRouteToHost) • Set Read-Ahead Cache (blockdev) • $(uname -r) >= 2.6.32-504.16.2 • Older Kernels Buggy • du/xfsaild Blocks Datanodes • Bad Network Drivers
- 29. Proprietary & Confidential. Copyright © 2015. Short Circuits Shock
- 30. Proprietary & Confidential. Copyright © 2015. HDFS Obscurities • Input Reads Do NOT Timeout (HDFS-7005) • Namenode Improvements • dfs.namenode.fslock.fair => false • dfs.namenode.audit.log.async => true • Enable Percentile Latency Metrics • dfs.metrics.percentiles.intervals
- 31. Proprietary & Confidential. Copyright © 2015. Gotcha • Disk Failures? Inevitable! • Tolerate Failed Volumes On DN • RM Down For a While • NMs Shutdown • YARN-3644 • Simple Typo Brought Cluster Down • Do xmllint on *-site.xml • YARN-3403
- 32. Fair Share Preemption. MR-5900, YARN-2181
- 33. Proprietary & Confidential. Copyright © 2015. What Next? • Scale and Availability • HDFS Federation • RM HA • More Applications on YARN • Kafka • Docker/Kubernetes • Alerts and Monitoring 3.0 • App History & Timeline Services • Self Service Alerting with OpenTSDB
- 34. Proprietary & Confidential. Copyright © 2015. We are Hiring!! rocketfuel.com/careers Abhijit Pol Nikhil Mulley Contributors: Shrijeet Paliwal Kishore Yellamraju
Editor's Notes
- Abhijit: Advertisers use our services to connect with potential customers visiting web sites and using mobile applications all over the world. We participate in the RTB ecosystem, serve a large number of impressions, and continuously learn from our past performance so that we can keep getting better and better at satisfying the needs of our customers.
- Accomodation for all types of neighbours.
- This is the bird’s eye view of YARN cluster setup. All the slaves have 12 disks. Each disk is 7.2k rpm disk with 3TB capacity with about 64GB physical memory Master nodes both run namenode with higher end hardware capacity configuration and use QJM based configuration for high availability – we will talk more about it later Note that QJM and ZK nodes are typical worker nodes running slave worker services except that there is a separate and dedicate disk for these services.
- As part of the operations, we understand the fundamental need for a configuration management tool and the flexibility it provides during the setup and maintenance of a cluster. We were fortunate to have this tools at our disposal for a very long period of time since the beginning of the firm, forming core fundamentals of our infrastructure. These tools help us automate pretty much everything from configuration, deployment and monitoring With the configuration management, it becomes really easy to provision a cluster or machine or service Everything is in source control (git) so there is a change management and flexibility for ‘oops’ times.
- We maintain an internal mirror of upstream git repository with a couple of small patches. The idea is to be able to pick any community patches that are not available in installed version and to be able to build on top of it. Combination of build and publish scripts are used as part of Jenkins and FPM tools. FPM is a package management utility to wrap around building rpms Puppet manifests are then used to deploy the hadoop packages across. It is to be noted that multiple hadoop versions are installed, the directory hierarchy does not conflict or overwrite the previous installations with the new ones. They sit next to each other. Again, puppet manifests come in aid to pick up the more stable version for our needs.
- HDFS High availability setup and configuration Two namenodes are configured to be active and standby using the QJM style configuration. QJM runs on 5 nodes with appropriate level of resources and a dedicated disk Zookeeper dedicated disks Appropriate HA and session timeouts for HA and ZKFC – this is important you do not want to see too much failover and failback activity in the cluster Clients use configuredfailoverproxyprovider class to get redirected to the active namenode
- YARN configuration tended to be simple, we started using the default configuration for most of the properties and they are designed to scale and work anyways. However, some of the resource allocation related properties had to be configured Mapreduce.preserve.failedtasks set this to false otherwise the users’ staging directory will grow and that will take a lot of HDFS space unnecessarily Vmem-check-enabled false; this is after noticing that YARN kills containers even though the physical memory usage is less than the virtual memory usage Other properties include configuring the node manager resources and mapreduce memory settings. The script yarn-utils available in hortonworks’ github is a good reference to start with what are the good values for these properties. Enable Log aggregation so all the user application logs are available via HDFS, do away with maintaining local logs on individual nodes.
- Schedulers work best only when it is tuned according to the workloads in the cluster. Spend time in understandiing the patterns of the workload and their requirements in the cluster before venturing out to tune the scheduler. It helps us to use weighted queues for better SLAs for production jobs ACLs keep prod queues safe from abusing users.
- When there is a fresh data lake and all the while new improved environment, naturally hadoop tends to be happy and that’s what we did We saw that MR in YARN is much better than older world. It has no static slots everything is a container resource in terms of memory and cpu Resource Management is split away from JobTracker, so that makes it lean for arbitration of resources and scheduling among the cluster consumption Application Master for managing the life cycle of the job, launching and managing mapper and reducer containers. Since there is no fixed slots, reducer containers can start after the mapper containers in the same nodemanager, it allows for greater utilization and throughput Job History for MapReduce is again seperated now into a service so that makes it more easy to look at the past jobs and no more hogging the master services Log Aggregation makes application container logs available on HDFS for users and retain for longer periods.
- It was only beginning that mapreduce use case was seeing the benefits With the framework YARN it is, it was also easy to have other beasts in the jungle onboarded onto the platform Majority of the applications on the cluster is MapReduce based Tez -- version 0.4.1 Spark -- version 1.0.1 Storm -- -- storm cluster in still testing with in a same cluster sharing hdfs -- use of slider -- few topologies, together they run 350 odd workers with totall allocation of 1TB of memory -- data processing statistics: -- 1 million event / sec, typical event is of about 3kb. 3GB/sec -- 2% cluster utilization Hbase on YARN -- number of hbase clusters with in single big yarn cluster : 7 prod + 1 hangar -- multi-tenancy : different clusters for different application needs -- 5% cluster utilization -- there is a detailed talk on how we use multi-tenant HBase on single YARN platform tomorrow at 1:30pm. Do Not Miss it!
- Results A glance at the scale of our current YARN cluster. The new BCP cluster currently has more than 1500 nodes although with a humble beginnings of 150 node when setup about 1 and half year ago.
- Some graphs for YARN cluster and queue metrics YARN is still relatively new and evolving, with that in mind, we tend to monitor and measure all things possible. We use a plethora of tools, such as opentsdb and also YARN REST API to collect the information from resource manager and nodemanagers about the running state of applications and containers.
- OpenTSDB is our Doctor Who Tardis! A separate cluster running Hbase 0.96 and opentsdb 2.0 Helps to look at the metrics and unravel the crime stories that happened in the past. Only Past. A look at some of the tidbits that we have for collecting, storing the metrics
- Our preferred tool for alerting
- Hadoop is still not IPv6 friendly. Disable ipv6 on your hosts or you will often run into NoRouteToHost Exceptions Setting readahead using blockdev to about 4MB helped improve the read IO on the slaves Check if your kernel is running stable and without any performance problems Datanode forks du command often to generate disk usage reports and they are found to be stuck process Bug in xfs caused du to stuck, morever reports spurious system activity numbers Bad networking drivers tend to have packet drops resulting in choking network throughput
- On the HDFS side there are many stories as well, one of the story is that datanodes were found often to be running out of xceiver threads. Looking closely we found that there is a consistent leakage of sockets in datanodes. The stack traces of datanodes showed threads were hung in short circuit reads. Disabling short circuit read feature got the datanodes back to sanity.
- On an occasion of network partitions, servers close the socket but the dfs clients do not realize that and tend to get blocked on the reads almost forever. This affects all the systems using dfsclient including resourcemanager/NM/oozie HDFS-7005 provides a patch for setting the peer timeouts in dfsclient code Some of the obscure namenode parameters we enabled on our large cluster is to disable fair fslock, this has simply improved the metadata lookups and the read ops throughput Enable async audit log, do not get the namenode blocked in logging Enable percentile level metrics to get a granular view of the rpc latencies
- 1) Disk failures are quite a norm in a 18 thousands of disks cluster farm Instead of dealing with it, have the tools handy that help in detecting the issues with disks early Also have datanode property to tolerate the max volume failures – do not keep it too high either! 2) Its observed during a maintenance that if the resource manager is down for about 30 minutes or so, all the node managers in the cluster shutdown This results in work loss, better handle this maintenance. There is a patch also available in YARN-3644 3) Some of the things are still fragile in YARN, for example inducing a small typo in mapred-site.xml file on the disk will crash the running nodemanagers Ideally nodemanagers are not supposed to pick up the configs until the restart, but this has proven to be costly especially when deploying the configs on a wide scale in the cluster. Better do a valid syntax checks before deploying the configuration
- Couple of notes on Fair Share preemption: Preemption is meant to kill the containers when running for long and the resources are no more supposed to be available for the container by the scheduler. We found that there is not much visibility into containers when containers getting killed due to preemption were being treated as Failed tasks and eventually failed jobs. MR-5900 fixes that and also YARN-2181 provides visibility into the preemption of the resources to the application manager.
- I would like to call out big efforts of our Big Data team at RocketFuel for collaborating and realizing the true potential of the next generation big data platform. Also I would like to thank Shrijeet and Kishore for their huge contributions in this journey. If you like our work and what we are doing, come join us. We are hiring!!