Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hadoop scheduler

4,620 views

Published on

Hadoop

Published in: Data & Analytics
  • My special guest's 3-Step "No Product Funnel" can be duplicated to start earning a significant income online. ★★★ http://dwz1.cc/G9GauKYg
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Earn $90/day Working Online. You won't get rich, but it is going to make you some money! ■■■ https://tinyurl.com/y4urott2
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Hadoop scheduler

  1. 1. Scheduling
  2. 2. Hadoopas batch processing system •Hadoopwas designed mainly for running large batch jobs such as web indexing and log mining. •Users submitted jobs to a queue, and the cluster ran them in order. •Soon, another use case became attractive: –Sharing a MapReducecluster between multiple users.
  3. 3. The benefits of sharing •With all the data in one place, users can run queries that they may never have been able to execute otherwise, and •Costs go down because system utilization is higher than building a separate Hadoopcluster for each group. •However, sharing requires support from the Hadoopjob scheduler to –provide guaranteed capacity to production jobs and –good response time to interactive jobs while allocating resources fairly between users.
  4. 4. Approaches to Sharing •FIFO : In FIFO scheduling, a JobTrackerpulls jobs from a work queue, oldest job first. This schedule had no concept of the priority or size of the job •Fair : Assign resources to jobs such that on average over time, each job gets an equal share of the available resources. The result is that jobs that require less time are able to access the CPU and finish intermixed with the execution of jobs that require more time to execute. This behavior allows for some interactivity among Hadoopjobs and permits greater responsiveness of the Hadoopcluster to the variety of job types submitted. •Capacity : In capacity scheduling, instead of pools, several queues are created, each with a configurable number of map and reduce slots. Each queue is also assigned a guaranteed capacity (where the overall capacity of the cluster is the sum of each queue's capacity). Queues are monitored; if a queue is not consuming its allocated capacity, this excess capacity can be temporarily allocated to other queues.
  5. 5. FIFO Scheduling Job Queue
  6. 6. FIFO Scheduling Job Queue
  7. 7. FIFO Scheduling Job Queue
  8. 8. Hadoop default scheduler (FIFO) –Problem:short jobs get stuck behind long ones •Separate clusters –Problem 1:poor utilization –Problem 2:costly data replication •Full replication across clusters nearly infeasible at Facebook/Yahoo! scale •Partial replication prevents cross-dataset queries
  9. 9. Fair Scheduling Job Queue
  10. 10. Fair Scheduling Job Queue
  11. 11. Fair Scheduler Basics •Group jobs into “pools” •Assign each pool a guaranteed minimum share •Divide excess capacity evenly between pools
  12. 12. Pools •Determined from a configurable job property –Default in 0.20: user.name (one pool per user) •Pools have properties: –Minimum map slots –Minimum reduce slots –Limit on # of running jobs
  13. 13. Example Pool Allocations entire cluster100 slots emp1 emp2 finance min share = 40 emp6 min share = 30 job 2 15 slots job 3 15 slots job 1 30 slots job 4 40 slots
  14. 14. Scheduling Algorithm •Split each pool’s min share among its jobs •Split each pool’s total share among its jobs •When a slot needs to be assigned: –If there is any job below its min share, schedule it –Else schedule the job that we’ve been most unfair to (based on “deficit”)
  15. 15. Scheduler Dashboard
  16. 16. Scheduler Dashboard Change priority Change pool FIFO mode (for testing)
  17. 17. Additional Features •Weights for unequal sharing: –Job weights based on priority (each level = 2x) –Job weights based on size –Pool weights •Limits for # of running jobs: –Per user –Per pool
  18. 18. Installing the Fair Scheduler •Build it: –ant package •Place it on the classpath: –cp build/contrib/fairscheduler/*.jar lib
  19. 19. Configuration Files •Hadoop config(conf/mapred-site.xml) –Contains scheduler options, pointer to pools file •Pools file (pools.xml) –Contains min share allocations and limits on pools –Reloaded every 15 seconds at runtime
  20. 20. Minimal hadoop-site.xml <property> <name>mapred.jobtracker.taskScheduler</name> <value>org.apache.hadoop.mapred.FairScheduler</ value> </property> <property> <name>mapred.fairscheduler.allocation.file</name> <value>/path/to/pools.xml</value> </property>
  21. 21. Minimal pools.xml <?xml version="1.0"?> <allocations> </allocations>
  22. 22. Configuring a Pool <?xml version="1.0"?> <allocations> <pool name=“emp4"> <minMaps>10</minMaps> <minReduces>5</minReduces> </pool> </allocations>
  23. 23. Setting Running Job Limits <?xml version="1.0"?> <allocations> <pool name=“emp4"> <minMaps>10</minMaps> <minReduces>5</minReduces> <maxRunningJobs>3</maxRunningJobs> </pool> <user name=“emp1"> <maxRunningJobs>1</maxRunningJobs> </user> </allocations>
  24. 24. Default Per-User Running Job Limit <?xml version="1.0"?> <allocations> <pool name=“emp4"> <minMaps>10</minMaps> <minReduces>5</minReduces> <maxRunningJobs>3</maxRunningJobs> </pool> <user name=“emp1"> <maxRunningJobs>1</maxRunningJobs> </user> <userMaxJobsDefault>10</userMaxJobsDefault> </allocations>
  25. 25. Other Parameters mapred.fairscheduler.assignmultiple: •Assign a map and a reduce on each heartbeat; improves ramp-up speed and throughput; recommendation: set to true
  26. 26. Other Parameters mapred.fairscheduler.poolnameproperty: •Which JobConfproperty sets what pool a job is in -Default: user.name (one pool per user) -Can make up your own, e.g. “pool.name”, and pass in JobConfwith conf.set(“pool.name”, “mypool”)
  27. 27. Useful Setting <property> <name>mapred.fairscheduler.poolnameproperty</name> <value>pool.name</value> </property> <property> <name>pool.name</name> <value>${user.name}</value> </property> Make pool.name default to user.name
  28. 28. Issues with Fair Scheduler –Fine-grained sharing at level of map & reduce tasks –Predictable response times and user isolation •Problem:data locality –For efficiency, must run tasks near their input data –Strictly following any job queuing policy hurts locality: job picked by policy may not have data on free nodes •Solution:delay scheduling –Relax queuing policy for limited time to achieve locality
  29. 29. The Problem Job 2 Master Job 1 Scheduling order Slave Slave Slave Slave Slave Slave 4 2 1 1 2 2 3 3 9 5 3 3 6 7 5 6 9 4 8 7 8 2 1 1 Task 2 Task 5 Task 3 Task 1 Task 7 Task 4 File 1: File 2:
  30. 30. The Problem Job 1 Master Job 2 Scheduling order Slave Slave Slave Slave Slave Slave 4 2 1 2 2 3 9 5 3 3 6 7 5 6 9 4 8 7 8 2 1 1 Task 2 Task 5 3 Task 1 File 1: File 2: Task 1 7 Task 2 Task 4 3 Problem: Fair decision hurts locality Especially bad for jobs with small input files 1 3
  31. 31. Solution: Delay Scheduling •Relax queuing policy to make jobs wait for a limited time if they cannot launch local tasks •Result: Very short wait time (1-5s) is enough to get nearly 100% locality
  32. 32. Delay Scheduling Example Job 1 Master Job 2 Scheduling order Slave Slave Slave Slave Slave Slave 4 2 1 1 2 2 3 3 9 5 3 3 6 7 5 6 9 4 8 7 8 2 1 1 Task 2 3 File 1: File 2: Task 8 7 Task 2 Task 4 6 Idea: Wait a short time to get data-local scheduling opportunities 5 Task 1 1 Task 3 Wait!
  33. 33. Delay Scheduling Details •Scan jobs in order given by queuing policy, picking first that is permitted to launch a task •Jobs must wait before being permitted to launch non-local tasks –If wait < T1, only allow node-local tasks –If T1< wait < T2, also allow rack-local –If wait > T2, also allow off-rack •Increase a job’s time waited when it is skipped
  34. 34. Capacity Scheduler •Organizes jobs into queues •Queue shares as %’s of cluster •FIFO scheduling within each queue •Supports preemption •Queues are monitored; if a queue is not consuming its allocated capacity, this excess capacity can be temporarily allocated to other queues.
  35. 35. End of session Day –1: Scheduling

×