Tune hadoop

How to Debug and Tune Hadoop Alex Rovner Proclivity Systems

Tune Your Cluster

Tune Your Cluster
Choose optimal number of mappers / reducers per node
mapred.tasktracker.map.tasks.maximum
mapred.tasktracker.reduce.tasks.maximum
Oversubscribe the CPU by 20-30% (8 Cores can generally handle 10 slots)
Mappers to reducers ratio 4:3

Tune Hadoop
Adjust memory allocations
mapred.child.javaopts=-Xmx512M
Use 80% of available memory
Do not oversubscribe memory to avoid swapping
Total Memory = Map Slots + Reduce Slots + TT + DN + Other Services + OS

Tune Hadoop
Increase buffers for sorting and shuffling
io.sort.mb & fs.inmemorysize.mb
Set to 60-70% of Java heap size
Set it large enough to avoid disk spills
Compress intermediate data
mapred.compress.map.output
Install native libraries for performance
Use LZO to minimize CPU cycles
Set compression to use BLOCK compression

Tune Your Job
Use Combiner where possible!
Combiner is a mini reduce phase on the map side
Reduces the amount of data sent to the reducers
Does not need to be the same class as the reducer

Tune Your Job
Setup an appropriate number of reducers
Check job stats to figure out how many reducers are needed
Map output bytes will drive how many reducers you need
Rule of thumb is 1 Gig per reducer

Tune Your Job

Tune Your Job 7.4GB = 7 Reducers

Tune Your Job

Tune hadoop

by Jason Shao on Jan 03, 2012

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Tune hadoop — Presentation Transcript