This document provides tips for tuning Hadoop clusters and jobs. It recommends: 1) Choosing optimal numbers of mappers and reducers per node and oversubscribing CPUs slightly. 2) Adjusting memory allocations for tasks and ensuring they do not exceed total memory available. 3) Increasing buffers for sorting and shuffling, compressing intermediate data, and using combiners to reduce data sent to reducers.

1. How to Debug and Tune Hadoop Alex Rovner Proclivity Systems

3. Tune Your Cluster

4. Tune Your Cluster Choose optimal number of mappers / reducers per node mapred.tasktracker.map.tasks.maximum

5. mapred.tasktracker.reduce.tasks.maximum

6. Oversubscribe the CPU by 20-30% (8 Cores can generally handle 10 slots)

7. Mappers to reducers ratio 4:3

8. Tune Hadoop Adjust memory allocations mapred.child.javaopts=-Xmx512M

9. Use 80% of available memory

10. Do not oversubscribe memory to avoid swapping

11. Total Memory = Map Slots + Reduce Slots + TT + DN + Other Services + OS

12. Tune Hadoop Increase buffers for sorting and shuffling io.sort.mb & fs.inmemorysize.mb

13. Set to 60-70% of Java heap size

14. Set it large enough to avoid disk spills Compress intermediate data mapred.compress.map.output