Tune hadoop

7,289 views

Published on

Overview for beginners on common tunings and some of the statistics and diagnostics which are exposed allowing you to tune a cluster.

Published in: Technology, Business

Tune hadoop

  1. 1. How to Debug and Tune Hadoop Alex Rovner Proclivity Systems
  2. 3. Tune Your Cluster
  3. 4. Tune Your Cluster <ul><li>Choose optimal number of mappers / reducers per node </li><ul><li>mapred.tasktracker.map.tasks.maximum
  4. 5. mapred.tasktracker.reduce.tasks.maximum
  5. 6. Oversubscribe the CPU by 20-30% (8 Cores can generally handle 10 slots)
  6. 7. Mappers to reducers ratio 4:3 </li></ul></ul>
  7. 8. Tune Hadoop <ul><li>Adjust memory allocations </li><ul><li>mapred.child.javaopts=-Xmx512M
  8. 9. Use 80% of available memory
  9. 10. Do not oversubscribe memory to avoid swapping
  10. 11. Total Memory = Map Slots + Reduce Slots + TT + DN + Other Services + OS </li></ul></ul>
  11. 12. Tune Hadoop <ul><li>Increase buffers for sorting and shuffling </li><ul><li>io.sort.mb & fs.inmemorysize.mb
  12. 13. Set to 60-70% of Java heap size
  13. 14. Set it large enough to avoid disk spills </li></ul><li>Compress intermediate data </li><ul><li>mapred.compress.map.output
  14. 15. Install native libraries for performance
  15. 16. Use LZO to minimize CPU cycles
  16. 17. Set compression to use BLOCK compression </li></ul></ul>
  17. 18. Tune Your Job <ul><li>Use Combiner where possible! </li><ul><li>Combiner is a mini reduce phase on the map side
  18. 19. Reduces the amount of data sent to the reducers
  19. 20. Does not need to be the same class as the reducer </li></ul></ul>
  20. 21. Tune Your Job <ul><li>Setup an appropriate number of reducers </li><ul><li>Check job stats to figure out how many reducers are needed
  21. 22. Map output bytes will drive how many reducers you need
  22. 23. Rule of thumb is 1 Gig per reducer </li></ul></ul>
  23. 24. Tune Your Job
  24. 25. Tune Your Job 7.4GB = 7 Reducers
  25. 26. Tune Your Job
  26. 27. Tune Your Job
  27. 28. Tune Your Job
  28. 29. Demo

×