Advanced Hadoop Tuning and Optimization - Hadoop Consulting


Published on

Impetus provides expert consulting services around Hadoop implementations, including R&D, assessment, deployment (on private and public clouds), optimizations for enhanced static shared data implementations.

This presentation speaks about Advanced Hadoop Tuning and Optimisation.

Published in: Technology
  • Before any tuning, you need to find out what is going on with your cluster. Programmers from Etsy developed a StatsD JVM Profiler which is really cool. Here is their article and this is my instruction how to install it:
    Are you sure you want to  Yes  No
    Your message goes here
  • how to download
    Are you sure you want to  Yes  No
    Your message goes here
  • hadoop tunning
    Are you sure you want to  Yes  No
    Your message goes here
  • Practical and to the point
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • e.g. a node has 8GB main memory + 8 core CPU + swap space Maximum memory required by a task ~ 500MB Memory required by Tasktracker, Datanode and other processes ~ (1 + 1 +1) = 3GB Maximum tasks that can be run = (8-3) GB/500MB = 10 Number of map or reduce task (out of the maximum tasks) can be decided on the basis of memory usage and computation complexities of the tasks. The memory available to each task (JVM) is controlled by property. The default is –Xmx200m (200 MB). Other JVM options can also be provided in this property.
  • Advanced Hadoop Tuning and Optimization - Hadoop Consulting

    1. 1. Presented By: Sanjay Sharma Advanced Hadoop Tuning and Optimizations
    2. 2. Hadoop- The Good/Bad/Ugly <ul><li>Hadoop is GOOD- that is why we all are here </li></ul><ul><li>Hadoop is not BAD- else we would not be here </li></ul><ul><li>Hadoop is sometimes Ugly- why? </li></ul><ul><ul><li>Out of the box configuration not friendly </li></ul></ul><ul><ul><li>Difficult to debug </li></ul></ul><ul><ul><li>Performance – tuning/optimizations is a black art </li></ul></ul>
    3. 3. Configuration parameters
    4. 4. Compression <ul><li> Map Output Compression </li></ul><ul><li>Default: False </li></ul><ul><li>Pros: Faster disk writes, lower disk space usage, lesser time spent on data transfer (from mappers to reducers). </li></ul><ul><li>Cons: Overhead in compression at Mappers and decompression at Reducers. </li></ul><ul><li>Suggestions: For large cluster and large jobs this property should be set true. The compression codec can also be set through the property (Default is </li></ul>
    5. 5. Speculative Execution <ul><li> Enable/Disable task (map/reduce) speculative Execution </li></ul><ul><li>Default: True </li></ul><ul><li>Pros: Reduces the job time if the task progress is slow due to memory unavailability or hardware degradation. </li></ul><ul><li>Cons: Increases the job time if the task progress is slow due to complex and large calculations. On a busy cluster speculative execution can reduce overall throughput, since redundant tasks are being executed in an attempt to bring down the execution time for a single job. </li></ul><ul><li>Suggestions: In large jobs where average task completion time is significant (> 1 hr) due to complex and large calculations and high throughput is required the speculative execution should be set to false. </li></ul>
    6. 6. Number of Maps/Reducers <ul><li> Maximum tasks (map/reduce) for a tasktracker </li></ul><ul><li>Default: 2 </li></ul><ul><li>Suggestions: </li></ul><ul><ul><li>Recommended range - (cores_per_node)/2 to 2x(cores_per_node), especially for large clusters. </li></ul></ul><ul><ul><li>This value should be set according to the hardware specification of cluster nodes and resource requirements of tasks (map/reduce). </li></ul></ul>
    7. 7. File block size <ul><li>dfs.block.size: File system block size </li></ul><ul><li>Default : 67108864 (bytes) </li></ul><ul><li>Suggestions : </li></ul><ul><ul><li>Small cluster and large data set: default block size will create a large number of map tasks. </li></ul></ul><ul><ul><ul><li>e.g. Input data size = 160 GB and dfs.block.size = 64 MB then the minimum no. of maps= (160*1024)/64 = 2560 maps. </li></ul></ul></ul><ul><ul><ul><li>If dfs.block.size = 128 MB minimum no. of maps= (160*1024)/128 = 1280 maps. </li></ul></ul></ul><ul><ul><ul><li>If dfs.block.size = 256 MB minimum no. of maps= (160*1024)/256 = 640 maps. </li></ul></ul></ul><ul><ul><li>  In a small cluster (6-10 nodes) the map task creation overhead is considerable. So dfs.block.size should be large in this case but small enough to utilize all the cluster resources. </li></ul></ul><ul><ul><li>  The block size should be set according to size of the cluster, map task complexity, map task capacity of cluster and average size of input files. </li></ul></ul>
    8. 8. Sort size <ul><li>io.sort.mb: Buffer size (MBs) for sorting </li></ul><ul><li>Default: 100 </li></ul><ul><li>Suggestions: </li></ul><ul><ul><li>For Large jobs (the jobs in which map output is very large), this value should be increased keeping in mind that it will increase the memory required by each map task. So the increment in this value should be according to the available memory at the node. </li></ul></ul><ul><ul><li>Greater the value of io.sort.mb , lesser will be the spills to the disk, saving write to the disk. </li></ul></ul>
    9. 9. Sort factor <ul><li>io.sort.factor: Stream merge factor </li></ul><ul><li>Default: 10 </li></ul><ul><li>Suggestions: </li></ul><ul><ul><li>For Large jobs (the jobs in which map output is very large and number of maps are also large) which have large number of spills to disk, value of this property should be increased. </li></ul></ul><ul><ul><li>The number of input streams (files) to be merged at once in the map/reduce tasks, as specified by io.sort.factor, should be set to a sufficiently large value (for example, 100) to minimize disk accesses. </li></ul></ul><ul><ul><li>Increment in io.sort.factor, benefits in merging at reducers since the last batch of streams (equal to io.sort.factor) are sent to the reduce function without merging, thus saving time in merging. </li></ul></ul>
    10. 10. JVM reuse <ul><li>mapred.job.reuse.jvm.num.tasks: Reuse single JVM </li></ul><ul><li>Default: 1 </li></ul><ul><li>Suggestions: The minimum overhead of JVM creation for each task is around 1 second. So for the tasks which live for seconds or a few minutes and have lengthy initialization, this value can be increased to gain performance. </li></ul>
    11. 11. Reduce parallel copies <ul><li>mapred.reduce.parallel.copies: Threads for parallel copy at reducer </li></ul><ul><li>Default: 5 </li></ul><ul><li>Description: The number of threads used to copy map outputs to the reducer. </li></ul><ul><li>Suggestions: For Large jobs (the jobs in which map output is very large), value of this property can be increased keeping in mind that it will increase the total CPU usage. </li></ul>
    12. 12. The Other Threads <ul><ul><li>dfs.namenode{/mapred.job.tracker}.handler.count : server threads that handle remote procedure calls (RPCs) </li></ul></ul><ul><li>Default: 10 </li></ul><ul><li>Suggestions: This can be increased for larger server (50-64). </li></ul><ul><ul><li>dfs.datanode.handler.count : server threads that handle remote procedure calls (RPCs) </li></ul></ul><ul><li>Default: 3 </li></ul><ul><li>Suggestions: This can be increased for larger number of HDFS clients (6-8). </li></ul><ul><ul><li>tasktracker.http.threads : number of worker threads on the HTTP server on each TaskTracker </li></ul></ul><ul><li>Default: 40 </li></ul><ul><li>Suggestions: The can be increased for larger clusters (50). </li></ul>
    13. 13. Other hotspots
    14. 14. Revelation-Temporary space <ul><li>Temporary space allocation: </li></ul><ul><li>Jobs which generate large intermediate data (map output) should have enough temporary space controlled by property mapred.local.dir. This property specifies list directories where the MapReduce stores intermediate data for jobs. The data is cleaned-up after the job completes. </li></ul><ul><li>By default, replication factor for file storage on HDFS is 3, which means that every file has three replicas. As a rule of thumb, at least 25% of the total hard disk should be allocated for intermediate temporary output. So effectively, only ¼ hard disk space is available for business use. </li></ul><ul><li>The default value for mapred.local.dir is ${hadoop.tmp.dir}/mapred/local. So if mapred.local.dir is not set, hadoop.tmp.dir must have enough space to hold job’s intermediate data. If the node doesn’t have enough temporary space the task attempt will fail and starts a new attempt, thus impacting the performance. </li></ul>
    15. 15. Java- JVM <ul><li>JVM tuning: </li></ul><ul><li>Besides normal java code optimizations, JVM settings for each child task also affects the processing time. On slave node end, the task tracker and data node use 1 GB RAM each. Effective use of the remaining RAM as well as choosing the right GC mechanism for each Map or Reduce task is very important for maximum utilization of hardware resources. The default max RAM for child tasks is 200MB which might be insufficient for many production grade jobs. The JVM settings for child tasks are governed by property. </li></ul><ul><li>Use JDK 1.6 64 BIT– </li></ul><ul><ul><li>+ +XX:CompressedOops helpful in dealing with OOM errors </li></ul></ul><ul><li>Do remember changing Linux open file descriptor </li></ul><ul><li>Set set to true, to avoid timeouts in cases where the OS/JVM picks up an IPv6 address and must resolve the hostname. </li></ul>
    16. 16. Logging <ul><li>Is a friend to developers, Foe in production </li></ul><ul><ul><li>Default - INFO level </li></ul></ul><ul><ul><ul><li>dfs.namenode.logging.level </li></ul></ul></ul><ul><ul><ul><li>hadoop.job.history </li></ul></ul></ul><ul><ul><ul><li>hadoop.logfile.size/count </li></ul></ul></ul>
    17. 17. Static Data strategies <ul><li>Available Approaches </li></ul><ul><ul><li>JobConf.set(“key”,”value”) </li></ul></ul><ul><ul><li>Distributed cache </li></ul></ul><ul><ul><li>HDFS shared file </li></ul></ul><ul><li>Suggested approaches if above ones not efficient </li></ul><ul><ul><li>Memcached </li></ul></ul><ul><ul><li>Tokyocabinet/TokyoTyrant </li></ul></ul><ul><ul><li>Berkley DB </li></ul></ul><ul><ul><li>HBase </li></ul></ul>
    18. 18. Debugging and profiling - Arun C Murthy <ul><li>Hadoop Map-Reduce – Tuning and Debugging- from Arun C Murthy presentation </li></ul><ul><li>Debugging </li></ul><ul><ul><li>Log files/UI view </li></ul></ul><ul><ul><li>Local runner </li></ul></ul><ul><ul><li>Single machine mode </li></ul></ul><ul><ul><li>Set keep.failed.task.files to true and use the IsolationRunner </li></ul></ul><ul><li>Profiling </li></ul><ul><ul><li>Set mapred.task.profile to true </li></ul></ul><ul><ul><li>Use mapred.task.profile.{maps|reduces} </li></ul></ul><ul><ul><li>hprof support is built-in </li></ul></ul><ul><ul><li>Use mapred.task.profile.params to set options for the debugger </li></ul></ul><ul><ul><li>Possibly DistributedCache for the profiler’s agent </li></ul></ul>
    19. 19. Tuning - Arun C Murthy <ul><li>Hadoop Map-Reduce – Tuning and Debugging- from Arun C Murthy presentation </li></ul><ul><li>Tuning </li></ul><ul><ul><li>Tell HDFS and Map-Reduce about your network! – Rack locality script: </li></ul></ul><ul><ul><li>Number of maps – Data locality </li></ul></ul><ul><ul><li>Number of reduces – You don’t need a single output file!Log files/UI view </li></ul></ul><ul><ul><li>Amount of data processed per Map - Consider fatter maps, Custom input format </li></ul></ul><ul><ul><li>Combiner - multi-level combiners at both Map and Reduce </li></ul></ul><ul><ul><li>Check to ensure the combiner is useful! </li></ul></ul><ul><ul><li>Map-side sort -io.sort.mb, io.sort.factor, io.sort.record.percent, io.sort.spill.percent </li></ul></ul><ul><ul><li>Shuffle </li></ul></ul><ul><ul><ul><li>Compression for map-outputs – , , lzo via, tasktracker.http.threads </li></ul></ul></ul><ul><ul><ul><li>mapred.reduce.parallel.copies, mapred.reduce.copy.backoff, mapred.job.shuffle.input.buffer.percent, mapred.job.shuffle.merge.percent, mapred.inmem.merge.threshold, mapred.job.reduce.input.buffer.percent </li></ul></ul></ul><ul><ul><li>Compress the job output </li></ul></ul><ul><ul><li>Miscellaneous -Speculative execution, Heap size for the child, Re-use jvm for maps/reduces, Raw Comparators </li></ul></ul>
    20. 20. Next steps <ul><li>Hadoop Vaidya (since 0.20.0) </li></ul><ul><li>Job configuration analyzer (WIP-to be contributed back to Hadoop) </li></ul><ul><ul><li>Part of Analyze Job web ui </li></ul></ul><ul><ul><li>Analyze and suggest config parameters from job.xml </li></ul></ul><ul><ul><li>Smart suggestion engine/auto-correction </li></ul></ul>
    21. 21. Conclusion <ul><li>Performance of Hadoop MapReduce jobs can be improved without increasing the hardware costs, by tuning several key configuration parameters for cluster specifications, input data size and processing complexity. </li></ul>
    22. 22. References <ul><li> </li></ul><ul><li>Hadoop-performance tuning--white paper v1 1.pdf – Arun C Murthy </li></ul><ul><li>Intel_White_Paper_Optimizing_Hadoop_Deployments.pdf </li></ul>