Advanced Hadoop Tuning and Optimization - Hadoop Consulting
Report
Impetus TechnologiesFollow
Dec. 1, 2009•0 likes•23,399 views
107 likes
Be the first to like this
Show More
views
Total views
0
On Slideshare
0
From embeds
0
Number of embeds
0
Advanced Hadoop Tuning and Optimization - Hadoop Consulting
Dec. 1, 2009•0 likes•23,399 views
107 likes
Be the first to like this
Show More
views
Total views
0
On Slideshare
0
From embeds
0
Number of embeds
0
Download to read offline
Report
Technology
Impetus provides expert consulting services around Hadoop implementations, including R&D, assessment, deployment (on private and public clouds), optimizations for enhanced static shared data implementations. This presentation speaks about Advanced Hadoop Tuning and Optimisation.
Impetus TechnologiesFollow
Recommended
Hadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your ApplicationYahoo Developer Network
Optimizing MapReduce Job performanceDataWorks Summit
Hadoop Summit 2012 | Optimizing MapReduce Job PerformanceCloudera, Inc.
How to Increase Performance of Your Hadoop ClusterAltoros
Advanced Hadoop Tuning and Optimization Shivkumar Babshetty
Hw09 Monitoring Best PracticesCloudera, Inc.
Advanced Hadoop Tuning and Optimization - Hadoop Consulting
- Presented By: Sanjay Sharma Advanced Hadoop Tuning and Optimizations Download the Whitepaper: Deriving Intelligence from Large Data Using Hadoop and Applying Analytics at http://bit.ly/cNCCGj
- Configuration parameters
- Other hotspots
Editor's Notes
- e.g. a node has 8GB main memory + 8 core CPU + swap space Maximum memory required by a task ~ 500MB Memory required by Tasktracker, Datanode and other processes ~ (1 + 1 +1) = 3GB Maximum tasks that can be run = (8-3) GB/500MB = 10 Number of map or reduce task (out of the maximum tasks) can be decided on the basis of memory usage and computation complexities of the tasks. The memory available to each task (JVM) is controlled by mapred.child.java.opts property. The default is –Xmx200m (200 MB). Other JVM options can also be provided in this property.