SlideShare a Scribd company logo
1 of 19
Download to read offline
Apache Yarn Upgrade
● Example upgrade
● From V1 -> Yarn
● Environment
● Approach
● Install steps
● Install check
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Yarn Upgrade Environment
● Java OpenJDK 1.6.0_27
● Ubuntu 12.04
● Maven 3.0.4
● Hadoop 1.2.0
● Mahout 0.9
● Hadoop to install
– 2.0.6-alpha
Full details are available from our web site site
under guides folder
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Yarn Upgrade Approach
● Install along side existing Hadoop on all nodes
● Use existing hdfs
● Change cfg files on all nodes
● Set up as single nodes and test via mapreduce
● Create cluster and test via mapreduce
● Check web GUI access
Full details are available from our web site site
under guides folder
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Yarn Upgrade Install
● Build with Maven into a distribution directory
mvn clean package -Pdist -Dtar -DskipTests -Pnative
release created under ./hadoop-dist/target/hadoop-2.0.6-alpha
● Only skip tests after first build to speed things up
● Configure $HOME/.bashrc
– HADOOP_COMMON_HOME
– HADOOP_HDFS_HOME
– HADOOP_MAPRED_HOME
– HADOOP_YARN_HOME
– HADOOP_CONF_DIR
– YARN_CONF_DIR
– MAPRED_CONF_DIR
– HADOOP_PREFIX
– PATH
– YARN_CLASSPATH
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Yarn Upgrade Install
● Set up core-site.xml
cd $HADOOP_COMMON_HOME/etc/hadoop
● Alter values for
– fs.default.name
– hadoop.tmp.dir
– fs.checkpoint.dir
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Yarn Upgrade Install
● Set up hdfs-site.xml
cd $HADOOP_HDFS_HOME/etc/hadoop
● Alter values for
– dfs.name.dir
– dfs.data.dir
– dfs.http.address
– dfs.secondary.http.address
– dfs.https.address
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Yarn Upgrade Install
● Set up yarn-site.xml
cd $YARN_CONF_DIR
● Alter values for
– yarn.resourcemanager.resource-tracker.address
– yarn.resourcemanager.scheduler.address
– yarn.resourcemanager.scheduler.class
– yarn.resourcemanager.address
– yarn.nodemanager.local-dirs
– yarn.nodemanager.address
– yarn.nodemanager.resource.memory-mb
– yarn.nodemanager.remote-app-log-dir
– yarn.nodemanager.log-dirs
– yarn.nodemanager.aux-services
– yarn.web-proxy.address
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Yarn Upgrade Install
● Set up mapred-site.xml
cd $MAPRED_CONF_DIR
● Alter values for
– mapreduce.cluster.temp.dir
– mapreduce.cluster.local.dir
– mapreduce.jobhistory.address
– mapreduce.jobhistory.webapp.address
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Yarn Upgrade Install
● Set up capcity-scheduler.xml
cd $HADOOP_YARN_HOME/etc/hadoop
● Alter values for
– yarn.scheduler.capacity.maximum-applications
– yarn.scheduler.capacity.maximum-am-resource-percent
– yarn.scheduler.capacity.resource-calculator
– yarn.scheduler.capacity.root.queues
– yarn.scheduler.capacity.child.queues
– yarn.scheduler.capacity.child.unfunded.capacity
– yarn.scheduler.capacity.child.default.capacity
– yarn.scheduler.capacity.root.capacity
– yarn.scheduler.capacity.root.unfunded.capacity
– yarn.scheduler.capacity.root.default.capacity
– yarn.scheduler.capacity.root.default.user-limit-factor
– yarn.scheduler.capacity.root.default.maximum-capacity
– yarn.scheduler.capacity.root.default.state
– yarn.scheduler.capacity.root.default.acl_submit_applications
– yarn.scheduler.capacity.root.default.acl_administer_queue
– yarn.scheduler.capacity.node-locality-delay
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Yarn Upgrade Install
● Start Resource Manager
cd $HADOOP_YARN_HOME
sbin/yarn-deamon.sh start resourcemanager
● Start Node Manager
cd $HADOOP_YARN_HOME
sbin/yarn-deamon.sh start ndemanager
● Test via map reduce job
cd $HADOOP_MAPRED_HOME/share/hadoop/mapreduce
$HADOOP_COMMON_HOME/bin/hadoop jar 
hadoop-mapreduce-examples-2.0.6-alpha.jar randomwriter out
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Yarn Upgrade Install
● Mapreduce job should end with
BYTES_WRITTEN=1073750341
RECORDS_WRITTEN=102099
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=1085699265
Job ended: Sun Aug 25 12:45:35 NZST 2013
The job took 89 seconds.
● Run this test on each node being upgraded
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Yarn Upgrade Install
● Stop the servers
cd $HADOOP_YARN_HOME
sbin/yarn-daemon.sh stop resourcemanager
stopping resourcemanager
sbin/yarn-daemon.sh stop nodemanager
stopping nodemanager
● Alter Hadoop env
cd $HADOOP_CONF_DIR
vi hadoop-env.sh
add a JAVA_HOME definition at the end. i.e.
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-i386
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Yarn Upgrade Install
● Alter $HADOOP_CONF_DIR/slaves file
– Add details ( one per line ) for slave nodes
● Format the cluster
– DONT have the cluster running else you will lose data
– hdfs namenode -format
● Now proceed to start the cluster
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Yarn Upgrade Install
cd $HADOOP_COMMON_HOME
sbin/hadoop-daemon.sh --config $HADOOP_COMMON_HOME/etc/hadoop
--script hdfs start namenode
cd $HADOOP_COMMON_HOME
sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode
cd $HADOOP_YARN_HOME
sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager
cd $HADOOP_YARN_HOME
sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager
cd $HADOOP_YARN_HOME
bin/yarn start proxyserver --config $HADOOP_CONF_DIR
cd $HADOOP_MAPRED_HOME
sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Yarn Upgrade Install
● Use jps to check servers running
jps
5856 DataNode
6434 Jps
5776 NameNode
6181 NodeManager
6255 WebAppProxyServer
5927 ResourceManager
6352 JobHistoryServer
● Then run the same mapreduce job on the cluster
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Web Access
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Web Access
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Web Access
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
Contact Us
● Feel free to contact us at
– www.semtech-solutions.co.nz
– info@semtech-solutions.co.nz
● We offer IT project consultancy
● We are happy to hear about your problems
● You can just pay for those hours that you need
● To solve your problems

More Related Content

More from Mike Frampton

An introduction to Apache Mesos
An introduction to Apache MesosAn introduction to Apache Mesos
An introduction to Apache MesosMike Frampton
 
An introduction to Pentaho
An introduction to PentahoAn introduction to Pentaho
An introduction to PentahoMike Frampton
 
An introduction to Apache Thrift
An introduction to Apache ThriftAn introduction to Apache Thrift
An introduction to Apache ThriftMike Frampton
 
An introduction to Apache Cassandra
An introduction to Apache CassandraAn introduction to Apache Cassandra
An introduction to Apache CassandraMike Frampton
 
An example Hadoop Install
An example Hadoop InstallAn example Hadoop Install
An example Hadoop InstallMike Frampton
 
An Introduction to Apache Hadoop Yarn
An Introduction to Apache Hadoop YarnAn Introduction to Apache Hadoop Yarn
An Introduction to Apache Hadoop YarnMike Frampton
 
An Introduction to Cloud Computing
An Introduction to Cloud ComputingAn Introduction to Cloud Computing
An Introduction to Cloud ComputingMike Frampton
 
An Introduction to Hadoop Hue Gui
An Introduction to Hadoop Hue GuiAn Introduction to Hadoop Hue Gui
An Introduction to Hadoop Hue GuiMike Frampton
 
An introduction to Apache Hadoop Hive
An introduction to Apache Hadoop HiveAn introduction to Apache Hadoop Hive
An introduction to Apache Hadoop HiveMike Frampton
 

More from Mike Frampton (20)

Prometheus
PrometheusPrometheus
Prometheus
 
Apache Tephra
Apache TephraApache Tephra
Apache Tephra
 
Apache Kudu
Apache KuduApache Kudu
Apache Kudu
 
Apache Bahir
Apache BahirApache Bahir
Apache Bahir
 
Apache Arrow
Apache ArrowApache Arrow
Apache Arrow
 
JanusGraph DB
JanusGraph DBJanusGraph DB
JanusGraph DB
 
Apache Ignite
Apache IgniteApache Ignite
Apache Ignite
 
Apache Samza
Apache SamzaApache Samza
Apache Samza
 
Apache Flink
Apache FlinkApache Flink
Apache Flink
 
Apache Edgent
Apache EdgentApache Edgent
Apache Edgent
 
Apache CouchDB
Apache CouchDBApache CouchDB
Apache CouchDB
 
An introduction to Apache Mesos
An introduction to Apache MesosAn introduction to Apache Mesos
An introduction to Apache Mesos
 
An introduction to Pentaho
An introduction to PentahoAn introduction to Pentaho
An introduction to Pentaho
 
An introduction to Apache Thrift
An introduction to Apache ThriftAn introduction to Apache Thrift
An introduction to Apache Thrift
 
An introduction to Apache Cassandra
An introduction to Apache CassandraAn introduction to Apache Cassandra
An introduction to Apache Cassandra
 
An example Hadoop Install
An example Hadoop InstallAn example Hadoop Install
An example Hadoop Install
 
An Introduction to Apache Hadoop Yarn
An Introduction to Apache Hadoop YarnAn Introduction to Apache Hadoop Yarn
An Introduction to Apache Hadoop Yarn
 
An Introduction to Cloud Computing
An Introduction to Cloud ComputingAn Introduction to Cloud Computing
An Introduction to Cloud Computing
 
An Introduction to Hadoop Hue Gui
An Introduction to Hadoop Hue GuiAn Introduction to Hadoop Hue Gui
An Introduction to Hadoop Hue Gui
 
An introduction to Apache Hadoop Hive
An introduction to Apache Hadoop HiveAn introduction to Apache Hadoop Hive
An introduction to Apache Hadoop Hive
 

Recently uploaded

Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInThousandEyes
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdfThe Good Food Institute
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingFrancesco Corti
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1DianaGray10
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTxtailishbaloch
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Alkin Tezuysal
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxNeo4j
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNeo4j
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxSatishbabu Gunukula
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2DianaGray10
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxNeo4j
 
Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)Muhammad Tiham Siddiqui
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4DianaGray10
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxKaustubhBhavsar6
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechProduct School
 

Recently uploaded (20)

Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is going
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptx
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
 
Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
 

An example Apache Hadoop Yarn upgrade

  • 1. Apache Yarn Upgrade ● Example upgrade ● From V1 -> Yarn ● Environment ● Approach ● Install steps ● Install check www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 2. Yarn Upgrade Environment ● Java OpenJDK 1.6.0_27 ● Ubuntu 12.04 ● Maven 3.0.4 ● Hadoop 1.2.0 ● Mahout 0.9 ● Hadoop to install – 2.0.6-alpha Full details are available from our web site site under guides folder www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 3. Yarn Upgrade Approach ● Install along side existing Hadoop on all nodes ● Use existing hdfs ● Change cfg files on all nodes ● Set up as single nodes and test via mapreduce ● Create cluster and test via mapreduce ● Check web GUI access Full details are available from our web site site under guides folder www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 4. Yarn Upgrade Install ● Build with Maven into a distribution directory mvn clean package -Pdist -Dtar -DskipTests -Pnative release created under ./hadoop-dist/target/hadoop-2.0.6-alpha ● Only skip tests after first build to speed things up ● Configure $HOME/.bashrc – HADOOP_COMMON_HOME – HADOOP_HDFS_HOME – HADOOP_MAPRED_HOME – HADOOP_YARN_HOME – HADOOP_CONF_DIR – YARN_CONF_DIR – MAPRED_CONF_DIR – HADOOP_PREFIX – PATH – YARN_CLASSPATH www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 5. Yarn Upgrade Install ● Set up core-site.xml cd $HADOOP_COMMON_HOME/etc/hadoop ● Alter values for – fs.default.name – hadoop.tmp.dir – fs.checkpoint.dir www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 6. Yarn Upgrade Install ● Set up hdfs-site.xml cd $HADOOP_HDFS_HOME/etc/hadoop ● Alter values for – dfs.name.dir – dfs.data.dir – dfs.http.address – dfs.secondary.http.address – dfs.https.address www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 7. Yarn Upgrade Install ● Set up yarn-site.xml cd $YARN_CONF_DIR ● Alter values for – yarn.resourcemanager.resource-tracker.address – yarn.resourcemanager.scheduler.address – yarn.resourcemanager.scheduler.class – yarn.resourcemanager.address – yarn.nodemanager.local-dirs – yarn.nodemanager.address – yarn.nodemanager.resource.memory-mb – yarn.nodemanager.remote-app-log-dir – yarn.nodemanager.log-dirs – yarn.nodemanager.aux-services – yarn.web-proxy.address www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 8. Yarn Upgrade Install ● Set up mapred-site.xml cd $MAPRED_CONF_DIR ● Alter values for – mapreduce.cluster.temp.dir – mapreduce.cluster.local.dir – mapreduce.jobhistory.address – mapreduce.jobhistory.webapp.address www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 9. Yarn Upgrade Install ● Set up capcity-scheduler.xml cd $HADOOP_YARN_HOME/etc/hadoop ● Alter values for – yarn.scheduler.capacity.maximum-applications – yarn.scheduler.capacity.maximum-am-resource-percent – yarn.scheduler.capacity.resource-calculator – yarn.scheduler.capacity.root.queues – yarn.scheduler.capacity.child.queues – yarn.scheduler.capacity.child.unfunded.capacity – yarn.scheduler.capacity.child.default.capacity – yarn.scheduler.capacity.root.capacity – yarn.scheduler.capacity.root.unfunded.capacity – yarn.scheduler.capacity.root.default.capacity – yarn.scheduler.capacity.root.default.user-limit-factor – yarn.scheduler.capacity.root.default.maximum-capacity – yarn.scheduler.capacity.root.default.state – yarn.scheduler.capacity.root.default.acl_submit_applications – yarn.scheduler.capacity.root.default.acl_administer_queue – yarn.scheduler.capacity.node-locality-delay www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 10. Yarn Upgrade Install ● Start Resource Manager cd $HADOOP_YARN_HOME sbin/yarn-deamon.sh start resourcemanager ● Start Node Manager cd $HADOOP_YARN_HOME sbin/yarn-deamon.sh start ndemanager ● Test via map reduce job cd $HADOOP_MAPRED_HOME/share/hadoop/mapreduce $HADOOP_COMMON_HOME/bin/hadoop jar hadoop-mapreduce-examples-2.0.6-alpha.jar randomwriter out www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 11. Yarn Upgrade Install ● Mapreduce job should end with BYTES_WRITTEN=1073750341 RECORDS_WRITTEN=102099 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=1085699265 Job ended: Sun Aug 25 12:45:35 NZST 2013 The job took 89 seconds. ● Run this test on each node being upgraded www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 12. Yarn Upgrade Install ● Stop the servers cd $HADOOP_YARN_HOME sbin/yarn-daemon.sh stop resourcemanager stopping resourcemanager sbin/yarn-daemon.sh stop nodemanager stopping nodemanager ● Alter Hadoop env cd $HADOOP_CONF_DIR vi hadoop-env.sh add a JAVA_HOME definition at the end. i.e. export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-i386 www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 13. Yarn Upgrade Install ● Alter $HADOOP_CONF_DIR/slaves file – Add details ( one per line ) for slave nodes ● Format the cluster – DONT have the cluster running else you will lose data – hdfs namenode -format ● Now proceed to start the cluster www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 14. Yarn Upgrade Install cd $HADOOP_COMMON_HOME sbin/hadoop-daemon.sh --config $HADOOP_COMMON_HOME/etc/hadoop --script hdfs start namenode cd $HADOOP_COMMON_HOME sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode cd $HADOOP_YARN_HOME sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager cd $HADOOP_YARN_HOME sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager cd $HADOOP_YARN_HOME bin/yarn start proxyserver --config $HADOOP_CONF_DIR cd $HADOOP_MAPRED_HOME sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 15. Yarn Upgrade Install ● Use jps to check servers running jps 5856 DataNode 6434 Jps 5776 NameNode 6181 NodeManager 6255 WebAppProxyServer 5927 ResourceManager 6352 JobHistoryServer ● Then run the same mapreduce job on the cluster www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  • 19. Contact Us ● Feel free to contact us at – www.semtech-solutions.co.nz – info@semtech-solutions.co.nz ● We offer IT project consultancy ● We are happy to hear about your problems ● You can just pay for those hours that you need ● To solve your problems