Hadoop Lab 2 - MapReduce

Lab 2 –MapReduce and the Web ConsoleLab objectivesIn this lab you will practice what you have learned in this lesson, specifically you will practice withMapReduce jobs and learn about the BigInsights Web Console.Lab instructionsThis lab has been developed as a tutorial. Simply execute the commands provided, and analyze theresults.BigInsights should be already started before working on this lab (Refer to “Lab 0 – Setup” forinstructions on starting BigInsights)What is the WordCount job?WordCount is an example on Hadoop MapReduce that is included with the Apache open sourcedocumentation.Running the WordCount job 1. First we need to copy the data files to HDFS: > hadoop fs -put /BigDataUniversity/input/statsFed/ /input 2. Now we can run the wordcount job with the following command, where “/input” is where the input files are, and “output” is the directory where the output of the job will be stored. > hadoop jar /opt/ibm/biginsights/IHC/hadoop-0.20.2-examples.jar wordcount /input output 1©2011 BigDataUniversity.com

Note:If you are working on the IBM SmartCloud Enterprise, wordcount is in another path, so execute thiscommand instead:> hadoop jar /mnt/biginsights/opt/ibm/biginsights/IHC/hadoop-0.20.2-examples.jar wordcount /input output … 3. Now review the output of step 2: 2©2011 BigDataUniversity.com

4. You can also inspect the job further by navigating to the BigInsights Web Console. Open a Web Browser (e.g. Firefox) and navigate to: http://localhost:8080/BigInsights Note: If you are working on the Cloud, ensure you replace “localhost” with the appropriate IP address of your Cloud instance. 5. Then click the jobs tab at the top-left of the page 4©2011 BigDataUniversity.com

6. You should see a table containing a summary of jobs you’ve ran. Select the row with Name word count (this should be the top row) then click the View Job button. You should then be presented with a screen like the following: Top pink section displays all the general information about the job (e.g. Start/Finish time). The blue section displays the summary of all tasks run for said job. For example you can see that there were 4 Map, and 8 Reduce jobs ran. Moreover, the 8 Reduce jobs correspond to the 8 output files from Step 3. 7. Click on the Job Counters button to display details about the number of bytes read and written, the number of various types of input and output records produced by the MapReduce framework, and so on. Scroll through the pop-up window, if needed, to become familiar with 5©2011 BigDataUniversity.com

the various statistical data collected. 8. Finally, click on the Job Conf… button to display information about the configuration parameters associated with this job. Inspect the .xml file displayed in the lower pane, scrolling down as needed to review the information collected. Configuration information is provided through property elements that consist of name/value pairs. About midway through the file, you’ll find a property with a name of mapred.job.name and a value of word count. This corresponds to your job’s name. 6©2011 BigDataUniversity.com

Working in the Web ConsoleInspecting the overall health of your system 1. Click on the Administration tab in the upper left corner. Most administrative tasks are performed from this view. 2. Inspect the contents of the Dashboard Summary immediately beneath the Administration tab. Note that this dashboard reports the number of nodes in your cluster as well as the number of errors and warnings reported. The final line reports on the overall status of your system. In this example, the dashboard reports a healthy status for a single-node environment. 3. Inspect the Start Stop Summary to the right of the dashboard. This indicates if all installed components are up and running. Note that your BigInsights environment may be healthy even if one or more optional components have been stopped, as shown in this figure below. 8©2011 BigDataUniversity.com

4. Inspect the Server Administration pane at the bottom. Click on the drop-down list (Select View) to see the Components view. This view summarizes the installed components and their start/stop status. 5. Select the checkbox of a component of your choice and click on the Status Details button. A pop-up window appears with information about your selected component, as shown in the following figure. (The report you see will vary depending on the component you selected and its operational status.) 9©2011 BigDataUniversity.com

Starting and stopping a component 1. In the Components section on the lower left, highlight HBase and click the Stop button to stop the HBase server. 2. After the console reports that HBase has been stopped, click on the Start button to start the service again. ------ This is the end of this lab ------ 10©2011 BigDataUniversity.com

Hadoop Lab 2 - MapReduce

by Leons Petražickis on Nov 08, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Hadoop Lab 2 - MapReduce — Document Transcript