Hands-on Hadoop: An intro for Web developers

Hands-on Hadoop: An intro for Web developers Erik Eldridge Engineer/Evangelist Yahoo! Developer Network Photo credit: http://www.flickr.com/photos/exfordy/429414926/sizes/l/

Goals

Gain familiarity with Hadoop

Approach Hadoop from a web dev's perspective

Prerequisites

VMWare

Hadoop will be demonstrated using a VMWare virtual machine

I’ve found the use of a virtual machine to be the easiest way to get started with Hadoop

Curl installed

Setup VM

Download VM from YDN

http://developer.yahoo.com/hadoop/tutorial/module3.html#vm-setup

Note:

user name: hadoop-user

password: hadoop

Launch vm

Log in

Note ip of machine

Start Hadoop

Run the util to launch hadoop: $ ~/start-hadoop

If it's already running, we'll get an error like "172.16.83.132: datanode running as process 6752. Stop it first. 172.16.83.132: secondarynamenode running as process 6845. Stop it first. ..."

Saying “hi” to Hadoop

Call hadoop command line util: $ hadoop

Hadoop should have been launched on boot. verify this is the case: $ hadoop dfs -ls /

Saying “hi” to Hadoop

If hadoop has not been started, you'll see something like: "09/07/22 10:01:24 INFO ipc.Client: Retrying connect to server: /172.16.83.132:9000. Already tried 0 time(s). 09/07/22 10:01:24 INFO ipc.Client: Retrying connect to server: /172.16.83.132:9000. Already tried 1 time(s)...”

If hadoop has been launched, the dfs -ls command should show the contents of hdfs

Before continuing, view all the hadoop utilities and sample files: $ ls

Install Apache

Why? In the interest of creating a relevant example, I'm going to work on Apache access logs

Update apt-get so it can find apache2: $ sudo apt-get update

Install apache2 so we can generate access log data: $ sudo apt-get install apache2

Generate data

Jump into the directory containing the apache logs: $ cd /var/log/apache2

Show the top n lines of the access log: $ tail -f -n 10 access.log

Generate data

Put this script, or something similar, in an executable file on your local machine

Edit the IP address to that of your VM

Generate data

Set executable permissions on the file: $ chmod +x generate.sh

Run the file: $ ./generate.sh

Note log data in tail output in VM

Exploring HDFS

Show home dir structure:

$ hadoop dfs -ls /user

Create a directory:

$ hadoop dfs -mkdir /user/foo

Exploring HDFS

Attempt to re-create new dir and note error:

$ hadoop dfs -mkdir /user/foo

Create a destination directory using implicit path:

$ hadoop dfs -mkdir bar

Auto-create nested destination directories:

$ hadoop dfs -mkdir dir1/dir2/dir3

Remove dir:

$ hadoop dfs -rmr /user/foo

Remove dir:

$ hadoop dfs -rmr bar dir1

Try to re-remove dir and note error:

$ hadoop dfs -rmr bar

Browse HDFS using web UI

Open http://{VM IP address}:50030 in browser

Import access log data

Load access log into hdfs:

$ hadoop dfs -put /var/log/apache2/access.log input/access.log

Verify it's in there:

$ hadoop dfs -ls input/access.log

View the contents:

$ hadoop dfs -cat input/access.log

Count words in data using Hadoop Streaming

Hadoop Streaming refers to the ability to use an arbitrary language to define a job’s map and reduce processes

Python wordcount mapper Credit: http://www.michael-noll.com/wiki/Writing_An_Hadoop_MapReduce_Program_In_Python#Map:_mapper.py

Python wordcount reducer Credit: http://www.michael-noll.com/wiki/Writing_An_Hadoop_MapReduce_Program_In_Python#Map:_mapper.py

Test run mapper and reducer

$ cat data | mapper.py | sort | reducer.py

Run Hadoop

Stream data through these two files, saving the output back to HDFS: $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar -input /data/ydata/ydata-ysm-keyphrase-bid-imp-click-v1_0 -output /user/{username}/output -mapper /home/{username}/mapper.py -reducer /home/{username}/reducer.py

View output