Daniel Sikar: Hadoop MapReduce - 06/09/2010

QUICK AND DIRTY PARALLEL PROCESSING ON THE CLOUD Daniel Sikar

EC2 S3

Tools
AWS Command line tools
Elastic MapReduce Ruby library
Hadoop
s3cmd

Hadoop MapReduce Job Tracker + Task Tracker + Slaves HDFS – Distributed file system

Hadoop MapReduce usage Data crunching in general Clicks Statistics etc

Hadoop Project Mgmt Committee

MapReduce ?

MapReduce Key Pairs <key,value>

MapReduce

HTTP Logs Log file A: (...) FreeTouchScreenNokia5230 (...) (...) GetRidofAllSpeedCameras(...) (...) USManWinsLottery (...) (...) BNPToLaunchElectionManifesto (...) Log file B: (...) FreeTouchScreenNokia5230 (...) (...) BodyLanguageTellsAll (...)

MapReduce <FreeTouchScreenNokia5230, 1> + <FreeTouchScreenNokia5230, 1> = <FreeTouchScreenNokia5230, 2>

Hadoop Streaming Running MapReduce jobs with .exe fiels and scripts $ <list> | mapper | reducer

Real life example of Hadoop Streaming usage

Wikipedia Page Access Logs

Wine Grape Varieties

Wikipedia WGV Page Access Stats

Business Decisions

Launching a virtual Hadoop Cluster $ elastic-mapreduce --create --name "Wiki log crunch" --alive --num-instances –instance-type c1.medium 20 Created job flow <job flow id> $ ec2din (...)

Hadoop
Standalone Operation
Pseudo-Distributed Operation
Fully-Distributed Operation
NameNode
JobTracker
DataNode + TaskTracker

Add a step $ elastic-mapreduce --jobflow <jfid> --stream --step-name "Wiki log crunch" --input s3n://dsikar-wikilogs-2009/dec/ --output s3n://dsikar-wikilogs-output/21 --mapper s3n://dsikar-wiki-scripts/wikidictionarymap.pl --reducer s3n://dsikar-wiki-scripts/wikireduce.pl http://<instance public dns>:9100

s3cmd # make bucket $ s3cmd mb s3://dsikar-wikilogs # put log files $ s3cmd put pagecounts-200912*.gz s3://dsikar-wikilogs/dec $ s3cmd put pagecounts-201004*.gz s3://dsikar-wikilogs/apr # list log files $ s3cmd ls s3://dsikar-wikilogs/ # put scripts $ s3cmd put *.pl s3://dsikar-wiki-scripts/ # delete log files $ s3cmd del --recursive --force s3://dsikar-wikilogs/ # remove bucket $ s3cmd rb s3://dsikar-wikilogs/

Elastic MapReduce --create --list --jobflow --describe --stream --terminate

Output files part-00000 part-00001 part-00002 (...)

Further aggregation

Conclusion Hadoop MapReduce provides out-of-the-box ready-to-go distributed computing.

Daniel Sikar: Hadoop MapReduce - 06/09/2010

by Skills Matter

Accessibility

Categories

Upload Details

Usage Rights

Statistics