Hadoop 101 Mohit Soni eBay
Inc. BarCamp Chennai - 5 Mohit Soni

About Me • I work
as a Software Engineer at eBay • Worked on large-scale data processing with eBay Research Labs BarCamp Chennai - 5 Mohit Soni

First Things First BarCamp Chennai
- 5 Mohit Soni

MapReduce • Inspired from functional
operations – Map – Reduce • Functional operations do not modify data, they generate new data • Original data remains unmodified BarCamp Chennai - 5 Mohit Soni

Functional Operations Map Reduce def
sqr(n): def add(i, j): return n * n return i + j list = [1,2,3,4] list = [1,2,3,4] map(sqr, list) -> [1,4,9,16] reduce(add, list) -> 10 MapReduce def MapReduce(data, mapper, reducer): return reduce(reducer, map(mapper, data)) MapReduce(list, sqr, add) -> 30 Python code BarCamp Chennai - 5 Mohit Soni

BarCamp Chennai - 5 Mohit
Soni

What is Hadoop ? •
Framework for large-scale data processing • Based on Google’s MapReduce and GFS • An Apache Software Foundation project • Open Source! • Written in Java • Oh, btw BarCamp Chennai - 5 Mohit Soni

Why Hadoop ? • Need
to process lots of data (PetaByte scale) • Need to parallelize processing across multitude of CPUs • Achieves above while KeepIng Software Simple • Gives scalability with low-cost commodity hardware BarCamp Chennai - 5 Mohit Soni

Hadoop fans Source: Hadoop Wiki
BarCamp Chennai - 5 Mohit Soni

When to use and not-use
Hadoop ? Hadoop is a good choice for: • Indexing data • Log Analysis • Image manipulation • Sorting large-scale data • Data Mining Hadoop is not a good choice: • For real-time processing • For processing intensive tasks with little data • If you have Jaguar or RoadRunner in your stock BarCamp Chennai - 5 Mohit Soni

HDFS – Overview • Hadoop
Distributed File System • Based on Google’s GFS (Google File System) • Write once read many access model • Fault tolerant • Efficient for batch-processing BarCamp Chennai - 5 Mohit Soni

HDFS – Blocks Block 1
Block 2 Input Data Block 3 • HDFS splits input data into blocks • Block size in HDFS: 64/128MB (configurable) • Block size *nix: 4KB BarCamp Chennai - 5 Mohit Soni

HDFS – Replication Block 1
Block 1 Block 2 Block 3 Block 2 Block 3 • Blocks are replicated across nodes to handle hardware failure • Node failure is handled gracefully, without loss of data BarCamp Chennai - 5 Mohit Soni

HDFS – Architecture NameNode Client
Cluster DataNodes BarCamp Chennai - 5 Mohit Soni

HDFS – NameNode • NameNode
(Master) – Manages filesystem metadata – Manages replication of blocks – Manages read/write access to files • Metadata – List of files – List of blocks that constitutes a file – List of DataNodes on which blocks reside, etc • Single Point of Failure (candidate for spending $$) BarCamp Chennai - 5 Mohit Soni

HDFS – DataNode • DataNode
(Slave) – Contains actual data – Manages data blocks – Informs NameNode about block IDs stored – Client read/write data blocks from DataNode – Performs block replication as instructed by NameNode • Block Replication – Supports various pluggable replication strategies – Clients read blocks from nearest DataNode • Data Pipelining – Client write block to first DataNode – First DataNode forwards data to next DataNode in pipeline – When block is replicated across all replicas, next block is chosen BarCamp Chennai - 5 Mohit Soni

Hadoop - Architecture User JobTracker
TaskTracker TaskTracker NameNode DataNode DataNode DataNode DataNode DataNode DataNode BarCamp Chennai - 5 Mohit Soni

Hadoop - Terminology • JobTracker
(Master) – 1 Job Tracker per cluster – Accepts job requests from users – Schedule Map and Reduce tasks for TaskTrackers – Monitors tasks and TaskTrackers status – Re-execute task on failure • TaskTracker (Slave) – Multiple TaskTrackers in a cluster – Run Map and Reduce tasks BarCamp Chennai - 5 Mohit Soni

MapReduce – Flow Input Map
Shuffle + Sort Reduce Output Map Reduce Input Output Data Map Data Reduce Map BarCamp Chennai - 5 Mohit Soni

Word Count Hadoop’s HelloWorld BarCamp
Chennai - 5 Mohit Soni

Word Count Example • Input
– Text files • Output – Single file containing (Word <TAB> Count) • Map Phase – Generates (Word, Count) pairs – [{a,1}, {b,1}, {a,1}] [{a,2}, {b,3}, {c,5}] [{a,3}, {b,1}, {c,1}] • Reduce Phase – For each word, calculates aggregate – [{a,7}, {b,5}, {c,6}] BarCamp Chennai - 5 Mohit Soni

Word Count – Mapper public
class WordCountMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> out, Reporter reporter) throws Exception { String l = value.toString(); StringTokenizer t = new StringTokenizer(l); while(t.hasMoreTokens()) { word.set(t.nextToken()); out.collect(word, one); } } } BarCamp Chennai - 5 Mohit Soni

Word Count – Reducer public
class WordCountReducer extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterator<IntWriter> values, OutputCollector<Text, IntWritable> out, Reporter reporter) throws Exception { int sum = 0; while(values.hasNext()) { sum += values.next().get(); } out.collect(key, new IntWritable(sum)); } } BarCamp Chennai - 5 Mohit Soni

Word Count – Config public
class WordCountConfig { public static void main(String[] args) throws Exception { if (args.length() != 2) { System.exit(1); } JobConf conf = new JobConf(WordCountConfig.class); conf.setJobName(“Word Counter”); FileInputFormat.addInputPath(conf, new Path(args[0]); FileInputFormat.addOutputPath(conf, new Path(args[1])); conf.setMapperClass(WordCountMapper.class); conf.setCombinerClass(WordCountReducer.class); conf.setReducerClass(WordCountReducer.class); conf.setInputFormat(TextInputFormat.class); conf.setOutputFormat(TextOutputFormat.class); JobClient.runJob(conf); } } BarCamp Chennai - 5 Mohit Soni

Diving Deeper • http://hadoop.apache.org/ •
Jeffrey Dean and Sanjay Ghemwat, MapReduce: Simplified Data Processing on Large Clusters • Tom White, Hadoop: The Definitive Guide, O’Reilly • Setting up a Single-Node Cluster: http://bit.ly/glNzs4 • Setting up a Multi-Node Cluster: http://bit.ly/f5KqCP BarCamp Chennai - 5 Mohit Soni

Catching-Up • Follow me on
twitter @mohitsoni • http://mohitsoni.com/ BarCamp Chennai - 5 Mohit Soni