Downloaded 759 times
![public class SimpleWordCount
extends Configured implements Tool {
public static class MapClass
extends Mapper<Object, Text, Text, IntWritable> {
...
}
public static class Reduce
extends Reducer<Text, IntWritable, Text, IntWritable> {
...
}
public int run(String[] args) throws Exception { ... }
public static void main(String[] args) { ... }
}](https://image.slidesharecdn.com/hadoop-110509234250-phpapp02/85/Hadoop-34-320.jpg)
![public int run(String[] args) throws Exception {
Configuration conf = getConf();
Job job = new Job(conf, "Counting Words");
job.setJarByClass(SimpleWordCount.class);
job.setMapperClass(MapClass.class);
job.setReducerClass(Reduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
return job.waitForCompletion(true) ? 0 : 1;
}](https://image.slidesharecdn.com/hadoop-110509234250-phpapp02/85/Hadoop-37-320.jpg)
![public static void main(String[] args) throws Exception {
int result = ToolRunner.run(new Configuration(),
new SimpleWordCount(),
args);
System.exit(result);
}](https://image.slidesharecdn.com/hadoop-110509234250-phpapp02/85/Hadoop-38-320.jpg)
Uploaded byScott Leberknight
Hadoop
Hadoop is an open-source framework for distributed storage and processing of large datasets across clusters of computers. It consists of HDFS for distributed storage and MapReduce for distributed processing. HDFS stores large files across multiple machines and provides high throughput access to application data. MapReduce allows processing of large datasets in parallel by splitting the work into independent tasks called maps and reduces. Companies use Hadoop for applications like log analysis, data warehousing, machine learning, and scientific computing on large datasets.
More Related Content
![[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)](https://cdn.slidesharecdn.com/ss_thumbnails/cs264hadooplecture2011-110322172329-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Hadoop
- 1.
- 4.
- 5. e Hadoop users.. Notabl Yahoo! LinkedIn Facebook New York Times Twitter Rackspace Baidu eHarmony eBay Powerset http://wiki.apache.org/hadoop/PoweredBy
- 6. Hadoop in theReal World..
- 7. Recommendation Financial analysis systems Natural Language Correlation engines Processing (NLP) Data warehousing Image/video processing Market research/forecasting Log analysis
- 8. Finance Social networking Health & Academic research Life Sciences Government Telecommunications
- 9.
- 10. Inspired by GoogleBigTable and MapReduce papers circa 2004 Created by Doug Cutting Originally built to support distribution for Nutch search engine Named after a stuffed elephant
- 11. OK, So whatexactly is Hadoop?
- 12. An open source... batch/offline oriented... data & I/O intensive... general purpose framework for creating distributed applications that process huge amounts of data.
- 13. One definition of"huge" 25,000 machines More than 10 clusters 3 petabytes of data (compressed, unreplicated) 700+ users 10,000+ jobs/week
- 14. Had oop M ajornts: C omp one Distributed File System (HDFS) Map/Reduce System
- 15. But first, what isn't Hadoop?
- 16. doop is NOT: Ha ...a relational database! ...an online transaction processing (OLTP) system! ...a structured data store of any kind!
- 17.
- 18. Hadoop Relational Scale-out Scale-up(*) Key/value pairs Tables Say how to process Say what you want the data (SQL) Offline/batch Online/real-time (*) Sharding attempts to horizontally scale RDBMS, but is difficult at best
- 19.
- 20. Data is distributedand replicated over multiple machines Designed for large files (where "large" means GB to TB) Block oriented Linux-style commands, e.g. ls, cp, mv, rm, etc.
- 21. NameNode File Block Mappings: /user/aaron/data1.txt -> 1, 2, 3 /user/aaron/data2.txt -> 4, 5 /user/andrew/data3.txt -> 6, 7 DataNode(s) 5 1 4 2 2 3 7 4 6 1 4 6 2 3 6 1 3 7 5 7 5
- 22. fault tolerant whennodes fail Self-healing rebalances files across cluster scalable just by adding new nodes!
- 23.
- 24. Split input files(e.g. by HDFS blocks) Operate on key/value pairs Mappers filter & transform input data Reducers aggregate mapper output
- 25. move code todata
- 26. map: (K1, V1) list(K2, V2) reduce: (K2, list(V2)) list(K3, V3)
- 27. Word Count (the canonicalMap/Reduce example)
- 28. the quick brownfox jumped over the lazy brown dog
- 29. m ap phase- inputs (K1, V1) (0, "the quick brown fox") (20, "jumped over") (32, "the lazy brown dog")
- 30. map ph ase - list(K2, V2) outpu ts ("the", 1) ("quick", 1) ("brown", 1) ("fox", 1) ("jumped", 1) ("over", 1) ("the", 1) ("lazy", 1) ("brown", 1) ("dog", 1)
- 31. redu ce phase- inputs (K2, list(V2)) ("brown", (1, 1)) ("dog", (1)) ("fox", (1)) ("jumped", (1)) ("lazy", (1)) ("over", (1)) ("quick", (1)) ("the", (1, 1))
- 32. reduce phase outpu - list(K3, V3) ts ("brown", 2) ("dog", 1) ("fox", 1) ("jumped", 1) ("lazy", 1) ("over", 1) ("quick", 1) ("the", 2)
- 33.
- 34. public class SimpleWordCount extends Configured implements Tool { public static class MapClass extends Mapper<Object, Text, Text, IntWritable> { ... } public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { ... } public int run(String[] args) throws Exception { ... } public static void main(String[] args) { ... } }
- 35. public static classMapClass extends Mapper<Object, Text, Text, IntWritable> { private static final IntWritable ONE = new IntWritable(1L); private Text word = new Text(); @Override protected void map(Object key, Text value, Context context) throws IOException, InterruptedException { StringTokenizer st = new StringTokenizer(value.toString()); while (st.hasMoreTokens()) { word.set(st.nextToken()); context.write(word, ONE); } } }
- 36. public static classReduce extends Reducer<Text, IntWritable, Text, IntWritable> { private IntWritable count = new IntWritable(); @Override protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable value : values) { sum += value.get(); } count.set(sum); context.write(key, count); } }
- 37. public int run(String[]args) throws Exception { Configuration conf = getConf(); Job job = new Job(conf, "Counting Words"); job.setJarByClass(SimpleWordCount.class); job.setMapperClass(MapClass.class); job.setReducerClass(Reduce.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(Text.class); FileInputFormat.setInputPaths(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); return job.waitForCompletion(true) ? 0 : 1; }
- 38. public static voidmain(String[] args) throws Exception { int result = ToolRunner.run(new Configuration(), new SimpleWordCount(), args); System.exit(result); }
- 39. aF low uce Dat p/Red M a (Image from Hadoop in Action...great book!)
- 40. Partitioning Deciding whichkeys go to which reducer Desire even distribution across reducers Skewed data can overload a single reducer!
- 41. Map/Reduce Partitioning &Shuffling (Image from Hadoop in Action...great book!)
- 42. Combiner Effectively a reducein the mappers a.k.a. "Local Reduce"
- 43. Shuffling WordCount data # k/v pairs shuffled without combiner ("the", 1) 1000 with combiner ("the", 1000) 1 (looking at one mapper that sees the word "the" 1000 times)
- 44. Advanced Map/Reduce Hadoop Streaming Chaining Map/Reduce jobs Joining data Bloom filters
- 45.
- 46. HDFS NameNode Secondary NameNode Map/Reduce DataNode JobTracker TaskTracker
- 47. Secondary NameNode NameNode JobTracker DataNode1 DataNode2 DataNodeN TaskTracker1 TaskTracker2 TaskTrackerN map map map reduce reduce reduce
- 48. NameNode Bookkeeper for HDFS Manages DataNodes Should not store data or run jobs Single point of failure!
- 51. DataNode Store actual file blocks on disk Does not store entire files! Report block info to NameNode Receive instructions from NameNode
- 52. Secondary NameNode Snapshot of NameNode Not a failover server for NameNode! Help minimize downtime/data loss if NameNode fails
- 53. JobTracker Partition tasksacross HDFS cluster Track map/reduce tasks Re-start failed tasks on different nodes Speculative execution
- 56. TaskTracker Track individual map& reduce tasks Report progress to JobTracker
- 58.
- 59.
- 60. Logs View task logs on machine where specific task was processed (or via web UI) $HADOOP_HOME/logs/userlogs on task tracker
- 62. Counters Define one or more counters Increment counters during map/reduce tasks Counter values displayed in job tracker UI
- 64. IsolationRunner Re-run failed taskswith original input data Must set keep.failed.tasks.files to 'true'
- 65. Skipping Bad Records Data may not always be clean New data may have new interesting twists Can you pre-process to filter & validate input?
- 66.
- 67. Speculative execution Use a Combiner (on by default) Reduce amount of JVM Re-use input data (be careful) Refactor code/ Data compression algorithms
- 68.
- 69. Lots of knobs Trash can Needs active Add/remove management data nodes Network topology/ "Fair" scheduling rack awareness NameNode/SNN Permissions/quotas management
- 70.
- 71. Simulate structure fordata stored in Hadoop Query language analogous to SQL (Hive QL) Translates queries into Map/Reduce job(s)... ...so not for real-time processing!
- 72. Queries: Projection Joins (inner, outer, semi) Grouping Aggregation Sub-queries Multi-table insert Customizable: User-defined functions Input/output formats with SerDe
- 73. /user/sleberkn/nber-patent/tables/patent_citation/cite75_99.txt "CITING","CITED" 3858241,956203 3858241,1324234 3858241,3398406 3858241,3557384 3858241,3634889 3858242,1515701 3858242,3319261 Patent citation dataset 3858242,3668705 3858242,3707004 3858243,2949611 3858243,3146465 3858243,3156927 3858243,3221341 3858243,3574238 ... http://www.nber.org/patents
- 74. create external tablepatent_citations (citing string, cited string) row format delimited fields terminated by ',' stored as textfile location '/user/sleberkn/nber-patent/tables/patent_citation'; create table citation_histogram (num_citations int, count int) stored as sequencefile;
- 75. insert overwrite tablecitation_histogram select num_citations, count(num_citations) from (select cited, count(cited) as num_citations from patent_citations group by cited) citation_counts group by num_citations order by num_citations;
- 77. Hadoop in theclouds
- 78. Amazon EC2 +S3 EC2 instances are compute nodes (Map/Reduce) Storage options: HDFS on EC2 nodes HDFS on EC2 nodes loading data from S3 Native S3 (bypasses HDFS)
- 79. Amazon Elastic MapReduce Interact via web-based console Submit Map/Reduce job (streaming, Hive, Pig, or JAR) EMR configures & launches Hadoop cluster for job Uses S3 for data input/output
- 80.
- 81. Hadoop = HDFS+ Map/Reduce Distributed, parallel processing Designed for fault tolerance Horizontal scale-out Structure & queries via Hive
- 82.
- 83. http://hadoop.apache.org/ http://hadoop.apache.org/hive/ Hadoop in Action http://www.manning.com/lam/ Definitive Guide to Hadoop, 2nd ed. http://oreilly.com/catalog/0636920010388 Yahoo! Hadoop blog http://developer.yahoo.net/blogs/hadoop/ Cloudera http://www.cloudera.com/
- 84.
- 85.