Downloaded 27 times
Uploaded byShinpei Ohtani
サンプルから見るMap reduceコード
This document provides an overview of MapReduce and its components in Hadoop. It discusses key classes like Mapper, Reducer, MapTask, ReduceTask, and JobConf that are used to define and run MapReduce jobs. It also describes how mappers process input splits in parallel and generate output which is shuffled and sorted for input to the reducers.
More Related Content
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
サンプルから見るMap reduceコード
- 1.
- 2. Cloudera Avro Sqoop Desktop Pig Hive HBase Chukwa Map Zoo HDFS Reduce Keeper Core
- 3. Cloudera Avro Sqoop Desktop Pig Hive HBase Chukwa Map Zoo HDFS Reduce Keeper Core
- 4. • MapReduce – Mapper/Reducer •
- 5. MapReduce • WordCount • • – Mapper/Reducer Job ⾏行行 – InputFormat/OutputFormat ⽅方 – HDFS(FileSystem) – Writable ⽅方
- 6. WordCount • Hadoop Hello World • API (org.apache.hadoop.mapreduce) • API
- 7. Grep • grep – grepJob/sortJob 2 ⾏行行 – JobConf/Mapper/Reducer ⽅方 – Mapper RegexMapper ⾏行行 <Text, Long> SequenceFileFormat – sortJob – ⼒力力 –
- 8. Grep - • JobConf • Mapper • Reducer
- 9. o.a.hadoop.mapred.JobConf • – mapred-default.xml – conf/mapred-site.xml – XML ⾝身 DOM – ⾃自 ⽬目 ⼿手 – ⼦子 • JobConf child = new JobConf( Conf, jar );
- 10. mapred-site.xml <configuration> <!– --> <property> <key>mapred.job.tracker</key> <value>your-site:9001</value> </property> </configuration>
- 11. o.a.hadoop.mapred.Mapper • Mapper • InputSplit Mapper • MapTask/MapRunner • map(KEY, VALUE, COLLECTOR, REPORTER) – KEY:Map VALUE:Map – COLLECTOR: – REPORTER: API • MapReduceBase
- 12. o.a.hadoop.mapred.MapTask • Map • initiazlize (Task Reducer ) – ⽣生 – (o.a.h.mapred.TaskStatus.State) • RUNNING, SUCCEEDED, FAILED, UNASSIGNED, KILLED, COMMIT_PENDING, FAILED_UNCLEAN, KILLED_UNCLEAN – OutputCommiter ⽣生 • Task ⼒力力 ⾏行行 • ⼒力力 – mapred.work.output.dir
- 13. o.a.h.mapred.MapTask cont • run runOldMapper • JobClient InputSplit • RecordReader
- 14. o.a.h.mapred.MapTask cont2 • Reduce – spill (* ) • $mapred.local.dir/taskTracker/jobcache/$ {taskid}/output/spill${spillNumber}.out – Reducer ⼒力力 • Combiner min.num.spills.for.combine combiner – RecordWriter ⼒力力 • MapRunner
- 15. o.a.h.mapred.MapRunner • MapRunnable – mapred.map.runner.class – Hadoop PipeMapRunner – Map MultiThreadedMapRunner
- 16. o.a.h.mapred.MapRunner cont • run(RecordReader, OutputCollector, Reporter) – RecordReader: InputFormat Split Reader(InputFormat/RecordReader ) • – RecordReader – ⾝身 –
- 17. MapTask MapRunner Mapper Record Output Reader Collector Input Split⽣生 Spill & run createKey() SpillThread createValue() next(key, value) EOF Map(key, value, Spill outputCollector, reporter)
- 18.
- 19. • Mapper – JobConf – Mapper/MapRunner/MapTask • – Reducer • Reducer ⾏行行 • Reducer ⾏行行 – InputFormat/RecordReader
- 20. o.a.h.mapred.Reducer • Reducer • InputSplit Mapper • ReduceTask/ReduceRunner • reduce(KEY, Iterator<VALUE>, COLLECTOR, REPORTER) – KEY: Iterator<VALUE>: – COLLECTOR: – REPORTER: API • MapReduceBase
- 21. o.a.h.mapred.ReduceTask • SHUFFLE • ReduceTask.ReduceCopier – fetchOutputs( Merger.MergeQueue) • Map x mapred.reduce.parallel.copies – MapOutputCopier • Map ⾏行行 LocalFSMerger • ⾏行行 InMemFSMergeThread • GetMapEventsThread – Map – < , MapOutputLocation(taskId, host, httpUrl)> • ⼀一 TaskTracker ⼯工
- 22. o.a.h.mapred.ReduceTask • run(RecordReader, OutputCollector, Reporter) • SORT – Memory, disk ⽣生 • RowKeyValueItetator – Reducer ⽣生 – RecordWriter ⽣生 – ReduceValuesIterator ⾏行行