Your SlideShare is downloading. ×

Map reduce模型

242

Published on

Map reduce模型

Map reduce模型

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
242
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 《MapReduce模型》丁海亮 2013-02-27
  • 2. 议程l 计算模型l 基础架构l HDFSl MapReducel 集群部署
  • 3. 典型例子
  • 4. 计算模型
  • 5. 基础架构
  • 6. 基础架构
  • 7. 基础架构
  • 8. HDFS
  • 9. 读取数据
  • 10. 写入数据
  • 11. MapReduce
  • 12. MapReduce
  • 13. MapReducepublic static class TokenizerMapperextends Mapper<Object, Text, Text, IntWritable>{private final static IntWritable one = new IntWritable(1);private Text word = new Text();public void map(Object key, Text value, Context context) throws IOException, InterruptedException {StringTokenizer itr = new StringTokenizer(value.toString());while (itr.hasMoreTokens()) {word.set(itr.nextToken());context.write(word, one);}}}
  • 14. MapReducepublic static class IntSumReducerextends Reducer<Text,IntWritable,Text,IntWritable> {private IntWritable result = new IntWritable();public void reduce(Text key, Iterable<IntWritable> values,Context context) throws IOException, InterruptedException {int sum = 0;for (IntWritable val : values) {sum += val.get();}result.set(sum);context.write(key, result);}}
  • 15. MapReducepublic static void main(String[] args) throws Exception {Configuration conf = new Configuration();String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();if (otherArgs.length != 2) {System.err.println("Usage: wordcount <in> <out>");System.exit(2);}Job job = new Job(conf, "word count");job.setJarByClass(WordCount.class);job.setMapperClass(TokenizerMapper.class);job.setCombinerClass(IntSumReducer.class);job.setReducerClass(IntSumReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);FileInputFormat.addInputPath(job, new Path(otherArgs[0]));FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));System.exit(job.waitForCompletion(true) ? 0 : 1);}
  • 16. 实施流程硬件选型(商用机)操作系统选型(Linux)内核调优硬盘配置DataNode无须RAID和LVM网络配置Hadoop环境配置运维支撑 rsync自动化同步配置部署vm.overcommit_memory
  • 17. 集群部署
  • 18. 集群部署
  • 19. 集群部署
  • 20. 论文参考
  • 21. Thank You

×