Map reduce模型
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Map reduce模型

on

  • 594 views

Map reduce模型

Map reduce模型

Statistics

Views

Total Views
594
Views on SlideShare
594
Embed Views
0

Actions

Likes
0
Downloads
2
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Map reduce模型 Presentation Transcript

  • 1. 《MapReduce模型》丁海亮 2013-02-27
  • 2. 议程l 计算模型l 基础架构l HDFSl MapReducel 集群部署
  • 3. 典型例子
  • 4. 计算模型
  • 5. 基础架构
  • 6. 基础架构
  • 7. 基础架构
  • 8. HDFS
  • 9. 读取数据
  • 10. 写入数据
  • 11. MapReduce
  • 12. MapReduce
  • 13. MapReducepublic static class TokenizerMapperextends Mapper<Object, Text, Text, IntWritable>{private final static IntWritable one = new IntWritable(1);private Text word = new Text();public void map(Object key, Text value, Context context) throws IOException, InterruptedException {StringTokenizer itr = new StringTokenizer(value.toString());while (itr.hasMoreTokens()) {word.set(itr.nextToken());context.write(word, one);}}}
  • 14. MapReducepublic static class IntSumReducerextends Reducer<Text,IntWritable,Text,IntWritable> {private IntWritable result = new IntWritable();public void reduce(Text key, Iterable<IntWritable> values,Context context) throws IOException, InterruptedException {int sum = 0;for (IntWritable val : values) {sum += val.get();}result.set(sum);context.write(key, result);}}
  • 15. MapReducepublic static void main(String[] args) throws Exception {Configuration conf = new Configuration();String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();if (otherArgs.length != 2) {System.err.println("Usage: wordcount <in> <out>");System.exit(2);}Job job = new Job(conf, "word count");job.setJarByClass(WordCount.class);job.setMapperClass(TokenizerMapper.class);job.setCombinerClass(IntSumReducer.class);job.setReducerClass(IntSumReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);FileInputFormat.addInputPath(job, new Path(otherArgs[0]));FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));System.exit(job.waitForCompletion(true) ? 0 : 1);}
  • 16. 实施流程硬件选型(商用机)操作系统选型(Linux)内核调优硬盘配置DataNode无须RAID和LVM网络配置Hadoop环境配置运维支撑 rsync自动化同步配置部署vm.overcommit_memory
  • 17. 集群部署
  • 18. 集群部署
  • 19. 集群部署
  • 20. 论文参考
  • 21. Thank You