Map reduce模型

  • 207 views
Uploaded on

Map reduce模型

Map reduce模型

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
207
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
2
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. 《MapReduce模型》丁海亮 2013-02-27
  • 2. 议程l 计算模型l 基础架构l HDFSl MapReducel 集群部署
  • 3. 典型例子
  • 4. 计算模型
  • 5. 基础架构
  • 6. 基础架构
  • 7. 基础架构
  • 8. HDFS
  • 9. 读取数据
  • 10. 写入数据
  • 11. MapReduce
  • 12. MapReduce
  • 13. MapReducepublic static class TokenizerMapperextends Mapper<Object, Text, Text, IntWritable>{private final static IntWritable one = new IntWritable(1);private Text word = new Text();public void map(Object key, Text value, Context context) throws IOException, InterruptedException {StringTokenizer itr = new StringTokenizer(value.toString());while (itr.hasMoreTokens()) {word.set(itr.nextToken());context.write(word, one);}}}
  • 14. MapReducepublic static class IntSumReducerextends Reducer<Text,IntWritable,Text,IntWritable> {private IntWritable result = new IntWritable();public void reduce(Text key, Iterable<IntWritable> values,Context context) throws IOException, InterruptedException {int sum = 0;for (IntWritable val : values) {sum += val.get();}result.set(sum);context.write(key, result);}}
  • 15. MapReducepublic static void main(String[] args) throws Exception {Configuration conf = new Configuration();String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();if (otherArgs.length != 2) {System.err.println("Usage: wordcount <in> <out>");System.exit(2);}Job job = new Job(conf, "word count");job.setJarByClass(WordCount.class);job.setMapperClass(TokenizerMapper.class);job.setCombinerClass(IntSumReducer.class);job.setReducerClass(IntSumReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);FileInputFormat.addInputPath(job, new Path(otherArgs[0]));FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));System.exit(job.waitForCompletion(true) ? 0 : 1);}
  • 16. 实施流程硬件选型(商用机)操作系统选型(Linux)内核调优硬盘配置DataNode无须RAID和LVM网络配置Hadoop环境配置运维支撑 rsync自动化同步配置部署vm.overcommit_memory
  • 17. 集群部署
  • 18. 集群部署
  • 19. 集群部署
  • 20. 论文参考
  • 21. Thank You