• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Map reduce模型
 

Map reduce模型

on

  • 446 views

Map reduce模型

Map reduce模型

Statistics

Views

Total Views
446
Views on SlideShare
446
Embed Views
0

Actions

Likes
0
Downloads
2
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Map reduce模型 Map reduce模型 Presentation Transcript

    • 《MapReduce模型》丁海亮 2013-02-27
    • 议程l 计算模型l 基础架构l HDFSl MapReducel 集群部署
    • 典型例子
    • 计算模型
    • 基础架构
    • 基础架构
    • 基础架构
    • HDFS
    • 读取数据
    • 写入数据
    • MapReduce
    • MapReduce
    • MapReducepublic static class TokenizerMapperextends Mapper<Object, Text, Text, IntWritable>{private final static IntWritable one = new IntWritable(1);private Text word = new Text();public void map(Object key, Text value, Context context) throws IOException, InterruptedException {StringTokenizer itr = new StringTokenizer(value.toString());while (itr.hasMoreTokens()) {word.set(itr.nextToken());context.write(word, one);}}}
    • MapReducepublic static class IntSumReducerextends Reducer<Text,IntWritable,Text,IntWritable> {private IntWritable result = new IntWritable();public void reduce(Text key, Iterable<IntWritable> values,Context context) throws IOException, InterruptedException {int sum = 0;for (IntWritable val : values) {sum += val.get();}result.set(sum);context.write(key, result);}}
    • MapReducepublic static void main(String[] args) throws Exception {Configuration conf = new Configuration();String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();if (otherArgs.length != 2) {System.err.println("Usage: wordcount <in> <out>");System.exit(2);}Job job = new Job(conf, "word count");job.setJarByClass(WordCount.class);job.setMapperClass(TokenizerMapper.class);job.setCombinerClass(IntSumReducer.class);job.setReducerClass(IntSumReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);FileInputFormat.addInputPath(job, new Path(otherArgs[0]));FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));System.exit(job.waitForCompletion(true) ? 0 : 1);}
    • 实施流程硬件选型(商用机)操作系统选型(Linux)内核调优硬盘配置DataNode无须RAID和LVM网络配置Hadoop环境配置运维支撑 rsync自动化同步配置部署vm.overcommit_memory
    • 集群部署
    • 集群部署
    • 集群部署
    • 论文参考
    • Thank You