The document discusses MapReduce model, including its computing model, basic architecture, HDFS, MapReduce and cluster deployment. It provides code examples of Mapper, Reducer and main functions in MapReduce job. It also talks about hardware selection, operating system choice, kernel tuning, disk configuration, network setup and Hadoop environment configuration for cluster deployment.

1. 《MapReduce模型》
丁海亮 2013-02-27

2. 议程
l 计算模型
l 基础架构
l HDFS
l MapReduce
l 集群部署

3. 典型例子

4. 计算模型

5. 基础架构

6. 基础架构

7. 基础架构

8. HDFS

9. 读取数据

10. 写入数据

11. MapReduce

12. MapReduce

13. MapReduce
public static class TokenizerMapper
extends Mapper<Object, Text, Text, IntWritable>{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}

14. MapReduce
public static class IntSumReducer
extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values,
Context context
) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}

15. MapReduce
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
if (otherArgs.length != 2) {
System.err.println("Usage: wordcount <in> <out>");
System.exit(2);
}
Job job = new Job(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}

16. 实施流程
硬件选型（商用机）
操作系统选型（Linux）
内核调优
硬盘配置
DataNode无须RAID和LVM
网络配置
Hadoop环境配置
运维支撑 rsync自动化同步配置部署
vm.overcommit_memory

17. 集群部署

18. 集群部署

19. 集群部署

20. 论文参考

21. Thank You