This document provides an overview of Hadoop and related technologies. It discusses Hadoop components like HDFS, MapReduce, Pig and Hive. It also describes how MapReduce works, examples of using MapReduce for tasks like grep and analyzing Wikipedia data. Finally it discusses the vision for Hadoop applications and becoming a Hadooper.

1. Hadoop Tenjin
2010 06 25
2010 6 27

2. 2010 6 27

3. 3
2010 6 27

4. 4
2010 6 27

5. 5
2010 6 27

6. MPI
MPI=Message Passing Interface
6
2010 6 27

7. Hadoop
2010 6 27

8. OSS
Hadoop
8
2010 6 27

9. Hadoop
9
2010 6 27

10. Hadoop
JDK
OS
10
2010 6 27

11. Hadoop
Master Slave
job task
tracker tracker
Slave
task
tracker
11
2010 6 27

12. Hadoop
Hadoop
Zookeeper Chukwa Pig Hive Mahout
HBase MapReduce Avro
HDFS
12
2010 6 27

13. Hadoop
2010 6 27

14. Hadoop
WikiPedia
KeyValueStore NoSQL
14
2010 6 27

15. Hadoop
15
2010 6 27

16. MapReduce
2010 6 27

17. MapReduce
Hadoop
17
2010 6 27

18. MapReduce
MapReduce
input Map Reduce
hoge, hoge, hoge, fuga, fuga
[hoge, [1, 1, 1]]
[fuga, [1, 1]]
[hoge, 1]
[hoge, 1]
[hoge, 1] [hoge, [1, 1, 1]] [hoge, 3]
[fuga, 1] [fuga, [1, 1]] [fuga, 2]
output
[fuga, 1]
Shuffle & Sort
key value
18
2010 6 27

19. MapReduce
HDFS Key Value
MapReduce
Map
XML CSV
TextInputFormat Key Value
Reduce
Key Value
TextOutputFormat
19
2010 6 27

20. MapReduce
2010 6 27

21. MapReduce
MapReduce
Input
Job1 Job2
Map Reduce KeyValue
Map Reduce
Output
21
2010 6 27

22. Grep
Hadoop Grep
$ hadoop jar hadoop-example.jar grep [ ][ ][ ]
grep-serch Job grep-sort Job
Map Map Key Value
Reduce Reduce
22
2010 6 27

23. Grep
Hadoop Grep
23
2010 6 27

24. MapReduce
2010 6 27

25. Wikipedia
2010 6 27

26. Wikipedia
Wikipedia
Wik-IE
Wik-IE
Map
ID
D/L
Reduce
Map
KeyValue
26
2010 6 27

27. Hadoop
2010 6 27

28. add Hadoop OUTPUT
28
2010 6 27

29. Hadoop Application
vision of Hadoop-based Application
29
2010 6 27

30. Hadoop
2010 6 27

31. Hadoop
31
2010 6 27

32. Hadoop
32
2010 6 27

33. Hadoop
Hadooper Hadoop
33
2010 6 27