1. Hadoop

2. http://hadoop.apache.org/

3. Hadooooo
• Google MapReduce
•
•

5. • PC …

6. • PC
PC /

8. PC

11. Yahoo! Search Assist
•
•

12. Hadoop

13. •
7000 …
• Hadoop

15. 816
30 3 1

17. •
DB
• Hadoop SQL

18. Hive
• Hadoop
• SQL(HiveQL) SQL
• SQL

19. Hive
• (each do ... end)
• Hive DB,
• (HiveQL)
• MySQL
EXISTS
…

20. Hadoop

21. Hadoop

22. 1) Map
2) Shuffle & Sort
3) Reduce

23. Map

24. aaa
bbb
ccc
ddd
eee
Mapper
2
※

25. aaa
bbb
ccc
ddd
eee
2 aaa
0 bbb
1 ccc
1 ddd
0 eee

26. aaa
bbb
ccc
ddd
eee
2 aaa
0 bbb
key 1 ccc value
1 ddd
0 eee

27. Shuffle & Sort

28. key
Reducer

29. Map
2 aaa
0 bbb
key 1 ccc value
1 ddd
0 eee

30. Map
2 aaa
0 bbb
1 ccc Reducer
1 ddd
0 eee

31. Map
key

32. Map Reduce
Reduce

33. Map
2 aaa
0 bbb
1 ccc
1 ddd
0 eee
Reducer
1
※

34. key value
2 aaa
0 bbb
1 ccc
1 ddd
0 eee
Reducer 3

35. 2 aaa
0 bbb
1 ccc
1 ddd
0 eee

36. 2 aaa
0 bbb
1 ccc
1 ddd
0 eee

37. 2 aaa
0 bbb
1 ccc
1 ddd
0 eee
key Reducer

38. Hadoop
Google MapReduce

39. Reduce
• Reduce
•
…
• Google MapReduce Reducer

40. Hadoop
Iterater

41. id:naoya
http://d.hatena.ne.jp/naoya/20080513/1210684438

42. Hadoop

43. Hadoop
• Hadoop Streaming (Ruby)
• EC2 Hadoop
• S3
• 50

44. EC2 S3
Amazon
• EC2 •••
※
• S3 •••

45. •
DB
• Hadoop SQL

47. 1. (CSV or
Marshal) S3
2. EC2 Hadoop 1.
S3
3. S3 2.
MySQL

48. DB
1. (CSV or
Marshal) S3
2. EC2 Hadoop 1.
S3
3. S3 2.
MySQL

49. Hadoop
1. (CSV or
Marshal) S3
2. EC2 Hadoop 1.
S3
3. S3 2.
MySQL

50. DB
1. (CSV or
Marshal) S3
2. EC2 Hadoop 1.
S3
3. S3 2.
MySQL

52. MySQL
…orz

54. 1taaa,bbb,ccc aaa,bbb,ccc
1thoge,fuga,foo hoge,fuga,foo

55. Mapper, Reducer

56. •
• Mapper, Reducer
•

57. Hadoop S3
`hadoop dfs -cat s3://xxx/
input/user_info`

58. failed to allocate memory
(NoMemoryError)

59. Mapper
or

61. 7000 →
30

62. • Hadoop
• MapReduce
MapReduce
• Hadoop