【エンジニアカフェEvent×gumiStudy】ソーシャルゲームの解析を支える技術-Hadoop編- http://www.facebook.com/event.php?eid=245262765524522 の発表資料です。

1. Hadoop
gumi
1

2. :
Twitter:@CkReal
gumi 2
AWS S3,EMR
2

3. 1.
2. Elastic MapReduce EMR
3.
4. EMR
3

4. 4

5. 2
( 1 )
fluentd
5

6. 1 1,000 /
AP
APAP
AP DB
d fluentd fluentd mongos mongod(PRIMARY)
DB
config
mongod(SECONDARY)
DB
fluentd mongos
mongod(SECONDARY)
config ReplicaSets & Sharding
NFS
6

7. EMR
7

8. 8.5GB 1.4GB /
ID
Nov 1 23:59:59 hogehoge-ap1 hogehoge ADD_MONEY 12345
[BeforeMoney] 67979 [AfterMoney] 68024 [Money] 45
Nov 1 23:59:59 hogehoge-ap2 hogehoge CONSUME_POWER 12345
[BeforePower] 25 [AfterPower] 20 [ConsumePower] 5
8

9. NFS
NFS
9

10. MongoDB
MongoDB ( )
"app" : "hogehoge",
"userid" : "12345",
ID
"dateint" : 20111101,
"hourint" : 23,
"actions" : [
"CONSUME_POWER", MongoDB Sharding
"ADD_MONEY"
],
"records" : [
"action" : "ADD_MONEY",
"timeint" : 235959,
]
10

11. EMR
Hive Pig Hadoop Streaming
Hadoop
Streaming
(Python)
11

12. m2.4xlarge × 1 4.9GB 85
EMR(m2.xlarge) × 5 4.9GB 44
m2.4xlarge × 1 7.2GB 138
EMR(m2.xlarge) × 5 7.2GB 69
(Macbook Air) 3.6GB 30 …
12

13. EMR CPU
13

14. 14

15. ( )
NFS Amazon S3 EMR
S3
EMR
S3 EMR
config
MongoDB mongos
15

16. S3
boto
S3
c3cmd S3
EMR
Mapper,Reducer,Python2.7
MongoDB
pymongo MongoDB
EMR
Client Tool(Ruby) EMR
16

17. EMR
17

18. EMR
S3
EC2⇔S3 20MB/sec
Hadoop
HadoopStreaming
EMR
18

19. GB/
19

20. 20

21. 21