ソーシャルゲームログ解析基盤のHadoop活用事例

Hadoopgumi 1

:Twitter:@CkRealgumi 2 AWS S3,EMR 2

1.2. Elastic MapReduce EMR3.4. EMR 3

2 ( 1 )ﬂuentd 5

1 1,000 / AP APAP AP DBd fluentd fluentd mongos mongod(PRIMARY) DB config mongod(SECONDARY) DB fluentd mongos mongod(SECONDARY) config ReplicaSets & Sharding NFS 6

8.5GB 1.4GB / IDNov 1 23:59:59 hogehoge-ap1 hogehoge ADD_MONEY 12345[BeforeMoney] 67979 [AfterMoney] 68024 [Money] 45Nov 1 23:59:59 hogehoge-ap2 hogehoge CONSUME_POWER 12345[BeforePower] 25 [AfterPower] 20 [ConsumePower] 5 8

NFSNFS 9

MongoDB MongoDB ( ) "app" : "hogehoge", "userid" : "12345", ID "dateint" : 20111101, "hourint" : 23, "actions" : [ "CONSUME_POWER", MongoDB Sharding "ADD_MONEY" ], "records" : [ "action" : "ADD_MONEY", "timeint" : 235959, ] 10

EMR Hive Pig Hadoop Streaming Hadoop Streaming (Python) 11

m2.4xlarge × 1 4.9GB 85EMR(m2.xlarge) × 5 4.9GB 44 m2.4xlarge × 1 7.2GB 138EMR(m2.xlarge) × 5 7.2GB 69 (Macbook Air) 3.6GB 30 … 12

EMR CPU 13

( )NFS Amazon S3 EMRS3 EMR S3 EMR conﬁg MongoDB mongos 15

S3 botoS3 c3cmd S3EMR Mapper,Reducer,Python2.7MongoDB pymongo MongoDBEMR Client Tool(Ruby) EMR 16

EMR 17

EMRS3 EC2⇔S3 20MB/sec Hadoop HadoopStreamingEMR 18

GB/ 19

http://d.hatena.ne.jp	2125
http://paper.li	24
http://a0.twimg.com	17
http://webcache.googleusercontent.com	3

ソーシャルゲームログ解析基盤のHadoop活用事例

by Tomonori Homma on Nov 09, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 2,169

Statistics

ソーシャルゲームログ解析基盤のHadoop活用事例 — Presentation Transcript