ソーシャルゲームログ解析基盤のHadoop活用事例

10,451 views

Published on

【エンジニアカフェEvent×gumiStudy】ソーシャルゲームの解析を支える技術-Hadoop編-
http://www.facebook.com/event.php?eid=245262765524522
の発表資料です。

Published in: Technology, Business
0 Comments
25 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
10,451
On SlideShare
0
From Embeds
0
Number of Embeds
2,978
Actions
Shares
0
Downloads
62
Comments
0
Likes
25
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • ソーシャルゲームログ解析基盤のHadoop活用事例

    1. 1. Hadoopgumi 1
    2. 2. :Twitter:@CkRealgumi 2 AWS S3,EMR 2
    3. 3. 1.2. Elastic MapReduce EMR3.4. EMR 3
    4. 4. 4
    5. 5. 2 ( 1 )fluentd 5
    6. 6. 1 1,000 / AP APAP AP DBd fluentd fluentd mongos mongod(PRIMARY) DB config mongod(SECONDARY) DB fluentd mongos mongod(SECONDARY) config ReplicaSets & Sharding NFS 6
    7. 7. EMR 7
    8. 8. 8.5GB 1.4GB / IDNov 1 23:59:59 hogehoge-ap1 hogehoge ADD_MONEY 12345[BeforeMoney] 67979 [AfterMoney] 68024 [Money] 45Nov 1 23:59:59 hogehoge-ap2 hogehoge CONSUME_POWER 12345[BeforePower] 25 [AfterPower] 20 [ConsumePower] 5 8
    9. 9. NFSNFS 9
    10. 10. MongoDB MongoDB ( ) "app" : "hogehoge", "userid" : "12345", ID "dateint" : 20111101, "hourint" : 23, "actions" : [ "CONSUME_POWER", MongoDB Sharding "ADD_MONEY" ], "records" : [ "action" : "ADD_MONEY", "timeint" : 235959, ] 10
    11. 11. EMR Hive Pig Hadoop Streaming Hadoop Streaming (Python) 11
    12. 12. m2.4xlarge × 1 4.9GB 85EMR(m2.xlarge) × 5 4.9GB 44 m2.4xlarge × 1 7.2GB 138EMR(m2.xlarge) × 5 7.2GB 69 (Macbook Air) 3.6GB 30 … 12
    13. 13. EMR CPU 13
    14. 14. 14
    15. 15. ( )NFS Amazon S3 EMRS3 EMR S3 EMR config MongoDB mongos 15
    16. 16. S3 botoS3 c3cmd S3EMR Mapper,Reducer,Python2.7MongoDB pymongo MongoDBEMR Client Tool(Ruby) EMR 16
    17. 17. EMR 17
    18. 18. EMRS3 EC2⇔S3 20MB/sec Hadoop HadoopStreamingEMR 18
    19. 19. GB/ 19
    20. 20. 20
    21. 21. 21

    ×