Hadoopgumi           1
:Twitter:@CkRealgumi 2        AWS S3,EMR                     2
1.2. Elastic MapReduce EMR3.4. EMR                      3
4
2             ( 1   )fluentd         5
1                        1,000      /      AP     APAP       AP                                           DBd         fluen...
EMR      7
8.5GB             1.4GB /                                                       IDNov 1 23:59:59 hogehoge-ap1 hogehoge ADD...
NFSNFS      9
MongoDB     MongoDB                   (           )     "app" : "hogehoge",     "userid" : "12345",                       ...
EMR Hive Pig        Hadoop Streaming                             Hadoop                            Streaming              ...
m2.4xlarge × 1         4.9GB           85EMR(m2.xlarge) × 5       4.9GB           44  m2.4xlarge × 1         7.2GB   138EM...
EMR        CPU      13
14
( )NFS   Amazon S3                   EMRS3                                 EMR                          S3             EMR...
S3     botoS3     c3cmd               S3EMR     Mapper,Reducer,Python2.7MongoDB     pymongo                  MongoDBEMR   ...
EMR      17
EMRS3 EC2⇔S3            20MB/sec Hadoop HadoopStreamingEMR                      18
GB/      19
20
21
Upcoming SlideShare
Loading in...5
×

ソーシャルゲームログ解析基盤のHadoop活用事例

9,221

Published on

【エンジニアカフェEvent×gumiStudy】ソーシャルゲームの解析を支える技術-Hadoop編-
http://www.facebook.com/event.php?eid=245262765524522
の発表資料です。

Published in: Technology, Business
0 Comments
25 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
9,221
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
60
Comments
0
Likes
25
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • ソーシャルゲームログ解析基盤のHadoop活用事例

    1. 1. Hadoopgumi 1
    2. 2. :Twitter:@CkRealgumi 2 AWS S3,EMR 2
    3. 3. 1.2. Elastic MapReduce EMR3.4. EMR 3
    4. 4. 4
    5. 5. 2 ( 1 )fluentd 5
    6. 6. 1 1,000 / AP APAP AP DBd fluentd fluentd mongos mongod(PRIMARY) DB config mongod(SECONDARY) DB fluentd mongos mongod(SECONDARY) config ReplicaSets & Sharding NFS 6
    7. 7. EMR 7
    8. 8. 8.5GB 1.4GB / IDNov 1 23:59:59 hogehoge-ap1 hogehoge ADD_MONEY 12345[BeforeMoney] 67979 [AfterMoney] 68024 [Money] 45Nov 1 23:59:59 hogehoge-ap2 hogehoge CONSUME_POWER 12345[BeforePower] 25 [AfterPower] 20 [ConsumePower] 5 8
    9. 9. NFSNFS 9
    10. 10. MongoDB MongoDB ( ) "app" : "hogehoge", "userid" : "12345", ID "dateint" : 20111101, "hourint" : 23, "actions" : [ "CONSUME_POWER", MongoDB Sharding "ADD_MONEY" ], "records" : [ "action" : "ADD_MONEY", "timeint" : 235959, ] 10
    11. 11. EMR Hive Pig Hadoop Streaming Hadoop Streaming (Python) 11
    12. 12. m2.4xlarge × 1 4.9GB 85EMR(m2.xlarge) × 5 4.9GB 44 m2.4xlarge × 1 7.2GB 138EMR(m2.xlarge) × 5 7.2GB 69 (Macbook Air) 3.6GB 30 … 12
    13. 13. EMR CPU 13
    14. 14. 14
    15. 15. ( )NFS Amazon S3 EMRS3 EMR S3 EMR config MongoDB mongos 15
    16. 16. S3 botoS3 c3cmd S3EMR Mapper,Reducer,Python2.7MongoDB pymongo MongoDBEMR Client Tool(Ruby) EMR 16
    17. 17. EMR 17
    18. 18. EMRS3 EC2⇔S3 20MB/sec Hadoop HadoopStreamingEMR 18
    19. 19. GB/ 19
    20. 20. 20
    21. 21. 21
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×