Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Hadoopgumi           1
:Twitter:@CkRealgumi 2        AWS S3,EMR                     2
1.2. Elastic MapReduce EMR3.4. EMR                      3
4
2             ( 1   )fluentd         5
1                        1,000      /      AP     APAP       AP                                           DBd         fluen...
EMR      7
8.5GB             1.4GB /                                                       IDNov 1 23:59:59 hogehoge-ap1 hogehoge ADD...
NFSNFS      9
MongoDB     MongoDB                   (           )     "app" : "hogehoge",     "userid" : "12345",                       ...
EMR Hive Pig        Hadoop Streaming                             Hadoop                            Streaming              ...
m2.4xlarge × 1         4.9GB           85EMR(m2.xlarge) × 5       4.9GB           44  m2.4xlarge × 1         7.2GB   138EM...
EMR        CPU      13
14
( )NFS   Amazon S3                   EMRS3                                 EMR                          S3             EMR...
S3     botoS3     c3cmd               S3EMR     Mapper,Reducer,Python2.7MongoDB     pymongo                  MongoDBEMR   ...
EMR      17
EMRS3 EC2⇔S3            20MB/sec Hadoop HadoopStreamingEMR                      18
GB/      19
20
21
Upcoming SlideShare
Loading in …5
×
Upcoming SlideShare
Hadoopを業務で使ってみた
Next
Download to read offline and view in fullscreen.

25

Share

Download to read offline

ソーシャルゲームログ解析基盤のHadoop活用事例

Download to read offline

【エンジニアカフェEvent×gumiStudy】ソーシャルゲームの解析を支える技術-Hadoop編-
http://www.facebook.com/event.php?eid=245262765524522
の発表資料です。

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

ソーシャルゲームログ解析基盤のHadoop活用事例

  1. 1. Hadoopgumi 1
  2. 2. :Twitter:@CkRealgumi 2 AWS S3,EMR 2
  3. 3. 1.2. Elastic MapReduce EMR3.4. EMR 3
  4. 4. 4
  5. 5. 2 ( 1 )fluentd 5
  6. 6. 1 1,000 / AP APAP AP DBd fluentd fluentd mongos mongod(PRIMARY) DB config mongod(SECONDARY) DB fluentd mongos mongod(SECONDARY) config ReplicaSets & Sharding NFS 6
  7. 7. EMR 7
  8. 8. 8.5GB 1.4GB / IDNov 1 23:59:59 hogehoge-ap1 hogehoge ADD_MONEY 12345[BeforeMoney] 67979 [AfterMoney] 68024 [Money] 45Nov 1 23:59:59 hogehoge-ap2 hogehoge CONSUME_POWER 12345[BeforePower] 25 [AfterPower] 20 [ConsumePower] 5 8
  9. 9. NFSNFS 9
  10. 10. MongoDB MongoDB ( ) "app" : "hogehoge", "userid" : "12345", ID "dateint" : 20111101, "hourint" : 23, "actions" : [ "CONSUME_POWER", MongoDB Sharding "ADD_MONEY" ], "records" : [ "action" : "ADD_MONEY", "timeint" : 235959, ] 10
  11. 11. EMR Hive Pig Hadoop Streaming Hadoop Streaming (Python) 11
  12. 12. m2.4xlarge × 1 4.9GB 85EMR(m2.xlarge) × 5 4.9GB 44 m2.4xlarge × 1 7.2GB 138EMR(m2.xlarge) × 5 7.2GB 69 (Macbook Air) 3.6GB 30 … 12
  13. 13. EMR CPU 13
  14. 14. 14
  15. 15. ( )NFS Amazon S3 EMRS3 EMR S3 EMR config MongoDB mongos 15
  16. 16. S3 botoS3 c3cmd S3EMR Mapper,Reducer,Python2.7MongoDB pymongo MongoDBEMR Client Tool(Ruby) EMR 16
  17. 17. EMR 17
  18. 18. EMRS3 EC2⇔S3 20MB/sec Hadoop HadoopStreamingEMR 18
  19. 19. GB/ 19
  20. 20. 20
  21. 21. 21
  • TatsuyaTsuruoka

    May. 1, 2014
  • TatsuroFujii

    Apr. 22, 2013
  • ssuser82d97a

    Apr. 11, 2013
  • yatchi24

    Apr. 3, 2013
  • ShujiOchi

    Mar. 15, 2013
  • uekkee

    Mar. 12, 2013
  • j5ik2o

    Mar. 12, 2013
  • ryuichifutamura

    Feb. 21, 2013
  • iino

    Dec. 6, 2012
  • ikko615

    Aug. 22, 2012
  • MasakazuShinya

    Jul. 30, 2012
  • TetsuyaMatsuda

    Jun. 27, 2012
  • kishir1

    Feb. 17, 2012
  • madokanez

    Feb. 17, 2012
  • totobook

    Feb. 15, 2012
  • KentaroMatsunaga

    Feb. 1, 2012
  • hrsm1979

    Dec. 13, 2011
  • tednag

    Dec. 6, 2011
  • invent

    Dec. 4, 2011
  • matsumana0101

    Nov. 23, 2011

【エンジニアカフェEvent×gumiStudy】ソーシャルゲームの解析を支える技術-Hadoop編- http://www.facebook.com/event.php?eid=245262765524522 の発表資料です。

Views

Total views

11,283

On Slideshare

0

From embeds

0

Number of embeds

3,187

Actions

Downloads

62

Shares

0

Comments

0

Likes

25

×