ソーシャルアプリでの Amazon Elastic MapReduce 活用事例

  • 9,685 views
Uploaded on

TokyoWebmining #7 …

TokyoWebmining #7
buhii

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
9,685
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
41
Comments
0
Likes
11

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Amazon Elastic MapReduce Takahiro Kamatani gumi, Inc. 2010/09/26 Sunday, September 26, 2010
  • 2. • • • Amazon Elastic MapReduce • Sunday, September 26, 2010
  • 3. • Twitter: @buhii • • gumi @ http://www.kansei.tsukuba.ac.jp/~uchiyamalab/beacon • • beacon • gumi @ynil Sunday, September 26, 2010
  • 4. Sunday, September 26, 2010
  • 5. gumi • mixi, , GREE • python Django • Amazon Web Services (EC2 + RDS) • • DB Sunday, September 26, 2010
  • 6. • PV , UU • DAU Daily Active Users • • • ÷ DAU • ARPU Average Revenue Per User • Sunday, September 26, 2010
  • 7. Amazon Web Service AWS Sunday, September 26, 2010
  • 8. Sunday, September 26, 2010
  • 9. Sunday, September 26, 2010
  • 10. Sunday, September 26, 2010
  • 11. Sunday, September 26, 2010
  • 12. Sunday, September 26, 2010
  • 13. Amazon Elastic MapReduce Sunday, September 26, 2010
  • 14. MapReduce Mapper Key, Value Mapper key Sort / Shuffle Reducer Reducer key, value Mapper, Reducer Sunday, September 26, 2010
  • 15. Amazon Elastic MapReduce • Hadoop • Hadoop Streaming Mapper Reducer Ruby, Perl, Python, PHP, R, Bash, C++ • EC2 job • Sunday, September 26, 2010
  • 16. Example Task • • • Sunday, September 26, 2010
  • 17. • Mapper • Apache Log • ID key value Reducer Sunday, September 26, 2010
  • 18. • Reducer • sort/shuffle ID Reducer • ID Sunday, September 26, 2010
  • 19. Reducer 31758623 2010-08-20 42346572 2010-09-05,2010-09-06 31977736 2010-08-11,2010-08-12,2010-08-13,2010-08-14 14007991 2010-08-16 35995849 2010-08-12,2010-08-13,2010-08-14 34246688 2010-08-21,2010-08-22,2010-08-23,2010-08-27 ... PC Sunday, September 26, 2010
  • 20. Amazon Elastic Mapreduce • AWS • S3 Mapper, Reducer → s3cmd, S3Fox Organizer, Cyberduck • Job OK • Sunday, September 26, 2010
  • 21. Sunday, September 26, 2010
  • 22. Sunday, September 26, 2010
  • 23. Streaming Sunday, September 26, 2010
  • 24. {Input, Output} Location, Mapper, Reducer S3 gzip Hadoop Extra Args -jobconf stream.recordreader.compression=gzip input Location Extra Args -input s3n://(bucket )/( )/access_log.* Sunday, September 26, 2010
  • 25. Sunday, September 26, 2010
  • 26. Sunday, September 26, 2010
  • 27. Sunday, September 26, 2010
  • 28. Debug Sunday, September 26, 2010
  • 29. Sunday, September 26, 2010
  • 30. • Hadoop • MapReduce • • Sunday, September 26, 2010
  • 31. Hadoop • • S3 gzip • • hadoop EC2 • ( 20 ...) Sunday, September 26, 2010
  • 32. @ynil MapReduce http://nlpyutori.g.hatena.ne.jp/yaruki_nil/20100911/1284089305 Sunday, September 26, 2010
  • 33. Sunday, September 26, 2010
  • 34. Sunday, September 26, 2010
  • 35. Sunday, September 26, 2010
  • 36. MapReduce MapReduce Google Map Reduce Map Reduce MapReduce C++ Java Python Wikipedia “MapReduce” http://ja.wikipedia.org/wiki/MapReduce Sunday, September 26, 2010
  • 37. cron • PV, UU NFS CSV • DB → DB • PV, UU Sunday, September 26, 2010