961万人の食卓を支えるデータ解析
Upcoming SlideShare
Loading in...5
×
 

961万人の食卓を支えるデータ解析

on

  • 10,330 views

2010/10/18のJJUG CCC 2010 Fallの講演で使用したスライドです

2010/10/18のJJUG CCC 2010 Fallの講演で使用したスライドです

Statistics

Views

Total Views
10,330
Views on SlideShare
9,338
Embed Views
992

Actions

Likes
34
Downloads
175
Comments
0

5 Embeds 992

http://blog.livedoor.jp 973
https://twitter.com 13
http://webcache.googleusercontent.com 4
http://paper.li 1
http://us-w1.rockmelt.com 1

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />

961万人の食卓を支えるデータ解析 Presentation Transcript

  • 1. 961
  • 2. • • • Hadoop (Cloudera) • Elastic MapReduce • •
  • 3. • • • Hadoop (Cloudera) • Elastic MapReduce • •
  • 4. • (@sasata299) • 2009 8 JOIN • • Hadoop
  • 5. • • • Hadoop (Cloudera) • Elastic MapReduce • •
  • 6. • 961 • 30 3 1 • - -
  • 7. • • ( , , ...) - - ( , , , …) -
  • 8. • Hadoop MySQL • - GROUP BY • 7000 … • (´Д` )
  • 9. • MySQL - • - -
  • 10. • • • Hadoop (Cloudera) • Elastic MapReduce • •
  • 11. Hadoop • Google MapReduce OSS • - - - -
  • 12. Hadoop master ( ) slave ( )
  • 13. Hadoop master ( ) slave ( ) Map
  • 14. Hadoop master ( ) slave ( ) <key,value> Map Shuffle & Sort
  • 15. Hadoop master ( ) slave ( ) <key,value> Map Reduce Shuffle & Sort
  • 16. • Hadoop Streaming (Ruby ) • EC2 Cloudera Hadoop - Cloudera CDH1 - Hadoop 0.18.3 • S3
  • 17. MySQL → Hadoop • • GROUP BY MapReduce - ( ) - key • JOIN MapReduce •
  • 18. (1) master (2) S3
  • 19. (1) master (2) S3
  • 20. (1) master master slave scp (2) S3
  • 21. (1) master master slave scp (2) S3 S3 slave scp
  • 22. MySQL vs Hadoop 7000 MySQL Hadoop MySQL Hadoop
  • 23. MySQL vs Hadoop ( Д ) 7000 30 MySQL Hadoop MySQL Hadoop
  • 24. Hadoop++ ←Hadoop ↓MySQL
  • 25. • • • Hadoop (Cloudera) • Elastic MapReduce • •
  • 26. • Hadoop - • Hadoop (HADOOP-6254) - S3 - SocketTimeoutException
  • 27. • EMR (Elastic MapReduce) - Amazon Hadoop • Cloudera CDH2 -
  • 28. AMI (Amazon Machine UP Image) EMR CDH2
  • 29. AMI (Amazon Machine UP Image) EMR CDH2
  • 30. EMR Job Flow ( )
  • 31. EMR BootStrap Action Job Flow ( )
  • 32. EMR BootStrap Action Step (Hadoop Job) Job Flow ( )
  • 33. EMR BootStrap Action Step (Hadoop Job) Job Flow ( )
  • 34. • - - --alive • AMI - AMI - BootStrap Action
  • 35. Created job flow j-8IXS98OW1WEE ID
  • 36. Hadoop
  • 37. • - mapred.child.java.opts - streaming • - - ElasticMapReduce-master 5100
  • 38. • • • Hadoop (Cloudera) • Elastic MapReduce • •
  • 39. • Map - • Reduce - key Reduce -
  • 40. UU Map Reduce
  • 41. UU Map ID Reduce
  • 42. UU Map Reduce ID
  • 43. UU Map Reduce ID
  • 44. Map Reduce
  • 45. Map ID key Reduce
  • 46. Map Reduce key Reduce
  • 47. Map 100 100 Reduce key Reduce
  • 48. × Map 100 × 100 Reduce key Reduce
  • 49. × Map 100 ×100 Reduce Reduce key sort
  • 50. × Map 100 ×100 Reduce Reduce key sort
  • 51. Hadoop • - Hadoop
  • 52. Hadoop • - Hadoop
  • 53. Hadoop • - Hadoop
  • 54. Hadoop • - Hadoop
  • 55. Hadoop • - Hadoop
  • 56. • • • Hadoop (Cloudera) • Elastic MapReduce • •
  • 57. • Hadoop - - - Reduce