Your SlideShare is downloading. ×
961万人の食卓を支えるデータ解析
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

961万人の食卓を支えるデータ解析

9,682
views

Published on

2010/10/18のJJUG CCC 2010 Fallの講演で使用したスライドです

2010/10/18のJJUG CCC 2010 Fallの講演で使用したスライドです

Published in: Technology, Spiritual

0 Comments
35 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
9,682
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
176
Comments
0
Likes
35
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide












































































































  • Transcript

    • 1. 961
    • 2. • • • Hadoop (Cloudera) • Elastic MapReduce • •
    • 3. • • • Hadoop (Cloudera) • Elastic MapReduce • •
    • 4. • (@sasata299) • 2009 8 JOIN • • Hadoop
    • 5. • • • Hadoop (Cloudera) • Elastic MapReduce • •
    • 6. • 961 • 30 3 1 • - -
    • 7. • • ( , , ...) - - ( , , , …) -
    • 8. • Hadoop MySQL • - GROUP BY • 7000 … • (´Д` )
    • 9. • MySQL - • - -
    • 10. • • • Hadoop (Cloudera) • Elastic MapReduce • •
    • 11. Hadoop • Google MapReduce OSS • - - - -
    • 12. Hadoop master ( ) slave ( )
    • 13. Hadoop master ( ) slave ( ) Map
    • 14. Hadoop master ( ) slave ( ) <key,value> Map Shuffle & Sort
    • 15. Hadoop master ( ) slave ( ) <key,value> Map Reduce Shuffle & Sort
    • 16. • Hadoop Streaming (Ruby ) • EC2 Cloudera Hadoop - Cloudera CDH1 - Hadoop 0.18.3 • S3
    • 17. MySQL → Hadoop • • GROUP BY MapReduce - ( ) - key • JOIN MapReduce •
    • 18. (1) master (2) S3
    • 19. (1) master (2) S3
    • 20. (1) master master slave scp (2) S3
    • 21. (1) master master slave scp (2) S3 S3 slave scp
    • 22. MySQL vs Hadoop 7000 MySQL Hadoop MySQL Hadoop
    • 23. MySQL vs Hadoop ( Д ) 7000 30 MySQL Hadoop MySQL Hadoop
    • 24. Hadoop++ ←Hadoop ↓MySQL
    • 25. • • • Hadoop (Cloudera) • Elastic MapReduce • •
    • 26. • Hadoop - • Hadoop (HADOOP-6254) - S3 - SocketTimeoutException
    • 27. • EMR (Elastic MapReduce) - Amazon Hadoop • Cloudera CDH2 -
    • 28. AMI (Amazon Machine UP Image) EMR CDH2
    • 29. AMI (Amazon Machine UP Image) EMR CDH2
    • 30. EMR Job Flow ( )
    • 31. EMR BootStrap Action Job Flow ( )
    • 32. EMR BootStrap Action Step (Hadoop Job) Job Flow ( )
    • 33. EMR BootStrap Action Step (Hadoop Job) Job Flow ( )
    • 34. • - - --alive • AMI - AMI - BootStrap Action
    • 35. Created job flow j-8IXS98OW1WEE ID
    • 36. Hadoop
    • 37. • - mapred.child.java.opts - streaming • - - ElasticMapReduce-master 5100
    • 38. • • • Hadoop (Cloudera) • Elastic MapReduce • •
    • 39. • Map - • Reduce - key Reduce -
    • 40. UU Map Reduce
    • 41. UU Map ID Reduce
    • 42. UU Map Reduce ID
    • 43. UU Map Reduce ID
    • 44. Map Reduce
    • 45. Map ID key Reduce
    • 46. Map Reduce key Reduce
    • 47. Map 100 100 Reduce key Reduce
    • 48. × Map 100 × 100 Reduce key Reduce
    • 49. × Map 100 ×100 Reduce Reduce key sort
    • 50. × Map 100 ×100 Reduce Reduce key sort
    • 51. Hadoop • - Hadoop
    • 52. Hadoop • - Hadoop
    • 53. Hadoop • - Hadoop
    • 54. Hadoop • - Hadoop
    • 55. Hadoop • - Hadoop
    • 56. • • • Hadoop (Cloudera) • Elastic MapReduce • •
    • 57. • Hadoop - - - Reduce