Hadoop導入事例 in クックパッド

  • 18,201 views
Uploaded on

4/2, 4/3に #urapad で使った発表資料

4/2, 4/3に #urapad で使った発表資料

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
18,201
On Slideshare
0
From Embeds
0
Number of Embeds
6

Actions

Shares
Downloads
345
Comments
0
Likes
15

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide


































Transcript

  • 1. Hadoop in
  • 2. • id:sasata299 ( ) • Hadoop • • http://blog.livedoor.jp/sasata299/
  • 3. Hadoop
  • 4. Hadoop MySQL
  • 5. … Hadoop MySQL
  • 6. :-)
  • 7. 1. Hadoop 2. 3. 4. 5.
  • 8. Hadoop
  • 9. 915 30 3 1
  • 10. ( )
  • 11. ( )
  • 12. ‣ ‣ GROUP BY ( ( Д`) ‣ 7000 ( )
  • 13. !!
  • 14. Hadoop
  • 15. ‣ Google MapReduce ‣ ‣
  • 16. mapper reducer ( ) ( )
  • 17. ‣ Hadoop Streaming ‣ Ruby ‣ Cloudera CDH1 (0.18.3) ‣ EC2 Hadoop ( 10 50 ) ‣ Hadoop S3
  • 18. S3 Native FileSystem (s3n://) ‣ ‣ 5GB S3 Block FileSystem (s3://) ‣ ‣ HDFS ‣
  • 19. ( ) mapper ( )
  • 20. mapper, reducer
  • 21. master ‣ -file master slave hadoop jar xxx.jar -mapper hoge.rb -reducer fuga.rb -file hoge.rb -file fuga.rb -file outdata ‣ mapper, reducer File.open File.open(‘outdata’) {|f| ...}
  • 22. S3 ‣ hadoop cat ※ ‣ -cacheFile S3 slave ( File.open) ※ hadoop jar xxx.jar -mapper hoge.rb -reducer fuga.rb -file hoge.rb -file fuga.rb -cacheFile s3n://path/to/outdata#othername mapper reducer
  • 23. 7000 ( )→
  • 24. 7000 ( )→ 30
  • 25. Hadoop++
  • 26. Hadoop … mapper, reducer … Hadoop …
  • 27. 1. 2. Hadoop 3. Hadoop 4. Hadoop 5. Hadoop ( ) 6. (Excel ) 7. Hadoop
  • 28. Hadoop !!
  • 29. • MySQL Hadoop • Hadoop EC2/S3 • Hadoop Streaming Hadoop