Hadoop導入事例 in クックパッド

19,527 views
19,303 views

Published on

4/2, 4/3に #urapad で使った発表資料

Published in: Technology
0 Comments
15 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
19,527
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
362
Comments
0
Likes
15
Embeds 0
No embeds

No notes for slide


































  • Hadoop導入事例 in クックパッド

    1. 1. Hadoop in
    2. 2. • id:sasata299 ( ) • Hadoop • • http://blog.livedoor.jp/sasata299/
    3. 3. Hadoop
    4. 4. Hadoop MySQL
    5. 5. … Hadoop MySQL
    6. 6. :-)
    7. 7. 1. Hadoop 2. 3. 4. 5.
    8. 8. Hadoop
    9. 9. 915 30 3 1
    10. 10. ( )
    11. 11. ( )
    12. 12. ‣ ‣ GROUP BY ( ( Д`) ‣ 7000 ( )
    13. 13. !!
    14. 14. Hadoop
    15. 15. ‣ Google MapReduce ‣ ‣
    16. 16. mapper reducer ( ) ( )
    17. 17. ‣ Hadoop Streaming ‣ Ruby ‣ Cloudera CDH1 (0.18.3) ‣ EC2 Hadoop ( 10 50 ) ‣ Hadoop S3
    18. 18. S3 Native FileSystem (s3n://) ‣ ‣ 5GB S3 Block FileSystem (s3://) ‣ ‣ HDFS ‣
    19. 19. ( ) mapper ( )
    20. 20. mapper, reducer
    21. 21. master ‣ -file master slave hadoop jar xxx.jar -mapper hoge.rb -reducer fuga.rb -file hoge.rb -file fuga.rb -file outdata ‣ mapper, reducer File.open File.open(‘outdata’) {|f| ...}
    22. 22. S3 ‣ hadoop cat ※ ‣ -cacheFile S3 slave ( File.open) ※ hadoop jar xxx.jar -mapper hoge.rb -reducer fuga.rb -file hoge.rb -file fuga.rb -cacheFile s3n://path/to/outdata#othername mapper reducer
    23. 23. 7000 ( )→
    24. 24. 7000 ( )→ 30
    25. 25. Hadoop++
    26. 26. Hadoop … mapper, reducer … Hadoop …
    27. 27. 1. 2. Hadoop 3. Hadoop 4. Hadoop 5. Hadoop ( ) 6. (Excel ) 7. Hadoop
    28. 28. Hadoop !!
    29. 29. • MySQL Hadoop • Hadoop EC2/S3 • Hadoop Streaming Hadoop

    ×