Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hadoop導入事例 in クックパッド

20,155 views

Published on

4/2, 4/3に #urapad で使った発表資料

Published in: Technology
  • Be the first to comment

Hadoop導入事例 in クックパッド

  1. 1. Hadoop in
  2. 2. • id:sasata299 ( ) • Hadoop • • http://blog.livedoor.jp/sasata299/
  3. 3. Hadoop
  4. 4. Hadoop MySQL
  5. 5. … Hadoop MySQL
  6. 6. :-)
  7. 7. 1. Hadoop 2. 3. 4. 5.
  8. 8. Hadoop
  9. 9. 915 30 3 1
  10. 10. ( )
  11. 11. ( )
  12. 12. ‣ ‣ GROUP BY ( ( Д`) ‣ 7000 ( )
  13. 13. !!
  14. 14. Hadoop
  15. 15. ‣ Google MapReduce ‣ ‣
  16. 16. mapper reducer ( ) ( )
  17. 17. ‣ Hadoop Streaming ‣ Ruby ‣ Cloudera CDH1 (0.18.3) ‣ EC2 Hadoop ( 10 50 ) ‣ Hadoop S3
  18. 18. S3 Native FileSystem (s3n://) ‣ ‣ 5GB S3 Block FileSystem (s3://) ‣ ‣ HDFS ‣
  19. 19. ( ) mapper ( )
  20. 20. mapper, reducer
  21. 21. master ‣ -file master slave hadoop jar xxx.jar -mapper hoge.rb -reducer fuga.rb -file hoge.rb -file fuga.rb -file outdata ‣ mapper, reducer File.open File.open(‘outdata’) {|f| ...}
  22. 22. S3 ‣ hadoop cat ※ ‣ -cacheFile S3 slave ( File.open) ※ hadoop jar xxx.jar -mapper hoge.rb -reducer fuga.rb -file hoge.rb -file fuga.rb -cacheFile s3n://path/to/outdata#othername mapper reducer
  23. 23. 7000 ( )→
  24. 24. 7000 ( )→ 30
  25. 25. Hadoop++
  26. 26. Hadoop … mapper, reducer … Hadoop …
  27. 27. 1. 2. Hadoop 3. Hadoop 4. Hadoop 5. Hadoop ( ) 6. (Excel ) 7. Hadoop
  28. 28. Hadoop !!
  29. 29. • MySQL Hadoop • Hadoop EC2/S3 • Hadoop Streaming Hadoop

×