800万人の"食べたい"をHadoopで分散処理
Upcoming SlideShare
Loading in...5
×
 

800万人の"食べたい"をHadoopで分散処理

on

  • 6,313 views

 

Statistics

Views

Total Views
6,313
Views on SlideShare
4,339
Embed Views
1,974

Actions

Likes
5
Downloads
67
Comments
0

10 Embeds 1,974

http://techlife.cookpad.com 1893
http://webmemo.uzuralife.com 66
http://webcache.googleusercontent.com 6
http://www.slideshare.net 2
http://d.hatena.ne.jp 2
url_unknown 1
http://livedoor.blogcms.jp 1
http://static.slideshare.net 1
resource://brief-content 1
http://cookpadtech.wordpress.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

800万人の"食べたい"をHadoopで分散処理 800万人の"食べたい"をHadoopで分散処理 Presentation Transcript

  • 800 ” ” Hadoop
  • • id:sasata299 ( ) • Ruby Perl • • http://blog.livedoor.jp/sasata299/
  • 1. Hadoop 2. Hadoop 3. 4. 5.
  • Hadoop
  • 816 30 3 1
  • ( )
  • ( )
  • • • GROUP BY ( ( Д`) • 7000 ( )
  • !!
  • Hadoop
  • Hadoop
  • • Google MapReduce • • • HDFS
  • ( ) ( ) Mapper Reducer ( ) ( )
  • ‣ Hadoop Streaming ‣ Ruby ‣ EC2 Hadoop ( 50 ) ‣ HDFS S3 (s3fs)
  • ( ) ( ) Mapper ( ) ( )
  • HDFS Mapper, Reducer
  • Hadoop cat `hadoop dfs -cat s3://xxx/user/root/in/hoge` ※
  • require ‘csv’ path = ‘s3://xxx/user/root/in/user_info’ # user_info = `hadoop dfs -cat #{path}` ARGF.each_line do |line| # line.chomp! csv = CSV.parse(line) # user_info end
  • 7000 ( )→
  • 7000 ( )→ 30
  • Hadoop !!
  • • Mapper, Reducer HDFS (Hadoop cat) • • DB