• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
961万人の食卓を支えるデータ解析
 

961万人の食卓を支えるデータ解析

on

  • 10,024 views

2010/10/18のJJUG CCC 2010 Fallの講演で使用したスライドです

2010/10/18のJJUG CCC 2010 Fallの講演で使用したスライドです

Statistics

Views

Total Views
10,024
Views on SlideShare
9,041
Embed Views
983

Actions

Likes
33
Downloads
174
Comments
0

5 Embeds 983

http://blog.livedoor.jp 964
https://twitter.com 13
http://webcache.googleusercontent.com 4
http://paper.li 1
http://us-w1.rockmelt.com 1

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />

961万人の食卓を支えるデータ解析 961万人の食卓を支えるデータ解析 Presentation Transcript

  • 961
  • • • • Hadoop (Cloudera) • Elastic MapReduce • •
  • • • • Hadoop (Cloudera) • Elastic MapReduce • •
  • • (@sasata299) • 2009 8 JOIN • • Hadoop
  • • • • Hadoop (Cloudera) • Elastic MapReduce • •
  • • 961 • 30 3 1 • - -
  • • • ( , , ...) - - ( , , , …) -
  • • Hadoop MySQL • - GROUP BY • 7000 … • (´Д` )
  • • MySQL - • - -
  • • • • Hadoop (Cloudera) • Elastic MapReduce • •
  • Hadoop • Google MapReduce OSS • - - - -
  • Hadoop master ( ) slave ( )
  • Hadoop master ( ) slave ( ) Map
  • Hadoop master ( ) slave ( ) <key,value> Map Shuffle & Sort
  • Hadoop master ( ) slave ( ) <key,value> Map Reduce Shuffle & Sort
  • • Hadoop Streaming (Ruby ) • EC2 Cloudera Hadoop - Cloudera CDH1 - Hadoop 0.18.3 • S3
  • MySQL → Hadoop • • GROUP BY MapReduce - ( ) - key • JOIN MapReduce •
  • (1) master (2) S3
  • (1) master (2) S3
  • (1) master master slave scp (2) S3
  • (1) master master slave scp (2) S3 S3 slave scp
  • MySQL vs Hadoop 7000 MySQL Hadoop MySQL Hadoop
  • MySQL vs Hadoop ( Д ) 7000 30 MySQL Hadoop MySQL Hadoop
  • Hadoop++ ←Hadoop ↓MySQL
  • • • • Hadoop (Cloudera) • Elastic MapReduce • •
  • • Hadoop - • Hadoop (HADOOP-6254) - S3 - SocketTimeoutException
  • • EMR (Elastic MapReduce) - Amazon Hadoop • Cloudera CDH2 -
  • AMI (Amazon Machine UP Image) EMR CDH2
  • AMI (Amazon Machine UP Image) EMR CDH2
  • EMR Job Flow ( )
  • EMR BootStrap Action Job Flow ( )
  • EMR BootStrap Action Step (Hadoop Job) Job Flow ( )
  • EMR BootStrap Action Step (Hadoop Job) Job Flow ( )
  • • - - --alive • AMI - AMI - BootStrap Action
  • Created job flow j-8IXS98OW1WEE ID
  • Hadoop
  • • - mapred.child.java.opts - streaming • - - ElasticMapReduce-master 5100
  • • • • Hadoop (Cloudera) • Elastic MapReduce • •
  • • Map - • Reduce - key Reduce -
  • UU Map Reduce
  • UU Map ID Reduce
  • UU Map Reduce ID
  • UU Map Reduce ID
  • Map Reduce
  • Map ID key Reduce
  • Map Reduce key Reduce
  • Map 100 100 Reduce key Reduce
  • × Map 100 × 100 Reduce key Reduce
  • × Map 100 ×100 Reduce Reduce key sort
  • × Map 100 ×100 Reduce Reduce key sort
  • Hadoop • - Hadoop
  • Hadoop • - Hadoop
  • Hadoop • - Hadoop
  • Hadoop • - Hadoop
  • Hadoop • - Hadoop
  • • • • Hadoop (Cloudera) • Elastic MapReduce • •
  • • Hadoop - - - Reduce