961万人の食卓を支えるデータ解析

• • • Hadoop (Cloudera) • Elastic MapReduce • •

• (@sasata299) • 2009 8 JOIN • • Hadoop

• 961 • 30 3 1 • - -

• • ( , , ...) - - ( , , , …) -

• Hadoop MySQL • - GROUP BY • 7000 … • (´Д` )

• MySQL - • - -

Hadoop • Google MapReduce OSS • - - - -

Hadoop master ( ) slave ( )

Hadoop master ( ) slave ( ) Map

Hadoop master ( ) slave ( ) <key,value> Map Shufﬂe & Sort

Hadoop master ( ) slave ( ) <key,value> Map Reduce Shufﬂe & Sort

• Hadoop Streaming (Ruby ) • EC2 Cloudera Hadoop - Cloudera CDH1 - Hadoop 0.18.3 • S3

MySQL → Hadoop • • GROUP BY MapReduce - ( ) - key • JOIN MapReduce •

(1) master (2) S3

(1) master master slave scp (2) S3

(1) master master slave scp (2) S3 S3 slave scp

MySQL vs Hadoop 7000 MySQL Hadoop MySQL Hadoop

MySQL vs Hadoop ( Д ) 7000 30 MySQL Hadoop MySQL Hadoop

Hadoop++ ←Hadoop ↓MySQL

• Hadoop - • Hadoop (HADOOP-6254) - S3 - SocketTimeoutException

• EMR (Elastic MapReduce) - Amazon Hadoop • Cloudera CDH2 -

AMI (Amazon Machine UP Image) EMR CDH2

EMR Job Flow ( )

EMR BootStrap Action Job Flow ( )

EMR BootStrap Action Step (Hadoop Job) Job Flow ( )

• - - --alive • AMI - AMI - BootStrap Action

Created job ﬂow j-8IXS98OW1WEE ID

Hadoop

• - mapred.child.java.opts - streaming • - - ElasticMapReduce-master 5100

• Map - • Reduce - key Reduce -

UU Map Reduce

UU Map ID Reduce

UU Map Reduce ID

Map Reduce

Map ID key Reduce

Map Reduce key Reduce

Map 100 100 Reduce key Reduce

× Map 100 × 100 Reduce key Reduce

× Map 100 ×100 Reduce Reduce key sort

Hadoop • - Hadoop

• Hadoop - - - Reduce

http://blog.livedoor.jp	564
http://webcache.googleusercontent.com	3
http://paper.li	1
http://us-w1.rockmelt.com	1

961万人の食卓を支えるデータ解析

by Tatsuya Sasaki on Oct 19, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 569

Statistics

961万人の食卓を支えるデータ解析 — Presentation Transcript