Like this presentation? Why not share!

# MapReduce

## on Aug 02, 2011

• 1,520 views

2011년 아꿈사 시작발표.

2011년 아꿈사 시작발표.
MapReduce

### Views

Total Views
1,520
Views on SlideShare
1,380
Embed Views
140

Likes
3
Downloads
2
Comments
0

### 9 Embeds140

 http://soomong.net 107 http://www.hanrss.com 9 http://andstudy.com 6 http://feeds2.feedburner.com 6 http://www.andstudy.com 4 http://www.samsung.net 4 http://www.slideshare.net 2 http://soomong.github.io 1 http://feeds.feedburner.com 1
More...

### Upload Details

Uploaded via as Microsoft PowerPoint

### Usage Rights

© All Rights Reserved

### Report content

• Comment goes here.
Are you sure you want to
Your message goes here
Edit your comment

## MapReducePresentation Transcript

• MapReduce
아키텍트를 꿈꾸는 사람들cafe.naver.com/architect1
현수명 soomong.net
#soomong
• MapReduce?
Map + Reduce
Execution Overview
• MapReduce?
mechanism for processing large data
developed within Google
from a functional language
<key,value> pair
• Execution Overview
Google
• Execution Overview
Hadoop
• Map
<Only value>
Map takes as input a function and
a sequence of values.
It then applies the function
to each value in the sequence
func
Map
(func,values)
applied
values
values
• Map
<Only value>
f(x)=x*x
Map
(func,values)
(1,4,9)
(1,2,3)
Clojure
(map (fn [x] (x*x)) [1 2 3])
• Reduce
<Only value>
A reduce combines
all the elements of a sequence
using a binary operation
func
Reduce
(func,values)
applied
value
values
• Reduce
<Only value>
+
Reduce
(func,values)
(14)
(1,4,9)
Clojure
(reduce + [1 4 9])
• Map
<key,value>
f(x)=x*x
Map
(func,values)
(keyA,1)
(keyB,4)
(keyA,9)
(1,2,3)
• Reduce
<key,value>
+
Reduce
(func,(key,value))
(keyA,10)
(keyB,4)
(keyA,1)
(keyB,4)
(keyA,9)
• Map + Reduce
(1,2,3…99998,99999,100000)
f(x)=x*x
f(x)=x*x
f(x)=x*x
Map
(func,values)
Map
(func,values)
Map
(func,values)
+
+
Reduce
(func,values)
Reduce
(func,values)
Reduce
(func,values)
(keyA,107)
(keyB,29)
(…,…,…)
(…)
(result)
(keyA,1)
(keyB,4)
(keyA,9)
(1,2,3)
(4,5,6)
(99998,99999,…)
(keyA,16)
(keyB,25)
(keyA,81)
+

• Execution Example
“Stay Hungry Stay Foolish
Don’t settle”
Word Counter
Stay 2
Hungry 1
Foolish 1
Don’t 1
Settle 1
• Map
map(String key, String value):
// key: document name
// value: document contents
for each word w in value:
EmitIntermediate(w, "1");
• Reduce
reduce(String key, Iterator values):
// key: a word
// values: a list of counts
int result = 0;
for each v in values:
result +=ParseInt(v);
Emit(AsString(result));
• Stay Hungry Stay Foolish
Don’t settle
Stay Hungry Stay Foolish
Don’t settle
(1) Shards the input files
• Stay Hungry Stay Foolish
Don’t settle
(2) Master pick idle workers
assign a map or reduce task
• <Stay,1>
<Hungry,1>
<Stay,1>
<Foolish,1>
Stay Hungry Stay Foolish
Don’t settle
<Don’t,1>
<settle,1>
(3) Map worker reads input shard,
do Map func -> intermediate <key,value>
• <Stay,1>
<Hungry,1>
<Stay,1>
<Foolish,1>
<Don’t,1>
<settle,1>
(4) Write intermediate <key,value> on local disk
• <Stay,1>
<Hungry,1>
<Stay,1>
<Foolish,1>
<Foolish,1>
<Hungry,1>
<Stay,(1,1)>
<Don’t,1>
<settle,1>
<Don’t,1>
<settle,1>
(5) Reduce worker reads intermediate data
sort by key
• <Foolish,1>
<Hungry,1>
<Stay,(1,1)>
<Foolish,1>
<Hungry,1>
<Stay,2>
<Don’t,1>
<settle,1>
<Don’t,1>
<settle,1>
(6) Reduce worker do reduce func
-> wrtie output
• <Foolish,1>
<Hungry,1>
<Stay,2>
<Don’t,1>
<settle,1>
<Foolish,1>
<Hungry,1>
<Stay,2>
<Don’t,1>
<settle,1>
(7) All map tasks and reduce tasks have been completed
master wakes up the user program -> return Result
• Map
map = function () {
for (var key in this) {
emit(key, {count:1});
}
• Reduce
reduce = function (key, emits) {
total = 0;
for (vari in emits) {
total += emits[i].count;
}
return {count:total};
• Execution
mr = db.foo.mapReduce
(map,reduce,{out:"mongoDBmapReduce"})
{
"result" : "mongoDBmapReduce",
"timeMillis" : 11,
"counts" : {
"input" : 4,
"emit" : 8,
"output" : 5
},
"ok" : 1,
}
• Reference
Google MapReduce
http://code.google.com/intl/ko-KR/edu/parallel/mapreduce-tutorial.html
MongoDBMapReduce
http://kylebanker.com/blog/2009/12/mongodb-map-reduce-basics/
HadoopMapReduce
http://hadoop.apache.org/common/docs/current/mapred_tutorial.html
• 감사합니다