2011년 아꿈사 시작발표. MapReduce

1. MapReduce 아키텍트를 꿈꾸는 사람들cafe.naver.com/architect1 현수명 soomong.net #soomong

2. MapReduce? Map + Reduce Execution Overview

3. MapReduce? mechanism for processing large data developed within Google from a functional language <key,value> pair

5. Execution Overview Google

6. Execution Overview Hadoop

7. Map <Only value> Map takes as input a function and a sequence of values. It then applies the function to each value in the sequence func Map (func,values) applied values values

8. Map <Only value> f(x)=x*x Map (func,values) (1,4,9) (1,2,3) Clojure (map (fn [x] (x*x)) [1 2 3])

9. Reduce <Only value> A reduce combines all the elements of a sequence using a binary operation func Reduce (func,values) applied value values

10. Reduce <Only value> + Reduce (func,values) (14) (1,4,9) Clojure (reduce + [1 4 9])

11. Map <key,value> f(x)=x*x Map (func,values) (keyA,1) (keyB,4) (keyA,9) (1,2,3)

12. Reduce <key,value> + Reduce (func,(key,value)) (keyA,10) (keyB,4) (keyA,1) (keyB,4) (keyA,9)

13. Map + Reduce (1,2,3…99998,99999,100000) f(x)=x*x f(x)=x*x f(x)=x*x Map (func,values) Map (func,values) Map (func,values) + + Reduce (func,values) Reduce (func,values) Reduce (func,values) (keyA,107) (keyB,29) (…,…,…) (…) (result) (keyA,1) (keyB,4) (keyA,9) (1,2,3) (4,5,6) (99998,99999,…) (keyA,16) (keyB,25) (keyA,81) + …

14. Execution Example “Stay Hungry Stay Foolish Don’t settle” Word Counter Stay 2 Hungry 1 Foolish 1 Don’t 1 Settle 1

15. Map map(String key, String value): // key: document name // value: document contents for each word w in value: EmitIntermediate(w, "1");

16. Reduce reduce(String key, Iterator values): // key: a word // values: a list of counts int result = 0; for each v in values: result +=ParseInt(v); Emit(AsString(result));

17. Stay Hungry Stay Foolish Don’t settle Stay Hungry Stay Foolish Don’t settle (1) Shards the input files

18. Stay Hungry Stay Foolish Don’t settle (2) Master pick idle workers assign a map or reduce task

19. <Stay,1> <Hungry,1> <Stay,1> <Foolish,1> Stay Hungry Stay Foolish Don’t settle <Don’t,1> <settle,1> (3) Map worker reads input shard, do Map func -> intermediate <key,value>

20. <Stay,1> <Hungry,1> <Stay,1> <Foolish,1> <Don’t,1> <settle,1> (4) Write intermediate <key,value> on local disk

21. <Stay,1> <Hungry,1> <Stay,1> <Foolish,1> <Foolish,1> <Hungry,1> <Stay,(1,1)> <Don’t,1> <settle,1> <Don’t,1> <settle,1> (5) Reduce worker reads intermediate data sort by key

22. <Foolish,1> <Hungry,1> <Stay,(1,1)> <Foolish,1> <Hungry,1> <Stay,2> <Don’t,1> <settle,1> <Don’t,1> <settle,1> (6) Reduce worker do reduce func -> wrtie output

23. <Foolish,1> <Hungry,1> <Stay,2> <Don’t,1> <settle,1> <Foolish,1> <Hungry,1> <Stay,2> <Don’t,1> <settle,1> (7) All map tasks and reduce tasks have been completed master wakes up the user program -> return Result

24. Map map = function () { for (var key in this) { emit(key, {count:1}); }

25. Reduce reduce = function (key, emits) { total = 0; for (vari in emits) { total += emits[i].count; } return {count:total};

26. Execution mr = db.foo.mapReduce (map,reduce,{out:"mongoDBmapReduce"}) { "result" : "mongoDBmapReduce", "timeMillis" : 11, "counts" : { "input" : 4, "emit" : 8, "output" : 5 }, "ok" : 1, }

27. Reference Google MapReduce http://code.google.com/intl/ko-KR/edu/parallel/mapreduce-tutorial.html MongoDBMapReduce http://kylebanker.com/blog/2009/12/mongodb-map-reduce-basics/ HadoopMapReduce http://hadoop.apache.org/common/docs/current/mapred_tutorial.html

28. 감사합니다