MapReduce

2,287 views

Published on

2011년 아꿈사 시작발표.
MapReduce

Published in: Technology
1 Comment
6 Likes
Statistics
Notes
  • http://dbmanagement.info/Tutorials/MapReduce.htm
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
2,287
On SlideShare
0
From Embeds
0
Number of Embeds
355
Actions
Shares
0
Downloads
2
Comments
1
Likes
6
Embeds 0
No embeds

No notes for slide

MapReduce

  1. 1. MapReduce<br />아키텍트를 꿈꾸는 사람들cafe.naver.com/architect1<br />현수명 soomong.net<br />#soomong<br />
  2. 2. MapReduce?<br />Map + Reduce<br />Execution Overview<br />
  3. 3. MapReduce?<br />mechanism for processing large data<br />developed within Google <br />from a functional language<br /><key,value> pair<br />
  4. 4.
  5. 5. Execution Overview<br /> Google<br />
  6. 6. Execution Overview<br />Hadoop<br />
  7. 7. Map<br /><Only value><br />Map takes as input a function and<br />a sequence of values. <br />It then applies the function <br />to each value in the sequence<br />func<br />Map<br />(func,values)<br />applied<br />values<br />values<br />
  8. 8. Map<br /><Only value><br />f(x)=x*x<br />Map<br />(func,values)<br />(1,4,9)<br />(1,2,3)<br />Clojure<br />(map (fn [x] (x*x)) [1 2 3]) <br />
  9. 9. Reduce<br /><Only value><br />A reduce combines<br />all the elements of a sequence <br />using a binary operation<br />func<br />Reduce<br />(func,values)<br />applied<br />value<br />values<br />
  10. 10. Reduce<br /><Only value><br />+<br />Reduce<br />(func,values)<br />(14)<br />(1,4,9)<br />Clojure<br />(reduce + [1 4 9])<br />
  11. 11. Map<br /><key,value><br />f(x)=x*x<br />Map<br />(func,values)<br />(keyA,1)<br />(keyB,4)<br />(keyA,9)<br />(1,2,3)<br />
  12. 12. Reduce<br /><key,value><br />+<br />Reduce<br />(func,(key,value))<br />(keyA,10)<br />(keyB,4)<br />(keyA,1)<br />(keyB,4)<br />(keyA,9)<br />
  13. 13. Map + Reduce<br />(1,2,3…99998,99999,100000)<br />f(x)=x*x<br />f(x)=x*x<br />f(x)=x*x<br />Map<br />(func,values)<br />Map<br />(func,values)<br />Map<br />(func,values)<br />+<br />+<br />Reduce<br />(func,values)<br />Reduce<br />(func,values)<br />Reduce<br />(func,values)<br />(keyA,107)<br />(keyB,29)<br />(…,…,…)<br />(…)<br />(result)<br />(keyA,1)<br />(keyB,4)<br />(keyA,9)<br />(1,2,3)<br />(4,5,6)<br />(99998,99999,…)<br />(keyA,16)<br />(keyB,25)<br />(keyA,81)<br />+<br />…<br />
  14. 14. Execution Example<br />“Stay Hungry Stay Foolish<br /> Don’t settle”<br />Word Counter<br />Stay 2<br />Hungry 1<br />Foolish 1<br />Don’t 1<br />Settle 1<br />
  15. 15. Map<br />map(String key, String value): <br />// key: document name <br />// value: document contents <br />for each word w in value: <br />EmitIntermediate(w, "1");<br />
  16. 16. Reduce<br />reduce(String key, Iterator values): <br />// key: a word <br /> // values: a list of counts <br />int result = 0; <br /> for each v in values: <br /> result +=ParseInt(v);<br /> Emit(AsString(result));<br />
  17. 17. Stay Hungry Stay Foolish<br />Don’t settle<br />Stay Hungry Stay Foolish<br />Don’t settle<br />(1) Shards the input files<br />
  18. 18. Stay Hungry Stay Foolish<br />Don’t settle<br />(2) Master pick idle workers<br /> assign a map or reduce task<br />
  19. 19. <Stay,1><br /><Hungry,1><br /><Stay,1><br /><Foolish,1><br />Stay Hungry Stay Foolish<br />Don’t settle<br /><Don’t,1><br /><settle,1><br />(3) Map worker reads input shard,<br />do Map func -> intermediate <key,value><br />
  20. 20. <Stay,1><br /><Hungry,1><br /><Stay,1><br /><Foolish,1><br /><Don’t,1><br /><settle,1><br />(4) Write intermediate <key,value> on local disk<br />
  21. 21. <Stay,1><br /><Hungry,1><br /><Stay,1><br /><Foolish,1><br /><Foolish,1><br /><Hungry,1><br /><Stay,(1,1)><br /><Don’t,1><br /><settle,1><br /><Don’t,1><br /><settle,1><br />(5) Reduce worker reads intermediate data<br />sort by key<br />
  22. 22. <Foolish,1><br /><Hungry,1><br /><Stay,(1,1)><br /><Foolish,1><br /><Hungry,1><br /><Stay,2><br /><Don’t,1><br /><settle,1><br /><Don’t,1><br /><settle,1><br />(6) Reduce worker do reduce func<br /> -> wrtie output<br />
  23. 23. <Foolish,1><br /><Hungry,1><br /><Stay,2><br /><Don’t,1><br /><settle,1><br /><Foolish,1><br /><Hungry,1><br /><Stay,2><br /><Don’t,1><br /><settle,1><br />(7) All map tasks and reduce tasks have been completed<br /> master wakes up the user program -> return Result<br />
  24. 24. Map<br />map = function () {<br /> for (var key in this) {<br /> emit(key, {count:1});<br /> }<br />
  25. 25. Reduce<br />reduce = function (key, emits) {<br /> total = 0;<br /> for (vari in emits) {<br /> total += emits[i].count;<br /> }<br /> return {count:total};<br />
  26. 26. Execution<br />mr = db.foo.mapReduce<br />(map,reduce,{out:"mongoDBmapReduce"})<br />{<br /> "result" : "mongoDBmapReduce",<br /> "timeMillis" : 11,<br /> "counts" : {<br /> "input" : 4,<br /> "emit" : 8,<br /> "output" : 5<br /> },<br /> "ok" : 1,<br />}<br />
  27. 27. Reference<br />Google MapReduce<br />http://code.google.com/intl/ko-KR/edu/parallel/mapreduce-tutorial.html<br />MongoDBMapReduce<br />http://kylebanker.com/blog/2009/12/mongodb-map-reduce-basics/<br />HadoopMapReduce<br />http://hadoop.apache.org/common/docs/current/mapred_tutorial.html<br />
  28. 28. 감사합니다<br />

×