• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
MapReduce
 

MapReduce

on

  • 1,520 views

2011년 아꿈사 시작발표.

2011년 아꿈사 시작발표.
MapReduce

Statistics

Views

Total Views
1,520
Views on SlideShare
1,380
Embed Views
140

Actions

Likes
3
Downloads
2
Comments
0

9 Embeds 140

http://soomong.net 107
http://www.hanrss.com 9
http://andstudy.com 6
http://feeds2.feedburner.com 6
http://www.andstudy.com 4
http://www.samsung.net 4
http://www.slideshare.net 2
http://soomong.github.io 1
http://feeds.feedburner.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    MapReduce MapReduce Presentation Transcript

    • MapReduce
      아키텍트를 꿈꾸는 사람들cafe.naver.com/architect1
      현수명 soomong.net
      #soomong
    • MapReduce?
      Map + Reduce
      Execution Overview
    • MapReduce?
      mechanism for processing large data
      developed within Google
      from a functional language
      <key,value> pair
    • Execution Overview
      Google
    • Execution Overview
      Hadoop
    • Map
      <Only value>
      Map takes as input a function and
      a sequence of values.
      It then applies the function
      to each value in the sequence
      func
      Map
      (func,values)
      applied
      values
      values
    • Map
      <Only value>
      f(x)=x*x
      Map
      (func,values)
      (1,4,9)
      (1,2,3)
      Clojure
      (map (fn [x] (x*x)) [1 2 3])
    • Reduce
      <Only value>
      A reduce combines
      all the elements of a sequence
      using a binary operation
      func
      Reduce
      (func,values)
      applied
      value
      values
    • Reduce
      <Only value>
      +
      Reduce
      (func,values)
      (14)
      (1,4,9)
      Clojure
      (reduce + [1 4 9])
    • Map
      <key,value>
      f(x)=x*x
      Map
      (func,values)
      (keyA,1)
      (keyB,4)
      (keyA,9)
      (1,2,3)
    • Reduce
      <key,value>
      +
      Reduce
      (func,(key,value))
      (keyA,10)
      (keyB,4)
      (keyA,1)
      (keyB,4)
      (keyA,9)
    • Map + Reduce
      (1,2,3…99998,99999,100000)
      f(x)=x*x
      f(x)=x*x
      f(x)=x*x
      Map
      (func,values)
      Map
      (func,values)
      Map
      (func,values)
      +
      +
      Reduce
      (func,values)
      Reduce
      (func,values)
      Reduce
      (func,values)
      (keyA,107)
      (keyB,29)
      (…,…,…)
      (…)
      (result)
      (keyA,1)
      (keyB,4)
      (keyA,9)
      (1,2,3)
      (4,5,6)
      (99998,99999,…)
      (keyA,16)
      (keyB,25)
      (keyA,81)
      +

    • Execution Example
      “Stay Hungry Stay Foolish
      Don’t settle”
      Word Counter
      Stay 2
      Hungry 1
      Foolish 1
      Don’t 1
      Settle 1
    • Map
      map(String key, String value):
      // key: document name
      // value: document contents
      for each word w in value:
      EmitIntermediate(w, "1");
    • Reduce
      reduce(String key, Iterator values):
      // key: a word
      // values: a list of counts
      int result = 0;
      for each v in values:
      result +=ParseInt(v);
      Emit(AsString(result));
    • Stay Hungry Stay Foolish
      Don’t settle
      Stay Hungry Stay Foolish
      Don’t settle
      (1) Shards the input files
    • Stay Hungry Stay Foolish
      Don’t settle
      (2) Master pick idle workers
      assign a map or reduce task
    • <Stay,1>
      <Hungry,1>
      <Stay,1>
      <Foolish,1>
      Stay Hungry Stay Foolish
      Don’t settle
      <Don’t,1>
      <settle,1>
      (3) Map worker reads input shard,
      do Map func -> intermediate <key,value>
    • <Stay,1>
      <Hungry,1>
      <Stay,1>
      <Foolish,1>
      <Don’t,1>
      <settle,1>
      (4) Write intermediate <key,value> on local disk
    • <Stay,1>
      <Hungry,1>
      <Stay,1>
      <Foolish,1>
      <Foolish,1>
      <Hungry,1>
      <Stay,(1,1)>
      <Don’t,1>
      <settle,1>
      <Don’t,1>
      <settle,1>
      (5) Reduce worker reads intermediate data
      sort by key
    • <Foolish,1>
      <Hungry,1>
      <Stay,(1,1)>
      <Foolish,1>
      <Hungry,1>
      <Stay,2>
      <Don’t,1>
      <settle,1>
      <Don’t,1>
      <settle,1>
      (6) Reduce worker do reduce func
      -> wrtie output
    • <Foolish,1>
      <Hungry,1>
      <Stay,2>
      <Don’t,1>
      <settle,1>
      <Foolish,1>
      <Hungry,1>
      <Stay,2>
      <Don’t,1>
      <settle,1>
      (7) All map tasks and reduce tasks have been completed
      master wakes up the user program -> return Result
    • Map
      map = function () {
      for (var key in this) {
      emit(key, {count:1});
      }
    • Reduce
      reduce = function (key, emits) {
      total = 0;
      for (vari in emits) {
      total += emits[i].count;
      }
      return {count:total};
    • Execution
      mr = db.foo.mapReduce
      (map,reduce,{out:"mongoDBmapReduce"})
      {
      "result" : "mongoDBmapReduce",
      "timeMillis" : 11,
      "counts" : {
      "input" : 4,
      "emit" : 8,
      "output" : 5
      },
      "ok" : 1,
      }
    • Reference
      Google MapReduce
      http://code.google.com/intl/ko-KR/edu/parallel/mapreduce-tutorial.html
      MongoDBMapReduce
      http://kylebanker.com/blog/2009/12/mongodb-map-reduce-basics/
      HadoopMapReduce
      http://hadoop.apache.org/common/docs/current/mapred_tutorial.html
    • 감사합니다