Upcoming SlideShare
×

# MapReduce

685 views
615 views

Published on

Published in: Technology
1 Like
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total views
685
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
41
0
Likes
1
Embeds 0
No embeds

No notes for slide

### MapReduce

1. 1. MapReduce Ahmed Elmorsy
2. 2. What is MapReduce?● MapReduce is a programming model for processing and generating large data sets.● Inspired by the map and reduce primitives present in Lisp and many other functional languages● Use of a functional model with user-specified map and reduce operations allows us to parallelize large computations easily.
3. 3. Map functionMap, written by the user, takes an input pairand produces a set of intermediate key/valuepairs. The MapReduce library groups togetherall intermediate values associated with thesame intermediate key I and passes them to theReduce function. map (k1,v1) → list(k2,v2)
4. 4. Reduce functionThe Reduce function, also written by the user,accepts an intermediate key I and a set ofvalues for that key. It merges together thesevalues to form a possibly smaller set of values.Typically just zero or one output value isproduced per Reduce invocation.reduce (k2,list(v2)) → list(v2)
5. 5. Example (Word Count) Problem Counting the number of occurrences of each word in a large collection of documents
6. 6. Example (Word Count)Map function:map(String key, String value):// key: document name// value: document contentsfor each word w in value:EmitIntermediate(w, "1");
7. 7. Example (Word Count)Reduce function:reduce(String key, Iterator values):// key: a word// values: a list of countsint result = 0;for each v in values:result += ParseInt(v);Emit(AsString(result));
8. 8. Execution Overview● The Map invocations are distributed across multiple machines by automatically partitioning the input data into a set of M splits.● Reduce invocations are distributed by partitioning the intermediate key space into R pieces using a partitioning function.● The number of partitions (R) and the partitioning function are specified by the user.
9. 9. How Master works?● The master picks idle workers and assigns each one a map task or a reduce task.● For each map task and reduce task, it stores the state (idle, in-progress, or completed), and the identity of the worker machine (for non-idle tasks).
10. 10. Fault Tolerance● Worker Failure● Master Failure
11. 11. Worker Failure● The master pings every worker periodically.● If no response is received from a worker, the master marks the worker as failed.● Any map task or reduce task in progress on a failed worker is also reset to idle and becomes eligible for rescheduling.
12. 12. Worker Failure● Any map tasks completed by the worker are reset back to their initial idle state, and therefore become eligible for scheduling on other workers (WHY?!).● Completed reduce tasks do not need to be re- executed (WHY?!).
13. 13. Master FailureThere are two options:1. Make the master write periodic checkpoints of the master data structures described above. If the master task dies, a new copy can be started from the last checkpointed state.2. Abort the MapReduce computation if the master fails.
14. 14. Backup Tasks● One of the common causes that lengthens the total time taken for a MapReduce operation is a “straggler”.● When a MapReduce operation is close to completion, the master schedules backup executions of the remaining in-progress tasks.● The task is marked as completed whenever either the primary or the backup execution completes
15. 15. Refinements1. Partitioning Function2. Combiner Function3. Input and Output Types4. Skipping Bad Records5. Status Information6. Counters
16. 16. More Examples● Distributed Grep● Count of URL Access Frequency● Reverse Web-Link Graph● Inverted Index● Distributed Sort
17. 17. Apache HadoopOpen Source Implementation of MapReduce