MapReduce is a programming model and associated implementation for processing large datasets in parallel across clusters of computers. It allows for automatic parallelization, distribution, fault tolerance, and monitoring of large-scale computations. The MapReduce model consists of a map function that processes input key-value pairs to generate intermediate key-value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. A MapReduce job is made up of many map and reduce tasks that are executed in parallel by the framework on a distributed computing infrastructure.

1. MapReduce
Presented by – Somesh maliye

2. Content
•Motivation for MapReduce.
•What is MapReduce.
•Map() & Reduce() functions.
•MapReduce - Example.
•Dataflow.
•MapReduce Job.
•Job Tracker & Task tracker.
•Characteristics of MapReduce.
•Real Time uses.
•Failure in MapReduce.
•Conclusion.

3. Motivation For MapReduce
•Large scale data processing.
◦ Want to use 1000s of CPUs
•MapReduce Architecture provides
◦ Automatic parallelization & distribution
◦ Fault tolerance
◦ I/O scheduling
◦ Monitoring & status updates

4. What is MapReduce
•MapReduce is programming model and an associated implementation for
processing and generating large data sets with parallel and distributed algorithm
on clusters.

5. Map() function
• Reads in input pair <Key, Value>
• Outputs a pair <K’, V’>
• Let’s count number of each word in user queries (or Tweets/Blogs)
• The input to the map() will be <queryID, QueryText>:
• <Q1,“The teacher went to the store. The store was closed; the store opens in
the morning. The store opens at 9am.” >
• The output would be:
<The, 1> <teacher, 1> <went, 1> <to, 1> <the, 1> <store,1> <the, 1> <store, 1> <was, 1>
<closed, 1> <the, 1> <store,1> <opens, 1> <in, 1> <the, 1> <morning, 1> <the 1> <store, 1>
<opens, 1> <at, 1> <9am, 1>

6. Reduce() function
•Accepts the Map() output, and aggregates values on the key
•For our example, the reducer input would be:
• <The, 1> <teacher, 1> <went, 1> <to, 1> <the, 1> <store, 1> <the, 1> <store, 1> <was, 1> <closed, 1>
<the, 1> <store, 1> <opens,1> <in, 1> <the, 1> <morning, 1> <the 1> <store, 1> <opens, 1> <at, 1>
<9am, 1>
• The output would be:
• <The, 6> <teacher, 1> <went, 1> <to, 1> <store, 3> <was, 1> <closed, 1> <opens, 1> <morning, 1> <at, 1>
<9am, 1>

7. MapReduce - Example

8. Dataflow
Dataflow can be determine through the following function:
• an input reader
• a Map function
• a partition function
• a compare function
• a Reduce function
• an output writer

9. Dataflow(Cont.)
•Input reader
The input reader divides the input into appropriate size 'splits' (in practice typically 64 MB to 128 MB)
and the framework assigns one split to each Map function. The input reader reads data from stable
storage (typically a distributed file system) and generates key/value pairs.
•Map function
The Map function takes a series of key/value pairs, processes each, and generates zero or more output
key/value pairs.
• Partition function
Each Map function output is allocated to a particular reducer by the application's partition function for
sharding purposes. The partition function is given the key and the number of reducers and returns the
index of the desired reducer.
•Comparison function
The input for each Reduce is pulled from the machine where the Map ran and sorted using the
application's comparison function.

10. Dataflow(Cont.)
•Reduce function
The framework calls the application's Reduce function once for each unique key in the sorted order. The
Reduce can iterate through the values that are associated with that key and produce zero or more
outputs.
•Output writer
The Output Writer writes the output of the Reduce to the stable storage.

11. MapReduce Job
A job is a full MapReduce program , which typically will cause multiple Map
and Reduce functions to be run in parallel over the life of program. A task is a
map or reduce function executed on a subset of data.

12. Job tracker & Task tracker

13. Characteristics of MapReduce

14. Real time uses

15. Failures in MapReduce
• Failures are norm in commodity hardware
• Worker failure
• Detect failure via periodic heartbeats
• Re-execute in-progress map/reduce tasks
• Master failure
• Single point of failure; Resume from Execution Log

16. Conclusion.
•Simplifies large-scale computations that fit this model
•Allows user to focus on the problem without worrying about details

17. Thank You