MapReduce is a programming model and framework for processing large datasets in a distributed manner. It allows users to write map and reduce functions to extract and aggregate data across large clusters of machines. The MapReduce workflow involves mapping data to extract key-value pairs, shuffling and sorting the data, and reducing the data by key to aggregate results. It is implemented by Google's distributed file system GFS which splits files into chunks across many machines for scalability and fault tolerance.

1. MapReduce
Mostafa Abbas
American University of Culture and
Education (AUCE)
Nabatieh Campus
Distributed Operating Systems
CSI510
Dr. Omar Falou

2. Problem Statement / Motivation
01
MapReduce definition
02
A Typical MapReduce Program
03
MapReduce workflow
04
Outline
MapReduce Examples
05
Conclusion
07
GFS
06

3. 1. Why MapReduce?
 Considerations
 Threading is hard!
 How do you scale to more machines?
 How do you handle machine failures?
 How do you facilitate communication between nodes?
 Does your solution scale?
 Before MapReduce
 Large Concurrent Systems
 Grid Computing
 Rolling Your Own Solution
 Scale out, not up!
Map Reduce
1

4. 2. What is MapReduce?
“MapReduce is a programming model and an associated
implementation for processing and generating large data sets”
 Programming model
 Abstractions to express simple computations
 Library
 Takes care of the gory stuff: Parallelization, Fault Tolerance,
Data Distribution and Load Balancing
Map Reduce
2

5. 3. A Typical MapReduce Program
Map Reduce
3
1. Read a lot of data.
2. Map: extract something you
care about from each record.
3. Shuffle and Sort.
4. Reduce: aggregate,
summarize, filter, or transform.
5. Write the results.

6. 4. MapReduce workflow 4
Map Reduce
Worker
Worker
Worker
Worker
Worker
read
local
write
remote
read,
sort
Output
File 0
Output
File 1
write
Split 0
Split 1
Split 2
Input Data Output Data
Map Reduce

7. 5. MapReduce Examples
 Mapper
 Reads in input pair <K,V> Outputs a pair <K’, V’>
 Ex. For our Word Count example, with the following input: “The teacher went to
the store. The store was closed; the store opens in the morning. The store opens
at 9am.”
The output would be:
 <The, 1> <teacher, 1> <went, 1> <to, 1> <the, 1> <store, 1> <the, 1> <store,
1> <was, 1> <closed, 1> <the, 1> <store, 1> <opens, 1> <in, 1> <the, 1>
<morning, 1> <the 1> <store, 1> <opens, 1> <at, 1> <9am, 1>
Map Reduce
5

8.  Reducer
 Accepts the Mapper output, and collects values on the key
 For our example, the reducer input would be:
<The, 1> <teacher, 1> <went, 1> <to, 1> <the, 1> <store, 1>
<the, 1> <store, 1> <was, 1> <closed, 1> <the, 1> <store, 1>
<opens, 1> <in, 1> <the, 1> <morning, 1> <the 1> <store, 1>
<opens, 1> <at, 1> <9am, 1>
The output would be:
• <The, 6> <teacher, 1> <went, 1> <to, 1> <store, 3> <was, 1> <closed, 1>
<opens, 1> <morning, 1> <at, 1> <9am, 1>
Map Reduce
6

9. 6. GFS Distributed File System
HOW DATA IS SAVED?!
Each file is split into contiguous 64 Mbyte chunks
GFS cluster is made up of two types of nodes:
1. Large number of Chunkservers
2. One Master node
Link GFS client library into applications
Map Reduce
7

10. 7. Conclusion
 MapReduce provides a simple
way to scale your application.
8
Map Reduce
 Scales out to more machines,
rather than scaling up.
 Effortlessly scale from a single machine
to thousands.
 Fault tolerant & High performance.
 If you can fit your use case to its
paradigm, scaling is handled by the
framework.