MapReduce in Distributed Operating System

1. MapReduce
Mostafa Abbas
American University of Culture and
Education (AUCE)
Nabatieh Campus
Distributed Operating Systems
CSI510
Dr. Omar Falou

2. Problem Statement / Motivation
01
MapReduce definition
02
A Typical MapReduce Program
03
MapReduce workflow
04
Outline
MapReduce Examples
05
Conclusion
07
GFS
06

3. 1. Why MapReduce?
 Considerations
 Threading is hard!
 How do you scale to more machines?
 How do you handle machine failures?
 How do you facilitate communication between nodes?
 Does your solution scale?
 Before MapReduce
 Large Concurrent Systems
 Grid Computing
 Rolling Your Own Solution
 Scale out, not up!
Map Reduce
1

4. 2. What is MapReduce?
“MapReduce is a programming model and an associated
implementation for processing and generating large data sets”
 Programming model
 Abstractions to express simple computations
 Library
 Takes care of the gory stuff: Parallelization, Fault Tolerance,
Data Distribution and Load Balancing
Map Reduce
2

5. 3. A Typical MapReduce Program
Map Reduce
3
1. Read a lot of data.
2. Map: extract something you
care about from each record.
3. Shuffle and Sort.
4. Reduce: aggregate,
summarize, filter, or transform.
5. Write the results.

6. 4. MapReduce workflow 4
Map Reduce
Worker
Worker
Worker
Worker
Worker
read
local
write
remote
read,
sort
Output
File 0
Output
File 1
write
Split 0
Split 1
Split 2
Input Data Output Data
Map Reduce

7. 5. MapReduce Examples
 Mapper
 Reads in input pair <K,V> Outputs a pair <K’, V’>
 Ex. For our Word Count example, with the following input: “The teacher went to
the store. The store was closed; the store opens in the morning. The store opens
at 9am.”
The output would be:
 <The, 1> <teacher, 1> <went, 1> <to, 1> <the, 1> <store, 1> <the, 1> <store,
1> <was, 1> <closed, 1> <the, 1> <store, 1> <opens, 1> <in, 1> <the, 1>
<morning, 1> <the 1> <store, 1> <opens, 1> <at, 1> <9am, 1>
Map Reduce
5

8.  Reducer
 Accepts the Mapper output, and collects values on the key
 For our example, the reducer input would be:
<The, 1> <teacher, 1> <went, 1> <to, 1> <the, 1> <store, 1>
<the, 1> <store, 1> <was, 1> <closed, 1> <the, 1> <store, 1>
<opens, 1> <in, 1> <the, 1> <morning, 1> <the 1> <store, 1>
<opens, 1> <at, 1> <9am, 1>
The output would be:
• <The, 6> <teacher, 1> <went, 1> <to, 1> <store, 3> <was, 1> <closed, 1>
<opens, 1> <morning, 1> <at, 1> <9am, 1>
Map Reduce
6

9. 6. GFS Distributed File System
HOW DATA IS SAVED?!
Each file is split into contiguous 64 Mbyte chunks
GFS cluster is made up of two types of nodes:
1. Large number of Chunkservers
2. One Master node
Link GFS client library into applications
Map Reduce
7

10. 7. Conclusion
 MapReduce provides a simple
way to scale your application.
8
Map Reduce
 Scales out to more machines,
rather than scaling up.
 Effortlessly scale from a single machine
to thousands.
 Fault tolerant & High performance.
 If you can fit your use case to its
paradigm, scaling is handled by the
framework.