This document discusses MapReduce using Python. It defines MapReduce as reading large amounts of data, extracting relevant information from each record using a Map function, sorting and shuffling the data, and then aggregating, summarizing, filtering or transforming the data using a Reduce function. It describes Mappers as processing input data line-by-line to create smaller chunks of data, and Reducers as accepting output from Mappers to aggregate values by key and produce a new output set. An example counts the number of occurrences of each word in a text by having Mappers output word counts and Reducers sum the counts by word.

1. Whirl Data
MAP REDUCE USING PYTHON
By: Manivannan
Date: 8 July, 2016
1

2. Whirl Data
WHAT IS MAPREDUCE
Read a lot of data
Map: extract something you care about from each record Shuffle and Sort
Reduce: aggregate, summarize, filter, or transform
Write the results
2

3. Whirl Data
MapReduce Workflow
3

4. Whirl Data
Mapper
What is Mapper:
The map or mapper’s job is to process the
input data. Generally the input data is in the form of file or directory and is stored
in the Hadoop file system (HDFS). The input file is passed to the mapper function
line by line. The mapper processes the data and creates several small chunks of
data.
4

5. Whirl Data
Reducer
What is Reducer:
This stage is the combination of the Shuffle
stage and the Reduce stage. The Reducer’s job is to process the data that comes
from the mapper. After processing, it produces a new set of output, which will be
stored in the HDFS.
5

6. Whirl Data
Key and Values
6

7. Whirl Data
Mapper Work
Reads in input pair <Key,Value>
Outputs a pair <K’, V’>
->Let’s count number of each word in user queries (or Tweets/Blogs)
->The input to the mapper will be <queryID, QueryText>:
<Q1,“The teacher went to the store. The store was closed; the store opens in the morning. The store opens at
9am.” >
->The output would be:
<The, 1> <teacher, 1> <went, 1> <to, 1> <the, 1> <store,1> <the, 1> <store, 1> <was, 1> <closed, 1> <the, 1>
<store,1> <opens, 1> <in, 1> <the, 1> <morning, 1> <the 1> <store, 1> <opens, 1> <at, 1> <9am, 1>
7

8. Whirl Data
Reducer Work
Accepts the Mapper output, and aggregates values on the key
->For our example, the reducer input would be:
<The, 1> <teacher, 1> <went, 1> <to, 1> <the, 1> <store, 1> <the, 1> <store, 1> <was, 1> <closed, 1> <the, 1>
<store, 1> <opens,1> <in, 1> <the, 1> <morning, 1> <the 1> <store, 1> <opens, 1> <at, 1> <9am, 1>
->The output would be:
<The, 6> <teacher, 1> <went, 1> <to, 1> <store, 3> <was, 1> <closed, 1> <opens, 1> <morning, 1> <at, 1>
<9am, 1>
8

9. Whirl Data
Flow Chart
Divide into Single Word
Count the Single Word
9
Input Data Mapper.py
Reducer.pyOutput Data

10. Whirl Data
Source File
LINK:http://bit.ly/whirl-mapreducer
10

11. Whirl Data
Contact us
Whirl Data
www.citrisys.com
11