This document provides an overview of MapReduce, including: what MapReduce is and how it works through mapping and reducing large datasets in parallel across clusters; how data flows through Map and Reduce functions; comparisons of different MapReduce models like Google and Hadoop; and a demonstration using Java streams and lambda expressions. References are also provided for further reading.

1. © RAGINIJAIN CC SA 4.0
Ragini Jain
MSc CA 1st
Year (2015 - 2017)
Map Reduce

2. © RAGINIJAIN CC SA 4.0
Overview
● What is Map Reduce
● Map Reduce schematic
● Map Reduce in detail
● Comparison of Map Reduce models
● Demo
● References

3. © RAGINIJAIN CC SA 4.0
What is Map Reduce
● A software framework which supports
– Parallel
– Distributed computing
on large data sets.
● The framework abstracts the data flow of running a parallel
program on a distributed computing system by providing users
with two interfaces in the form of functions:
– Map
– Reduce
● Users can control and manipulate the data flow of their programs
by overriding the Map() and Reduce() function
● Map Reduce library is the controller.

4. © RAGINIJAIN CC SA 4.0
Map – reduce schematic
Source: jeremykyun

5. © RAGINIJAIN CC SA 4.0
Map – reduce schematic (2)
Source: hadoop project

6. © RAGINIJAIN CC SA 4.0
Map Reduce (in detail)
● The Map function is applied in parallel to every input (key, value)
pair and produces new set of intermediate (key, value) pairs
(key1, val1) ------(map function)---> List (key2, val2)
● Then the MapReduce library collects all the produced intermediate
(key, value) pairs from all input (key, val) pairs and sorts them
based on the key part
● Finally Reduce function is applied in parallel to each group
producing the collection of values
(key2, List(val2)) -----(reduce function) ---> List (val2)

7. © RAGINIJAIN CC SA 4.0
Map Reduce (as a query framework)
● SQL clauses that are the building block for Map Reduce
operations on structured data and data warehouses
– GROUP BY
– ORDER BY
● On a very large set of demographic data
SELECT age, AVG(contacts)
FROM social.person
GROUP BY age
ORDER BY age

8. © RAGINIJAIN CC SA 4.0
GROUP BY (SQL vs Pig)

9. © RAGINIJAIN CC SA 4.0
Comparison Map Reduce models
● Google Map Reduce
– Prog Model: Map Reduce
– Data handling: Google file system
● Apache Hadoop
– Prog Model: Map Reduce
– Data Handling: HDFS (Hadoop Distributed File system)
● Microsoft Dryad
– Prog Model: DAG (Directed Acyclic Graph) execution
– Data Handling: Shared directories, Local disks
● Twister
– Prog Model: Iterative Map Reduce
– Data Handling: Local disks

10. © RAGINIJAIN CC SA 4.0
Demo
● Java program
– Utilizes concepts from Java 8 programming language platform.
● Lambda expressions
● Streams
– JDK ref
● java.util.Collection.stream()
● java.lang.Iterable.forEach( )
● java.util.List

11. © RAGINIJAIN CC SA 4.0
References
● Jeffrey Dean et' al
MapReduce: Simplified Data Processing on Large Clusters
http://research.google.com/archive/mapreduce.html
● Michelle Stonebraker et' al
MapReduce and Parallel DBMSs: Friends or Foes ?
http://dl.acm.org/citation.cfm?id=1629197
● Java Lambda expressions
https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexp
● PostgreSQL GROUP BY and ORDER BY
http://www.postgresql.org/docs/devel/static/sql­select.html

12. © RAGINIJAIN CC SA 4.0
Thank you.
● Questions
● Clarifications
● Suggestions
● Feedback
Ragini Jain
15030142023@sicsr.ac.in