The document discusses the necessity of efficient data management for big data analysis, particularly through the MapReduce programming model and distributed file systems like HDFS. It outlines the structure of MapReduce computations involving the processing of <key, value> pairs, and provides a detailed example of word count to illustrate its execution. Additionally, it briefly touches on matrix calculations and their data model as applied within the MapReduce framework.

1. MapReduce and the New
Software Stack
Maruf Aytekin
PhD Student
BAU Computer Engineering Department
Besiktas/Istanbul
January 5, 2015

2. Outline
• Introduction
• DFS
• MapReduce
• Examples
• Matrix Calculation on Hadoop

3. Introduction
Modern data-mining or ML applications,
called «big-data analysis» requires us to
manage massive amounts of data quickly.

4. Important Examples
• The ranking of Web pages by importance,
which involves an iterated matrix-vector
multiplication where the dimension is
many billions.
• Searches in social-networking sites, which
involve graphs with hundreds of millions
of nodes and many billions of edges.
• Processing large amount of text or
streams such as news recommendation.

5. New software stack
• Not a “supercomputer” (Beowulf etc.)
• “computing clusters” – large collections of
commodity hardware, including conventional
processors (“compute nodes”) connected by
Ethernet cables or inexpensive switches.

6. Distributed File System
• The new form of file system which features
much larger units than the disk blocks in a
conventional operating system.
• Files can be enormous, possibly a terabytes
in size.
• Files are rarely updated.

7. Physical Organization
• Files are divided into chunks
• Chunks are replicated

8. DFS Implementations
• The Google File System (GFS)
• Hadoop Distributed File System (HDFS)
• CloudStore, by Kosmix

9. HDFS Architecture

10. Block Replication

11. MapReduce
Style of computing/framework/pattern.
Implementations:
• MapReduce by Google (internal)
• Hadoop by the Apache Foundation.

12. MapReduce
Operates exclusively on <key, value> pairs.
(input) <k1, v1>
-> map -> <k2, v2>
-> combine -> <k2, v2>
-> reduce -> <k3, v3> (output)

13. MapReduce Computation

14. MapReduce
In brief, a MapReduce computation executes as follows:
• Chunks from a DFS are given to Map tasks.
• These Map tasks turn the chunks into a sequence of
<key, value> pairs.
• The <key,value> pairs from each Map task are
collected by a master controller and sorted by key.
(Combine)
• The keys are divided among all the Reduce tasks, so
all <key,value> pairs with the same key wind up at
the same Reduce task.
• The Reduce tasks work on one key at a time and
processes values for that key then outputs the results
as <key,value> pairs.

15. Execution of MapReduce

16. Hello World
Word Count
• file01:
Hello World Bye World
• file02:
Hello Hadoop Goodbye Hadoop

17. Word Count
For the given sample input
the first map emits:
< Hello, 1 >
< World, 1 >
< Bye, 1 >
< World, 1 >
The second map emits:
< Hello, 1 >
< Hadoop, 1 >
< Goodbye, 1 >
< Hadoop, 1 >
Combiner:
After being sorted on the keys:
The output of the first map:
< Bye, 1 >
< Hello, 1 >
< World, 2 >
The output of the second map:
< Goodbye, 1 >
< Hadoop, 2 >
< Hello, 1 >

18. Word Count
Thus the output of the job is:
< Bye, 1 >
< Goodbye, 1 >
< Hadoop, 2 >
< Hello, 2 >
< World, 2 >
The Reducer implementation, via the reduce method just sums up
the values, which are the occurrence counts for each key.

19. M
j
İ
N
k
j
Matrix Calculation
P = M N
k
i
Matrix Data Model for MapReduce:
M (i, j,mij )
N (j, k, njk)
P(1,1) P(1,2)

20. Matrix Data Files
for MapReduce
M,0,0,10.0
M,0,2,9.0
M,0,3,9.0
M,1,0,1.0
M,1,1,3.0
M,1,2,18.0
M,1,3,25.2
.
.
.
M, i, j, mij
N,0,0,1.0
N,0,2,3.0
N,0,4,2.0
N,1,0,2.0
N,3,2,-1.0
N,3,6,4.0
N,4,6,5.0
.
.
.
N (j, k, njk)

21. Map

22. Reduce

23. Example

24. Map Task
Matrix M
key, value pairs produced as
follows:
Matrix N
key, value pairs produced as
follows:

25. Map Task Output

26. Reduce Task
P =

27. Application
• Run the application on Hadoop

28. Thank you!
Q & A