mr dbms

1. CSci 5707, Fall 2013
MapReduce
vs.
Parallel DBMS
Hamid Safizadeh, Otelia Buffington
University of Minnesota

2. 2
MapReduce Idea
 Mapping
map (k1, v1)
 list (k2, v2)
 Reducing
reduce (k2, list(v2))
 list (v2)
Pseudo-code for counting the number of occurrences of each
word in a large collection of documents
Jeffrey Dean and Sanjay Ghemawat, MapReduce: Simplified Data Processing on Large Clustering, OSDI’08

3. 3
MapReduce Example
Calculation of the number of occurrences of each word
http://aimotion.blogspot.com/2010/08/mapreduce-with-mongodb-and-python.html

4. 4
MapReduce Architecture
Execution overview
Jeffrey Dean and Sanjay Ghemawat, MapReduce: Simplified Data Processing on Large Clustering, OSDI’08

5. 5
MapReduce or Parallel DBMS
 Pavlo, A., Paulson, E., Rasin, A., Abadi, D.J., DeWitt, D.J., Madden, S., and
Stonebraker, M., “A comparison of approaches to large-scale data analysis”,
ACM SIGMOD International Conference, 2009
(http://database.cs.brown.edu/projects/mapreduce-vs-dbms)
 Dean, J., and Ghemawat, S., “MapReduce: A flexible data processing tool”,
Communications of the ACM, Vol. 53, 2010 (DOI: 10.1145/1629175.1629198)

6. MapReduce Design Properties
6
 Heterogeneous Systems
 Processing and combining data from a wide variety of storage systems
(such as relational databases, file systems, etc.)
 Fault Tolerance
 Providing fine-grain fault tolerance for large jobs (Failure in middle of a
multi-hour execution does not require restarting the job from scratch)
 Complex Functions
 Simple Map and Reduce functions with straightforward SQL equivalents
 Offering a better framework for some complicated tasks
Jeffrey Dean and Sanjay Ghemawat, MapReduce: A Flexible Data Processing Tool, Communications of the ACM, Vol. 53, 2010

7. MapReduce Design Properties
7
 Performance
 Loading data: Startup overhead for MapReduce
 Reading data: Full scan over large data files
 Merging results: A MapReduce as the next consumer
Jeffrey Dean and Sanjay Ghemawat, MapReduce: A Flexible Data Processing Tool, Communications of the ACM, Vol. 53, 2010
 Cost
 Hardware: Network workstations
 Software: Open source (Hodoop)
 Communication: Network system

8. Companies Using Hodoop
8
 Facebook
 Yahoo!
 Google
 Amazon
 Twitter