• What is MapReduce? • What are MapReduce implementations? Facing these questions I have make a personal research, and realize a synthesis, which has help me to clarify some ideas. The attached presentation does not intend to be exhaustive on the subject, but could perhaps bring you some useful insights.

1. MapReduce
michel.bruley@teradata.com
Extract from various presentations: Sudarshan, Chungnam, Teradata Aster, …
April 2012
www.decideo.fr/bruley

2. What is MapReduce?
Restricted parallel programming model meant for large
clusters
– User implements Map() and Reduce() functions
Parallel computing framework
– Libraries take care of EVERYTHING else
• Parallelization
• Fault Tolerance
• Data Distribution
• Load Balancing
Useful model for many practical tasks
www.decideo.fr/bruley

3. Map and Reduce
The idea of Map, and Reduce is 40+ year old
– Present in all Functional Programming Languages.
– See, e.g., APL, Lisp and ML
Alternate names for Map: Apply-All
Higher Order Functions
– take function definitions as arguments, or
– return a function as output
Map and Reduce are higher-order functions.
www.decideo.fr/bruley

4. Map and Reduce Functions
Functions borrowed from functional programming
languages (eg. Lisp)
Map()
– Process a key/value pair to generate intermediate
key/value pairs
Reduce()
– Merge all intermediate values associated with the same
key
www.decideo.fr/bruley

5. Example: Counting Words
Map()
– Input <filename, file text>
– Parses file and emits <word, count> pairs
• eg. <”hello”, 1>
Reduce()
– Sums all values for the same key and emits <word,
TotalCount>
• eg. <”hello”, (3 5 2 7)> => <”hello”, 17>
www.decideo.fr/bruley

6. Execution on Clusters
1.
Input files split (M splits)
2.
Assign Master & Workers
3.
Map tasks
4.
Writing intermediate data to disk (R regions)
5.
Intermediate data read & sort
6.
Reduce tasks
7.
Return
www.decideo.fr/bruley

7. Map/Reduce Cluster
Implementation
Input
files
M map Intermediate
tasks
files
R reduce
tasks
split 0
split 1
split 2
split 3
split 4
Several map or
reduce tasks can
run on a single
computer
www.decideo.fr/bruley
Output
files
Output 0
Output 1
Each intermediate
file is divided into R
partitions, by
partitioning function
Each reduce task
corresponds to
one partition

8. Map Reduce vs. Parallel
Databases
Map Reduce widely used for parallel processing
– Google, Yahoo, and 100’s of other companies
– Example uses: compute PageRank, build keyword indices,
do data analysis of web click logs, ….
Database people say:
– but parallel databases have been doing this for decades
Map Reduce people say:
– we operate at scales of 1000’s of machines
– We handle failures seamlessly
– We allow procedural code in map and reduce and allow
data of any type
www.decideo.fr/bruley

9. Typical MapReduce Cluster
www.decideo.fr/bruley

10. Map Reduce Implementations
Google
– Not available outside Google
Hadoop
– An open-source implementation in Java
– Uses HDFS for stable storage
– Download: http://lucene.apache.org/hadoop/
Teradata Aster
– Cluster-optimized SQL Database that also implements
MapReduce
• IITB alumnus among founders
And several others, such as Cassandra at Facebook, etc.
www.decideo.fr/bruley

11. MapReduce v. Hadoop
MapReduce
Hadoop
Org
Google
Yahoo/Apache
Impl
C++
Java
Distributed
GFS
File Sys
HDFS
Data Base Bigtable
HBase
Distributed
Chubby
lock mgr
ZooKeeper
www.decideo.fr/bruley

12. Solutions Stack for Teradata Aster
Data
Integration
/ ETL
Business
Intelligence
Tools
Query
Tools
Analytics
Specialists
Systems Management
Aster Data
Ecosystem
Security
Aster Data nCluster
Operating System
Servers
Cloud Infrastructure
Storage
www.decideo.fr/bruley
Aster Data
Platform
Infrastructure

13. Teradata Aster Platform Infrastructure
For physical infrastructure (non-cloud) deployments
Aster Data
Analytic
Platform
nCluster
nCluster
Aster Data nCluster packaged software
Operating
System
Certified Linux operating system
Server
Hardware
Certified commodity (x86) server
hardware with internal storage
www.decideo.fr/bruley

14. Teradata Aster Infrastructure
For cloud deployments
Aster Data
Analytic
Platform
nCluster
nCluster
Aster Data nCluster packaged software
Operating
System
Compute
Instance
Storage
www.decideo.fr/bruley
Linux operating system
CC
CC
xLarge
xLarge
EBS
EBS
Ephemeral
Ephemeral
Compute instance from cloud provider
(e.g. Amazon Web Services EC2)
Storage connected to cloud computing
capacity

15. Teradata Aster Architecture for
Analytics
Your Analytics & Advanced Reporting
Applications
App
App
App
App
• Support for in-database processing of custom
applications written in broad variety of languages
• Integration with third-party packaged software via
ODBC/JDBC or in-database integration
Aster Data nCluster
Analytic Functions and Frameworks
• Rich libraries of MapReduce analytics from Aster
Data and partners
• Visual development environment--develop in hours
Unified Interface
• Standard SQL interface
• MapReduce processing integrated with SQL via
SQL-MapReduce interface
SQL
SQL-MapReduce
Analytics Processing Engines
SQL
MapReduce
Massively Parallel Data Stores
www.decideo.fr/bruley
…
• Optimized SQL engine
• Fully-integrated in-database MapReduce
• Hybrid row/column DBMS
• Linear, incremental scalability
• Commodity hardware

16. Teradata Aster Ecosystem
Partner
Product
Product
release
Platform for Certification
MicroStrategy
Intelligence Server
9.2.1 32-bit
Windows 7, Enterprise Edition SP1, 32-bit, 64-bit
SAP
Business Objects
XI 3.1
Windows 2008, 32-bit
Informatica
Powercenter
9.0.1
Client: Windows 2003/2008 Server 32 bit.
Server: Windows 2003/2008 Server 32 bit and 64 bit
IBM
Cognos
10.1FP1
n/a
Tableau
Tableau Server
6
Windows (SS: TBU)
Microsoft
SSLS, SSAS,
SSFS, SSIS
SQL Server
2008
.NET Framework 2.0
Windows Server, 2008 64-bit
Windows 2003, 32-bit
*Oracle BIEE certification currently in process
www.decideo.fr/bruley