This document provides an introduction to Hadoop and MapReduce. It describes how Hadoop is composed of MapReduce and HDFS. MapReduce is a programming model that allows processing of large datasets in a distributed, parallel manner. It works by breaking the processing into mappers that perform filtering and sorting, and reducers that perform summarization. HDFS provides a distributed file system that stores data across clusters of machines. An example of word count using MapReduce is described to illustrate how it works.

1. Introduction to Hadoop and MapReduce
Colin Su, Tagtoo

2. Advertisement System Architecture (now)

3. Advertisement System Architecture (future)
• Grid
• Ad Server
• Data Highway
• Steaming Computing

4. Grid
• Core:
• Data mining
• Machine Learning
• Collecting data from users, logs and calculate out the strategy
• Sort our data in a proper form, them we could use it anytime
Data -> Information

5. Ad Server
• Ranking
• According the “information” in Grid, decide which AD should be advertised
• show proper ads to website visitors

6. Data Highway
• Transfer your data to the proper place

7. Stream Computing
• Core:
• logging
• feedback
• anti-cheating
• pricing
• post-process everything thrown out from Ad Server, and feedback useful information to Grid
• be the entrance of advertisement system

8. Hadoop
• an open-source software framework for data scientists
• derives from Google’s MapReduce and Google File System (GFS) papers
• written in Java
• could be divided in to 2 components:
• MapReduce
• HDFS (Hadoop distributed file system)
• a yellow elephant

9. Why Hadoop?
• moving computation is much cheaper and easier than moving data
• “Big Data”, the amount of data becomes too large, need a effective way to manage it
• so does computation
• high fault-tolerance
• developed by Yahoo!

10. MapReduce
• a programming model for processing “large data sets” with a “parallel, distributed” algorithm on a cluster
• different from map/reduce, the conception of functional programming, but actually they have the same idea,
“divide and conquer”
• proposed by Google

11. Functional “map/reduce”
• map()/reduce() in Python
• map(function(elem), list) -> list
• reduce(function(elem1, elem2), list) -> single result
• e.g.
• map(lambda x: x*2, [1,2,3,4]) => [2,4,6,8]
• reduce(lambda x,y: x+y, [1,2,3,4]) => 10

12. Parallel “MapReduce” 5 Steps
•
prepare the map() input for mappers
•
mappers run the map() code -> generated intermediate pairs
•
dispatch intermediate pairs to reducers
•
reducers run the reduce() code, aggregate the results
•
prepare output from the result of reduce()

13. Example of “MapReduce” Word Count
map()
reduce()

14. Example of “MapReduce” Word Count
• Original Input
Apple Orange Mongo
Orange Grapes Plum
...

15. Example of “MapReduce” Word Count
• Prepare data for mappers
Apple Orange Mongo
Orange Grapes Plum
...

16. Example of “MapReduce” Word Count
• map() to useful record
(Apple, 1)
Apple Orange Mongo
(Orange, 1)
(Mongo, 1)
Intermediate key/value pair

17. Example of “MapReduce” Word Count
• sort and shuffle
(Apple, 1)
(Mongo, 1)
(Apple, 1)
(Orange, 1)
Reducer
(Apple, 1)
(Mongo, 1)
(Apple, 1)
(Orange, 1)
(Orange, 1)
Shuffle to Reducers
(Orange, 1)
(Orange, 1)
(Apple, 1)
(Mongo, 1)
(Apple, 1)
(Mongo, 1)
unsorted
Sorted
(Orange, 1)
Reducer
(Mongo, 1)
(Mongo, 1)
Reducer

18. Example of “MapReduce” Word Count
• Reduce()
(Apple, 1)
(Apple, 1)
(Apple, 2)
Reducer
(Orange, 1)
(Orange, 1)
(Orange, 1)
Reducer
(Orange, 3)

19. Example of “MapReduce” Word Count
• Generate Output
(Apple, 2)
(Orange, 3)
(Grapes, 1)
(Plum, 5)
Apple 2
Orange 3
Grapes 1
Plum 5
WordCount.txt

20. Hadoop Infrastructure
• Pig: Programming Language for MapReduce
• Thrift: cross-language communication, just like Google’s ProtoBuffer
• Zookeeper: cluster management
Hadoop
Hadoop
Other Services
Thrift
MapReduce
Pig
Hadoop
HDFS
Hadoop
Hadoop
ZooKeeper