This presentation explains how to build a simple map reduce algorithm step by step.

1. A Hands-on Introduction to
MapReduce in Python
David Massart, PhD

2. Who Am I ?

3. Outline
• Set-up and requirements
• Counting words
• Limitations
• Map / Reduce
– Mapping
– Shuffling
– Reducing
• Hadoop

4. Environment Set-up
• Required
– Unix-like shell
• Linux
• Mac OS X
• Windows + Cygwin
– Python (e.g., anaconda)
• Good to have
– Java 8
– Hadoop 2.6

5. Moby Dick by Herman Melville
• Download Moby Dick:
wget
https://www.gutenberg.org/cache/epub/2701/p
g2701.txt
• Rename it input.txt:
mv pg2701.txt input.txt

6. cat input.txt

7. Counting Words

8. ./counter.py < input.txt

9. Limitations
• Processing time is, at best, proportional to the
size of the text
• Actually, performance decreases with the size
of the dictionary
• Very large texts can require more than one
disk

10. MapReduce, Part 1: Mapping

11. ./mapper.py < input.txt

12. MapReduce, Part 2: Shuffling
• Redistribute data based on the output keys
produced by the "mapper”
• So that all data belonging to one key is
grouped together

13. ./mapper.py < input.txt | sort

14. MapReduce, Part 3: Reducing

15. ./mapper.py < input.txt | sort | ./reducer.py

16. Hadoop

17. More details available at
http://zettadatanet.wordpress.com
These slides are available at
http://www.slideshare.net/dmassart
/mapreduce-20150315