This document provides an overview of MapReduce in Python for analyzing text. It discusses setting up the environment, counting the words in Moby Dick as an example, the mapping, shuffling, and reducing steps of MapReduce, and limitations when processing very large texts. Requirements include a Unix-like system and Python. The example counts words by processing the input text with a mapper, sorting the output, and then reducing the counts with a reducer. Hadoop is also introduced as a MapReduce framework.

1. A Hands-on Introduction to
MapReduce in Python
David Massart, PhD

2. Who Am I ?

3. Outline
• Set-up and requirements
• Counting words
• Limitations
• Map / Reduce
– Mapping
– Shuffling
– Reducing
• Hadoop

4. Environment Set-up
• Required
– Unix-like shell
• Linux
• Mac OS X
• Windows + Cygwin
– Python (e.g., anaconda)
• Good to have
– Java 8
– Hadoop 2.6

5. Moby Dick by Herman Melville
• Download Moby Dick:
wget
https://www.gutenberg.org/cache/epub/2701/p
g2701.txt
• Rename it input.txt:
mv pg2701.txt input.txt

6. cat input.txt

7. Counting Words

8. ./counter.py < input.txt

9. Limitations
• Processing time is, at best, proportional to the
size of the text
• Actually, performance decreases with the size
of the dictionary
• Very large texts can require more than one
disk

10. MapReduce, Part 1: Mapping

11. ./mapper.py < input.txt

12. MapReduce, Part 2: Shuffling
• Redistribute data based on the output keys
produced by the "mapper”
• So that all data belonging to one key is
grouped together

13. ./mapper.py < input.txt | sort

14. MapReduce, Part 3: Reducing

15. ./mapper.py < input.txt | sort | ./reducer.py

16. Hadoop

17. More details available at
http://zettadatanet.wordpress.com
These slides are available at
http://www.slideshare.net/dmassart
/mapreduce-20150315