This document discusses MapReduce and provides an example implementation in Python using the MinceMeat library to solve the Netflix Prize problem. It first explains the MapReduce model developed by Google with key-value pairs and separate mapping and reducing functions. It then shows sample mapping and reducing functions to count word frequencies in web pages as an example. Finally, it shows code using MinceMeat to map and reduce movie rating data to find the number of movies each user rated.

1. Mi Primer Map/Reduce
Rubén Orta @agileando

2. 1
historia
2
implementación
3
netflix prize en python
4
enlaces

3. 1
Big Data = Contar

4. CON
TAR
1

5. 1
Jeff
Dean
Sanjay
Ghemawat

6. map (key , value)
new_value = a_function(value)
return new_key, new_value
2
reduce (key, value)
new_value = another_function(value)
return key, new_value

7. Dataset:
2
Millones de páginas web
Map
f()
f()
f()
f()
f()
f’()
f’()
f’()
f’()
f’()
for each word in document:
return (word, 1);
Reduce
total = 0
for each item in value:
total++
return (key, total);

8. 2

9. 3

10. import mincemeat
3
data = dict((f, read_data(f)) for f in data_files)
s = mincemeat.Server()
s.datasource = data
s.mapfn = mapfn
s.reducefn = reducefn
results = s.run_server (password = "ruben")

11. def mapfn(key, value):
lines = value.splitlines()
film_id = lines[0][:-1]
for line in lines[1:]:
items = line.split(",")
user_id = items[0]
rating = items[1]
date = items[2]
yield user_id, film_id
3

12. def reducefn(key, values):
number_of_films = 0
for value in values:
number_of_films += 1
return number_of_films
3

13. Papers
4
GFS
MapReduce
BigTable
http://research.google.com/archive/gfs.html
http://research.google.com/archive/mapreduce.html
http://research.google.com/archive/bigtable.html
Dynamo
http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/decandia07dynamo.pdf
Dremel
Spanner
http://research.google.com/pubs/pub36632.html
http://research.google.com/archive/spanner.html
Python
MinceMeat.py https://github.com/michaelfairley/mincemeatpy
Octo.py
http://code.google.com/p/octopy/
Netflix DataSet http://www.lifecrunch.biz/archives/207

14. Rubén Orta
http://www.slideshare.net/agileando/mi-primer-map-reduce
Blog
Twitter
GitHub
http://devspoke.com/
https://twitter.com/agileando
https://github.com/rubenorta
4

15. BUSCAMOS GENTE
PARA NUESTRO
EQUIPO
¿Quieres unirte?
*unix, scripting (python, perl)
devops