MapReduce: teoria e prática
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

MapReduce: teoria e prática

on

  • 376 views

Hoje em dia é fácil juntar quantidades absurdamente grandes de dados. Mas, uma vez de posse deles, como fazer para extrair informações dessas montanhas amorfas de dados? Nesse minicurso vamos ...

Hoje em dia é fácil juntar quantidades absurdamente grandes de dados. Mas, uma vez de posse deles, como fazer para extrair informações dessas montanhas amorfas de dados? Nesse minicurso vamos apresentar o modelo de programação MapReduce: entender como ele funciona, para que serve e como construir aplicações usando-o. Vamos ver também como usar o Elastic MapReduce, o serviço da Amazon que cria clusters MapReduce sob-demanda, para que você não se preocupe em administrar e conseguir acesso a um cluster de máquinas, mas em como fazer seu código digerir de forma distribuída os dados que você possui. Veremos exemplos práticos em ação e codificaremos juntos alguns desafios.

Statistics

Views

Total Views
376
Views on SlideShare
376
Embed Views
0

Actions

Likes
1
Downloads
9
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial LicenseCC Attribution-NonCommercial License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

MapReduce: teoria e prática Presentation Transcript

  • 1. MapReduce 101 ms ste Sy dic or by ha C
  • 2. Brought to you by...
  • 3. Big Data, what's the big deal? Why is this talk relevant to you? ● we have too much data to process in a single computer ● we make too few informed decision based on the data we have ● we have too little {time|CPU|memory} to analyze all this data ● 'cuz not everything needs to be on-line It's 2013 but doing batch processing is still OK
  • 4. Map-what? And why MapReduce and not, say MPI? ● Simple computation model MapReduce exposes a simple (and limited) computational model. It can be a restraining at times but it is a trade off. ● Fault-tolerance, parallelization and distribution among machines for free The framework deals with this for you so you don't have to ● Because it is the bread-and-butter of Big Data processing It is available in all major cloud computing platforms, and it is against what other Big Data systems compare themselves against.
  • 5. Outline ● Fast recap on python and whatnot ● Introduction to MapReduce ● Counting Words ● MrJob and EMR ● Real-life examples
  • 6. Fast recap
  • 7. Fast recap Let's assume you know what the following is: ● JSON ● Python's yield keyword ● Generators in Python ● Amazon S3 ● Amazon EC2 If you don't, raise your hand now. REALLY
  • 8. Recap JSON JSON (JavaScript Object Notation) is a lightweight data-interchange format. It's like if XML and JavaScript slept together and gave birth a bastard but goodlooking child. {"timestamp": "2011-08-15 22:17:31.334057", "track_id": "TRACCJA128F149A144", "tags": [["Bossa Nova", "100"], ["jazz", "20"], ["acoustic", "20"], ["romantic", "20"],], "title": "Segredo", "artist": "Jou00e3o Gilberto"}
  • 9. Recap Python generators From Python's wiki: “Generators functions allow you to declare a function that behaves like an iterator, i.e. it can be used in a for loop.” The difference is: a generator can be iterated (or read) only once as you don't store things in memory but create them on the fly [2]. You can create generators using the yield keyword.
  • 10. Recap Python yield keyword It's just like a return, but turns your function into a generator. Your function will suspend its execution after yielding a value and resume its execution for after the request for the next item in the generator (next loop). def count_from_1(): i = 1 while True: yield i i += 1 for j in count_from_1(): print j
  • 11. Recap Amazon S3 From Wikipedia: “Amazon S3 (Simple Storage Service) is an online storage web service offered by Amazon Web Services.” Its like a distributed filesystem that is easy to use from other Amazon services, specially from Amazon Elastic MapReduce.
  • 12. Recap EC2 - Elastic Cloud Computing From Wikipedia: “EC2 allows users to rent virtual computers on which to run their own computer applications” So you can rent clusters on demand, no need to maintain, keep fixing and up-to-date your ever breaking cluster of computers. Less headache, moar action. Instances can be purchased on demand for fixed prices or you can bid on those.
  • 13. MapReduce: a quick introduction
  • 14. MapReduce MapReduce builds on the observation that many tasks have the same structure: computation is applied over a large number of records to generate partial results, which are then aggregated in some fashion.
  • 15. MapReduce MapReduce builds on the observation that many tasks have the same structure: computation is applied over a large number of records to generate partial results, which are then aggregated in some fashion. Map
  • 16. MapReduce MapReduce builds on the observation that many tasks have the same structure: computation is applied over a large number of records to generate partial results, which are then aggregated in some fashion. Map Reduce
  • 17. Typical (big data) problem ● Iterate over a large number of records Map something of interest from each ● Extract ● Shuffle and sort intermediate results uce Red ● Aggregate intermediate results ● Generate final output
  • 18. Phases of a MapReduction MapReduce have the following steps: map(key, value) -> [(key1, value1), (key1, value2)] combine May happen in parallel, in multiple machines! sort + shuffle reduce(key1, [value1, value2]) -> [(keyX, valueY)]
  • 19. Notice: Reduce phase only starts after all mappers have completed. Yes, there is a synchronization barrier right there. There is no global knowledge Neither mappers nor reducers know what other mappers (or reducers) are processing
  • 20. Counting Words Counting the number of occurrences of a word in a document collection is quite a big deal. Let's try with a small example: "Me gusta correr, me gustas tu. Me gusta la lluvia, me gustas tu."
  • 21. Counting Words "Me gusta correr, me gustas tu. Me gusta la lluvia, me gustas tu." me 4 gusta 2 correr 1 gustas 2 tu 2 la 1 lluvia 1
  • 22. Counting word - in Python doc = open('input') count = {} for line in doc: words = line.split() for w in words: count[w] = count.get(w, 0) + 1 Easy, right? Yeah... too easy. Let's split what we do for each line and aggregate, shall we?
  • 23. Counting word - in MapReduce def map_get_words(self, key, line): for word in line.split(): yield word, 1 def reduce_sum_words(self, word, occurrences): yield word, sum(occurrences)
  • 24. What is Map's output? def map_get_words(self, key, line): for word in line.split(): yield word, 1 key=1 key=2 line="me gusta correr me gustas tu" line="me gusta la lluvia me gustas tu" ('me', 1) ('gusta', 1) ('correr', 1) ('me', 1) ('gustas', 1) ('tu', 1) ('me', 1), ('gusta', 1) ('la', 1) ('lluvia', 1) ('me', 1) ('gustas', 1) ('tu', 1)
  • 25. What about shuffle?
  • 26. What about shuffle? Think of it as a distributed group by operation. In the local map instance/node: ● it sorts map output values, ● groups them by their key, ● send this group of key and associated values to the reduce node responsible for this key. In the reduce instance/node: ● the framework joins all values associated with this key in a single list - for you, for free.
  • 27. What's Shuffle output? or What's Reducer input? Key (input) Values correr [1] Notice: gusta [1, 1] gustas [1, 1] la [1] lluvia [1] me [1, 1, 1, 1] tu [1, 1] This table represents a global view. "In real life", each reducer instance only knows about its own key and values.
  • 28. What's Reducer output? def reduce_sum_words(self, word, occurrences): yield word, sum(occurrences) word occurrences output correr [1] (correr, 1) gusta [1, 1] (gusta, 2) gustas [1, 1] (gustas, 2) la [1] (la, 1) lluvia [1] (lluvia, 1) me [1, 1, 1, 1] (me, 4) tu [1, 1] (tu, 2)
  • 29. MapReduce (main) Implementations Google MapReduce ● C++ ● Proprietary Apache Hadoop ● Java ● ○ interfaces for anything that runs in the JVM ○ Hadoop streamming for a pipe-like programming language agnostic interface Open source Nobody really cares about the others (for now... ;)
  • 30. Amazon Elastic MapReduce (EMR) Amazon Elastic MapReduce ● Uses Hadoop with extra sauces ● creates a hadoop cluster on demand ● It's magical -- except when it fails ● Can be a sort of unpredictable sometimes ○ Installing python modules can fail for no clear reason
  • 31. MrJob It's a python interface for hadoop streaming jobs with a really easy to use interface ● Can run jobs locally or in EMR. ● Takes care of uploading your python code to EMR. ● Deals better if everything is in a single python module. ● Easy interface to chain sequences of M/R steps. ● Some basic tools to aid debugging.
  • 32. Counting words Full MrJob Example from mrjob.job import MRJob class MRWordCounter(MRJob): def get_words(self, key, line): for word in line.split(): yield word, 1 def sum_words(self, word, occurrences): yield word, sum(occurrences) def steps(self): return [self.mr(self.get_words, self.sum_words),] if __name__ == '__main__': MRWordCounter.run()
  • 33. MrJob Lauching a job Running it locally python countwords.py --conf-path=mrjob.conf input.txt Running it in EMR Do not forget to set AWS_ env. vars! python countwords.py --conf-path=mrjob.conf -r emr 's3://ufcgplayground/data/words/*' --no-output --output-dir=s3://ufcgplayground/tmp/bla/
  • 34. MrJob Installing and Environment setup Install MrJob using pip or easy_install Do not, I repeat DO NOT install the version in Ubuntu/Debian. sudo pip install mrjob Setup your environment with AWS credentials export AWS_ACCESS_KEY_ID=... export AWS_SECRET_ACCESS_KEY=... Setup your environment to look for MrJob settings: export MRJOB_CONF=<path to mrjob.conf>
  • 35. MrJob Installing and Environment setup Use our sample MrJob app as your template git clone https://github.com/chaordic/mr101ufcg.git Modify the sample mrjob.conf so that your jobs are labeled to your team It's the Right Thing © to do. s3_logs_uri: s3://ufcgplayground/yournamehere/log/ s3_scratch_uri: s3://ufcgplayground/yournamehere/tmp/ Profit!
  • 36. Rea l
  • 37. Target Categories Objective: Find the most commonly viewed categories per user Input: ● views and orders Patterns used: ● simple aggregation
  • 38. zezin, fulano, [telefone, celulares, vivo] zezin, fulano, [telefone, celulares, vivo] Map input zezin, fulano, [eletro, caos, furadeira] lojaX, fulano, [livros, arte, anime] lojaX, fulano, [livros, arte, anime] lojaX, fulano, [livros, arte, anime]
  • 39. zezin, fulano, [telefone, celulares, vivo] zezin, fulano, [telefone, celulares, vivo] Map input zezin, fulano, [eletro, caos, furadeira] lojaX, fulano, [livros, arte, anime] lojaX, fulano, [livros, arte, anime] lojaX, fulano, [livros, arte, anime] Key
  • 40. zezin, fulano, [telefone, celulares, vivo] zezin, fulano, [telefone, celulares, vivo] Map input zezin, fulano, [eletro, caos, furadeira] lojaX, fulano, [livros, arte, anime] lojaX, fulano, [livros, arte, anime] lojaX, fulano, [livros, arte, anime] Key Sort + Shuffle [telefone, celulares, vivo] (zezin, fulano) [telefone, celulares, vivo] [eletro, caos, furadeira] Reduce Input [livros, arte, anime] (lojaX, fulano) [livros, arte, anime] [livros, arte, anime]
  • 41. [telefone, celulares, vivo] (zezin, fulano) [telefone, celulares, vivo] [eletro, caos, furadeira] Reduce Input [livros, arte, anime] (lojaX, fulano) [livros, arte, anime] [livros, arte, anime]
  • 42. [telefone, celulares, vivo] (zezin, fulano) [telefone, celulares, vivo] [eletro, caos, furadeira] Reduce Input [livros, arte, anime] (lojaX, fulano) [livros, arte, anime] [livros, arte, anime] (zezin, fulano) ([telefone, celulares, vivo], 2) ([eletro, caos, furadeira], 1) Reduce Output (lojaX, fulano) ([livros, arte, anime], 3)
  • 43. Filter Expensive Categories Objective: List all categories where a user purchased something expensive. Input: ● Orders (for price and user information) ● Products (for category information) Patterns used: ● merge using reducer
  • 44. BuyOrders Products Map Input lojaX livro fulano R$ 20 lojaX iphone deltrano R$ 1800 lojaX livro [livros, arte, anime] lojaX iphone [telefone, celulares, vivo] We have to merge those tables above!
  • 45. BuyOrders Products Map Input lojaX livro fulano R$ 20 lojaX iphone deltrano R$ 1800 lojaX livro [livros, arte, anime] lojaX iphone [telefone, celulares, vivo] common Key
  • 46. BuyOrders Products Map Input Map Output lojaX livro fulano R$ 20 (nada, é barato) lojaX iphone deltrano R$ 1800 {”usuario” : “deltrano”} lojaX livro [livros, arte, anime] {“cat”: [livros...]} lojaX iphone [telefone, celulares, vivo] {“cat”: [telefone...]} Key Value
  • 47. BuyOrders Products Map Input Map Output lojaX livro fulano R$ 20 (nada, é barato) lojaX iphone deltrano R$ 1800 {”usuario” : “deltrano”} lojaX livro [livros, arte, anime] {“cat”: [livros...]} lojaX iphone [telefone, celulares, vivo] {“cat”: [telefone...]} Reduce Input Key Value (lojaX, livro) {“cat”: [livros, arte, anime]} (lojaX, iphone) {”usuario” : “deltrano”} {“cat”: [telefone, celulares, vivo]}
  • 48. Reduce Input (lojaX, livro) {“cat”: [livros, arte, anime]} (lojaX, iphone) {”usuario” : “deltrano”} {“cat”: [telefone, celulares, vivo]} Key Values
  • 49. Reduce Input (lojaX, livro) {“cat”: [livros, arte, anime]} (lojaX, iphone) {”usuario” : “deltrano”} {“cat”: [telefone, celulares, vivo]} Key Values Those are the parts we care about!
  • 50. Reduce Input (lojaX, livro) {“cat”: [livros, arte, anime]} (lojaX, iphone) {”usuario” : “deltrano”} {“cat”: [telefone, celulares, vivo]} Reduce Output Key (lojaX, deltrano) Values [telefone, celulares, vivo]
  • 51. Rea l tasets Da
  • 52. Real datasets, real problems In the following hour we will write code to analyse some real datasets: ● Twitter Dataset (from an article published in WWW'10) ● LastFM Dataset, from The Million Song Datset Supporting code ● available at GitHub, under https://github. com/chaordic/mr101ufcg ● comes with sample data under data for local runs.
  • 53. Twitter Followers Dataset A somewhat big dataset ● 41.7 million profiles ● 1.47 billion social relations (who follows who) ● 25 Gb of uncompressed data Available at s3://mr101ufcg/data/twitter/ ... ● splitted/*.gz full dataset splitted in small compressed files ● numeric2screen.txt numerid id to original screen name mapping ● followed_by.txt original 25Gb dataset as a single file
  • 54. Twitter Followers Dataset Each line in followed_by.txt has the following format: user_id t follower_id For instance: 12 t 38 12 t 41 13 t 47 13 t 52 13 t 53 14 t 56
  • 55. Million Song Dataset project's Last.fm Dataset A not-so-big dataset ● 943,347 tracks ● 1.2G of compressed data Yeah, it is not all that big... Available at s3://mr101ufcg/data/lastfm/ ... ● metadata/*.gz Track metadata information, in JSONProtocol format. ● similars/*.gz Track similarity information, in JSONProtocol format.
  • 56. Million Song Dataset project's Last.fm Dataset JSONProcotol encodes key-pair information in a single line using json-encoded values separated by a tab character ( t ). <JSON encoded data> t <JSON encoded data> Exemple line: "TRACHOZ12903CCA8B3" t {"timestamp": "2011-09-07 22:12: 47.150438", "track_id": "TRACHOZ12903CCA8B3", "tags": [], "title": "Close Up", "artist": "Charles Williams"}
  • 57. tions? ues Q
  • 58. Stuff I didn't talk about but are sorta cool Persistent jobs Serialization (protocols in MrJob parlance) Amazon EMR Console Hadoop dashboard (and port 9100)
  • 59. Combiners Are just like reducers but take place just after a Map and just before data is sent to the network during shuffle. Combiners must... ● be associative {a.(b.c) == (a.b).c} ● commutative (a.b == b.a) ● have the same input and output types as yours Map output type. Caveats: ● Combiners can be executed zero, one or many times, so don't make your MR depend on them
  • 60. Reference & Further reading [1] MapReduce: A Crash Course [2] StackOverflow: The python yield keyword explained [3] Explicando iterables, generators e yield no python [4] MapReduce: Simplied Data Processing on Large Clusters
  • 61. Reference & Further reading [5] MrJob 4.0 - Quick start [6] Amazon EC2 Instance Types
  • 62. Life beyond MapReduce What reading about other frameworks for distributed processing with BigData? ● Spark ● Storm ● GraphLab And don't get me started on NoSQL...
  • 63. Many thanks to... for supporting this course. You know there will be some live, intense, groovy Elastic MapReduce action right after this presentation, right?
  • 64. Questions? Feel free to contact me at tiago. macambira@chaordicsystems.com.br Or follows us @chaordic
  • 65. So, lets write some code? Twitter Dataset ● Count how many followers each user has ● Discover the user with more followers ● What if I want the top-N most followed? LastFM ● Merge similarity and metadata for tracks ● What is the most "plain" song? ● What is the plainest rock song according only to rock songs?
  • 66. Extra slides