Hyderabad Scalability Meetup, profile picture
Uploaded byHyderabad Scalability Meetup
PDF, PPTX821 views

Map reduce and the art of Thinking Parallel - Dr. Shailesh Kumar

This document discusses parallelizing computation using MapReduce. It begins with an overview of MapReduce and how it works, breaking data into chunks that are processed in parallel by map tasks, and then combining results via reduce tasks. It then provides examples of using MapReduce to solve three problems: 1) calculating similarity between all pairs of documents, 2) parallelizing k-means clustering, and 3) finding all maximal cliques in a graph. For each problem it describes how to define the map and reduce functions to solve the problem in a data-parallel manner using MapReduce.

Embed presentation

Download as PDF, PPTX
MapReduce and the ā€Ø art of ā€œThinking Parallelā€ Shailesh Kumar Third Leap, Inc.
Three Iā€™s of a great product! Interface Intuitive |Functional | Elegant Infrastructur e Storage |Computation | Network Intelligence Learn |Predict | Adapt | Evolve
Drowning in Data, Starving for Knowledge ATATTAGGTTTTTACCTACCC AGGAAAAGCCAACCAACCTC GATCTCTTGTAGATCTGTTCT CTAAACGAACTTTAAAATCTG TGTAGCTGTCGCTCGGCTG CATGCCTAGTGCACCTACGC AGTATAAACAATAATAAATTTT ACTGTCGTTGACAAGAAACG AGTAACTCGTCCCTCTTCTG CAGACTGCTTATTACGCGAC CGTAAGCTACā€¦
How BIG is Big Data? 600 million tweets per DAY 100 hours per MINUTE 800+ websites per MINUTE 100 TB of data uploaded DAILY 3.5 Billion queries PER DAY 300 Million Active customers How BIG is BigData?
ā–Ŗ Better Sensors ā–Ŗ Higher resolution, Real-time, Diverse measurements, ā€¦ ā–Ŗ Faster Communication ā–Ŗ Network infrastructure, Compression Technologies, ā€¦ ā–Ŗ Cheaper Storage ā–Ŗ Cloud based storage, large warehouses, NoSQL databases ā–Ŗ Massive Computation ā–Ŗ Cloud computing, Mapreduce/Hadoop parallel processing paradigms ā–Ŗ Intelligent Decisions ā–Ŗ Advances in Machine Learning and Artificial Intelligence How did we get here?
The Evolution of ā€œComputingā€
Parallel Computing Basics ā–Ŗ Data Parallelism (distributed computing) ā–Ŗ Lots of data ! Break it into ā€œchunksā€, ā–Ŗ Process each ā€œchunkā€ of data in parallel, ā–Ŗ Combine results from each ā€œchunkā€ ā–Ŗ MAPREDUCE = Data Parallelism ā–Ŗ Process Parallelism (data flow computing) ā–Ŗ Lots of stages ! Set up process graph ā–Ŗ Pass data through all stages ā–Ŗ All stages running in parallel on different data ā–Ŗ Assembly line = process parallelism
Agenda MAPREDUCE Background Problem 1 ā€“ Similarity between all pairs of documents! Problem 2 ā€“ Parallelizing K-Means clustering Problem 3 ā€“ Finding all Maximal Cliques in a Graph
MAPREDUCE 101: A 4-stage ProcessLotsofdata Shard 1 Shard N Shard 2 Reduce 1 Reduce R Map 1 Map 2 Map K Combine 1 Combine 2 Combine K Shuffle 1 Shuffle 2 Shuffle K Output 1 Output R Each Map processes N/K shards
MAPREDUCE 101: An example Task ā–Ŗ Count total frequency of all words on the web ā–Ŗ Total number of documents > 20Billion ā–Ŗ Total number of unique words > 20Million ā–Ŗ Non-Parallel / Linear Implementation for each document d on the Web for each unique word w in d DocCount w d( )= # times w occurred in d WebCount w( ) += DocCount w d( )
MAPREDUCE ā€“ MAP/COMBINE Shard1 Key Value A 10 B 7 C 9 D 3 B 4 Key Value A 10 B 11 C 9 D 3 Shard2 Key Value A 3 D 1 C 4 D 9 B 6 Key Value A 3 B 6 C 4 D 10 Shard3 Key Value B 3 D 5 C 4 A 6 A 3 Map-1 Map-2 Map-3 Key Value A 9 B 3 C 4 D 5 Combine-1 Combine-2 Combine-3
MAPREDUCE ā€“ Shuffle/Reduce Key Value A 10 B 11 C 9 D 3 Key Value A 3 B 6 C 4 D 10 Key Value A 9 B 3 C 4 D 5 Key Value A 10 A 3 A 9 C 9 C 4 C 4 Key Value B 11 B 6 B 3 D 3 D 10 D 5 Shuffle 1 Shuffle 2 Shuffle 3 Key Value A 22 C 17 Key Value B 20 D 18 Reduce 1 Reduce 2
Key Questions in MAPREDUCE ā–Ŗ Is the task really ā€œdata-parallelizableā€? ā–Ŗ High dependence tasks (e.g. Fibonacci series) ā–Ŗ Recursive tasks (e.g. Binary Search) ā–Ŗ What is the key-value pair output for MAP step? ā–Ŗ Each map processes only one data record at a time ā–Ŗ It can generate none, one, or multiple key-value pairs ā–Ŗ How to combine values of a key in REDUCE step? ā–Ŗ The key for reduce is same as key for Map output ā–Ŗ The reduce function must be ā€œorder agnosticā€
Other considerations ā–Ŗ Reliability/Robustness ā–Ŗ A processor or disk might go bad during the process ā–Ŗ Optimization/Efficiency ā–Ŗ Allocate CPUā€™s near data shards to reduce network overhead ā–Ŗ Scale/Parallelism ā–Ŗ Parallelization linearly proportional to number of machines ā–Ŗ Simplicity/Usability ā–Ŗ Just specify the Map task and the Reduce task and be done! ā–Ŗ Generality ā–Ŗ Lots of parallelizable tasks can be written in MapReduce ā–Ŗ With some creativity, many more than you can imagine!
Agenda MAPREDUCE Background Problem 1 ā€“ Similarity between all pairs of documents! Problem 2 ā€“ Parallelizing K-Means clustering Problem 3 ā€“ Finding all Maximal Cliques in a Graph
Similarity between all pairs of docs. ā–Ŗ Why bother? ā–Ŗ Document Clustering, Similar document search, etc. ā–Ŗ Document represented as a ā€œBag-of-Tokensā€ ā–Ŗ A weight associated with each tokens in vocabulary. ā–Ŗ Most weights are zero ā€“ Sparsity ā–Ŗ Cosine Similarity between two documents di = w1 i ,w2 i ,...,wT i { }, dj = w1 j ,w2 j ,...,wT j { } Sim di ,dj( )= wt i t=1 T āˆ‘ Ɨ wt j
Non-Parallel / Linear Implementation For each document di For each document dj ( j > i) Sim di ,dj( )= wt i t=1 T āˆ‘ Ɨ wt j Complexity = O D2 TĻƒ( ) Ļƒ = Sparsity factor =10āˆ’5 = Average Fraction of vocabulary per document D = O(10B), T = O(10M ) Complexity = O 1020+7āˆ’5 ( )= O 1022 ( )
Toy Example for doc-doc similarity A classic ā€œJoinā€ Documents = W, X,Y, Z{ }, Words = a,b,c,d,e{ } W ā†’ a,1 , b,2 , e,5{ } X ā†’ a,3 , c,4 , d,5{ } Y ā†’ b,6 , c,7 , d,8{ } Z ā†’ a,9 , e,10{ } Input W, X ā†’ Sim W, X( )= 3 W,Y ā†’ Sim W,Y( )= 12 W,Z ā†’ Sim W,Z( )= 59 X,Y ā†’ Sim X,Y( )= 68 X,Z ā†’ Sim X,Z( )= 27 Y,Z ā†’ Sim Y,Z( )= 0 Output
Reverse Indexing to the rescue First convert the data to reverse index aā†’ W,1 , X,3 , Z,9{ } bā†’ W,2 , Y,6{ } cā†’ X,4 , Y,7{ } d ā†’ X,5 , Y,8{ } eā†’ W,5 , Z,10{ } W ā†’ a,1 , b,2 , e,5{ } X ā†’ a,3 , c,4 , d,5{ } Y ā†’ b,6 , c,7 , d,8{ } Z ā†’ a,9 , e,10{ }
Key/Value for the MAP-Step aā†’ W,1 , X,3 , Z,9{ } W, X ā†’ 3 W,Z ā†’ 9 X,Z ā†’ 27 bā†’ W,2 , Y,6{ } cā†’ X,4 , Y,7{ } W,Y ā†’12 eā†’ W,5 , Z,10{ } d ā†’ X,5 , Y,8{ } X,Y ā†’ 28 X,Y ā†’ 40 W,Z ā†’ 50 W, X ā†’ 3 W,Y ā†’12 W,Z ā†’ 9 W,Z ā†’ 50 X,Y ā†’ 40 X,Y ā†’ 28 X,Z ā†’ 27
Value combining in REDUCE-Step W, X ā†’ 3 W,Y ā†’12 W,Z ā†’ 9 W,Z ā†’ 50 X,Y ā†’ 40 X,Y ā†’ 28 X,Z ā†’ 27 W, X ā†’ Sim W, X( )= 3 W,Y ā†’ Sim W,Y( )= 12 W,Z ā†’ Sim W,Z( )= 59 X,Y ā†’ Sim X,Y( )= 68 X,Z ā†’ Sim X,Z( )= 27 Y,Z ā†’ Sim Y,Z( )= 0
Agenda MAPREDUCE Background Problem 1 ā€“ Similarity between all pairs of documents! Problem 2 ā€“ Parallelizing K-Means clustering Problem 3 ā€“ Finding all Maximal Cliques in a Graph
assignments ! centers K-Means Clustering mk (t+1) ā† Ī“n,k (t ) xn n=1 N āˆ‘ Ī“n,k (t) n=1 N āˆ‘ m1 (t+1) m2 (t+1) Ī“n,2 (t ) = 1 Ī“n,1 (t ) = 1 m1 (t ) m2 (t ) centers ! assignments Ī“n,k (t+1) = k == arg min j=1...K Ī” x n( ) ,mj (t) ( ){ }( )
K-means clustering 101 ā€“ Non-parallel E-Step ā€“ Update assignments from centers ā€Ø M-Step ā€“ Update centers from cluster assignments Ļ€n (t) ā† arg min k=1...K Ī” xn ,mk (t) ( ){ } mk (t+1) ā† Ī“ Ļ€n (t) = k( )xn n=1 N āˆ‘ Ī“ Ļ€n (t) = k( ) n=1 N āˆ‘ O NKD( ): N = Number of data points K = Number of clusters D = number of dimensions āŽ§ āŽØ āŽŖ āŽ© āŽŖ O ND( ): N = Number of data points D = number of dimensions āŽ§ āŽØ āŽ©
K-Means MapReduce mk (t) { }k=1 K Key = Ļ€n (t) ā†’ Value = xn Ļ€n (t) = arg min k=1...K Ī” xn ,mk (t) ( ){ } mk (t+1) ā† Ī“ Ļ€n (t) = k( )xn n=1 N āˆ‘ Ī“ Ļ€n (t) = k( ) n=1 N āˆ‘ mk (t+1) { }k=1 K Ļ€n (t) mk (t+1) Map Shuffle Reduce Iterative MapReduce: Update Cluster Centers/iteration
Agenda MAPREDUCE Background Problem 1 ā€“ Similarity between all pairs of documents! Problem 2 ā€“ Parallelizing K-Means clustering Problem 3 ā€“ Finding all Maximal Cliques in a Graph
Cliques: Useful structures in Graphs ā€¢ People ā€¢ Products ā€¢ Movies ā€¢ Keywords ā€¢ Documents ā€¢ Genes ā€¢ Neurons ā€¢ Co-Social ā€¢ Co-purchase ā€¢ Co-like ā€¢ Co-occurrence ā€¢ Similarity ā€¢ Co-expressions ā€¢ Co-firing
guitarist rock-music guitar song musician rock-band singer electric-guitar singing university school college student classroom school-teacher teacher teacher-student-relationship judge lawsuit trial lawyerfalse-persecution perjury courtroom Example Concepts in IMDB
Graph, Cliques, and Maximal Cliques Clique = a ā€œfully connectedā€ sub-graph Maximal Clique = a clique with no ā€œSuper-cliqueā€ Finding all Maximal Cliques is NP-hard: O(3n/3) a e b f c g d h
Neighborhood of a Clique a e b f c g d h f is connected to BOTH b and c g is connected to BOTH b and c N({b,c}) = {f,g} CLIQUEMAP: Clique (key) ! Its Neighbor (value) {a} ā†’ {b,e} {a,b} ā†’ {e} {b,c} ā†’ { f,g} {b,c, f } ā†’ {g} {h} ā†’ āˆ… {c,d} ā†’ āˆ… {a,b,e} ā†’ āˆ… {b,c, f ,g} ā†’ āˆ…
Growing Cliques from CliqueMap {b,c, f} ā†’ {g} a e b f c g d h {b,c, f} is a clique g is connected to all of them āŽ« āŽ¬ āŽ­ ā‡’ {b,c, f,g} is a clique
MapReduce for Maximal Cliques CliqueMap of size k ! size k + 1 {a,b} ā†’ {e} {a,e} ā†’ {b} {b,c} ā†’ { f,g} {b,e} ā†’ {a} {b, f } ā†’ {c,g} {b,g} ā†’ {c, f } {c, f } ā†’ {b,g} {c, g} ā†’ {b, f } { f, g} ā†’ {b,c} {c,d} ā†’ āˆ… Iteration 2 {a,b,e} ā†’ āˆ… {b,c, f } ā†’ {g} {b,c,g} ā†’ { f } {b, f ,g} ā†’ {c} {c, f ,g} ā†’ {b} Iteration 3 {b,c, f,g} ā†’ āˆ… Iteration 4 {a} ā†’ {b,e} {b} ā†’ {a,c,e, f ,g} {c} ā†’ {b,d, f ,g} {d} ā†’ {c} {e} ā†’ {a,b} { f } ā†’ {b,c,g} {g} ā†’ {b,c, f } {h} ā†’ āˆ… Iteration 1 Input: Adjacency List a e b f c g d h
Key/Value for the MAP-Step a e b f c g d h {a} ā†’ {b,e} {a,b} ā‡’ {e} {a,e} ā‡’ {b} {e} ā†’ {a,b} {b} ā†’ {a,c,e, f,g} {a,e} ā‡’ {b} {b,e} ā‡’ {a} {a,b} ā‡’ {c,e, f, g} {b,c} ā‡’ {a,e, f, g} {b,e} ā‡’ {a,c, f, g} {b, f } ā‡’ {a,c,e, g} {b,g} ā‡’ {a,c,e, f } {a,e} ā‡’ {b} {a,e} ā‡’ {b} {a,b} ā‡’ {e} {a,b} ā‡’ {c,e, f ,g} {b,e} ā‡’ {a.c, f,g} {b,e} ā‡’ {a} SHUFFLE MAP
Value combining in REDUCE-Step a e b f c g d h {a,e} ā‡’ {b} {a,e} ā‡’ {b} {a,b} ā‡’ {e} {a,b} ā‡’ {c,e, f ,g} {b,e} ā‡’ {a,c, f ,g} {b,e} ā‡’ {a} SHUFFLE {a,b} ā†’ {e}āˆ©{c,e, f,g} = {e} {b,e} ā†’ {a,c, f,g}āˆ©{a} = {a} {a,e} ā†’ {b}āˆ©{b} = {b} REDUCE Reduce = Intersection
Value combining in REDUCE-Step a e b f c g d h c,d{ }ā‡’ b, f ,g{ } c,d{ }ā‡’ āˆ… c{ }ā†’ b,d, f,g{ } d{ }ā†’ c{ } b,c{ }ā‡’ a,e, f,g{ } b,c{ }ā‡’ d, f,g{ } b{ }ā†’ a,c,e, f,g{ } c{ }ā†’ b,d, f ,g{ } c,d{ }ā†’ {b, f ,g}āˆ©āˆ… = āˆ… b,c{ }ā†’ a,e, f ,g{ }āˆ© d, f ,g{ } = f ,g{ }
ā€œArt of Thinking Parallelā€ is about ā–Ŗ Transforming the Input Data appropriately ā–Ŗ e.g. Reverse Indexing (doc-doc similarity) ā–Ŗ Breaking the problem into smaller ones ā–Ŗ e.g. Iterative MapReduce (clustering) ā–Ŗ Designing the Map step - Key/Value output ā–Ŗ e.g. CliqueMaps in Maximal Cliques ā–Ŗ Design the Reduce step ā€“ Combine values of key ā–Ŗ e.g. Intersections in Maximal Cliques

More Related Content

PDF
Interval Pattern Structures: An introdution
byINSA Lyon - L'Institut National des Sciences AppliquƩes de Lyon
Ā 
PPTX
Backtraking pic&def
bybalavigneshwari
Ā 
PDF
2017 RM-URISA Track: Spatial SQL - The Best Kept Secret in the Geospatial World
byGIS in the Rockies
Ā 
PDF
Polyadic systems and multiplace representations
bySteven Duplij (Stepan Douplii)
Ā 
DOCX
R-ggplot2 package Examples
byDr. Volkan OBAN
Ā 
PDF
High-Performance Approach to String Similarity using Most Frequent K Characters
byHolistic Benchmarking of Big Linked Data
Ā 
PDF
Spatial Analysis with R - the Good, the Bad, and the Pretty
byNoam Ross
Ā 
PDF
A new generalized lindley distribution
byAlexander Decker
Ā 
Interval Pattern Structures: An introdution
byINSA Lyon - L'Institut National des Sciences AppliquƩes de Lyon
Ā 
Backtraking pic&def
bybalavigneshwari
Ā 
2017 RM-URISA Track: Spatial SQL - The Best Kept Secret in the Geospatial World
byGIS in the Rockies
Ā 
Polyadic systems and multiplace representations
bySteven Duplij (Stepan Douplii)
Ā 
R-ggplot2 package Examples
byDr. Volkan OBAN
Ā 
High-Performance Approach to String Similarity using Most Frequent K Characters
byHolistic Benchmarking of Big Linked Data
Ā 
Spatial Analysis with R - the Good, the Bad, and the Pretty
byNoam Ross
Ā 
A new generalized lindley distribution
byAlexander Decker
Ā 

What's hot

PDF
An application of gd
bygraphhoc
Ā 
ODP
Geospatial Data in R
byBarry Rowlingson
Ā 
PDF
A Visual Exploration of Distance, Documents, and Distributions
byRebecca Bilbro
Ā 
PDF
Multi-scalar multiplication: state of the art and new ideas
byGus Gutoski
Ā 
PDF
The Lazy Traveling Salesman Memory Management for Large-Scale Link Discovery
byHolistic Benchmarking of Big Linked Data
Ā 
DOCX
k-means Clustering and Custergram with R
byDr. Volkan OBAN
Ā 
PDF
1452 86301000013 m
byPraveen Kumar
Ā 
PDF
Kumaraswamy distributin:
byPankaj Das
Ā 
PDF
Treewidth and Applications
byASPAK2014
Ā 
PDF
DISTANCE TWO LABELING FOR MULTI-STOREY GRAPHS
bygraphhoc
Ā 
PDF
Direct split-radix algorithm for fast computation of type-II discrete Hartley...
byTELKOMNIKA JOURNAL
Ā 
PPTX
Graph Analytics - From the Whiteboard to Your Toolbox - Sam Lerma
byPyData
Ā 
PDF
DocEng2013 Bilauca Healy - Splitting Wide Tables Optimally
bymbilauca
Ā 
PPTX
Sqlserver 2008 r2
byKashif Akram
Ā 
PDF
Distributed Support Vector Machines
byHarsha Vardhan Tetali
Ā 
PDF
S. Duplij. Polyadic algebraic structures and their applications
bySteven Duplij (Stepan Douplii)
Ā 
PDF
Multilayerity within multilayerity? On multilayer assortativity in social net...
byMoses Boudourides
Ā 
PPTX
An Interactive Introduction To R (Programming Language For Statistics)
byDataspora
Ā 
An application of gd
bygraphhoc
Ā 
Geospatial Data in R
byBarry Rowlingson
Ā 
A Visual Exploration of Distance, Documents, and Distributions
byRebecca Bilbro
Ā 
Multi-scalar multiplication: state of the art and new ideas
byGus Gutoski
Ā 
The Lazy Traveling Salesman Memory Management for Large-Scale Link Discovery
byHolistic Benchmarking of Big Linked Data
Ā 
k-means Clustering and Custergram with R
byDr. Volkan OBAN
Ā 
1452 86301000013 m
byPraveen Kumar
Ā 
Kumaraswamy distributin:
byPankaj Das
Ā 
Treewidth and Applications
byASPAK2014
Ā 
DISTANCE TWO LABELING FOR MULTI-STOREY GRAPHS
bygraphhoc
Ā 
Direct split-radix algorithm for fast computation of type-II discrete Hartley...
byTELKOMNIKA JOURNAL
Ā 
Graph Analytics - From the Whiteboard to Your Toolbox - Sam Lerma
byPyData
Ā 
DocEng2013 Bilauca Healy - Splitting Wide Tables Optimally
bymbilauca
Ā 
Sqlserver 2008 r2
byKashif Akram
Ā 
Distributed Support Vector Machines
byHarsha Vardhan Tetali
Ā 
S. Duplij. Polyadic algebraic structures and their applications
bySteven Duplij (Stepan Douplii)
Ā 
Multilayerity within multilayerity? On multilayer assortativity in social net...
byMoses Boudourides
Ā 
An Interactive Introduction To R (Programming Language For Statistics)
byDataspora
Ā 

Viewers also liked

PDF
Offline first geeknight
byHyderabad Scalability Meetup
Ā 
PDF
GeekNight: Evolution of Programming Languages
byHyderabad Scalability Meetup
Ā 
PDF
Serverless architectures
byHyderabad Scalability Meetup
Ā 
PPTX
Geeknight : Artificial Intelligence and Machine Learning
byHyderabad Scalability Meetup
Ā 
PPTX
Proving parallelism
bysalvie alvaro
Ā 
PDF
Understanding and building big data Architectures - NoSQL
byHyderabad Scalability Meetup
Ā 
PPTX
Parallelism and perpendicularity
bysalvie alvaro
Ā 
PPT
Pertemuan ii mankiw krugman
bystephaniejessey
Ā 
PPTX
Sim Photosynthesis
byeric
Ā 
PPTX
What is Parallelism?
byHussain Al-ghawi
Ā 
Offline first geeknight
byHyderabad Scalability Meetup
Ā 
GeekNight: Evolution of Programming Languages
byHyderabad Scalability Meetup
Ā 
Serverless architectures
byHyderabad Scalability Meetup
Ā 
Geeknight : Artificial Intelligence and Machine Learning
byHyderabad Scalability Meetup
Ā 
Proving parallelism
bysalvie alvaro
Ā 
Understanding and building big data Architectures - NoSQL
byHyderabad Scalability Meetup
Ā 
Parallelism and perpendicularity
bysalvie alvaro
Ā 
Pertemuan ii mankiw krugman
bystephaniejessey
Ā 
Sim Photosynthesis
byeric
Ā 
What is Parallelism?
byHussain Al-ghawi
Ā 

Similar to Map reduce and the art of Thinking Parallel - Dr. Shailesh Kumar

PPTX
introduction to Complete Map and Reduce Framework
byharikumar288574
Ā 
PDF
MapReduce
byAbe Arredondo
Ā 
PDF
Geoff Rothman Presentation on Parallel Processing
byGeoff Rothman
Ā 
PPT
Mapreduce in Search
byAmund Tveit
Ā 
PPT
Big Data, a space adventure - Mario Cartia - Codemotion Milan 2014
byCodemotion
Ā 
PPTX
COMPLETE MAP AND REDUCE FRAMEWORK INTRODUCTION
byharikumar288574
Ā 
PPTX
Introduction to Map-Reduce in Hadoop.pptx
bytest1miniproject
Ā 
PPTX
Introduction to Map-Reduce in Hadoop.pptx
bytest1miniproject
Ā 
PDF
Big Data, a space adventure - Mario Cartia - Codemotion Rome 2015
byCodemotion
Ā 
PPT
An Introduction To Map-Reduce
byFrancisco PĆ©rez-Sorrosal
Ā 
PPTX
Kansas City Big Data: The Future Of Insights - Keynote: "Big Data Technologie...
bykcitp
Ā 
PPTX
Map reduce programming model to solve graph problems
byNishant Gandhi
Ā 
PDF
lec8_ref.pdf
byvishal choudhary
Ā 
PDF
07 2
bya_b_g
Ā 
PDF
Intro to Map Reduce
byDoron Vainrub
Ā 
PDF
MapReduce technique in MongoDB and Can Tho university
bytamb2203579
Ā 
PPTX
Big data
byWaqas Nawaz
Ā 
PDF
Lecture 1 mapreduce
byShubham Bansal
Ā 
PDF
bigD3_mapReducebigD3_mapReducebigD3_mapReduce.pdf
bysh5701
Ā 
PDF
Start From A MapReduce Graph Pattern-recognize Algorithm
byYu Liu
Ā 
introduction to Complete Map and Reduce Framework
byharikumar288574
Ā 
MapReduce
byAbe Arredondo
Ā 
Geoff Rothman Presentation on Parallel Processing
byGeoff Rothman
Ā 
Mapreduce in Search
byAmund Tveit
Ā 
Big Data, a space adventure - Mario Cartia - Codemotion Milan 2014
byCodemotion
Ā 
COMPLETE MAP AND REDUCE FRAMEWORK INTRODUCTION
byharikumar288574
Ā 
Introduction to Map-Reduce in Hadoop.pptx
bytest1miniproject
Ā 
Introduction to Map-Reduce in Hadoop.pptx
bytest1miniproject
Ā 
Big Data, a space adventure - Mario Cartia - Codemotion Rome 2015
byCodemotion
Ā 
An Introduction To Map-Reduce
byFrancisco PĆ©rez-Sorrosal
Ā 
Kansas City Big Data: The Future Of Insights - Keynote: "Big Data Technologie...
bykcitp
Ā 
Map reduce programming model to solve graph problems
byNishant Gandhi
Ā 
lec8_ref.pdf
byvishal choudhary
Ā 
07 2
bya_b_g
Ā 
Intro to Map Reduce
byDoron Vainrub
Ā 
MapReduce technique in MongoDB and Can Tho university
bytamb2203579
Ā 
Big data
byWaqas Nawaz
Ā 
Lecture 1 mapreduce
byShubham Bansal
Ā 
bigD3_mapReducebigD3_mapReducebigD3_mapReduce.pdf
bysh5701
Ā 
Start From A MapReduce Graph Pattern-recognize Algorithm
byYu Liu
Ā 

More from Hyderabad Scalability Meetup

PPTX
Turbo charging v8 engine
byHyderabad Scalability Meetup
Ā 
PDF
Git internals
byHyderabad Scalability Meetup
Ā 
PPTX
Nlp
byHyderabad Scalability Meetup
Ā 
PPTX
Internet of Things - GeekNight - Hyderabad
byHyderabad Scalability Meetup
Ā 
PDF
Demystify Big Data, Data Science & Signal Extraction Deep Dive
byHyderabad Scalability Meetup
Ā 
PDF
Demystify Big Data, Data Science & Signal Extraction Deep Dive
byHyderabad Scalability Meetup
Ā 
PPTX
Java 8 Lambda Expressions
byHyderabad Scalability Meetup
Ā 
PPT
No SQL and MongoDB - Hyderabad Scalability Meetup
byHyderabad Scalability Meetup
Ā 
PPTX
Apache Spark - Lightning Fast Cluster Computing - Hyderabad Scalability Meetup
byHyderabad Scalability Meetup
Ā 
PPT
Docker by demo
byHyderabad Scalability Meetup
Ā 
Turbo charging v8 engine
byHyderabad Scalability Meetup
Ā 
Git internals
byHyderabad Scalability Meetup
Ā 
Nlp
byHyderabad Scalability Meetup
Ā 
Internet of Things - GeekNight - Hyderabad
byHyderabad Scalability Meetup
Ā 
Demystify Big Data, Data Science & Signal Extraction Deep Dive
byHyderabad Scalability Meetup
Ā 
Demystify Big Data, Data Science & Signal Extraction Deep Dive
byHyderabad Scalability Meetup
Ā 
Java 8 Lambda Expressions
byHyderabad Scalability Meetup
Ā 
No SQL and MongoDB - Hyderabad Scalability Meetup
byHyderabad Scalability Meetup
Ā 
Apache Spark - Lightning Fast Cluster Computing - Hyderabad Scalability Meetup
byHyderabad Scalability Meetup
Ā 
Docker by demo
byHyderabad Scalability Meetup
Ā 

Recently uploaded

PDF
Configure and Manage Systemd Timers- RHCSA (RH134).pdf
byLinuxCert Guru
Ā 
PPTX
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
Ā 
PPTX
Retrieval Augmented Generation- The Synergistic Power of Prompt Engineering
bySemantic SEO BD
Ā 
PDF
NTG -Company Profile_Software_Vendor_Company_in_MENA
byMustafa Kuğu
Ā 
PDF
UiPath Automation Developer Associate Training Series 2026 - Session 1
byDianaGray10
Ā 
PPTX
WEEK 1-introduction of industrial arts grade 7.pptx
byrosierufinomaliongan
Ā 
PDF
Traditional-Security-Models-No-Longer-Work.pptx (1).pdf
byOhhproJunction
Ā 
PPTX
TechSprint Inauguration at SJB Institute of Technology held on 18 December 2025
bysuhasspgdg
Ā 
PDF
Python Automation in Action: Real-World Use Cases and Practical Implementation
byDenys Smirnov
Ā 
PDF
Understanding Foldable 3-Wheel Electric Scooters for Everyday Use
byTopmate
Ā 
PDF
TravelTech Paris 2025 | The EU Digital Identity Wallet and itā€™s impact on Tra...
byapidays
Ā 
PDF
Mesh WiFi Router: The Smart Solution for Fast, Seamless, Whole-Home Internet
byAeroMesh Systems
Ā 
PDF
Explaining the flow of purpose-specific BOM
byakipii ogaoga
Ā 
PDF
GenerationAI Paris 2025 | How Agentic AI is Reinventing Organizations
byapidays
Ā 
PDF
Post-Hackathon-Learnings-Maximizing-Impact-Beyond-the-Event.pdf
byishantyadav1111
Ā 
PPTX
NTG - Data Center Management System Software
byMustafa Kuğu
Ā 
PDF
How does MES(Manufacturing Execution System) work?
byakipii ogaoga
Ā 
PDF
Analyze and Preserve Logs - RHCSA (RH134).pdf
byLinuxCert Guru
Ā 
PDF
AI for Risk Management & Fraud Detection ppt.pdf
byalexmartincaneda
Ā 
PPTX
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
Ā 
Configure and Manage Systemd Timers- RHCSA (RH134).pdf
byLinuxCert Guru
Ā 
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
Ā 
Retrieval Augmented Generation- The Synergistic Power of Prompt Engineering
bySemantic SEO BD
Ā 
NTG -Company Profile_Software_Vendor_Company_in_MENA
byMustafa Kuğu
Ā 
UiPath Automation Developer Associate Training Series 2026 - Session 1
byDianaGray10
Ā 
WEEK 1-introduction of industrial arts grade 7.pptx
byrosierufinomaliongan
Ā 
Traditional-Security-Models-No-Longer-Work.pptx (1).pdf
byOhhproJunction
Ā 
TechSprint Inauguration at SJB Institute of Technology held on 18 December 2025
bysuhasspgdg
Ā 
Python Automation in Action: Real-World Use Cases and Practical Implementation
byDenys Smirnov
Ā 
Understanding Foldable 3-Wheel Electric Scooters for Everyday Use
byTopmate
Ā 
TravelTech Paris 2025 | The EU Digital Identity Wallet and itā€™s impact on Tra...
byapidays
Ā 
Mesh WiFi Router: The Smart Solution for Fast, Seamless, Whole-Home Internet
byAeroMesh Systems
Ā 
Explaining the flow of purpose-specific BOM
byakipii ogaoga
Ā 
GenerationAI Paris 2025 | How Agentic AI is Reinventing Organizations
byapidays
Ā 
Post-Hackathon-Learnings-Maximizing-Impact-Beyond-the-Event.pdf
byishantyadav1111
Ā 
NTG - Data Center Management System Software
byMustafa Kuğu
Ā 
How does MES(Manufacturing Execution System) work?
byakipii ogaoga
Ā 
Analyze and Preserve Logs - RHCSA (RH134).pdf
byLinuxCert Guru
Ā 
AI for Risk Management & Fraud Detection ppt.pdf
byalexmartincaneda
Ā 
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
Ā 

Map reduce and the art of Thinking Parallel - Dr. Shailesh Kumar

  • 1.
    MapReduce and theā€Ø art of ā€œThinking Parallelā€ Shailesh Kumar Third Leap, Inc.
  • 2.
    Three Iā€™s ofa great product! Interface Intuitive |Functional | Elegant Infrastructur e Storage |Computation | Network Intelligence Learn |Predict | Adapt | Evolve
  • 3.
    Drowning in Data,Starving for Knowledge ATATTAGGTTTTTACCTACCC AGGAAAAGCCAACCAACCTC GATCTCTTGTAGATCTGTTCT CTAAACGAACTTTAAAATCTG TGTAGCTGTCGCTCGGCTG CATGCCTAGTGCACCTACGC AGTATAAACAATAATAAATTTT ACTGTCGTTGACAAGAAACG AGTAACTCGTCCCTCTTCTG CAGACTGCTTATTACGCGAC CGTAAGCTACā€¦
  • 4.
    How BIG isBig Data? 600 million tweets per DAY 100 hours per MINUTE 800+ websites per MINUTE 100 TB of data uploaded DAILY 3.5 Billion queries PER DAY 300 Million Active customers How BIG is BigData?
  • 5.
    ā–Ŗ Better Sensors ā–ŖHigher resolution, Real-time, Diverse measurements, ā€¦ ā–Ŗ Faster Communication ā–Ŗ Network infrastructure, Compression Technologies, ā€¦ ā–Ŗ Cheaper Storage ā–Ŗ Cloud based storage, large warehouses, NoSQL databases ā–Ŗ Massive Computation ā–Ŗ Cloud computing, Mapreduce/Hadoop parallel processing paradigms ā–Ŗ Intelligent Decisions ā–Ŗ Advances in Machine Learning and Artificial Intelligence How did we get here?
  • 6.
    The Evolution ofā€œComputingā€
  • 7.
    Parallel Computing Basics ā–ŖData Parallelism (distributed computing) ā–Ŗ Lots of data ! Break it into ā€œchunksā€, ā–Ŗ Process each ā€œchunkā€ of data in parallel, ā–Ŗ Combine results from each ā€œchunkā€ ā–Ŗ MAPREDUCE = Data Parallelism ā–Ŗ Process Parallelism (data flow computing) ā–Ŗ Lots of stages ! Set up process graph ā–Ŗ Pass data through all stages ā–Ŗ All stages running in parallel on different data ā–Ŗ Assembly line = process parallelism
  • 8.
    Agenda MAPREDUCE Background Problem 1ā€“ Similarity between all pairs of documents! Problem 2 ā€“ Parallelizing K-Means clustering Problem 3 ā€“ Finding all Maximal Cliques in a Graph
  • 9.
    MAPREDUCE 101: A4-stage ProcessLotsofdata Shard 1 Shard N Shard 2 Reduce 1 Reduce R Map 1 Map 2 Map K Combine 1 Combine 2 Combine K Shuffle 1 Shuffle 2 Shuffle K Output 1 Output R Each Map processes N/K shards
  • 10.
    MAPREDUCE 101: Anexample Task ā–Ŗ Count total frequency of all words on the web ā–Ŗ Total number of documents > 20Billion ā–Ŗ Total number of unique words > 20Million ā–Ŗ Non-Parallel / Linear Implementation for each document d on the Web for each unique word w in d DocCount w d( )= # times w occurred in d WebCount w( ) += DocCount w d( )
  • 11.
    MAPREDUCE ā€“ MAP/COMBINE Shard1 KeyValue A 10 B 7 C 9 D 3 B 4 Key Value A 10 B 11 C 9 D 3 Shard2 Key Value A 3 D 1 C 4 D 9 B 6 Key Value A 3 B 6 C 4 D 10 Shard3 Key Value B 3 D 5 C 4 A 6 A 3 Map-1 Map-2 Map-3 Key Value A 9 B 3 C 4 D 5 Combine-1 Combine-2 Combine-3
  • 12.
    MAPREDUCE ā€“ Shuffle/Reduce KeyValue A 10 B 11 C 9 D 3 Key Value A 3 B 6 C 4 D 10 Key Value A 9 B 3 C 4 D 5 Key Value A 10 A 3 A 9 C 9 C 4 C 4 Key Value B 11 B 6 B 3 D 3 D 10 D 5 Shuffle 1 Shuffle 2 Shuffle 3 Key Value A 22 C 17 Key Value B 20 D 18 Reduce 1 Reduce 2
  • 13.
    Key Questions inMAPREDUCE ā–Ŗ Is the task really ā€œdata-parallelizableā€? ā–Ŗ High dependence tasks (e.g. Fibonacci series) ā–Ŗ Recursive tasks (e.g. Binary Search) ā–Ŗ What is the key-value pair output for MAP step? ā–Ŗ Each map processes only one data record at a time ā–Ŗ It can generate none, one, or multiple key-value pairs ā–Ŗ How to combine values of a key in REDUCE step? ā–Ŗ The key for reduce is same as key for Map output ā–Ŗ The reduce function must be ā€œorder agnosticā€
  • 14.
    Other considerations ā–Ŗ Reliability/Robustness ā–ŖA processor or disk might go bad during the process ā–Ŗ Optimization/Efficiency ā–Ŗ Allocate CPUā€™s near data shards to reduce network overhead ā–Ŗ Scale/Parallelism ā–Ŗ Parallelization linearly proportional to number of machines ā–Ŗ Simplicity/Usability ā–Ŗ Just specify the Map task and the Reduce task and be done! ā–Ŗ Generality ā–Ŗ Lots of parallelizable tasks can be written in MapReduce ā–Ŗ With some creativity, many more than you can imagine!
  • 15.
    Agenda MAPREDUCE Background Problem 1ā€“ Similarity between all pairs of documents! Problem 2 ā€“ Parallelizing K-Means clustering Problem 3 ā€“ Finding all Maximal Cliques in a Graph
  • 16.
    Similarity between allpairs of docs. ā–Ŗ Why bother? ā–Ŗ Document Clustering, Similar document search, etc. ā–Ŗ Document represented as a ā€œBag-of-Tokensā€ ā–Ŗ A weight associated with each tokens in vocabulary. ā–Ŗ Most weights are zero ā€“ Sparsity ā–Ŗ Cosine Similarity between two documents di = w1 i ,w2 i ,...,wT i { }, dj = w1 j ,w2 j ,...,wT j { } Sim di ,dj( )= wt i t=1 T āˆ‘ Ɨ wt j
  • 17.
    Non-Parallel / LinearImplementation For each document di For each document dj ( j > i) Sim di ,dj( )= wt i t=1 T āˆ‘ Ɨ wt j Complexity = O D2 TĻƒ( ) Ļƒ = Sparsity factor =10āˆ’5 = Average Fraction of vocabulary per document D = O(10B), T = O(10M ) Complexity = O 1020+7āˆ’5 ( )= O 1022 ( )
  • 18.
    Toy Example fordoc-doc similarity A classic ā€œJoinā€ Documents = W, X,Y, Z{ }, Words = a,b,c,d,e{ } W ā†’ a,1 , b,2 , e,5{ } X ā†’ a,3 , c,4 , d,5{ } Y ā†’ b,6 , c,7 , d,8{ } Z ā†’ a,9 , e,10{ } Input W, X ā†’ Sim W, X( )= 3 W,Y ā†’ Sim W,Y( )= 12 W,Z ā†’ Sim W,Z( )= 59 X,Y ā†’ Sim X,Y( )= 68 X,Z ā†’ Sim X,Z( )= 27 Y,Z ā†’ Sim Y,Z( )= 0 Output
  • 19.
    Reverse Indexing tothe rescue First convert the data to reverse index aā†’ W,1 , X,3 , Z,9{ } bā†’ W,2 , Y,6{ } cā†’ X,4 , Y,7{ } d ā†’ X,5 , Y,8{ } eā†’ W,5 , Z,10{ } W ā†’ a,1 , b,2 , e,5{ } X ā†’ a,3 , c,4 , d,5{ } Y ā†’ b,6 , c,7 , d,8{ } Z ā†’ a,9 , e,10{ }
  • 20.
    Key/Value for theMAP-Step aā†’ W,1 , X,3 , Z,9{ } W, X ā†’ 3 W,Z ā†’ 9 X,Z ā†’ 27 bā†’ W,2 , Y,6{ } cā†’ X,4 , Y,7{ } W,Y ā†’12 eā†’ W,5 , Z,10{ } d ā†’ X,5 , Y,8{ } X,Y ā†’ 28 X,Y ā†’ 40 W,Z ā†’ 50 W, X ā†’ 3 W,Y ā†’12 W,Z ā†’ 9 W,Z ā†’ 50 X,Y ā†’ 40 X,Y ā†’ 28 X,Z ā†’ 27
  • 21.
    Value combining inREDUCE-Step W, X ā†’ 3 W,Y ā†’12 W,Z ā†’ 9 W,Z ā†’ 50 X,Y ā†’ 40 X,Y ā†’ 28 X,Z ā†’ 27 W, X ā†’ Sim W, X( )= 3 W,Y ā†’ Sim W,Y( )= 12 W,Z ā†’ Sim W,Z( )= 59 X,Y ā†’ Sim X,Y( )= 68 X,Z ā†’ Sim X,Z( )= 27 Y,Z ā†’ Sim Y,Z( )= 0
  • 22.
    Agenda MAPREDUCE Background Problem 1ā€“ Similarity between all pairs of documents! Problem 2 ā€“ Parallelizing K-Means clustering Problem 3 ā€“ Finding all Maximal Cliques in a Graph
  • 23.
    assignments ! centers K-MeansClustering mk (t+1) ā† Ī“n,k (t ) xn n=1 N āˆ‘ Ī“n,k (t) n=1 N āˆ‘ m1 (t+1) m2 (t+1) Ī“n,2 (t ) = 1 Ī“n,1 (t ) = 1 m1 (t ) m2 (t ) centers ! assignments Ī“n,k (t+1) = k == arg min j=1...K Ī” x n( ) ,mj (t) ( ){ }( )
  • 24.
    K-means clustering 101ā€“ Non-parallel E-Step ā€“ Update assignments from centers ā€Ø M-Step ā€“ Update centers from cluster assignments Ļ€n (t) ā† arg min k=1...K Ī” xn ,mk (t) ( ){ } mk (t+1) ā† Ī“ Ļ€n (t) = k( )xn n=1 N āˆ‘ Ī“ Ļ€n (t) = k( ) n=1 N āˆ‘ O NKD( ): N = Number of data points K = Number of clusters D = number of dimensions āŽ§ āŽØ āŽŖ āŽ© āŽŖ O ND( ): N = Number of data points D = number of dimensions āŽ§ āŽØ āŽ©
  • 25.
    K-Means MapReduce mk (t) { }k=1 K Key= Ļ€n (t) ā†’ Value = xn Ļ€n (t) = arg min k=1...K Ī” xn ,mk (t) ( ){ } mk (t+1) ā† Ī“ Ļ€n (t) = k( )xn n=1 N āˆ‘ Ī“ Ļ€n (t) = k( ) n=1 N āˆ‘ mk (t+1) { }k=1 K Ļ€n (t) mk (t+1) Map Shuffle Reduce Iterative MapReduce: Update Cluster Centers/iteration
  • 26.
    Agenda MAPREDUCE Background Problem 1ā€“ Similarity between all pairs of documents! Problem 2 ā€“ Parallelizing K-Means clustering Problem 3 ā€“ Finding all Maximal Cliques in a Graph
  • 27.
    Cliques: Useful structuresin Graphs ā€¢ People ā€¢ Products ā€¢ Movies ā€¢ Keywords ā€¢ Documents ā€¢ Genes ā€¢ Neurons ā€¢ Co-Social ā€¢ Co-purchase ā€¢ Co-like ā€¢ Co-occurrence ā€¢ Similarity ā€¢ Co-expressions ā€¢ Co-firing
  • 28.
  • 29.
    Graph, Cliques, andMaximal Cliques Clique = a ā€œfully connectedā€ sub-graph Maximal Clique = a clique with no ā€œSuper-cliqueā€ Finding all Maximal Cliques is NP-hard: O(3n/3) a e b f c g d h
  • 30.
    Neighborhood of aClique a e b f c g d h f is connected to BOTH b and c g is connected to BOTH b and c N({b,c}) = {f,g} CLIQUEMAP: Clique (key) ! Its Neighbor (value) {a} ā†’ {b,e} {a,b} ā†’ {e} {b,c} ā†’ { f,g} {b,c, f } ā†’ {g} {h} ā†’ āˆ… {c,d} ā†’ āˆ… {a,b,e} ā†’ āˆ… {b,c, f ,g} ā†’ āˆ…
  • 31.
    Growing Cliques fromCliqueMap {b,c, f} ā†’ {g} a e b f c g d h {b,c, f} is a clique g is connected to all of them āŽ« āŽ¬ āŽ­ ā‡’ {b,c, f,g} is a clique
  • 32.
    MapReduce for MaximalCliques CliqueMap of size k ! size k + 1 {a,b} ā†’ {e} {a,e} ā†’ {b} {b,c} ā†’ { f,g} {b,e} ā†’ {a} {b, f } ā†’ {c,g} {b,g} ā†’ {c, f } {c, f } ā†’ {b,g} {c, g} ā†’ {b, f } { f, g} ā†’ {b,c} {c,d} ā†’ āˆ… Iteration 2 {a,b,e} ā†’ āˆ… {b,c, f } ā†’ {g} {b,c,g} ā†’ { f } {b, f ,g} ā†’ {c} {c, f ,g} ā†’ {b} Iteration 3 {b,c, f,g} ā†’ āˆ… Iteration 4 {a} ā†’ {b,e} {b} ā†’ {a,c,e, f ,g} {c} ā†’ {b,d, f ,g} {d} ā†’ {c} {e} ā†’ {a,b} { f } ā†’ {b,c,g} {g} ā†’ {b,c, f } {h} ā†’ āˆ… Iteration 1 Input: Adjacency List a e b f c g d h
  • 33.
    Key/Value for theMAP-Step a e b f c g d h {a} ā†’ {b,e} {a,b} ā‡’ {e} {a,e} ā‡’ {b} {e} ā†’ {a,b} {b} ā†’ {a,c,e, f,g} {a,e} ā‡’ {b} {b,e} ā‡’ {a} {a,b} ā‡’ {c,e, f, g} {b,c} ā‡’ {a,e, f, g} {b,e} ā‡’ {a,c, f, g} {b, f } ā‡’ {a,c,e, g} {b,g} ā‡’ {a,c,e, f } {a,e} ā‡’ {b} {a,e} ā‡’ {b} {a,b} ā‡’ {e} {a,b} ā‡’ {c,e, f ,g} {b,e} ā‡’ {a.c, f,g} {b,e} ā‡’ {a} SHUFFLE MAP
  • 34.
    Value combining inREDUCE-Step a e b f c g d h {a,e} ā‡’ {b} {a,e} ā‡’ {b} {a,b} ā‡’ {e} {a,b} ā‡’ {c,e, f ,g} {b,e} ā‡’ {a,c, f ,g} {b,e} ā‡’ {a} SHUFFLE {a,b} ā†’ {e}āˆ©{c,e, f,g} = {e} {b,e} ā†’ {a,c, f,g}āˆ©{a} = {a} {a,e} ā†’ {b}āˆ©{b} = {b} REDUCE Reduce = Intersection
  • 35.
    Value combining inREDUCE-Step a e b f c g d h c,d{ }ā‡’ b, f ,g{ } c,d{ }ā‡’ āˆ… c{ }ā†’ b,d, f,g{ } d{ }ā†’ c{ } b,c{ }ā‡’ a,e, f,g{ } b,c{ }ā‡’ d, f,g{ } b{ }ā†’ a,c,e, f,g{ } c{ }ā†’ b,d, f ,g{ } c,d{ }ā†’ {b, f ,g}āˆ©āˆ… = āˆ… b,c{ }ā†’ a,e, f ,g{ }āˆ© d, f ,g{ } = f ,g{ }
  • 36.
    ā€œArt of ThinkingParallelā€ is about ā–Ŗ Transforming the Input Data appropriately ā–Ŗ e.g. Reverse Indexing (doc-doc similarity) ā–Ŗ Breaking the problem into smaller ones ā–Ŗ e.g. Iterative MapReduce (clustering) ā–Ŗ Designing the Map step - Key/Value output ā–Ŗ e.g. CliqueMaps in Maximal Cliques ā–Ŗ Design the Reduce step ā€“ Combine values of key ā–Ŗ e.g. Intersections in Maximal Cliques