Uploaded bycwensel
KEY, PDF620 views

Buzz words

The document summarizes common patterns for processing large datasets using MapReduce. It describes how MapReduce works by applying map and reduce functions to key-value pairs in parallel. Common patterns discussed include filtering, parsing, counting, merging, binning, distributed tasks, grouping, finding unique values, secondary sorting, and joining datasets. Real-world applications are described as chaining many MapReduce jobs together to process large amounts of data.

Embed presentation

Downloaded 66 times
Common MapReduce Patterns Chris K Wensel BuzzWords 2011
Engineer, Not Academic • Concurrent, Inc., Founder • Cascading support and tools • http://concurrentinc.com/ • Cascading, Lead Developer (started Sept 2007) • An alternative API to MapReduce • http://cascading.org/ • Formerly Hadoop mentoring and training • Sun - Apple - HP - LexisNexis - startups - etc • Formerly Systems Architect & Consultant • Thomson/Reuters - TeleAtlas - startups - etc Copyright Concurrent, Inc. 2011. All rights reserved.
Overview • MapReduce • Heavy Lifting • Analytics • Optimizations Copyright Concurrent, Inc. 2011. All rights reserved.
MapReduce • A “divide and conquer” strategy for parallelizing workloads against collections of data • Map & Reduce are two user defined functions chained via Key Value Pairs • It’s really Map->Group->Reduce where Group is built in Copyright Concurrent, Inc. 2011. All rights reserved.
Keys and Values • Map translates input to keys and values to new keys and values [K1,V1] Map [K2,V2]* • System Groups each unique [K2,V2] Group [K2,{V2,V2,....}] key with all its values [K2,{V2,V2,....}] Reduce [K3,V3]* • Reduce translates the values of each unique key to new keys and values * = zero or more Copyright Concurrent, Inc. 2011. All rights reserved.
Word Count Mapper [0, "when in the course of human events"] Map ["when",1] ["in",1] ["the",1] [...,1] ["when",1] ["when",1] ["when",1] ["when",1] Group ["when",{1,1,1,1,1}] ["when",1] Reducer ["when",{1,1,1,1,1}] Reduce ["when",5] Copyright Concurrent, Inc. 2011. All rights reserved.
Divide and Conquer Parallelism • Since the ‘records’ entering the Map and ‘groups’ entering the Reduce are independent • That is, there is no expectation of order or requirement to share state between records/ groups • Arbitrary numbers of Map and Reduce function instances can be created against arbitrary portions of input data Copyright Concurrent, Inc. 2011. All rights reserved.
Cluster Cluster Rack Rack Rack Node Node Node Node ... map map map map map reduce reduce reduce • Multiple instances of each Map and Reduce function are distributed throughout the cluster Copyright Concurrent, Inc. 2011. All rights reserved.
Another View [K1,V1] Map [K2,V2] Combine Group [K2,{V2,...}] Reduce [K3,V3] Mapper Task same code Mapper Reducer Shuffle Task Task Mapper Reducer Shuffle Task Task Mapper Reducer Shuffle Task Task Mapper Task Mappers must complete before Reducers can begin split1 split2 split3 split4 ... part-00000 part-00001 part-000N file directory Copyright Concurrent, Inc. 2011. All rights reserved.
Complex job assemblies • Real applications are many MapReduce jobs chained together • Linked by intermediate (usually temporary) files • Executed in order, by hand, from the ‘client’ application Count Job Sort Job [ k, [v] ] [ k, [v] ] Map Reduce Map Reduce [ k, v ] [ k, v ] [ k, v ] [ k, v ] File File File [ k, v ] = key and value pair [ k, [v] ] = key and associated values collection Copyright Concurrent, Inc. 2011. All rights reserved.
Real World Apps [37/75] map+reduce [54/75] map+reduce [41/75] map+reduce [43/75] map+reduce [42/75] map+reduce [45/75] map+reduce [44/75] map+reduce [39/75] map+reduce [36/75] map+reduce [46/75] map+reduce [40/75] map+reduce [50/75] map+reduce [38/75] map+reduce [49/75] map+reduce [51/75] map+reduce [47/75] map+reduce [52/75] map+reduce [53/75] map+reduce [48/75] map+reduce [23/75] map+reduce [25/75] map+reduce [24/75] map+reduce [27/75] map+reduce [26/75] map+reduce [21/75] map+reduce [19/75] map+reduce [28/75] map+reduce [22/75] map+reduce [32/75] map+reduce [20/75] map+reduce [31/75] map+reduce [33/75] map+reduce [29/75] map+reduce [34/75] map+reduce [35/75] map+reduce [30/75] map+reduce [7/75] map+reduce [2/75] map+reduce [8/75] map+reduce [10/75] map+reduce [9/75] map+reduce [5/75] map+reduce [3/75] map+reduce [11/75] map+reduce [6/75] map+reduce [13/75] map+reduce [4/75] map+reduce [16/75] map+reduce [14/75] map+reduce [15/75] map+reduce [17/75] map+reduce [18/75] map+reduce [12/75] map+reduce [60/75] map [62/75] map [61/75] map [58/75] map [55/75] map [56/75] map+reduce [57/75] map [71/75] map [72/75] map [59/75] map [64/75] map+reduce [63/75] map+reduce [65/75] map+reduce [68/75] map+reduce [67/75] map+reduce [70/75] map+reduce [69/75] map+reduce [73/75] map+reduce [66/75] map+reduce [74/75] map+reduce [75/75] map+reduce [1/75] map+reduce 1 app, 75 jobs green = map + reduce purple = map blue = join/merge orange = map split Copyright Concurrent, Inc. 2011. All rights reserved.
Heavy Lifting • Thing we must do because data can be heavy • These patterns are natural to MapReduce and easy to implement • But have some room for composition/aggregation within a Map/ Reduce (i.e., Filter + Binning) • (leading us to think of Hadoop as an ETL framework) • Record Filtering • Parsing, Conversion • Binning • Counting, Summing • Distributed Tasks • Unique Copyright Concurrent, Inc. 2011. All rights reserved.
Record Filtering • Think unix ‘grep’ • Filtering is discarding unwanted values (or preserving wanted) • Only uses a Map function, no Reducer Copyright Concurrent, Inc. 2011. All rights reserved.
Parsing, Conversion • Think unix ‘sed’ • A Map function that takes an input key and/or value and translates it into a new format • Examples: • raw logs to delimited text or archival efficient binary • entity extraction Copyright Concurrent, Inc. 2011. All rights reserved.
Counting, Summing • The same as SQL aggregation functions • Simply applying some function to the values collection seen in Reduce • Other examples: • average, max, min, unique Copyright Concurrent, Inc. 2011. All rights reserved.
Merging • Where many files of the same type are converted to one output path • Map side merges • One directory with as many part files as Mappers • Reduce side merges • Allows for removing duplicates or deleted items • One directory with as many part files as Reducers • Examples • Nutch • Normalizing log files (apache, log4j, etc) Copyright Concurrent, Inc. 2011. All rights reserved.
Binning • Where the values associated w/ unique keys are persisted together • Typically a directory path based on key’s value • Must be conscious of total open files, remember no appends • Examples: • web log files by year/month/day • trade data by symbol Copyright Concurrent, Inc. 2011. All rights reserved.
Distributed Tasks • Simply where a Map or Reduce function executes some ‘task’ based on the input key and value. • Examples: • web crawling, • load testing services, • rdbms/nosql updates, • file transfers (S3), • image to pdf (NYT on EC2) Copyright Concurrent, Inc. 2011. All rights reserved.
Basic Analytic Patterns • Some of these patterns are unnatural to MapReduce • We think in terms of columns/fields, not key value pairs • (leading us to think of Hadoop as a RDBMS) • Group By • Secondary Unique • Unique • CoGrouping and Joining • Secondary Sort Copyright Concurrent, Inc. 2011. All rights reserved.
Composite Keys/Values [K1,V1] <A1,B1,C1,...> • It is easier to think in columns/fields • e.g. “firstname” & “lastname”, not “line” • Whether a set of columns are Keys or Values is arbitrary • Keys become a means to piggyback the properties of MR and become an impl detail Copyright Concurrent, Inc. 2011. All rights reserved.
Group By GroupBy 1001 Jim dept_id Mary name Susan 1002 Fred Wilma Ernie Barny • Group By is where Value fields are grouped by Grouping fields • Above, Map output key is “dept_id” and value is “name” Copyright Concurrent, Inc. 2011. All rights reserved.
Group By Mapper Reducer Piggyback Code [K1,V1] [K2,{V2,V2,....}] [K1,V1] -> <A1,B1,C1,D1> [K2,V2] -> <A2,B2,{<C2,D2>,...}> User Code Map Reduce <A2,B2> -> K2, <C2,D2> -> V2 <A3,B3> -> K3, <C3,D3> -> V3 [K2,V2] [K3,V3] • So the K2 key becomes a composite Key of • key: [grouping], value: [values] Copyright Concurrent, Inc. 2011. All rights reserved.
Unique Mapper [0, "when in the course of human events"] Map ["when",null] ["in",null] [...,null] ["when",1] ["when",1] ["when",1] ["when",1] Group ["when",{nulls}] ["when",null] Reducer ["when",{nulls}] Reduce ["when",null] • Or Distinct (as in SQL) • Globally finding all the unique values in a dataset • Usually finding unique values in a column • Often used to filter a second dataset using a join Copyright Concurrent, Inc. 2011. All rights reserved.
Secondary Sort (group) (sorted value) (remaining value) Date Time Url 08/08/2008, 1:00:00, http://www.example.com/foo 08/08/2008, 1:01:00, http://www.example.com/bar 08/08/2008, 1:01:30, http://www.example.com/baz • Secondary Sorting is where • Some Fields are grouped on, and • Some of the remaining Fields are sorted within their grouping Copyright Concurrent, Inc. 2011. All rights reserved.
Secondary Sort Mapper Reducer [K1,V1] [K2,{V2,V2,....}] [K1,V1] -> <A1,B1,C1,D1> [K2,V2] -> <A2,B2,{<C2,D2>,...}> Map Reduce <A2,B2><C2> -> K2, <D2> -> V2 <A3,B3> -> K3, <C3,D3> -> V3 [K2,V2] [K3,V3] • So the K2 key becomes a composite Key of • key: [grouping, secondary], value: [remaining values] • The trick is to piggyback the Reduce sort yet not be compared during the unique key comparison Copyright Concurrent, Inc. 2011. All rights reserved.
Secondary Unique Mapper Assume Secondary Sorting magic happens here [0, "when in the course of human events"] Map [0, "when"] [0, "in"] [0,"the"] [0,...] ["when",1] ["when",1] ["when",1] ["when",1] Group [0,{"in","in","the","when","when",...}] [0,"when"] Reducer [0,{"in","in","the","when","when",...}] Reduce ["in",null] ["the",null] ["when",null] • Secondary Unique is where the grouping values are uniqued • .... in a “scale free” way • Perform a Secondary Sort... • Reducer removes duplicates by discarding every value that matches the previous value • since values are now ordered, no need to maintain a Set of values Copyright Concurrent, Inc. 2011. All rights reserved.
Joining lhs data rhs data 1001 dept_id Jim Accounting dept_name Mary Accounting name Susan Accounting 1002 Fred Shipping Wilma Shipping Ernie Shipping Barny Shipping • Where two or more input data sets are ‘joined’ by a common key • Like a SQL join Copyright Concurrent, Inc. 2011. All rights reserved.
Join Definitions • Consider the input data [key, value]: • LHS = [0,a] [1,b] [2,c] • RHS = [0,A] [2,C] [3,D] • Joins on the key: • Inner • [0,a,A] [2,c,C] • Outer (Left Outer, Right Outer) • [0,a,A] [1,b,null] [2,c,C] [3,null,D] • Left (Left Inner, Right Outer) • [0,a,A] [1,b,null] [2,c,C] • Right (Left Outer, Right Inner) • [0,a,A] [2,c,C] [3,null,D] Copyright Concurrent, Inc. 2011. All rights reserved.
CoGrouping • Before Joining, CoGrouping must happen • Simply concurrent GroupBy operations on each input data set Copyright Concurrent, Inc. 2011. All rights reserved.
GroupBy vs CoGroup lhs data rhs data GroupBy CoGroup 1001 1001 Jim Jim Accounting dept_id Mary Mary name Susan Susan dept_name 1002 1002 Fred Fred Shipping Wilma Wilma Ernie Ernie Barny Barny Independent collections of unordered values Copyright Concurrent, Inc. 2011. All rights reserved.
CoGroup Joined lhs data rhs data 1001 dept_id Jim Accounting dept_name Mary Accounting name Susan Accounting 1002 Fred Shipping Wilma Shipping Ernie Shipping Barny Shipping • Considering the previous data, a typical Inner Join Copyright Concurrent, Inc. 2011. All rights reserved.
CoGrouping Mapper [n] [n+1] Reducer [K1,V1] [K1',V1'] [K2,{V2,V2,....}] [K2,V2] -> <A2,B2,{<C2,D2,C2',D2'>,...}> [K1,V1] -> <A1,B1,C1,D1> [K1',V1'] -> <A1',B1',C1',D1'> Reduce Map <A3,B3> -> K3, <C3,D3> -> V3 <A2,B2> -> K2, [n]<C2,D2> -> V2 [K2,V2] [K3,V3] • Maps must run for each input set in same Job (n, n+1, etc) • CoGrouping must happen against each common key Copyright Concurrent, Inc. 2011. All rights reserved.
Joining Reducer [K2,{V2,V2,....}] <A2,B2,{[n]<C2,D2>,[n+1]..}> [K2,V2] -> <A2,B2,{<C2,D2,C2',D2'>,...}> <A2,B2,{<C2,D2>,...},{<C2',D2'>,...}> Reduce {<C2,D2>,...} Join {<C2',D2'>,...} <A3,B3> -> K3, <C3,D3> -> V3 <C2,D2,C2',D2'> [K3,V3] <A2,B2,{<C2,D2,C2',D2'>,...}> • The CoGroups must be joined • Finally the Reduce can be applied Copyright Concurrent, Inc. 2011. All rights reserved.
Optimizations • Patterns for reducing IO • Identity Mapper • Partial Aggregates • Map Side Join • Similarity Joins • Combiners Copyright Concurrent, Inc. 2011. All rights reserved.
Identity Mapper [head] Dfs['TextLine[['offset', 'line']->[ALL]]']['/logs/stumbles/short-stumbles-20090504.log']'] [{2}:'offset', 'line'] [{2}:'offset', 'line'] Each('import')[RegexParser[decl:'day', 'urlid', 'method'][args:1]] [{3}:'day', 'urlid', 'method'] [{3}:'day', 'urlid', 'method'] Each('import')[ExpressionFilter[decl:'day', 'urlid', 'method']] [{3}:'day', 'urlid', 'method'] [{3}:'day', 'urlid', 'method'] TempHfs['SequenceFile[['day', 'urlid', 'method']]'][import/71897/] [{3}:'day', 'urlid', 'method'] [{3}:'day', 'urlid', 'method'] [{3}:'day', 'urlid', 'method'] identity [{3}:'day', 'urlid', 'method'] Each('paidCount')[Not[decl:'day', 'urlid', 'method']] function [{3}:'day', 'urlid', 'method'] [{3}:'day', 'urlid', 'method'] Each('organicCount')[OrganicFilter[decl:'day', 'urlid', 'method']] [{3}:'day', 'urlid', 'method'] GroupBy('paidCount')[by:['day', 'urlid']] [{3}:'day', 'urlid', 'method'] paidCount[{2}:'day', 'urlid'] GroupBy('organicCount')[by:['day', 'urlid']] [{3}:'day', 'urlid', 'method'] organicCount[{2}:'day', 'urlid'] Every('paidCount')[Count[decl:'count']] [{3}:'day', 'urlid', 'method'] [{3}:'day', 'urlid', 'count'] Every('organicCount')[Count[decl:'count']] • [{3}:'day', 'urlid', 'method'] Move Map operations to the Each('paidCount')[Identity[decl:'paid_day', 'paid_urlid', 'paid_count']] [{3}:'day', 'urlid', 'count'] [{3}:'day', 'urlid', 'method'] [{3}:'paid_day', 'paid_urlid', 'paid_count'] [{3}:'paid_day', 'paid_urlid', 'paid_count'] TempHfs['SequenceFile[['paid_day', 'paid_urlid', 'paid_count']]'][paidCount/33072/] TempHfs['SequenceFile[['day', 'urlid', 'count']]'][organicCount/97544/] [{3}:'paid_day', 'paid_urlid', 'paid_count'] [{3}:'paid_day', 'paid_urlid', 'paid_count'] previous Reduce [{3}:'paid_day', 'paid_urlid', 'paid_count'] [{3}:'day', 'urlid', 'count'] [{3}:'paid_day', 'paid_urlid', 'paid_count'] [{3}:'day', 'urlid', 'count'] [{3}:'day', 'urlid', 'count'] [{3}:'day', 'urlid', 'count'] CoGroup('organicCount*paidCount')[by:organicCount:['day', 'urlid']paidCount:['paid_day', 'paid_urlid']] Each('paidDomainCount')[LookupDomainFunction[decl:'domainid'][args:1]] organicCount[{2}:'day', 'urlid'],paidCount[{2}:'paid_day', 'paid_urlid'] [{4}:'paid_day', 'paid_urlid', 'paid_count', 'domainid'] Each('organicDomainCount')[LookupDomainFunction[decl:'domainid'][args:1]] [{6}:'day', 'urlid', 'count', 'paid_day', 'paid_urlid', 'paid_count'] [{4}:'paid_day', 'paid_urlid', 'paid_count', 'domainid'] [{4}:'day', 'urlid', 'count', 'domainid'] Each('organicCount*paidCount')[ExpressionFunction[decl:'urlid_day']] GroupBy('paidDomainCount')[by:['paid_day', 'domainid']] [{4}:'day', 'urlid', 'count', 'domainid'] • [{7}:'day', 'urlid', 'count', 'paid_day', 'paid_urlid', 'paid_count', 'urlid_day'] paidDomainCount[{2}:'paid_day', 'domainid'] GroupBy('organicDomainCount')[by:['day', 'domainid']] [{7}:'day', 'urlid', 'count', 'paid_day', 'paid_urlid', 'paid_count', 'urlid_day'] [{4}:'paid_day', 'paid_urlid', 'paid_count', 'domainid'] Replace with an Identity organicDomainCount[{2}:'day', 'domainid'] Each('organicCount*paidCount')[ExpressionFunction[decl:'fixed_count']] Every('paidDomainCount')[Sum[decl:'sum'][args:1]] [{4}:'day', 'urlid', 'count', 'domainid'] [{8}:'day', 'urlid', 'count', 'paid_day', 'paid_urlid', 'paid_count', 'urlid_day', 'fixed_count'] [{3}:'paid_day', 'domainid', 'sum'] Every('organicDomainCount')[Sum[decl:'sum'][args:1]] [{8}:'day', 'urlid', 'count', 'paid_day', 'paid_urlid', 'paid_count', 'urlid_day', 'fixed_count'] [{4}:'paid_day', 'paid_urlid', 'paid_count', 'domainid'] function [{3}:'day', 'domainid', 'sum'] Each('organicCount*paidCount')[ExpressionFunction[decl:'fixed_paid_count']] Each('paidDomainCount')[Identity[decl:'paid_day', 'paid_domainid', 'paid_sum']] [{4}:'day', 'urlid', 'count', 'domainid'] [{9}:'day', 'urlid', 'count', 'paid_day', 'paid_urlid', 'paid_count', 'urlid_day', 'fixed_count', 'fixed_paid_count'] [{3}:'paid_day', 'paid_domainid', 'paid_sum'] TempHfs['SequenceFile[['day', 'domainid', 'sum']]'][organicDomainCount/49784/] [{9}:'day', 'urlid', 'count', 'paid_day', 'paid_urlid', 'paid_count', 'urlid_day', 'fixed_count', 'fixed_paid_count'] [{3}:'paid_day', 'paid_domainid', 'paid_sum'] Each('organicCount*paidCount')[Identity[decl:'urlid_day', 'default:organic_stumbles', 'default:paid_stumbles']] TempHfs['SequenceFile[['paid_day', 'paid_domainid', 'paid_sum']]'][paidDomainCount/54349/] [{3}:'paid_day', 'paid_domainid', 'paid_sum'] [{3}:'day', 'domainid', 'sum'] • [{3}:'paid_day', 'paid_domainid', 'paid_sum'] [{3}:'day', 'domainid', 'sum'] [{3}:'urlid_day', 'default:organic_stumbles', 'default:paid_stumbles'] [{3}:'urlid_day', 'default:organic_stumbles', 'default:paid_stumbles'] CoGroup('organicDomainCount*paidDomainCount')[by:organicDomainCount:['day', 'domainid']paidDomainCount:['paid_day', 'paid_domainid']] Assumes Map operations cascading.hbase.HBaseTap@d9a475a0 organicDomainCount[{2}:'day', 'domainid'],paidDomainCount[{2}:'paid_day', 'paid_domainid'] [{6}:'day', 'domainid', 'sum', 'paid_day', 'paid_domainid', 'paid_sum'] Each('organicDomainCount*paidDomainCount')[ExpressionFunction[decl:'domainid_day']] reduce the data [{7}:'day', 'domainid', 'sum', 'paid_day', 'paid_domainid', 'paid_sum', 'domainid_day'] [{7}:'day', 'domainid', 'sum', 'paid_day', 'paid_domainid', 'paid_sum', 'domainid_day'] [{3}:'urlid_day', 'default:organic_stumbles', 'default:paid_stumbles'] Each('organicDomainCount*paidDomainCount')[ExpressionFunction[decl:'fixed_sum']] [{3}:'urlid_day', 'default:organic_stumbles', 'default:paid_stumbles'] [{8}:'day', 'domainid', 'sum', 'paid_day', 'paid_domainid', 'paid_sum', 'domainid_day', 'fixed_sum'] [{8}:'day', 'domainid', 'sum', 'paid_day', 'paid_domainid', 'paid_sum', 'domainid_day', 'fixed_sum'] Each('organicDomainCount*paidDomainCount')[ExpressionFunction[decl:'fixed_paid_sum']] [{9}:'day', 'domainid', 'sum', 'paid_day', 'paid_domainid', 'paid_sum', 'domainid_day', 'fixed_sum', 'fixed_paid_sum'] [{9}:'day', 'domainid', 'sum', 'paid_day', 'paid_domainid', 'paid_sum', 'domainid_day', 'fixed_sum', 'fixed_paid_sum'] Each('organicDomainCount*paidDomainCount')[Identity[decl:'domainid_day', 'default:organic_stumbles', 'default:paid_stumbles']] [{3}:'domainid_day', 'default:organic_stumbles', 'default:paid_stumbles'] [{3}:'domainid_day', 'default:organic_stumbles', 'default:paid_stumbles'] cascading.hbase.HBaseTap@3d07f00 [{3}:'domainid_day', 'default:organic_stumbles', 'default:paid_stumbles'] [{3}:'domainid_day', 'default:organic_stumbles', 'default:paid_stumbles'] [tail] Copyright Concurrent, Inc. 2011. All rights reserved.
Map Side Joins • Bypasses the (immediate) need for a Reducer • Symmetrical • Where LHS and RHS are of equivalent size • Requires data to be sorted on key • Asymmetrical • One side is small enough to fit in memory • Typically a hashtable lookup Copyright Concurrent, Inc. 2011. All rights reserved.
Combiners Mapper [0, "when in the course of human events"] Map ["when",1] ["in",1] ["the",1] [...,1] Combiner ["when",1] ["when",1] Group ["when",{1,1}] ["when",{1,1}] Reduce ["when",2] Same Implementation ["when",1] ["when",1] Group ["when",{2,1,2}] ["when",2] Reducer ["when",{2,1,2}] Reduce ["when",5] • Where Reduce runs Map side, and again Reduce side • Only works if Reduce is commutative and associative • Reduces bandwidth by trading CPU for IO • Serialization/deserialization during local sorting before combining Copyright Concurrent, Inc. 2011. All rights reserved.
Partial Aggregates Mapper [0, "when in the course of human events"] ["when",1] ["in",1] ["the",1] [...,1] Map Partial Provides an opportunity to ["when",1] ["when",1] ["when",2] promote the functionality of the next Map to this Reduce ["when",1] ["when",1] Group ["when",{2,1,2}] ["when",2] Reducer ["when",{2,1,2}] Reduce ["when",5] • Supports any aggregate type, while being composable with other aggregates • Reduces bandwidth by trading Memory for IO • Very important for a CPU constrained cluster • Use a bounded LRU to keep constant memory (requires tuning) Copyright Concurrent, Inc. 2011. All rights reserved.
Partial Aggregates [a,b,c,a,a,b] [a,b,c,a,a,b] partial unique partial unique [a,b,c,a,b] [a,b,c,a,b] [a,b,c,a,a,b] [a,b,c,a,a,b] partial unique partial unique [a,b,c,a,b] [a,b,c,a,b] LRU* {_,_} *cache size of 2 a -> {a,_} -> _ b -> {b,a} -> _ incoming discarded c -> {c,b} -> a value value a -> {a,c} -> b a -> {a,c} b -> {b,a} -> c • OK that dupes emit from a Mapper and across Mappers (or prev Reducers!) • Final aggregation happens in Reducer • Larger the cache, fewer dupes Copyright Concurrent, Inc. 2011. All rights reserved.
Tradeoffs • CPU for IO == fault tolerance • Memory for IO == performance Copyright Concurrent, Inc. 2011. All rights reserved.
Similarity Join • Compare all values LHS to values RHS to find duplicates (or similar values) • Naive approaches • Cross Join (all data through one reducer) • In-common features (very common features will bottleneck) Copyright Concurrent, Inc. 2011. All rights reserved.
Set-Similarity Joining • “Efficient Parallel Set-Similarity Joins Using MapReduce” - R Vernica, M Carey, C Li • Only compare candidate pairs • Candidates share uncommon features Copyright Concurrent, Inc. 2011. All rights reserved.
4 1 1 4 2 2 4 3 3 2 4 4 3: order by least frequent 1: records 1 discard common 1 2: count tokens 1 1 1 3 3 5: candidate pairs 3 4: uncommon features 6: final compare in common • 1 and 3 share uncommon features • thus are candidates for a full comparison Copyright Concurrent, Inc. 2011. All rights reserved.
Tokenize Count Job Map Reduce Map Reduce File File File Join Tokens/Counts Job File Map Reduce File Sort/Prefix Filter Job Map Reduce File Match two sets Self Join Job Map Reduce using prefix File filtering Unique Pairs Job Map Reduce File Join LHS Job Map Reduce File Join RHS / Match Job Map Reduce File Copyright Concurrent, Inc. 2011. All rights reserved.
Duality • Note the use of the previous patterns to route data to implement a more efficient algorithm Copyright Concurrent, Inc. 2011. All rights reserved.
Use a Higher Abstraction • Command Line • Multitool - CLI for parallel sed, grep & joins • API • Cascading - Java Query API and Planner • Plume - “approximate clone of FlumeJava” • Interactive Shell • Cascalog - Clojure+Cascading query language (API also) • Pig - A text Syntax • Hive - Syntax + Infrastructure - SQL “like” Copyright Concurrent, Inc. 2011. All rights reserved.
References • Set Similarity • http://www.slideshare.net/ydn/4-similarity-joinshadoopsummit2010 • http://asterix.ics.uci.edu/fuzzyjoin-mapreduce/ • MapReduce Text Processing • http://www.umiacs.umd.edu/~jimmylin/book.html • Plume/FlumeJava • http://portal.acm.org/citation.cfm?id=1806596.1806638 • http://github.com/tdunning/Plume/wiki Copyright Concurrent, Inc. 2011. All rights reserved.
I’m Hiring • Enterprise Java server and web client • Language design, compilers, and interpreters • No Hadoop experience required • More info • http://www.concurrentinc.com/careers/ Copyright Concurrent, Inc. 2011. All rights reserved.
Resources • Chris K Wensel •chris@wensel.net •@cwensel • Cascading & Cascalog •http://cascading.org •@cascading • Concurrent, Inc. •http://concurrentinc.com •@concurrent Copyright Concurrent, Inc. 2011. All rights reserved.
Appendix Copyright Concurrent, Inc. 2011. All rights reserved.
Simple Total Sorting • Where lines in a result file should be sorted • Must set number of reducers to 1 • Sorting in MR is local per Reduce, not global across Reducers Copyright Concurrent, Inc. 2011. All rights reserved.
Why Sorting Isn’t “Total” [aaa,aab,aac] Mapper aaa Mapper aac Reducer [aaa,zzx] aab Mapper Reducer [aac,zzz] zzx Mapper zzz Reducer [aab,zzy] zzy [zzx,zzy,zzz] Mapper • Keys emitted from Map are naturally sorted at a given Reducer • But are Partitioned to Reducers in a random way • Thus, only one Reducer can be used for a total sort Copyright Concurrent, Inc. 2011. All rights reserved.
Distributed Total Sort • To work, the Shuffling phase must be modified with: • Custom Partitioner to partition on the distribution of ordered Keys • Custom Comparator for comparing Key types • Strings work by default Copyright Concurrent, Inc. 2011. All rights reserved.
Distributed Total Sort - Details a ... z ar ... ax za ... zo ara ... ari axe ... axi zag ... zap zon ... zoo aran aria axis zone • Sample all K2 values and build balanced distribution for num reducers • Sample all input keys and divide into partitions • Write out boundaries of partitions • Supply Partitioner that looks up partition for current K2 value • Read boundaries into a Trie (pronounced ‘try’) data structure • Use appropriate Comparator for Key type Copyright Concurrent, Inc. 2011. All rights reserved.

More Related Content

PDF
Map reduce
byHyosung Jeon
 
PPTX
High Dynamic Range color grading and display in Frostbite
byElectronic Arts / DICE
 
PDF
Rendering AAA-Quality Characters of Project A1
byKi Hyunwoo
 
PPT
Paris Master Class 2011 - 05 Post-Processing Pipeline
byWolfgang Engel
 
PPTX
FrameGraph: Extensible Rendering Architecture in Frostbite
byElectronic Arts / DICE
 
PPTX
More Performance! Five Rendering Ideas From Battlefield 3 and Need For Speed:...
byColin Barré-Brisebois
 
PPTX
GTC 2014 - DirectX 11 Rendering and NVIDIA GameWorks in Batman: Arkham Origins
byColin Barré-Brisebois
 
PPT
Paris Master Class 2011 - 01 Deferred Lighting, MSAA
byWolfgang Engel
 
Map reduce
byHyosung Jeon
 
High Dynamic Range color grading and display in Frostbite
byElectronic Arts / DICE
 
Rendering AAA-Quality Characters of Project A1
byKi Hyunwoo
 
Paris Master Class 2011 - 05 Post-Processing Pipeline
byWolfgang Engel
 
FrameGraph: Extensible Rendering Architecture in Frostbite
byElectronic Arts / DICE
 
More Performance! Five Rendering Ideas From Battlefield 3 and Need For Speed:...
byColin Barré-Brisebois
 
GTC 2014 - DirectX 11 Rendering and NVIDIA GameWorks in Batman: Arkham Origins
byColin Barré-Brisebois
 
Paris Master Class 2011 - 01 Deferred Lighting, MSAA
byWolfgang Engel
 

What's hot

PDF
Epic_GDC2011_Samaritan
byMinGeun Park
 
PPT
Paris Master Class 2011 - 07 Dynamic Global Illumination
byWolfgang Engel
 
PPTX
Photogrammetry and Star Wars Battlefront
byElectronic Arts / DICE
 
PPTX
GDC 2014 - Deformable Snow Rendering in Batman: Arkham Origins
byColin Barré-Brisebois
 
PPTX
The Rendering Pipeline - Challenges & Next Steps
byrepii
 
PPTX
A new Post-Processing Pipeline
byWolfgang Engel
 
PPTX
Lighting you up in Battlefield 3
byElectronic Arts / DICE
 
PPTX
Around the World in 80 Shaders
bystevemcauley
 
PPT
Terrain Rendering in Frostbite using Procedural Shader Splatting (Siggraph 2007)
byrepii
 
PPTX
Calibrating Lighting and Materials in Far Cry 3
bystevemcauley
 
PPTX
Triangle Visibility buffer
byWolfgang Engel
 
PPT
Paris Master Class 2011 - 04 Shadow Maps
byWolfgang Engel
 
PPTX
A Real-time Radiosity Architecture
byElectronic Arts / DICE
 
PPTX
SPU-Based Deferred Shading in BATTLEFIELD 3 for Playstation 3
byElectronic Arts / DICE
 
PPT
Crysis Next-Gen Effects (GDC 2008)
byTiago Sousa
 
PPT
A Bit More Deferred Cry Engine3
byguest11b095
 
Epic_GDC2011_Samaritan
byMinGeun Park
 
Paris Master Class 2011 - 07 Dynamic Global Illumination
byWolfgang Engel
 
Photogrammetry and Star Wars Battlefront
byElectronic Arts / DICE
 
GDC 2014 - Deformable Snow Rendering in Batman: Arkham Origins
byColin Barré-Brisebois
 
The Rendering Pipeline - Challenges & Next Steps
byrepii
 
A new Post-Processing Pipeline
byWolfgang Engel
 
Lighting you up in Battlefield 3
byElectronic Arts / DICE
 
Around the World in 80 Shaders
bystevemcauley
 
Terrain Rendering in Frostbite using Procedural Shader Splatting (Siggraph 2007)
byrepii
 
Calibrating Lighting and Materials in Far Cry 3
bystevemcauley
 
Triangle Visibility buffer
byWolfgang Engel
 
Paris Master Class 2011 - 04 Shadow Maps
byWolfgang Engel
 
A Real-time Radiosity Architecture
byElectronic Arts / DICE
 
SPU-Based Deferred Shading in BATTLEFIELD 3 for Playstation 3
byElectronic Arts / DICE
 
Crysis Next-Gen Effects (GDC 2008)
byTiago Sousa
 
A Bit More Deferred Cry Engine3
byguest11b095
 

Viewers also liked

PPT
An Integrated Marketing Plan
byLinda Dacosta
 
PPTX
Illinois Birds2
bypchambers61604
 
PDF
SAM SIG: Hadoop architecture, MapReduce patterns, and best practices with Cas...
bycwensel
 
PDF
Digital Marketing Lecture 2015
byDr. Ute Hillmer (PhD)
 
KEY
BigDataCamp 2011
bycwensel
 
PDF
Social Media Lecture Summer 2011
byDr. Ute Hillmer (PhD)
 
PDF
Real Social Media Recruitment ROI
byMikeVangel
 
PDF
Hadoop Summit EU 2014
bycwensel
 
KEY
Processing Big Data
bycwensel
 
PPTX
Illinois Birds
bypchambers61604
 
PDF
Hadoop User Group EU 2014
bycwensel
 
PDF
2015 Title Pckg_HEART YOUR LADY PARTS 5k
bylizloden
 
PDF
Engaging the customer
byDr. Ute Hillmer (PhD)
 
PPTX
Dialog Marketing with Digital Media
byDr. Ute Hillmer (PhD)
 
PDF
Making the Quantum Leap: UPS Social Media Recruitment ROI 2012
byMikeVangel
 
PDF
Building Scale Free Applications with Hadoop and Cascading
bycwensel
 
PDF
Digital Marketing Lecture 2016
byDr. Ute Hillmer (PhD)
 
PPTX
Isra' wal mikraj
byZuraimi Ali
 
An Integrated Marketing Plan
byLinda Dacosta
 
Illinois Birds2
bypchambers61604
 
SAM SIG: Hadoop architecture, MapReduce patterns, and best practices with Cas...
bycwensel
 
Digital Marketing Lecture 2015
byDr. Ute Hillmer (PhD)
 
BigDataCamp 2011
bycwensel
 
Social Media Lecture Summer 2011
byDr. Ute Hillmer (PhD)
 
Real Social Media Recruitment ROI
byMikeVangel
 
Hadoop Summit EU 2014
bycwensel
 
Processing Big Data
bycwensel
 
Illinois Birds
bypchambers61604
 
Hadoop User Group EU 2014
bycwensel
 
2015 Title Pckg_HEART YOUR LADY PARTS 5k
bylizloden
 
Engaging the customer
byDr. Ute Hillmer (PhD)
 
Dialog Marketing with Digital Media
byDr. Ute Hillmer (PhD)
 
Making the Quantum Leap: UPS Social Media Recruitment ROI 2012
byMikeVangel
 
Building Scale Free Applications with Hadoop and Cascading
bycwensel
 
Digital Marketing Lecture 2016
byDr. Ute Hillmer (PhD)
 
Isra' wal mikraj
byZuraimi Ali
 

Similar to Buzz words

PDF
Lecture 2: Data-Intensive Computing for Text Analysis (Fall 2011)
byMatthew Lease
 
PDF
Geoff Rothman Presentation on Parallel Processing
byGeoff Rothman
 
PPTX
Kansas City Big Data: The Future Of Insights - Keynote: "Big Data Technologie...
bykcitp
 
PPTX
MapReduce Paradigm
byDilip Reddy
 
PPTX
MapReduce Paradigm
byDilip Reddy
 
PDF
Lecture 3: Data-Intensive Computing for Text Analysis (Fall 2011)
byMatthew Lease
 
PDF
Hadoop
byScott Leberknight
 
PDF
Distributed computing the Google way
byEduard Hildebrandt
 
PPTX
EMC2, Владимир Суворов
byEYevseyeva
 
PDF
Mapreduce Osdi04
byJyotirmoy Dey
 
PDF
Mapreduce - Simplified Data Processing on Large Clusters
byAbhishek Singh
 
PDF
Brief introduction on Hadoop,Dremel, Pig, FlumeJava and Cassandra
bySomnath Mazumdar
 
PDF
Mapreduce Pact06 Keynote
byJeff Hammerbacher
 
PDF
Map reduce
byxydii
 
PDF
Map reduceoriginalpaper mandatoryreading
bycoolmirza143
 
PPT
Cloud computing-with-map reduce-and-hadoop
byVeda Vyas
 
PPTX
An introduction to Hadoop for large scale data analysis
byAbhijit Sharma
 
PDF
Map Reduce
bySri Prasanna
 
PDF
big data analytics introduction chapter 2
byterala1
 
PDF
Large Scale Data Processing & Storage
byIlayaraja P
 
Lecture 2: Data-Intensive Computing for Text Analysis (Fall 2011)
byMatthew Lease
 
Geoff Rothman Presentation on Parallel Processing
byGeoff Rothman
 
Kansas City Big Data: The Future Of Insights - Keynote: "Big Data Technologie...
bykcitp
 
MapReduce Paradigm
byDilip Reddy
 
MapReduce Paradigm
byDilip Reddy
 
Lecture 3: Data-Intensive Computing for Text Analysis (Fall 2011)
byMatthew Lease
 
Hadoop
byScott Leberknight
 
Distributed computing the Google way
byEduard Hildebrandt
 
EMC2, Владимир Суворов
byEYevseyeva
 
Mapreduce Osdi04
byJyotirmoy Dey
 
Mapreduce - Simplified Data Processing on Large Clusters
byAbhishek Singh
 
Brief introduction on Hadoop,Dremel, Pig, FlumeJava and Cassandra
bySomnath Mazumdar
 
Mapreduce Pact06 Keynote
byJeff Hammerbacher
 
Map reduce
byxydii
 
Map reduceoriginalpaper mandatoryreading
bycoolmirza143
 
Cloud computing-with-map reduce-and-hadoop
byVeda Vyas
 
An introduction to Hadoop for large scale data analysis
byAbhijit Sharma
 
Map Reduce
bySri Prasanna
 
big data analytics introduction chapter 2
byterala1
 
Large Scale Data Processing & Storage
byIlayaraja P
 

Recently uploaded

PPTX
Spacecraft Guidance Quick Research Guide by Arthur Morgan
byArthur Morgan
 
PDF
Ethical AI applied to publishing: Presenting wâsikan kisewâtisiwin, an AI too...
byBookNet Canada
 
PDF
5G Explained! A High Level Overview [Introduction]
byLuciano Motta
 
PDF
Peculiar Findings Around CEX Activity and ETH Price
bytobbymnbvcxz
 
PPTX
API Gateway Architecture - Technical Report 2026
byPowersoft2026
 
PPTX
Coded Agents – with UiPath SDK + LlamaIndex.pptx
bysuhanisingh58689
 
PDF
Skills to Pass the UiPath Agentic Automation Associate (UiAAA) Certification
byUiPathCommunity
 
PDF
How to Design Premium Health (Public Health) PPT Using AI? Top Strategies
byPublic Health Concern Nepal
 
PDF
MINDCTI Revenue Release Quarter 3 2025 - Financial Presentation
byMIND CTI
 
PDF
How PayPal Account Verification Works – Complete Guide for Online Businesses
byjhdhj3989
 
PPTX
AI and Digital Transformation Solutions.
byBuildingblocks
 
PDF
10 Best AI Tools for Fashion Brands in 2026
bymockitseo
 
PDF
Azure DevOps Managed Services: Driving Speed, Security, and Business Growth
byjohncarterjn
 
PDF
How the EU Ecolabel's Record Growth Creates ESG Opportunities for CRE
byWastify AI
 
PDF
Teaching Robots how to Read 1/2: AI Center & Classic Document Understanding (...
byanabulhac
 
DOCX
Best Web to Learn About Buying Verified Skrill Accounts (Los Angeles).docx
byhttps://topsellerit.com/product/buy-verified-binance-accounts/
 
PPTX
6G and Non-Terrestrial Networks (NTN) Testing.pptx
byXRComm
 
PDF
Deploying Windows Clients & Managing Identities
byVICTOR MAESTRE RAMIREZ
 
PPTX
Understanding-Search-Algorithms_09-12-25.pptx
byshaikmahammedtauheed
 
PDF
Why Are Cloud Migration Services Essential for Modern Business Growth?
byGeoPITS Global Pvt Ltd
 
Spacecraft Guidance Quick Research Guide by Arthur Morgan
byArthur Morgan
 
Ethical AI applied to publishing: Presenting wâsikan kisewâtisiwin, an AI too...
byBookNet Canada
 
5G Explained! A High Level Overview [Introduction]
byLuciano Motta
 
Peculiar Findings Around CEX Activity and ETH Price
bytobbymnbvcxz
 
API Gateway Architecture - Technical Report 2026
byPowersoft2026
 
Coded Agents – with UiPath SDK + LlamaIndex.pptx
bysuhanisingh58689
 
Skills to Pass the UiPath Agentic Automation Associate (UiAAA) Certification
byUiPathCommunity
 
How to Design Premium Health (Public Health) PPT Using AI? Top Strategies
byPublic Health Concern Nepal
 
MINDCTI Revenue Release Quarter 3 2025 - Financial Presentation
byMIND CTI
 
How PayPal Account Verification Works – Complete Guide for Online Businesses
byjhdhj3989
 
AI and Digital Transformation Solutions.
byBuildingblocks
 
10 Best AI Tools for Fashion Brands in 2026
bymockitseo
 
Azure DevOps Managed Services: Driving Speed, Security, and Business Growth
byjohncarterjn
 
How the EU Ecolabel's Record Growth Creates ESG Opportunities for CRE
byWastify AI
 
Teaching Robots how to Read 1/2: AI Center & Classic Document Understanding (...
byanabulhac
 
Best Web to Learn About Buying Verified Skrill Accounts (Los Angeles).docx
byhttps://topsellerit.com/product/buy-verified-binance-accounts/
 
6G and Non-Terrestrial Networks (NTN) Testing.pptx
byXRComm
 
Deploying Windows Clients & Managing Identities
byVICTOR MAESTRE RAMIREZ
 
Understanding-Search-Algorithms_09-12-25.pptx
byshaikmahammedtauheed
 
Why Are Cloud Migration Services Essential for Modern Business Growth?
byGeoPITS Global Pvt Ltd
 

Buzz words

  • 1.
    Common MapReduce Patterns Chris K Wensel BuzzWords 2011
  • 2.
    Engineer, Not Academic • Concurrent, Inc., Founder • Cascading support and tools • http://concurrentinc.com/ • Cascading, Lead Developer (started Sept 2007) • An alternative API to MapReduce • http://cascading.org/ • Formerly Hadoop mentoring and training • Sun - Apple - HP - LexisNexis - startups - etc • Formerly Systems Architect & Consultant • Thomson/Reuters - TeleAtlas - startups - etc Copyright Concurrent, Inc. 2011. All rights reserved.
  • 3.
    Overview • MapReduce • HeavyLifting • Analytics • Optimizations Copyright Concurrent, Inc. 2011. All rights reserved.
  • 4.
    MapReduce • A “divideand conquer” strategy for parallelizing workloads against collections of data • Map & Reduce are two user defined functions chained via Key Value Pairs • It’s really Map->Group->Reduce where Group is built in Copyright Concurrent, Inc. 2011. All rights reserved.
  • 5.
    Keys and Values • Map translates input to keys and values to new keys and values [K1,V1] Map [K2,V2]* • System Groups each unique [K2,V2] Group [K2,{V2,V2,....}] key with all its values [K2,{V2,V2,....}] Reduce [K3,V3]* • Reduce translates the values of each unique key to new keys and values * = zero or more Copyright Concurrent, Inc. 2011. All rights reserved.
  • 6.
    Word Count Mapper [0,"when in the course of human events"] Map ["when",1] ["in",1] ["the",1] [...,1] ["when",1] ["when",1] ["when",1] ["when",1] Group ["when",{1,1,1,1,1}] ["when",1] Reducer ["when",{1,1,1,1,1}] Reduce ["when",5] Copyright Concurrent, Inc. 2011. All rights reserved.
  • 7.
    Divide and Conquer Parallelism • Since the ‘records’ entering the Map and ‘groups’ entering the Reduce are independent • That is, there is no expectation of order or requirement to share state between records/ groups • Arbitrary numbers of Map and Reduce function instances can be created against arbitrary portions of input data Copyright Concurrent, Inc. 2011. All rights reserved.
  • 8.
    Cluster Cluster Rack Rack Rack Node Node Node Node ... map map map map map reduce reduce reduce • Multiple instances of each Map and Reduce function are distributed throughout the cluster Copyright Concurrent, Inc. 2011. All rights reserved.
  • 9.
    Another View [K1,V1] Map [K2,V2] Combine Group [K2,{V2,...}] Reduce [K3,V3] Mapper Task same code Mapper Reducer Shuffle Task Task Mapper Reducer Shuffle Task Task Mapper Reducer Shuffle Task Task Mapper Task Mappers must complete before Reducers can begin split1 split2 split3 split4 ... part-00000 part-00001 part-000N file directory Copyright Concurrent, Inc. 2011. All rights reserved.
  • 10.
    Complex job assemblies • Real applications are many MapReduce jobs chained together • Linked by intermediate (usually temporary) files • Executed in order, by hand, from the ‘client’ application Count Job Sort Job [ k, [v] ] [ k, [v] ] Map Reduce Map Reduce [ k, v ] [ k, v ] [ k, v ] [ k, v ] File File File [ k, v ] = key and value pair [ k, [v] ] = key and associated values collection Copyright Concurrent, Inc. 2011. All rights reserved.
  • 11.
    Real World Apps [37/75] map+reduce [54/75] map+reduce [41/75] map+reduce [43/75] map+reduce [42/75] map+reduce [45/75] map+reduce [44/75] map+reduce [39/75] map+reduce [36/75] map+reduce [46/75] map+reduce [40/75] map+reduce [50/75] map+reduce [38/75] map+reduce [49/75] map+reduce [51/75] map+reduce [47/75] map+reduce [52/75] map+reduce [53/75] map+reduce [48/75] map+reduce [23/75] map+reduce [25/75] map+reduce [24/75] map+reduce [27/75] map+reduce [26/75] map+reduce [21/75] map+reduce [19/75] map+reduce [28/75] map+reduce [22/75] map+reduce [32/75] map+reduce [20/75] map+reduce [31/75] map+reduce [33/75] map+reduce [29/75] map+reduce [34/75] map+reduce [35/75] map+reduce [30/75] map+reduce [7/75] map+reduce [2/75] map+reduce [8/75] map+reduce [10/75] map+reduce [9/75] map+reduce [5/75] map+reduce [3/75] map+reduce [11/75] map+reduce [6/75] map+reduce [13/75] map+reduce [4/75] map+reduce [16/75] map+reduce [14/75] map+reduce [15/75] map+reduce [17/75] map+reduce [18/75] map+reduce [12/75] map+reduce [60/75] map [62/75] map [61/75] map [58/75] map [55/75] map [56/75] map+reduce [57/75] map [71/75] map [72/75] map [59/75] map [64/75] map+reduce [63/75] map+reduce [65/75] map+reduce [68/75] map+reduce [67/75] map+reduce [70/75] map+reduce [69/75] map+reduce [73/75] map+reduce [66/75] map+reduce [74/75] map+reduce [75/75] map+reduce [1/75] map+reduce 1 app, 75 jobs green = map + reduce purple = map blue = join/merge orange = map split Copyright Concurrent, Inc. 2011. All rights reserved.
  • 12.
    Heavy Lifting • Thing we must do because data can be heavy • These patterns are natural to MapReduce and easy to implement • But have some room for composition/aggregation within a Map/ Reduce (i.e., Filter + Binning) • (leading us to think of Hadoop as an ETL framework) • Record Filtering • Parsing, Conversion • Binning • Counting, Summing • Distributed Tasks • Unique Copyright Concurrent, Inc. 2011. All rights reserved.
  • 13.
    Record Filtering • Thinkunix ‘grep’ • Filtering is discarding unwanted values (or preserving wanted) • Only uses a Map function, no Reducer Copyright Concurrent, Inc. 2011. All rights reserved.
  • 14.
    Parsing, Conversion • Think unix ‘sed’ • A Map function that takes an input key and/or value and translates it into a new format • Examples: • raw logs to delimited text or archival efficient binary • entity extraction Copyright Concurrent, Inc. 2011. All rights reserved.
  • 15.
    Counting, Summing • Thesame as SQL aggregation functions • Simply applying some function to the values collection seen in Reduce • Other examples: • average, max, min, unique Copyright Concurrent, Inc. 2011. All rights reserved.
  • 16.
    Merging • Where many files of the same type are converted to one output path • Map side merges • One directory with as many part files as Mappers • Reduce side merges • Allows for removing duplicates or deleted items • One directory with as many part files as Reducers • Examples • Nutch • Normalizing log files (apache, log4j, etc) Copyright Concurrent, Inc. 2011. All rights reserved.
  • 17.
    Binning • Where the values associated w/ unique keys are persisted together • Typically a directory path based on key’s value • Must be conscious of total open files, remember no appends • Examples: • web log files by year/month/day • trade data by symbol Copyright Concurrent, Inc. 2011. All rights reserved.
  • 18.
    Distributed Tasks • Simply where a Map or Reduce function executes some ‘task’ based on the input key and value. • Examples: • web crawling, • load testing services, • rdbms/nosql updates, • file transfers (S3), • image to pdf (NYT on EC2) Copyright Concurrent, Inc. 2011. All rights reserved.
  • 19.
    Basic Analytic Patterns • Some of these patterns are unnatural to MapReduce • We think in terms of columns/fields, not key value pairs • (leading us to think of Hadoop as a RDBMS) • Group By • Secondary Unique • Unique • CoGrouping and Joining • Secondary Sort Copyright Concurrent, Inc. 2011. All rights reserved.
  • 20.
    Composite Keys/Values [K1,V1] <A1,B1,C1,...> • It is easier to think in columns/fields • e.g. “firstname” & “lastname”, not “line” • Whether a set of columns are Keys or Values is arbitrary • Keys become a means to piggyback the properties of MR and become an impl detail Copyright Concurrent, Inc. 2011. All rights reserved.
  • 21.
    Group By GroupBy 1001 Jim dept_id Mary name Susan 1002 Fred Wilma Ernie Barny • Group By is where Value fields are grouped by Grouping fields • Above, Map output key is “dept_id” and value is “name” Copyright Concurrent, Inc. 2011. All rights reserved.
  • 22.
    Group By Mapper Reducer Piggyback Code [K1,V1] [K2,{V2,V2,....}] [K1,V1] -> <A1,B1,C1,D1> [K2,V2] -> <A2,B2,{<C2,D2>,...}> User Code Map Reduce <A2,B2> -> K2, <C2,D2> -> V2 <A3,B3> -> K3, <C3,D3> -> V3 [K2,V2] [K3,V3] • So the K2 key becomes a composite Key of • key: [grouping], value: [values] Copyright Concurrent, Inc. 2011. All rights reserved.
  • 23.
    Unique Mapper [0, "when in the course of human events"] Map ["when",null] ["in",null] [...,null] ["when",1] ["when",1] ["when",1] ["when",1] Group ["when",{nulls}] ["when",null] Reducer ["when",{nulls}] Reduce ["when",null] • Or Distinct (as in SQL) • Globally finding all the unique values in a dataset • Usually finding unique values in a column • Often used to filter a second dataset using a join Copyright Concurrent, Inc. 2011. All rights reserved.
  • 24.
    Secondary Sort (group) (sorted value) (remaining value) Date Time Url 08/08/2008, 1:00:00, http://www.example.com/foo 08/08/2008, 1:01:00, http://www.example.com/bar 08/08/2008, 1:01:30, http://www.example.com/baz • Secondary Sorting is where • Some Fields are grouped on, and • Some of the remaining Fields are sorted within their grouping Copyright Concurrent, Inc. 2011. All rights reserved.
  • 25.
    Secondary Sort Mapper Reducer [K1,V1] [K2,{V2,V2,....}] [K1,V1] -> <A1,B1,C1,D1> [K2,V2] -> <A2,B2,{<C2,D2>,...}> Map Reduce <A2,B2><C2> -> K2, <D2> -> V2 <A3,B3> -> K3, <C3,D3> -> V3 [K2,V2] [K3,V3] • So the K2 key becomes a composite Key of • key: [grouping, secondary], value: [remaining values] • The trick is to piggyback the Reduce sort yet not be compared during the unique key comparison Copyright Concurrent, Inc. 2011. All rights reserved.
  • 26.
    Secondary Unique Mapper Assume Secondary Sorting magic happens here [0, "when in the course of human events"] Map [0, "when"] [0, "in"] [0,"the"] [0,...] ["when",1] ["when",1] ["when",1] ["when",1] Group [0,{"in","in","the","when","when",...}] [0,"when"] Reducer [0,{"in","in","the","when","when",...}] Reduce ["in",null] ["the",null] ["when",null] • Secondary Unique is where the grouping values are uniqued • .... in a “scale free” way • Perform a Secondary Sort... • Reducer removes duplicates by discarding every value that matches the previous value • since values are now ordered, no need to maintain a Set of values Copyright Concurrent, Inc. 2011. All rights reserved.
  • 27.
    Joining lhs data rhs data 1001 dept_id Jim Accounting dept_name Mary Accounting name Susan Accounting 1002 Fred Shipping Wilma Shipping Ernie Shipping Barny Shipping • Where two or more input data sets are ‘joined’ by a common key • Like a SQL join Copyright Concurrent, Inc. 2011. All rights reserved.
  • 28.
    Join Definitions • Consider the input data [key, value]: • LHS = [0,a] [1,b] [2,c] • RHS = [0,A] [2,C] [3,D] • Joins on the key: • Inner • [0,a,A] [2,c,C] • Outer (Left Outer, Right Outer) • [0,a,A] [1,b,null] [2,c,C] [3,null,D] • Left (Left Inner, Right Outer) • [0,a,A] [1,b,null] [2,c,C] • Right (Left Outer, Right Inner) • [0,a,A] [2,c,C] [3,null,D] Copyright Concurrent, Inc. 2011. All rights reserved.
  • 29.
    CoGrouping • Before Joining,CoGrouping must happen • Simply concurrent GroupBy operations on each input data set Copyright Concurrent, Inc. 2011. All rights reserved.
  • 30.
    GroupBy vs CoGroup lhs data rhs data GroupBy CoGroup 1001 1001 Jim Jim Accounting dept_id Mary Mary name Susan Susan dept_name 1002 1002 Fred Fred Shipping Wilma Wilma Ernie Ernie Barny Barny Independent collections of unordered values Copyright Concurrent, Inc. 2011. All rights reserved.
  • 31.
    CoGroup Joined lhs data rhs data 1001 dept_id Jim Accounting dept_name Mary Accounting name Susan Accounting 1002 Fred Shipping Wilma Shipping Ernie Shipping Barny Shipping • Considering the previous data, a typical Inner Join Copyright Concurrent, Inc. 2011. All rights reserved.
  • 32.
    CoGrouping Mapper [n] [n+1] Reducer [K1,V1] [K1',V1'] [K2,{V2,V2,....}] [K2,V2] -> <A2,B2,{<C2,D2,C2',D2'>,...}> [K1,V1] -> <A1,B1,C1,D1> [K1',V1'] -> <A1',B1',C1',D1'> Reduce Map <A3,B3> -> K3, <C3,D3> -> V3 <A2,B2> -> K2, [n]<C2,D2> -> V2 [K2,V2] [K3,V3] • Maps must run for each input set in same Job (n, n+1, etc) • CoGrouping must happen against each common key Copyright Concurrent, Inc. 2011. All rights reserved.
  • 33.
    Joining Reducer [K2,{V2,V2,....}] <A2,B2,{[n]<C2,D2>,[n+1]..}> [K2,V2] -> <A2,B2,{<C2,D2,C2',D2'>,...}> <A2,B2,{<C2,D2>,...},{<C2',D2'>,...}> Reduce {<C2,D2>,...} Join {<C2',D2'>,...} <A3,B3> -> K3, <C3,D3> -> V3 <C2,D2,C2',D2'> [K3,V3] <A2,B2,{<C2,D2,C2',D2'>,...}> • The CoGroups must be joined • Finally the Reduce can be applied Copyright Concurrent, Inc. 2011. All rights reserved.
  • 34.
    Optimizations • Patternsfor reducing IO • Identity Mapper • Partial Aggregates • Map Side Join • Similarity Joins • Combiners Copyright Concurrent, Inc. 2011. All rights reserved.
  • 35.
    Identity Mapper [head] Dfs['TextLine[['offset', 'line']->[ALL]]']['/logs/stumbles/short-stumbles-20090504.log']'] [{2}:'offset', 'line'] [{2}:'offset', 'line'] Each('import')[RegexParser[decl:'day', 'urlid', 'method'][args:1]] [{3}:'day', 'urlid', 'method'] [{3}:'day', 'urlid', 'method'] Each('import')[ExpressionFilter[decl:'day', 'urlid', 'method']] [{3}:'day', 'urlid', 'method'] [{3}:'day', 'urlid', 'method'] TempHfs['SequenceFile[['day', 'urlid', 'method']]'][import/71897/] [{3}:'day', 'urlid', 'method'] [{3}:'day', 'urlid', 'method'] [{3}:'day', 'urlid', 'method'] identity [{3}:'day', 'urlid', 'method'] Each('paidCount')[Not[decl:'day', 'urlid', 'method']] function [{3}:'day', 'urlid', 'method'] [{3}:'day', 'urlid', 'method'] Each('organicCount')[OrganicFilter[decl:'day', 'urlid', 'method']] [{3}:'day', 'urlid', 'method'] GroupBy('paidCount')[by:['day', 'urlid']] [{3}:'day', 'urlid', 'method'] paidCount[{2}:'day', 'urlid'] GroupBy('organicCount')[by:['day', 'urlid']] [{3}:'day', 'urlid', 'method'] organicCount[{2}:'day', 'urlid'] Every('paidCount')[Count[decl:'count']] [{3}:'day', 'urlid', 'method'] [{3}:'day', 'urlid', 'count'] Every('organicCount')[Count[decl:'count']] • [{3}:'day', 'urlid', 'method'] Move Map operations to the Each('paidCount')[Identity[decl:'paid_day', 'paid_urlid', 'paid_count']] [{3}:'day', 'urlid', 'count'] [{3}:'day', 'urlid', 'method'] [{3}:'paid_day', 'paid_urlid', 'paid_count'] [{3}:'paid_day', 'paid_urlid', 'paid_count'] TempHfs['SequenceFile[['paid_day', 'paid_urlid', 'paid_count']]'][paidCount/33072/] TempHfs['SequenceFile[['day', 'urlid', 'count']]'][organicCount/97544/] [{3}:'paid_day', 'paid_urlid', 'paid_count'] [{3}:'paid_day', 'paid_urlid', 'paid_count'] previous Reduce [{3}:'paid_day', 'paid_urlid', 'paid_count'] [{3}:'day', 'urlid', 'count'] [{3}:'paid_day', 'paid_urlid', 'paid_count'] [{3}:'day', 'urlid', 'count'] [{3}:'day', 'urlid', 'count'] [{3}:'day', 'urlid', 'count'] CoGroup('organicCount*paidCount')[by:organicCount:['day', 'urlid']paidCount:['paid_day', 'paid_urlid']] Each('paidDomainCount')[LookupDomainFunction[decl:'domainid'][args:1]] organicCount[{2}:'day', 'urlid'],paidCount[{2}:'paid_day', 'paid_urlid'] [{4}:'paid_day', 'paid_urlid', 'paid_count', 'domainid'] Each('organicDomainCount')[LookupDomainFunction[decl:'domainid'][args:1]] [{6}:'day', 'urlid', 'count', 'paid_day', 'paid_urlid', 'paid_count'] [{4}:'paid_day', 'paid_urlid', 'paid_count', 'domainid'] [{4}:'day', 'urlid', 'count', 'domainid'] Each('organicCount*paidCount')[ExpressionFunction[decl:'urlid_day']] GroupBy('paidDomainCount')[by:['paid_day', 'domainid']] [{4}:'day', 'urlid', 'count', 'domainid'] • [{7}:'day', 'urlid', 'count', 'paid_day', 'paid_urlid', 'paid_count', 'urlid_day'] paidDomainCount[{2}:'paid_day', 'domainid'] GroupBy('organicDomainCount')[by:['day', 'domainid']] [{7}:'day', 'urlid', 'count', 'paid_day', 'paid_urlid', 'paid_count', 'urlid_day'] [{4}:'paid_day', 'paid_urlid', 'paid_count', 'domainid'] Replace with an Identity organicDomainCount[{2}:'day', 'domainid'] Each('organicCount*paidCount')[ExpressionFunction[decl:'fixed_count']] Every('paidDomainCount')[Sum[decl:'sum'][args:1]] [{4}:'day', 'urlid', 'count', 'domainid'] [{8}:'day', 'urlid', 'count', 'paid_day', 'paid_urlid', 'paid_count', 'urlid_day', 'fixed_count'] [{3}:'paid_day', 'domainid', 'sum'] Every('organicDomainCount')[Sum[decl:'sum'][args:1]] [{8}:'day', 'urlid', 'count', 'paid_day', 'paid_urlid', 'paid_count', 'urlid_day', 'fixed_count'] [{4}:'paid_day', 'paid_urlid', 'paid_count', 'domainid'] function [{3}:'day', 'domainid', 'sum'] Each('organicCount*paidCount')[ExpressionFunction[decl:'fixed_paid_count']] Each('paidDomainCount')[Identity[decl:'paid_day', 'paid_domainid', 'paid_sum']] [{4}:'day', 'urlid', 'count', 'domainid'] [{9}:'day', 'urlid', 'count', 'paid_day', 'paid_urlid', 'paid_count', 'urlid_day', 'fixed_count', 'fixed_paid_count'] [{3}:'paid_day', 'paid_domainid', 'paid_sum'] TempHfs['SequenceFile[['day', 'domainid', 'sum']]'][organicDomainCount/49784/] [{9}:'day', 'urlid', 'count', 'paid_day', 'paid_urlid', 'paid_count', 'urlid_day', 'fixed_count', 'fixed_paid_count'] [{3}:'paid_day', 'paid_domainid', 'paid_sum'] Each('organicCount*paidCount')[Identity[decl:'urlid_day', 'default:organic_stumbles', 'default:paid_stumbles']] TempHfs['SequenceFile[['paid_day', 'paid_domainid', 'paid_sum']]'][paidDomainCount/54349/] [{3}:'paid_day', 'paid_domainid', 'paid_sum'] [{3}:'day', 'domainid', 'sum'] • [{3}:'paid_day', 'paid_domainid', 'paid_sum'] [{3}:'day', 'domainid', 'sum'] [{3}:'urlid_day', 'default:organic_stumbles', 'default:paid_stumbles'] [{3}:'urlid_day', 'default:organic_stumbles', 'default:paid_stumbles'] CoGroup('organicDomainCount*paidDomainCount')[by:organicDomainCount:['day', 'domainid']paidDomainCount:['paid_day', 'paid_domainid']] Assumes Map operations cascading.hbase.HBaseTap@d9a475a0 organicDomainCount[{2}:'day', 'domainid'],paidDomainCount[{2}:'paid_day', 'paid_domainid'] [{6}:'day', 'domainid', 'sum', 'paid_day', 'paid_domainid', 'paid_sum'] Each('organicDomainCount*paidDomainCount')[ExpressionFunction[decl:'domainid_day']] reduce the data [{7}:'day', 'domainid', 'sum', 'paid_day', 'paid_domainid', 'paid_sum', 'domainid_day'] [{7}:'day', 'domainid', 'sum', 'paid_day', 'paid_domainid', 'paid_sum', 'domainid_day'] [{3}:'urlid_day', 'default:organic_stumbles', 'default:paid_stumbles'] Each('organicDomainCount*paidDomainCount')[ExpressionFunction[decl:'fixed_sum']] [{3}:'urlid_day', 'default:organic_stumbles', 'default:paid_stumbles'] [{8}:'day', 'domainid', 'sum', 'paid_day', 'paid_domainid', 'paid_sum', 'domainid_day', 'fixed_sum'] [{8}:'day', 'domainid', 'sum', 'paid_day', 'paid_domainid', 'paid_sum', 'domainid_day', 'fixed_sum'] Each('organicDomainCount*paidDomainCount')[ExpressionFunction[decl:'fixed_paid_sum']] [{9}:'day', 'domainid', 'sum', 'paid_day', 'paid_domainid', 'paid_sum', 'domainid_day', 'fixed_sum', 'fixed_paid_sum'] [{9}:'day', 'domainid', 'sum', 'paid_day', 'paid_domainid', 'paid_sum', 'domainid_day', 'fixed_sum', 'fixed_paid_sum'] Each('organicDomainCount*paidDomainCount')[Identity[decl:'domainid_day', 'default:organic_stumbles', 'default:paid_stumbles']] [{3}:'domainid_day', 'default:organic_stumbles', 'default:paid_stumbles'] [{3}:'domainid_day', 'default:organic_stumbles', 'default:paid_stumbles'] cascading.hbase.HBaseTap@3d07f00 [{3}:'domainid_day', 'default:organic_stumbles', 'default:paid_stumbles'] [{3}:'domainid_day', 'default:organic_stumbles', 'default:paid_stumbles'] [tail] Copyright Concurrent, Inc. 2011. All rights reserved.
  • 36.
    Map Side Joins •Bypasses the (immediate) need for a Reducer • Symmetrical • Where LHS and RHS are of equivalent size • Requires data to be sorted on key • Asymmetrical • One side is small enough to fit in memory • Typically a hashtable lookup Copyright Concurrent, Inc. 2011. All rights reserved.
  • 37.
    Combiners Mapper [0, "when in the course of human events"] Map ["when",1] ["in",1] ["the",1] [...,1] Combiner ["when",1] ["when",1] Group ["when",{1,1}] ["when",{1,1}] Reduce ["when",2] Same Implementation ["when",1] ["when",1] Group ["when",{2,1,2}] ["when",2] Reducer ["when",{2,1,2}] Reduce ["when",5] • Where Reduce runs Map side, and again Reduce side • Only works if Reduce is commutative and associative • Reduces bandwidth by trading CPU for IO • Serialization/deserialization during local sorting before combining Copyright Concurrent, Inc. 2011. All rights reserved.
  • 38.
    Partial Aggregates Mapper [0, "when in the course of human events"] ["when",1] ["in",1] ["the",1] [...,1] Map Partial Provides an opportunity to ["when",1] ["when",1] ["when",2] promote the functionality of the next Map to this Reduce ["when",1] ["when",1] Group ["when",{2,1,2}] ["when",2] Reducer ["when",{2,1,2}] Reduce ["when",5] • Supports any aggregate type, while being composable with other aggregates • Reduces bandwidth by trading Memory for IO • Very important for a CPU constrained cluster • Use a bounded LRU to keep constant memory (requires tuning) Copyright Concurrent, Inc. 2011. All rights reserved.
  • 39.
    Partial Aggregates [a,b,c,a,a,b] [a,b,c,a,a,b] partial unique partial unique [a,b,c,a,b] [a,b,c,a,b] [a,b,c,a,a,b] [a,b,c,a,a,b] partial unique partial unique [a,b,c,a,b] [a,b,c,a,b] LRU* {_,_} *cache size of 2 a -> {a,_} -> _ b -> {b,a} -> _ incoming discarded c -> {c,b} -> a value value a -> {a,c} -> b a -> {a,c} b -> {b,a} -> c • OK that dupes emit from a Mapper and across Mappers (or prev Reducers!) • Final aggregation happens in Reducer • Larger the cache, fewer dupes Copyright Concurrent, Inc. 2011. All rights reserved.
  • 40.
    Tradeoffs • CPU forIO == fault tolerance • Memory for IO == performance Copyright Concurrent, Inc. 2011. All rights reserved.
  • 41.
    Similarity Join • Compareall values LHS to values RHS to find duplicates (or similar values) • Naive approaches • Cross Join (all data through one reducer) • In-common features (very common features will bottleneck) Copyright Concurrent, Inc. 2011. All rights reserved.
  • 42.
    Set-Similarity Joining • “EfficientParallel Set-Similarity Joins Using MapReduce” - R Vernica, M Carey, C Li • Only compare candidate pairs • Candidates share uncommon features Copyright Concurrent, Inc. 2011. All rights reserved.
  • 43.
    4 1 1 4 2 2 4 3 3 2 4 4 3: order by least frequent 1: records 1 discard common 1 2: count tokens 1 1 1 3 3 5: candidate pairs 3 4: uncommon features 6: final compare in common • 1 and 3 share uncommon features • thus are candidates for a full comparison Copyright Concurrent, Inc. 2011. All rights reserved.
  • 44.
    Tokenize Count Job Map Reduce Map Reduce File File File Join Tokens/Counts Job File Map Reduce File Sort/Prefix Filter Job Map Reduce File Match two sets Self Join Job Map Reduce using prefix File filtering Unique Pairs Job Map Reduce File Join LHS Job Map Reduce File Join RHS / Match Job Map Reduce File Copyright Concurrent, Inc. 2011. All rights reserved.
  • 45.
    Duality • Note theuse of the previous patterns to route data to implement a more efficient algorithm Copyright Concurrent, Inc. 2011. All rights reserved.
  • 46.
    Use a Higher Abstraction • Command Line • Multitool - CLI for parallel sed, grep & joins • API • Cascading - Java Query API and Planner • Plume - “approximate clone of FlumeJava” • Interactive Shell • Cascalog - Clojure+Cascading query language (API also) • Pig - A text Syntax • Hive - Syntax + Infrastructure - SQL “like” Copyright Concurrent, Inc. 2011. All rights reserved.
  • 47.
    References • Set Similarity • http://www.slideshare.net/ydn/4-similarity-joinshadoopsummit2010 • http://asterix.ics.uci.edu/fuzzyjoin-mapreduce/ • MapReduce Text Processing • http://www.umiacs.umd.edu/~jimmylin/book.html • Plume/FlumeJava • http://portal.acm.org/citation.cfm?id=1806596.1806638 • http://github.com/tdunning/Plume/wiki Copyright Concurrent, Inc. 2011. All rights reserved.
  • 48.
    I’m Hiring • EnterpriseJava server and web client • Language design, compilers, and interpreters • No Hadoop experience required • More info • http://www.concurrentinc.com/careers/ Copyright Concurrent, Inc. 2011. All rights reserved.
  • 49.
    Resources • Chris K Wensel •chris@wensel.net •@cwensel • Cascading & Cascalog •http://cascading.org •@cascading • Concurrent, Inc. •http://concurrentinc.com •@concurrent Copyright Concurrent, Inc. 2011. All rights reserved.
  • 50.
    Appendix Copyright Concurrent, Inc. 2011. All rights reserved.
  • 51.
    Simple Total Sorting • Where lines in a result file should be sorted • Must set number of reducers to 1 • Sorting in MR is local per Reduce, not global across Reducers Copyright Concurrent, Inc. 2011. All rights reserved.
  • 52.
    Why Sorting Isn’t “Total” [aaa,aab,aac] Mapper aaa Mapper aac Reducer [aaa,zzx] aab Mapper Reducer [aac,zzz] zzx Mapper zzz Reducer [aab,zzy] zzy [zzx,zzy,zzz] Mapper • Keys emitted from Map are naturally sorted at a given Reducer • But are Partitioned to Reducers in a random way • Thus, only one Reducer can be used for a total sort Copyright Concurrent, Inc. 2011. All rights reserved.
  • 53.
    Distributed Total Sort •To work, the Shuffling phase must be modified with: • Custom Partitioner to partition on the distribution of ordered Keys • Custom Comparator for comparing Key types • Strings work by default Copyright Concurrent, Inc. 2011. All rights reserved.
  • 54.
    Distributed Total Sort- Details a ... z ar ... ax za ... zo ara ... ari axe ... axi zag ... zap zon ... zoo aran aria axis zone • Sample all K2 values and build balanced distribution for num reducers • Sample all input keys and divide into partitions • Write out boundaries of partitions • Supply Partitioner that looks up partition for current K2 value • Read boundaries into a Trie (pronounced ‘try’) data structure • Use appropriate Comparator for Key type Copyright Concurrent, Inc. 2011. All rights reserved.

Editor's Notes