Downloaded 49 times
![Pig examples
- - Inner Join:
A = JOIN comments BY userID, users BY userID;
- - Outer Join:
A = JOIN comments BY userID [LEFT | RIFGT| FULL] OUTER , users BY userID;
- - Binning:
SPLIT data INTO
eights IF col1 == 8,
bigs IF col1 > 8,
smalls IF (col1 < 8 and col1 > 0 );
- - Top Ten:
B = ORDER A BY col4 DESC’
C = limit B 10;
- - Filtering:
b = FILTER a BY value < 3;](https://image.slidesharecdn.com/mapreducedesignpatterns-140303100944-phpapp02/85/MapReduce-Design-Patterns-25-320.jpg)
Uploaded byAnastasiia Kornilova
MapReduce Design Patterns
This document discusses MapReduce design patterns. It describes the core MapReduce components including the mapper, reducer, and shuffle and sort. It then outlines several common MapReduce patterns such as filtering, summarization, joins, data organization, and input/output. Specific filtering patterns like bloom filtering and top-N are explained in more detail.
MapReduce Design Patterns
- 1. MapReduce Design Patterns Anastasiia Kornilova, SoftServeData Science Group
- 2. MapReduce Components ❖ record reader ❖ map ❖ Reader combiner ❖ partitioner ❖ Mapper Combiner Partitioner Shuffle andsort shuffle and sort ❖ reduce ❖ output format Reducer Output
- 4. MapReduce Patterns ❖ Filtering Patterns ❖ SummarizationPatterns ❖ Join Patterns ❖ Data Organization Patterns ❖ Metapatterns ❖ Input and Output Patterns
- 5.
- 6. Filtering ❖ Closer view ofdata ❖ Tracking a thread of events ❖ Distributed grep ❖ Data cleansing ❖ Simple random sampling ❖ Removing low scoring data
- 7.
- 8. Bloom filtering ❖ Removing mostof non watched values ❖ Prefiltering a data set for an expensive set membership check • • • Probabilistic data structure Hash functions comparing Answer: probably yes or now
- 9. Step 1 -Filter Training Bloom Filter Training Input split Output file Step 2 - Bloom Filtering via MapReduce Input split Bloom Filter Mapper Maybe Bloom Filter Test No Discarded Load filter from distributed cache Input split Output file Bloom Filter Mapper Maybe Bloom Filter Test Output file No Load filter from distributed cache Discarded
- 10. Top N ❖ Outlier analysis ❖ Selectinteresting data ❖ Catchy dashboards
- 11. Input split Top Ten Mapper local top10 Input split Top Ten Mapper local top 10 Top Ten Reducer Input split Top Ten Mapper local top 10 Input split Top Ten Mapper local top 10 final top 10 Top 10 Output
- 12. Distinct ❖ Deduplicate data ❖ Getting distinctvalues ❖ Protecting from inner join explosions
- 13. Summarization patterns ❖ Numerical summarization ❖ Invertedindex ❖ Counting with counters
- 14. Numerical summarization ❖ Word count ❖ Recordcount ❖ Min/Max/Count ❖ Average/Median/Standart deviation
- 15. Mapper Mapper Mapper (key, summary field) (key,summary field) (key, summary field) (key, summary field) (key, summary field) (key, summary field) Partitoner Reducer (group B, summary) (group D, summary) Reducer (group B, summary) (group D, summary) Partitoner Partitoner
- 16.
- 17. Mapper (keyword, unique ID) (keyword,unique ID) Partitoner Reducer Reducer (keyword, unique ID) (keyword, unique ID) (keyword A, list of IDs) (keyword D, list of IDs) Partitoner Mapper (keyword, unique ID) (keyword, unique ID) Mapper (keyword A, list of IDs) (keyword D, list of IDs) Partitoner
- 18. Data Organization Patterns ❖ Structuredto Hierarchical ❖ Partitioning ❖ Binning ❖ Total Order Sorting ❖ Shuffling
- 19. Join patterns ❖ Reduce SideJoin ❖ Replicated Join ❖ Composite Join ❖ Cartesian Product
- 21. Data Set A Input split Input split Input split Join Mapper Join Mapper Join Mapper (key,values A) (key, values A) Join Reducer Output part Join Reducer Output part Join Reducer Output part (key, values A) Shuffle and sort Data Set B Input split Input split Join Mapper Join Mapper (key, values B) (key, values B)
- 22. Node table id title tagnames authorized User table body nodetype parent id abs parent id added at score state string last edited id last activity id last activity at activity revision extra extra def extra count user id reputation gold silver bronze
- 25. Pig examples - -Inner Join: A = JOIN comments BY userID, users BY userID; - - Outer Join: A = JOIN comments BY userID [LEFT | RIFGT| FULL] OUTER , users BY userID; - - Binning: SPLIT data INTO eights IF col1 == 8, bigs IF col1 > 8, smalls IF (col1 < 8 and col1 > 0 ); - - Top Ten: B = ORDER A BY col4 DESC’ C = limit B 10; - - Filtering: b = FILTER a BY value < 3;