The document outlines the objectives and applications of MapReduce design patterns, particularly focusing on join patterns for merging datasets from multiple sources. It details various join types and their implementations, emphasizing the reduce side join as a common yet resource-intensive method. The guide is aimed at Java developers, MapReduce programmers, and aspiring Hadoop architects seeking to enhance their big data skills.

1. View MR Design Patterns course details at www.edureka.co/mapreduce-design-patterns
Application of JOIN Pattern
MAP Reduce Design PATTERN

2. Slide 2 www.edureka.co/mapreduce-design-patterns
Objectives
At the end of this module, you will be able to understand
Why Design Patterns in MR
Who should know Map-Reduce Design patterns
Available Design Patterns in MR
Join pattern

3. Slide 3 www.edureka.co/mapreduce-design-patternsSlide 3
Why Design Patterns in MR?
General reusable, optimized solutions to most common
problems
Template to solve problems used in different situations
Speed up the development process
Tried and tested design principles
An initial guideline to solve most common problems in MR
Help build sophisticated and best solution

4. Slide 4 www.edureka.co/mapreduce-design-patternsSlide 4
Who should know MR Design Pattern?
A Java developer who wants to explore world of Big Data
A MapReduce programmer who wants to develop expertise in his/her MR skills
One who aims to become a Hadoop Architect

5. Slide 5 www.edureka.co/mapreduce-design-patternsSlide 5
Available Design Patterns in MR
Summarization
Pattern
Filtering Pattern
Data Organization
Pattern
Join Pattern
Meta Pattern
Input & Output
Pattern

6. Slide 6 www.edureka.co/mapreduce-design-patterns
Join Patterns – What is it
 Datasets generally exist in multiple sources
 Deriving full-value requires merging them together
 Join Patterns are used for this purpose
 Performing joins on the fly on Big Data can be costly in terms of time
Example: Joining StackOverflow data from Comments & Posts on UserId

7. Slide 7 www.edureka.co/mapreduce-design-patterns
Join Patterns – What is it?
 Joining Patterns we will talk about are
» Reduce Side Join/Repartition Join
» Reduce Side Join with Bloom Filter
» Replicated Join
» Composite Join
» Cartesian Product

8. Slide 8 www.edureka.co/mapreduce-design-patterns
Join – Refresher
 Inner Join
 Outer Join
» Left Outer Join
» Right Outer Join
» Full Outer Join
 Anti Join
 Cartesian Product

9. Slide 9 www.edureka.co/mapreduce-design-patterns
Reduce Side Join – Description
 Easiest to implement but can be longest to execute
 Supports all types of join operation
 Can join multiple data sources, but expensive in terms of network resources & time
 All data transferred across network
Example : Join PostLinks table data in StackOverflow to Posts data

10. Slide 10 www.edureka.co/mapreduce-design-patterns
Reduce Side Join – Description (Contd.)
 Applicability – Use it when
» Multiple large data sets require to be joined
» If one of the data sources is small look at using replicated join
» Different data sources are linked by a foreign key
» You want all join operations to be supported

11. Slide 11 www.edureka.co/mapreduce-design-patterns
Reduce Side Join – Structure

12. Slide 12 www.edureka.co/mapreduce-design-patterns
Reduce Side Join – Structure (Contd.)
 Mapper
» Output key should reflect the foreign key
» Value can be the whole record and an identifier to identify the source
» Use projection and output only the required number of fields
 Combiner
» Not Required ; No additional benefit

13. Slide 13 www.edureka.co/mapreduce-design-patterns
Reduce Side Join – Structure (Contd.)
 Partitioner
» User Custom Partitioner if required;
 Reducer
» Reducer logic based on type of join required
» Reducer receives the data from all the different sources per key

14. Slide 14 www.edureka.co/mapreduce-design-patterns
Reduce Side Join – Analogy
 Resemblances
» SQL
» SELECT users.ID, users.Location, comments.upVotes
FROM users
[INNER|LEFT|RIGHT] JOIN comments
ON users.ID=comments.UserID
» Pig
» Supports inner & outer joins
» Inner Join
» A = JOIN comments BY userID, users BY userID;
» Outer Join
» A = JOIN comments BY userID [LEFT|RIGHT|FULL] OUTER, users BY userID

15. Slide 15 www.edureka.co/mapreduce-design-patterns
Reduce Side Join – Performance
 Performance
» The whole data moves across the network to reducers
» You can optimize by using projection and sending only the required fields
» Number of reducers typically higher than normal
» If you can use any other Join type for your problem, use that instead

16. Slide 16 www.edureka.co/mapreduce-design-patterns
Reduce Side Join – Use Cases
 Join tweets with user personal information for Behavioral Analysis
 Join PostLinks and Posts tables from StackOverflow to have all related posts in one place

17. Slide 17 www.edureka.co/mapreduce-design-patterns
Reduce Side Join Example – Problem
 Your dataset is the StackOverflow dataset. Look at the PostLinks.xml & Posts.xml file. Join the two tables based on
PostId in PostLinks & Id in Posts
» Use MultipleInputs class
» Projection on PostLinks to output only PostId & RelatedPostId fields

18. Slide 18 www.edureka.co/mapreduce-design-patterns
DEMO
Reduce Side Join Example

19. Slide 19 www.edureka.co/mapreduce-design-patterns
Questions

20. Slide 20 www.edureka.co/mapreduce-design-patterns