Skewed data is the enemy when joining tables using Spark. It shuffles a large proportion of the data onto a few overloaded nodes, bottlenecking Spark’s parallelism and resulting in out of memory errors. The go-to answer is to use broadcast joins; leaving the large, skewed dataset in place and transmitting a smaller table to every machine in the cluster for joining. But what happens when your second table is too large to broadcast, and does not fit into memory? Or even worse, when a single key is bigger than the total size of your executor? Firstly, we will give an introduction into the problem. Secondly, the current ways of fighting the problem will be explained, including why these solutions are limited. Finally, we will demonstrate a new technique – the iterative broadcast join – developed while processing ING Bank’s global transaction data. This technique, implemented on top of the Spark SQL API, allows multiple large and highly skewed datasets to be joined successfully, while retaining a high level of parallelism. This is something that is not possible with existing Spark join types.

1. Rob Keevil - KnowIT
Fokko Driesprong- GoDataDriven
Working with Skewed Data:
The Iterative Broadcast
#EUde11

2. #EUde11
About Rob Keevil
• Philosophy, Politics & Economics
• Big Data Architect
• Financial and Anti-Fraud Domains
• Scala Programmer

3. #EUde11
About Fokko Driesprong
• Software Engineering and Distributed Systems
• Data Engineer at GoDataDriven
• Committer & PPMC Member, Apache Airflow
• Open source enthusiast
https://godatadriven.com/players/fokko-driesprong

4. #EUde11
Overview
• The Skew Problem at ING
• Join Strategies in Spark
• Our Solution
• Performance Analysis
• Considerations
• Future Work

5. #EUde11
The Skew Problem at
• Billions of transactions…
• Billions of accounts…
• Millions of companies…
• …with a very uneven distribution!

6. #EUde11
Familiar?

7. #EUde11
Join Types Native to Spark
• Sort Merge Join
• Broadcast Hash Join
• Shuffled Hash Join
• Broadcast Nested Loop Join
• Cartesian Product

8. Key Column A
1 HELLO
The Sort Merge Join
8#EUde11
Stage 1:
Determine
partitions
by hashing
the key(s)
3 GOOD
Key Column A
Key Column BKey Column B
1 WORLD!
2 SUMMIT
1 DUBLIN
2 STREAMING
1 EVERYBODY
2 SQL
3 MORNING
3 EVENING
3 NIGHT
4 SPAM
4 SPAM
4 SPAM
2 SPARK
4 SPAM

9. Executor 1
Key Column B
1 WORLD!
1 DUBLIN
1 EVERYBODY
3 MORNING
3 EVENING
3 NIGHT
Key Column A Column B
1 WORLD!
1 DUBLIN
1 EVERYBODY
3 MORNING
3 EVENING
3 NIGHT
3 GOOD3 GOOD3 GOOD3 GOOD
Key Column A
Executor 2
9#EUde11
Stage 2:
Merge
Key Column B
2 SUMMIT
2 STREAMING
2 SQL
4 SPAM
4 SPAM
4 SPAM
Key Column AKey Column A
1 HELLO1 HELLO1 HELLO1 HELLO
Key Column A Column B
2 SUMMIT
2 STREAMING
2 SQL
4 SPAM
4 SPAM
4 SPAM
2 SPARK2 SPARK2 SPARK
4 SPAM4 SPAM4 SPAM
2 SPARK
4 SPAM

10. Executor 1 Executor 2
10#EUde11
Key Column A Column B
1 HELLO WORLD!
1 HELLO DUBLIN
1 HELLO EVERYBODY
3 GOOD MORNING
3 GOOD EVENING
3 GOOD NIGHT
Key Column A Column B
2 SPARK SUMMIT
2 SPARK STREAMING
2 SPARK SQL
4 SPAM SPAM
4 SPAM SPAM
4 SPAM SPAM

11. 11

12. Key Column A
1 HELLO
12#EUde11
Stage 1:
Determine
partitions
by hashing
the key(s)
3 GOOD
Key Column A
2 SPARK
4 SPAM
Key Column BKey Column B
4 SPAM
2 SUMMIT
3 MORNING
4 SPAM
1 WORLD!
4 SPAM
4 SPAM
4 SPAM
4 SPAM
4 SPAM
4 SPAM
4 SPAM
Regular Table Skewed Table

13. Executor 2Executor 1
Key Column B
1 WORLD!
3 MORNING
Key Column A Column B
1 WORLD!
3 MORNING
3 GOOD
Key Column A
13#EUde11
Stage 2:
Merge
Key Column B
2 SUMMIT
4 SPAM
4 SPAM
4 SPAM
4 SPAM
4 SPAM
4 SPAM
4 SPAM
4 SPAM
4 SPAM
Key Column AKey Column A
1 HELLO
Key Column A Column B
2 SUMMIT
4 SPAM
4 SPAM
4 SPAM
4 SPAM
4 SPAM
4 SPAM
4 SPAM
4 SPAM
4 SPAM
2 SPARK
4 SPAM
1 HELLO
3 GOOD 4 SPAM4 SPAM4 SPAM4 SPAM4 SPAM4 SPAM4 SPAM4 SPAM
2 SPARK
4 SPAM

14. Executor 1 Executor 2
14#EUde11
Key Column A Column B
1 HELLO WORLD!
3 GOOD MORNING
Key Column A Column B
2 SPARK SUMMIT
4 SPAM SPAM
4 SPAM SPAM
4 SPAM SPAM
4 SPAM SPAM
4 SPAM SPAM
4 SPAM SPAM
4 SPAM SPAM
4 SPAM SPAM
4 SPAM SPAM

15. Potential Solutions
• Give it more resources?
• Repartition the data?
15#EUde11

16. What about Broadcast Hash Join?
• Leaves the partitions larger table
untouched
• Copies the entire smaller table to
every executor
• Iterate over the large table, and
join using a HashMap
16#EUde11

17. Key Column B
4 SPAM
4 SPAM
4 SPAM
2 SUMMIT
3 MORNING
4 SPAM
Key Column A
1 HELLO
4 SPAM
2 SPARK
3 GOOD
Key Column A
1 HELLO
4 SPAM
2 SPARK
3 GOOD
Key Column A
1 HELLO
4 SPAM
2 SPARK
3 GOOD
Key Column B
4 SPAM
4 SPAM
1 WORLD
4 SPAM
4 SPAM
4 SPAM
17#EUde11
Smaller Table Larger Table
Stage 1:
Broadcast
smaller
dataset to
all
executors
Key Column B
4 SPAM
4 SPAM
4 SPAM
2 SUMMIT
3 MORNING
4 SPAM
4 SPAM
4 SPAM
1 WORLD!
4 SPAM
4 SPAM
4 SPAM

18. Executor 2
3 GOOD
2 SPARK
Executor 1
Key Column B
4 SPAM
4 SPAM
4 SPAM
2 SUMMIT
3 MORNING
4 SPAM
Key Column A Column B
4 SPAM
4 SPAM
4 SPAM
2 SUMMIT
3 MORNING
4 SPAM
3 GOOD
Key Column A
18#EUde11
Stage 2:
Left join
Key Column B
4 SPAM
4 SPAM
1 WORLD!
4 SPAM
4 SPAM
4 SPAM
Key Column A
Key Column A Column B
4 SPAM
4 SPAM
1 WORLD!
4 SPAM
4 SPAM
4 SPAM
2 SPARK
4 SPAM4 SPAM4 SPAM4 SPAM2 SPARK
4 SPAM
3 GOOD
Key Column AKey Column A
1 HELLO
4 SPAM4 SPAM4 SPAM4 SPAM4 SPAM
1 HELLO
4 SPAM
1 HELLO

19. Executor 1 Executor 2
19#EUde11
Key Column A Column B
4 SPAM SPAM
4 SPAM SPAM
4 SPAM SPAM
2 SPARK SUMMIT
3 GOOD MORNING
4 SPAM SPAM
Key Column A Column B
4 SPAM SPAM
4 SPAM SPAM
1 HELLO WORLD!
4 SPAM SPAM
4 SPAM SPAM
4 SPAM SPAM

20. But…
• …What if your “smaller” table
isn’t so small?
• The whole small table needs
to fit in driver memory and
every executor
20#EUde11

21. Introducing: The Iterative Broadcast!
• Divide the smaller table into “passes”
• Broadcast a pass & left join to the larger dataset
• Clear the broadcast partition from memory
• Repeat until all passes are processed
21#EUde11

22. 22#EUde11
Key Column A
1 HELLO
4 SPAM
2 SPARK
3 GOOD
Stage 1:
Assign Pass Number
to Smaller Table
Key Column A Pass
1 HELLO 1
4 SPAM 2
2 SPARK 2
3 GOOD 1

23. Executor 2Executor 1
23#EUde11
Stage 2:
Broadcast
first pass
Key Column B
4 SPAM
4 SPAM
1 WORLD!
4 SPAM
4 SPAM
4 SPAM
Key Column B
4 SPAM
4 SPAM
4 SPAM
2 SUMMIT
3 MORNING
4 SPAM
3 GOOD
1 HELLO
Key Column A Pass
1 HELLO 1
3 GOOD 1
Key Column A Pass
1 HELLO 1
3 GOOD 1

24. Executor 2Executor 1
24#EUde11
Stage 3:
Left join
first pass
Key Column B
4 SPAM
4 SPAM
1 WORLD!
4 SPAM
4 SPAM
4 SPAM
Key Column B
4 SPAM
4 SPAM
4 SPAM
2 SUMMIT
3 MORNING
4 SPAM
Key Column A Column B
4 SPAM
4 SPAM
4 SPAM
2 SUMMIT
3 MORNING
4 SPAM
Key Column A Column B
4 SPAM
4 SPAM
1 WORLD!
4 SPAM
4 SPAM
4 SPAM
3 GOOD
1 HELLO
Key Column A Pass
1 HELLO 1
3 GOOD 1
Key Column A Pass
1 HELLO 1
3 GOOD 1

25. Executor 2Executor 1
25#EUde11
Key Column B
4 SPAM
4 SPAM
1 WORLD!
4 SPAM
4 SPAM
4 SPAM
Key Column B
4 SPAM
4 SPAM
4 SPAM
2 SUMMIT
3 MORNING
4 SPAM
Key Column A Column B
4 SPAM
4 SPAM
4 SPAM
2 SUMMIT
3 GOOD MORNING
4 SPAM
Key Column A Column B
4 SPAM
4 SPAM
1 HELLO WORLD!
4 SPAM
4 SPAM
4 SPAM
1 HELLO
Stage 4:
Broadcast
second
pass
Key Column A Pass
1 HELLO 1
3 GOOD 1
Key Column A Pass
1 HELLO 1
3 GOOD 1
Key Column A Pass
2 SPARK 2
4 SPAM 2
Key Column A Pass
2 SPARK 2
4 SPAM 2

26. Executor 2Executor 1
26#EUde11
Key Column B
4 SPAM
4 SPAM
1 WORLD!
4 SPAM
4 SPAM
4 SPAM
Key Column B
4 SPAM
4 SPAM
4 SPAM
2 SUMMIT
3 MORNING
4 SPAM
Key Column A Column B
4 SPAM
4 SPAM
4 SPAM
2 SUMMIT
3 GOOD MORNING
4 SPAM
Key Column A Column B
4 SPAM
4 SPAM
1 HELLO WORLD!
4 SPAM
4 SPAM
4 SPAM
3 GOOD
1 HELLO
Stage 5:
Left join
second
pass
Key Column A Pass
1 HELLO 1
3 GOOD 1
3 SPARK
3 GOOD3 GOOD3 GOOD3 GOOD 3 GOOD3 GOOD3 GOOD3 GOOD3 GOOD3 GOOD
Key Column A Pass
2 SPARK 2
4 SPAM 2
Key Column A Pass
2 SPARK 2
4 SPAM 2

27. 27#EUde11
Performance Analysis
• Setup notes
– Tests on Amazon EMR using 4x m4.4xlarge workers
– 5 cores and 18gb memory per executor
– Spark 2.1
– Benchmark on GitHub
– Skewed dataset, 10000 keys
• Largest key 10000 rows, 2nd key 5000 rows, 3rd key 2500 rows … 10000th key 1 row
– Uniform dataset, evenly spread

28. 28#EUde11
Performance Analysis – Skewed Data

29. 29#EUde11
Performance Analysis – Uniform Data

30. Considerations
• Hyper parameters
– Size of the broadcast variable
– The number of passes
• Additional complexity
30#EUde11

31. Future Work
31#EUde11
• In-memory iterations
• Native to Spark

32. Recap
32#EUde11
• Skew hurts Spark parallelism and stability
• Default join types have issues with skewed data
• Iterative Broadcast Join can be used to process
skewed data while maintaining parallelism

33. That’s All Folks!
https://github.com/godatadriven/iterative-broadcast-join
#EUde11