Successfully reported this slideshow.

Spark SQL Join Improvement at Facebook

1

Share

Loading in …3
×
1 of 23
1 of 23

Spark SQL Join Improvement at Facebook

1

Share

Download to read offline

Join is one of most important and critical SQL operation in most data warehouses. This is essential when we want to get insights from multiple input datasets. Over the last year, we’ve added a series of join optimizations internally at Facebook, and we started to contribute back to upstream open source recently.

Join is one of most important and critical SQL operation in most data warehouses. This is essential when we want to get insights from multiple input datasets. Over the last year, we’ve added a series of join optimizations internally at Facebook, and we started to contribute back to upstream open source recently.

More Related Content

Spark SQL Join Improvement at Facebook

  1. 1. Spark SQL Join Improvement at Facebook Cheng Su Facebook
  2. 2. About Me ▪ Cheng Su ▪ Software Engineer at Facebook (Data Platform Team) ▪ Apache Spark Contributor (Spark SQL) ▪ Previously worked on Hive & Hadoop at Facebook
  3. 3. Agenda ▪ Overview for Spark SQL Joins ▪ Shuffled Hash Join Improvement (SPARK-32461) ▪ Leverage Bloom Filter for Join (under discussion with community) ▪ Stream-stream Join Improvement (SPARK-32883) ▪ Future Work
  4. 4. Overview for Spark SQL Joins ▪ SQL Join Physical Operators ▪ BroadcastHashJoinExec (broadcast hash join) ▪ ShuffledHashJoinExec (shuffled hash join) ▪ SortMergeJoinExec (sort merge join)
  5. 5. Overview for Spark SQL Joins Data Source A Data Source B . . . . . . . . . .JOIN
  6. 6. Spark Driver - Scan source B - Join with A Broadcast hash join - Ship smaller data source to all nodes - Stream the other side One side should be smaller than spark.sql.autoBroadcastJoinThreshold (default 10 MB). Pros: No shuffle and sort on both sides. No skew. Cons: OOM on driver. Broadcast - Broadcast hash relation on source A
  7. 7. Scan A Scan B . . . . . . . . . . ShuffleShuffleShuffle Join Sort merge join - Shuffle both sides, - Sort both sides, buffer one, stream the bigger one Pros: Handle large data size well on both sides. Cons: Need shuffle and sort. Skew. Sort Join Sort Join Sort
  8. 8. Shuffle hash join - Shuffle both sides, - Hash smaller one, stream the bigger one Disable by default via spark.sql.join.preferSortMergeJoin. One side should be smaller than (spark.sql.autoBroadcastJoinThreshold * spark.sql.shuffle.partitions) (default 10MB * 200), and 3x smaller than the other side Pros: Handle large data size well on one side. Cons: Need shuffle. Skew. OOM on task for building hash table. Scan A Scan B . . . . . . . . . . Build hash table ShuffleShuffleShuffle Join Build hash table Join
  9. 9. Shuffled Hash Join Improvement (SPARK-32461) ▪ Agenda ▪ Code-gen support ▪ Full outer join support ▪ Sort-based fallback mechanism
  10. 10. Code-gen Support for Shuffled Hash Join (SPARK- 32421) ▪ Why? ▪ Save compute resource, improve CPU (whole-stage code-gen) ▪ How? ▪ Refactor broadcast hash join code-gen logic ▪ Broadcast hash join already supports code-gen ▪ Refactor into common parent class for BHJ and SHJ - HashJoin.scala ▪ Performance Improvement ▪ 30% run-time improvement compared to non-code-gen for benchmark query ▪ PR Status ▪ Merged, will be available in Spark 3.1
  11. 11. Full Outer Shuffled Hash Join (SPARK-32399) ▪ Why? ▪ Save compute resource, improve CPU and IO ▪ Only sort merge join supports full outer, and sort is very expensive when table is large and needs to spill to disk. ▪ Shuffled hash join does hash table lookup join, instead of sorting. ▪ How? ▪ Need to record non-matched rows from both sides ▪ Stream side: trivial ▪ Build side: non-trivial, need extra data structure (e.g. hash set for matched rows)
  12. 12. Full outer shuffle hash join - Shuffle both sides, - Hash smaller one, stream the bigger one - Hash set for build side to record matched rows - Iterate build side hash table and output non-matched rows Scan A Scan B . . . . . . . . . . Build hash table ShuffleShuffleShuffle Join Build hash table Join Hash set Hash set
  13. 13. Full Outer Shuffled Hash Join (SPARK-32399) ▪ Performance Improvement ▪ 30% run-time improvement compared to full outer sort merge join for benchmark query ▪ PR Status ▪ Merged, will be available in Spark 3.1
  14. 14. Sort-based fallback mechanism for SHJ (SPARK-32634) ▪ Why? ▪ Build side hash table out of memory ▪ No fallback, no spill, task failure, query failure ▪ Hard to enable shuffled hash join by default given OOM limitation ▪ How? ▪ Introduce fallback when building hash table ▪ Whenever fail to get memory to insert current row to hash table, stop build hash table. Sort both sides and do sort merge join. ▪ PR Status ▪ WIP
  15. 15. Leverage Blook Filter for Join ▪ Why? ▪ Save compute resource, improve CPU and IO for shuffled hash join and sort merge join ▪ How? ▪ Build bloom filter on join key of smaller side ▪ Use bloom filter to filter out rows when scanning larger side ▪ Reduce amount of data to process in followed stages (less data to shuffle/sort/etc) ▪ PR Status ▪ Under discussion with community members, will submit JIRA later
  16. 16. Stream-stream Join Improvement (SPARK-32883) ▪ Agenda ▪ Left semi join support ▪ Full outer join support
  17. 17. Quick Refreshment for Stream-Stream Join ▪ Join Physical Operator ▪ StreamingSymmetricHashJoinExec (stream-stream join for Structured Streaming)
  18. 18. Scan A Scan B . . . . . . . . . . ShuffleShuffleShuffle Stream-stream join - Shuffle both sides, - Join both sides by looking up from each state store State Store B Join State Store A Join State Store B Join State Store A Join State Store B Join State Store A Join
  19. 19. Left semi stream-stream join (SPARK-32862) ▪ Why? ▪ Left semi is even more popular than left outer (observation on some FB streaming workload) ▪ Get all ads impression (left side) which has ads click (right side), but do not care what those ads clicks are ▪ How? ▪ For left side input row, check if there's a match on right side state store ▪ If there's a match, output the left side row, but do not put the row in left side state store (no need to put in state store). ▪ If there's no match, output nothing, but put the row in left side state store (with "matched" field to set to false in state store). ▪ For right side input row, check if there's a match on left side state store. ▪ For all matched left rows in state store, output the rows with "matched" field as false. Set all left rows with "matched" field to be true. Only output the left side rows matched for the first time to guarantee left semi join semantics. ▪ State store eviction: evict rows from left/right side state store below watermark, same as inner join. ▪ PR Status ▪ Merged, will be available in Spark 3.1
  20. 20. Full outer stream-stream join (SPARK-32863) ▪ How? ▪ for left side input row, check if there's a match on right side state store. If there's a match, output all matched rows. Put the row in left side state store. ▪ for right side input row, check if there's a match on left side state store. If there's a match, output all matched rows and update left side rows state with "matched" field to set to true. Put the row in right side state store. ▪ for left side row needs to be evicted from state store, output the row if "matched" field is false. ▪ for right side row needs to be evicted from state store, output the row if "matched" field is false. ▪ PR Status ▪ WIP
  21. 21. Future Work ▪ History-based optimization (HBO) to select best join strategy ▪ Decide between broadcast hash join, shuffled hash join, and sort merge join based on historical join input size
  22. 22. Summary ▪ Shuffled Hash Join Improvement (SPARK-32461) ▪ Leverage Bloom Filter for Join (under discussion with community) ▪ Stream-stream Join Improvement (SPARK-32883)
  23. 23. Thank you! Your feedback is important to us. Don’t forget to rate and review the sessions.

×