Processing data at scale usually results in struggling with performance, strict SLA, limited hardware etc. I've struggled with cutting Spark SQL query run-time and found the culprit! This culprit, and SOLUTION! I would like to share with you. Today in the world of Big Data and Spark we are processing high volume transactions. Catalyst is the Spark SQL query optimizer and in this talk, you will learn how to fully utilize Catalyst optimization power in order to make our queries as fast as possible, by pushing down actions and trying to avoid UDFs as much as possible and maximizing performance.

1. Light Up the Spark in Catalyst by
avoiding UDFs
Adi Polak
Milan | November 29 - 30, 2018

2. About me – Adi Polak
@adipolak
adi.polak@Microsoft.com
https://medium.com/@adipolak

3. Agenda
scala
Apache
Spark 2.3
UDF
Catalyst
Optimization

4. Apache Spark is distributed
general-purpose cluster-computing
framework
User-Defined Functions -
Column-based functions that
extend the vocabulary
of Spark SQL's DSL
Catalyst optimization allows some advanced
programming language features that allow
you to build an extensible query optimizer
Catalyst - Apache
Spark SQL query
optimizer
Concepts

5. Who use Apache Spark
Data scientists
Data engineers
Business analysts

6. Data analytics generic architecture

7. Structured
Streaming
MLib GraphFrame TensorFrames
SQL SparkSession / DataFrame / Dataset APIs
Data Source Connectors
Catalyst Optimization & Tungsten Execution
Spark Core (RDD APIs)
SQL

8. Lets look at the demo

10. But, why?

11. CATALYST

12. Fundamentals of Catalyst Optimizer
SUB
Attribute(x) SUB
some_func(1) some_func(2)
Tree Rules
SUB
Attribute(x) some_func(-1)

13. Spark SQL Execution Plan
Logical optimization –> Optimization rules
• Constant folding
• Predicate pushdown
• Projection pruning
• …
Physical Planning –> Planning strategies
Catalyst
Frontend Backend

14. SQL query with custom UDF

15. "Use the higher-level standard Column-based functions with
Dataset operators whenever possible before reverting to
using your own custom UDF functions since UDFs are a
blackbox for Spark and so it does not even try to optimize
them."
SQL query with custom UDF

16. "Use the higher-level standard Column-based functions with
Dataset operators whenever possible before reverting to
using your own custom UDF functions since UDFs are a
blackbox for Spark and so it does not even try to optimize
them."
SQL query with custom UDF

17. "Use the higher-level standard Column-based functions with
Dataset operators whenever possible before reverting to
using your own custom UDF functions since UDFs are a
blackbox for Spark and so it does not even try to optimize
them."
SQL query with custom UDF

18. What do we lose when
using Custom UDF ?
• Constant folding
• Predicate pushdown

19. What can we do ?

20. Use queryExecution & explain(true)
Catalyst
Frontend Backend

22. Use queryExecution & explain(true) API
My UDF
Register

23. Lost Push Down filter

24. What can be done instead?
sql functions DataFrame API:
Aggregate functions
Collection functions
Date time functions
Math functions
Non-aggregate functions
Sorting functions
String functions
Window functions
sql functions Column API
Expression operations..

25. How can I find what functions are available?
arrayContains, minute, round, rand, spark_partition_id, isin …
version

26. 0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Diamonds (53,940) users_filtered (115,134) BlackFriday (302,675) UserList (537,577)
Runtime Benchmarks
RunTime UDF (sec) RunTime Functions (sec)

27. Takeaways
• Use UDFs as a last resort
• Avoid UDFs or UDAFs that perform more than one thing
• Look under the hood –
• Analyzing Spark’s execution plan with .explain(true)

28. Reference
Azure Databricks Tutorial
• https://aka.ms/AA3e7rd
Apache Spark deep dive :
• https://aka.ms/AA3efgo
Kaggle:
• https://www.kaggle.com

29. GRATZIE
@adipolak
@adipolak
Adi.polak@Microsoft.com