Adi Polak, a Senior Software Engineer at Akamai, discusses the drawbacks of using custom User Defined Functions (UDFs) in Apache Spark and advocates for leveraging higher-level standard column functions whenever possible to enhance optimization. The document highlights the importance of understanding Spark's Catalyst optimization framework and provides recommendations for maintaining performance, including utilizing built-in functions and analyzing execution plans. Key conclusions urge developers to treat UDFs as a last resort and to verify their impact on performance using the 'explain' method.

1. Adi Polak, Sr. Software Engineer @ Akamai
Spark UDFs are EviL,
Catalyst to the rEsCue!

2. • Adi Polak
• Sr. Software Engineer @ Akamai
• Previous Security researcher
• Majored in Machine Learning
• Tel Avivian
• BGU alumni
• Co-funder of FLIP
• Spark & Scala enthusiast
• Foodie
Who am I
@adipolak
@adipolak

4. • Apache Spark with Scala
• Spark 1.6
• Catalyst optimization
• Spark custom UDFs
‫תאכלס‬..

6. CATALYST

7. Fundamentals of Catalyst Optimizer
SUB
Attribute(x) SUB
some_func(1) some_func(2)
SUB
Attribute(x) some_func(-1)

8. Spark SQL Execution Plan
Logical optimization –> Optimization rules
• Constant folding
• Predicate pushdown
• Projection pruning
• …
Physical Planning –> Planning strategies
Catalyst
Frontend Backend

9. What is Spark Custom UDF

10. What is Spark Custom UDF
"Use the higher-level standard Column-based functions with
Dataset operators whenever possible before reverting to
using your own custom UDF functions since UDFs are a
blackbox for Spark and so it does not even try to optimize
them."

11. What is Spark Custom UDF
"Use the higher-level standard Column-based functions with
Dataset operators whenever possible before reverting to
using your own custom UDF functions since UDFs are a
blackbox for Spark and so it does not even try to optimize
them."

12. What is Spark Custom UDF
"Use the higher-level standard Column-based functions with
Dataset operators whenever possible before reverting to
using your own custom UDF functions since UDFs are a
blackbox for Spark and so it does not even try to optimize
them."

13. What do we lose when
using Custom UDF ?
• Constant folding
• Predicate pushdown
• Null handling

14. What can we do ?

15. Use queryExecution & explain(true)
Catalyst
Frontend Backend

17. Use queryExecution & explain(true) API

18. Lost Push Down filter

19. What can be done instead?
sql functions DataFrame API:
Aggregate functions
Collection functions
Date time functions
Math functions
Non-aggregate functions
Sorting functions
String functions
Window functions
sql functions Column API
Expression operations..

20. How can I find what functions are available?
version
arrayContains, minute, round, rand, spark_partition_id, isin …

21. Can you show a complex example?

22. Using column functions ...

23. TEST - BENCHMARKS
Total output rows – 100
• Average execution time per row with UDF ~ 299.9 millis => 26sec
• Average execution time per row without one of the UDFs ~ 265.93 millis => 23sec

24. Takeaways
• Use UDFs as a last resort
• Always check yourself with df.explain(true)
• Act accordingly

25. THANK YOU
@adipolak
@adipolak