Dataset и Dataframe стали предпочтительными интерфейсами работы со Spark. Во многом благодаря активной разработке оптимизатора запросов Catalyst. В докладе мы рассмотрим мотивацию создания Spark.SQL и поймем, почему он так критически важен для работы PySpark. А так же подробно разберем как устроен Catalyst изнутри и как можно расширить его функциональность.

1. Павел Клеменков
p.klemenkov@rambler-co.ru
Погружаемся в Catalyst

3. rdd.map(lambda x: x)

4. 192.168.0.38 WARNING Something bad could
happen
192.168.0.88 INFO Just an info message
passing by
192.168.0.5 WARNING Something bad could
happen
192.168.0.36 ERROR When production
fails in despair,
whom you're gonna
call?
192.168.0.27 INFO Just an info message
passing by

5. 192.168.0.38 USA
192.168.0.88 RUSSIA
192.168.0.5 CHINA
192.168.0.36 USA
192.168.0.27 RUSSIA

6. SELECT country, code FROM table1
JOIN table2
WHERE table1.ip = table2.ip
AND table1.code != "INFO"

7. rdd1 = sc.textFile()
rdd2 = sc.textFile()
table1 = rdd1.map(lambda x: x.split("t"))
table2 = rdd2.map(lambda x: x.split("t"))
table1.join(table2)
.map(lambda (x, y): (y[1], y[0]))
.filter(lambda (x, y): y != "INFO")
.collect()
cхема
вычисления

8. SELECT country,
count(country) as messages
FROM table1 JOIN table2
WHERE table1.ip = table2.ip
AND table1.code != "INFO"
GROUP BY country

9. spark.sql(query).explain(True)
== Parsed Logical Plan ==
...
== Analyzed Logical Plan ==
...
== Optimized Logical Plan ==
...
== Physical Plan ==
...

10. Catalyst pipeline

11. abstract class TreeNode[BaseType <: TreeNode[BaseType]]
extends Product

12. case class Literal (value: Any, dataType: DataType)
extends LeafExpression with CodegenFallback
abstract class Attribute
extends LeafExpression with NamedExpression
with NullIntolerant
case class Add(left: Expression, right: Expression)
extends BinaryArithmetic with NullIntolerant

13. x + (1 + 2) Attrib(x) Add
Literal(1)Literal(2)
Add

14. abstract class TreeNode[BaseType <: TreeNode[BaseType]]
extends Product {
def transform(rule: PartialFunction[BaseType, BaseType]):
BaseType = {}
}

15. tree.transform {
case Add(Literal(c1), Literal(c2)) => Literal(c1+c2)
}
Attrib(x) Add
Literal(1)Literal(2)
Add
Attrib(x) Literal(3)
Add

16. SELECT country,
count(country) as messages
FROM table1 JOIN table2
WHERE table1.ip = table2.ip
AND table1.code != "INFO"
GROUP BY country
Scan
Join
Filter
Project
Aggregate
Scan

17. object PushDownPredicate extends Rule[LogicalPlan]
with PredicateHelper {
def apply(plan: LogicalPlan): LogicalPlan = plan transform {}
}

18. Scan
Join
Filter
Project
Aggregate
Scan Scan
Join
Filter
Project
Aggregate
Scan

19. Scan
Join
Filter
Project
Aggregate
Scan FileScan
BroadcastHashJoin
Filter
Project
HashAggregate
FileScan

20. Пример

21. SELECT country,
count(country) as messages
FROM table1 JOIN table2
WHERE table1.ip = table2.ip
AND table1.code != "INFO"
GROUP BY country
Scan
Join
Filter
Project
Aggregate
Scan

22. Catalyst pipeline

23. == Parsed Logical Plan ==
'Aggregate ['country], ['country, 'count('country) AS
messages#14]
+- 'Filter (('table1.ip = 'table2.ip) && NOT
('table1.code = INFO))
+- 'Join Inner
:- 'UnresolvedRelation `table1`
+- 'UnresolvedRelation `table2`

24. == Parsed Logical Plan ==
'Aggregate ['country], ['country, 'count('country) AS
messages#14]
+- 'Filter (('table1.ip = 'table2.ip) && NOT
('table1.code = INFO))
+- 'Join Inner
:- 'UnresolvedRelation `table1`
+- 'UnresolvedRelation `table2`

25. Catalyst pipeline

26. == Analyzed Logical Plan ==
country: string, messages: bigint
Aggregate [country#8], [country#8, count(country#8) AS
messages#14L]
+- Filter ((ip#0 = ip#7) && NOT (code#1 = INFO))
+- Join Inner
:- SubqueryAlias table1
: +- Relation[ip#0,code#1,msg#2] parquet
+- SubqueryAlias table2
+- Relation[ip#7,country#8] parquet

27. Catalyst pipeline

28. Filter pushdown

29. == Optimized Logical Plan ==
Aggregate [country#8], [country#8, count(country#8) AS
messages#14L]
+- Project [country#8]
+- Join Inner, (ip#0 = ip#7)
:- Project [ip#0]
: +- Filter ((isnotnull(code#1) && NOT (code#1 =
INFO)) && isnotnull(ip#0))
: +- Relation[ip#0,code#1,msg#2] parquet
+- Filter isnotnull(ip#7)
+- Relation[ip#7, country#8] parquet

30. Catalyst pipeline

31. == Physical Plan ==
*HashAggregate(
+- Exchange hashpartitioning(country#8, 200)
+- *HashAggregate
+- *Project [country#8]
+- *BroadcastHashJoin [ip#0], [ip#7], Inner,
BuildRight
:- *Project [ip#0]
: +- *Filter
: +- *FileScan parquet [ip#0,code#1],
PushedFilters: [IsNotNull(code),
Not(EqualTo(code,INFO)), IsNotNull(ip)]
+- BroadcastExchange
+- *Project [ip#7, country#8]
+- *Filter isnotnull(ip#7)
+- *FileScan parquet [ip#7,country#8],
PushedFilters: [IsNotNull(ip)]

32. Column pruning

33. == Physical Plan ==
*HashAggregate(
+- Exchange hashpartitioning(country#8, 200)
+- *HashAggregate
+- *Project [country#8]
+- *BroadcastHashJoin [ip#0], [ip#7], Inner,
BuildRight
:- *Project [ip#0]
: +- *Filter
: +- *FileScan parquet [ip#0,code#1],
PushedFilters: [IsNotNull(code),
Not(EqualTo(code,INFO)), IsNotNull(ip)]
+- BroadcastExchange
+- *Project [ip#7, country#8]
+- *Filter isnotnull(ip#7)
+- *FileScan parquet [ip#7,country#8],
PushedFilters: [IsNotNull(ip)]

34. СПАСИБО!