Spark 3.0 will include several new features including resolving over 2000 issues, a data source API with catalog support, adaptive query execution, dynamic partition pruning, ANSI SQL compliance, and running on Kubernetes. It will also improve query optimization using rules, costs, and runtime information. Koalas enables working with big data in Python using pandas APIs on Spark DataFrames. Delta Lake provides a standard for building data lakes using a transactional storage layer based on Parquet that ensures data reliability and supports both batch and streaming workloads.

1. New Developments
in Open Source Ecosystem:
Apache Spark 3.0, Koalas, Delta Lake
李 潇 (@gatorsmile)
2019-09-26 @ Hangzhou

2. Dr. Matei Zaharia

3. Spark 起源于加州伯克利分校
Spark 原创团队成立 Databricks
Spark 3.0
Spark 被捐赠给 Apache Foundation
Spark 开源
Spark 成为 Apache 顶级项目
2009 2010 2011 2012 2013 2015 2017 20182014 20192016
Spark 2.0 发布
10 Years

4. Stackoverflow: Q & A for professional and enthusiast programmers, 9.4 m visits/day
Source: https://insights.stackoverflow.com/trends?tags=pyspark%2Capache-flink%2Chadoop%2Capache-spark%2Chbase%2Capache-kafka%2Capache-beam%2Chive%2Ccassandra
Apache Spark
Apache Hadoop
PySpark
Apache Kafka
Apache Hive
Apache Cassandra
Apache Flink
Apache Hbase
Apache Beam
10年 累计排名
Apache Spark
PySpark
Apache Kafka
Apache Hive
Apache Hadoop
Apache Cassandra
Apache Beam
Apache Flink
Apache Hbase
当前 排名

5. 88,615 responses
最受欢迎的平台之一：
开 发 者 调 查 报 告

6. 3.0

7. Will resolve 2000+ JIRAs
Data Source API with
Catalog Supports
Adaptive Query
Execution
Spark Graph Accelerator-aware
Scheduling
Dynamic Partition
Pruning
ANSI SQL
Compliance
SQL Hints
Spark on
Kubernetes
JDK 11 Hadoop 3Vectorization in
SparkR
Scala 2.12

8. Query Optimization in Spark 2.X
Missing Statistics (一次性计算 ETL workloads)
Out-of-date Statistics (存储与计算分离)
Misestimated Costs (多样部署环境 + UDFs)

9. Query Optimization
Spark 1.x, Rule
Spark 2.x, Rule + Cost
Spark 3.0, Rule + Cost + Runtime

10. Dynamic Partition Pruning
Project
Join
Filter
Scan Scan
t1: a large fact table
with many partitions
t2.id < 2
t2: a dimension
table with a filter
SELECT t1.id, t2.pKey
FROM t1
JOIN t2
ON t1.pKey = t2.pKey
AND t2.id < 2
t1.pKey = t2.pKey

11. Project
Join
Filter + Scan Scan
t1.pKey = t2.pKey
Scan all the partitions
t2.id < 2
Filter pushdown
Filter
t1.pkey IN (
SELECT t2.pKey
FROM t2
WHERE t2.id < 2)
SELECT t1.id, t2.pKey
FROM t1
JOIN t2
ON t1.pKey = t2.pKey
AND t2.id < 2
Dynamic Partition Pruning

12. Project
Join
Filter + Scan Filter + Scan
Scan the affected partitions
t2.id < 2
Filter pushdown
t1.pKey in DPPFilterResult
Speed: 33 X faster Scan: 90+% less
Dynamic Partition Pruning
t1.pKey = t2.pKey

13. Adaptive Query Execution
adaptive planning

14. Adaptive Query Execution
SELECT *
FROM t1
JOIN t2
ON t1.key = t2.key
AND t2.col2 LIKE '9999%'
SortMergeJoin
Exchange
Scan Filter + Scan
t2.col2 LIKE
‘9999%’
t1.key = t2.key
SortSort
Exchange
QueryStrage0QueryStrage1

15. Adaptive Query Execution
SELECT *
FROM t1
JOIN t2
ON t1.key = t2.key
AND t2.col2 LIKE '9999%'
BroadcastHashJoin
Exchange
Scan Filter + Scan
t2.col2 LIKE
‘9999%’
t1.key = t2.key
BroadcastExchange
Exchange
QueryStrage0QueryStrage1
QueryStrage2

16. 生 态

17. Dr. Ion Stoica & Dr. Matei Zaharia

18. 18
BUSINESS INTELLIGENCE BIG DATA ANALYTICS UNIFYING DATA + AI
Autonomous Decisions
Advanced ML / AI - Make the
decision for me
Spark as a unifying technology -
Data processing at scale +
analytics + AI
Data Warehouses
Reactive Analytics - What just
happened?
Predictive Analytics
Proactive Analytics - Predict
what’s going to happen

19. 数 据 科 学数 据 工 程

20. Accelerate innovation by unifying data
science and engineering
数 据 科 学数 据 工 程

21. 数 据 科 学

22. Image source: Stack Overflow
Traffic for major programming languages

23. Image source: pypistats.org
koalas
koalas
PySpark

24. Column df['col'] df['col']
Mutability Mutable Immutable
Add a
column
df['c'] = df['a'] + df['b'] df.withColumn('c', df['a'] + df['b'])
Rename
columns
df.columns = ['a','b'] df.select(df['c1'].alias('a'), df['c2'].alias('b'))
Value
count
df['col'].value_counts()
df.groupBy(df['col']).count()
.orderBy('count', ascending = False)
pandas
DataFrame
PySpark
DataFrame

25. import pandas as pd
df = pd.read_csv("my_data.csv")
df.columns = ['x', 'y', 'z1']
df['x2'] = df.x * df.x
df = (spark.read
.option("inferSchema", "true")
.option("comment", True)
.csv("my_data.csv"))
df = df.toDF('x', 'y', 'z1')
df = df.withColumn('x2', df.x*df.x)
pandas PySpark

26. import databricks.koalas as ks
df = ks.read_csv("my_data.csv")
df.columns = ['x', 'y', 'z1']
df['x2'] = df.x * df.x
import pandas as pd
df = pd.read_csv("my_data.csv")
df.columns = ['x', 'y', 'z1']
df['x2'] = df.x * df.x

27. 数 据 工 程

28. A New Standard for Building Data Lakes
Open Format Based on Parquet
With Transactions
Apache Spark™ APIs

29. Instead of parquet … … simply say delta
dataframe
.write
.format("parquet")
.save("/data")
dataframe
.write
.format("delta")
.save("/data")

30. Delta Lake ensures data reliability
Transactional
Log
Parquet Files
Streaming
● ACID Transactions
● Schema Enforcement
● Unified Batch & Streaming
● Time Travel/Data Snapshots
Key Features
High Quality & Reliable Data
always ready for analytics
Batch
Updates
Deletes
Merges

31. Improved reliability:
Petabyte-scale jobs
10x lower compute:
640 服 务 器 → 64!
Simpler, faster ETL:
84 jobs → 3 jobs
halved data latency

32. 32
Easier transactional
updates:
No downtime or
consistency issues!
Simple CDC:
Easy with MERGE
Improved performance: Queries run faster
大 于 一 小 时 → 少 于 六 秒

33. 33
Databricks Cluster
Data consistency and
integrity:
not available before
Increased data quality:
name match accuracy up
80% → 95%
Faster data loads:
一 天 → 二十分钟

34. Databricks Delta in Numbers!
日均 38+ PB
在 客 户 生 产 环 境
处 理 数 据 量
日均 400+ TB
某 一 大 型 客 户 的 生 产 环 境
新 入 库 数 据 量

36. koalas

37. Thank You
李潇 (lixiao@databricks.com)