This document summarizes new directions for Spark in 2015, including developing high-level interfaces for data science similar to single-machine tools, platform interfaces to plug in external data sources and algorithms, machine learning pipelines inspired by scikit-learn, a R interface for Spark, and community packages of third-party libraries. The goal is to create a unified engine for Spark that can handle a variety of data sources, workloads, and environments.

1. New Directions for Spark in 2015
Matei Zaharia
March 18, 2015

2. 2014: an Amazing Year for Spark
Total contributors: 150 => 500
Lines of code: 190K => 370K
500+ active production deployments
2

3. 0
20
40
60
80
100
120
140
2011 2012 2013 2014 2015
Contributors per Month to Spark
Most active project in big data
3

4. 4
On-Disk Sort Record:
Time to sort 100TB
Source: Daytona GraySort benchmark, sortbenchmark.org
2100 machines2013 Record:
Hadoop
72 minutes
2014 Record:
Spark
207 machines
23 minutes

5. Major Additions in 2014
5
Spark SQL
Java 8 syntax
Python streaming
…
GraphX
Random forests
Streaming MLlib

6. 6
New Directions in 2015
Data Science
High-level interfaces similar
to single-machine tools
Platform Interfaces
Plug in data sources
and algorithms

7. 7
DataFrames
Similar API to data frames
in R and Pandas
Automatically optimized
via Spark SQL
Out in Spark 1.3
df = jsonFile(“tweets.json”)
df[df[“user”] == “matei”]
.groupBy(“date”)
.sum(“retweets”)
0
5
10
Python Scala DataFrame
RunningTime

8. 8
Machine Learning Pipelines
High-level API inspired by
SciKit-Learn
Featurization, evaluation,
parameter search
tokenizer = Tokenizer()
tf = HashingTF(numFeatures=1000)
lr = LogisticRegression()
pipe = Pipeline([tokenizer, tf, lr])
model = pipe.fit(df)
tokenizer TF LR
modelDataFrame

9. 9
R Interface (SparkR)
Targeting Spark 1.4 (June)
Exposes DataFrames,
RDDs, and ML library in R
df = jsonFile(“tweets.json”)
summarize(
group_by(
df[df$user == “matei”,],
“date”),
sum(“retweets”))

10. 10
New Directions in 2015
Data Science
High-level interfaces similar
to single-machine tools
Platform Interfaces
Plug in data sources
and algorithms

11. 11
External Data Sources
Platform API to plug smart
data sources into Spark
Returns DataFrames usable
in Spark apps or SQL
Pushes logic into sources
Spark
{JSON}

12. 12
External Data Sources
Platform API to plug smart
data sources into Spark
Returns DataFrames usable
in Spark apps or SQL
Pushes logic into sources
SELECT * FROM mysql_users u JOIN
hive_logs h
WHERE u.lang = “en”
Spark
{JSON}
SELECT * FROM users WHERE lang=“en”

13. 13
Spark Packages
Community index of third
party packages
bin/spark-shell --packages
databricks/spark-csv:0.2
spark-packages.org

14. 14
Spark Core
Spark
Streaming
Spark SQL MLlib GraphX

15. 15
Spark Core
DataFrames ML Pipelines
Spark
Streaming
Spark SQL MLlib GraphX

16. 16
{JSON}
Data Sources
Spark Core
DataFrames ML Pipelines
Spark
Streaming
Spark SQL MLlib GraphX

17. 17
{JSON}
Data Sources
Spark Core
DataFrames ML Pipelines
Spark
Streaming
Spark SQL MLlib GraphX

18. 18
Goal: unified engine across data sources,
workloads and environments

19. 19
Enjoy Spark Summit East!