SparkR allows users to perform large-scale data analysis from R. It provides a unified approach for working with big data using DataFrames. The current release of SparkR includes DataFrame support released in Spark 1.4 for improved performance. Future developments will expand SparkR's capabilities for large-scale machine learning, simple parallel programming APIs, and tighter integration with existing R packages. The open source project has an growing community of over 20 contributors working to advance SparkR.

1. SparkR
The Past, Present and Future
Shivaram Venkataraman

2. Big Data & R
DataFrames
Visualization
Libraries
Data+

3. Big Data & R
Big Data
Small Learning
Partition
Aggregate
Large Scale
Machine Learning

4. 1. Big Data, Small Learning
Data
Cleaning
Filtering
Aggregation
Collect
Subset
DataFrames
Visualization
Libraries

5. 2(a). Partition Aggregate
Data
Collect
Subset
Best
Model
Params
Parameter Tuning

6. 2(b). Partition Aggregate
Data Combine
Models
Model Averaging

7. 3. Large Scale Machine Learning
Data Featurize Learning Model

8. Big Data & R
Big Data
Small Learning
Partition
Aggregate
Large Scale
Machine Learning
SparkR:
Unified approach

9. Outline
Project History
Current Release
SparkR Future

10. Speed
Scalable
Flexible
Statistics
Visualization
DataFrames

11. RDD
Parallel Collection
Transformations
map
filter
groupBy
…
Actions
count
collect
saveAsTextFile
…

12. R + RDD =
RRDD
lapply
lapplyPartition
groupByKey
collect
cache
…
broadcast
includePackage
textFile

13. Example: Word Count
library(SparkR)
lines
<-­‐
textFile(sc,
“hdfs://my_text_file”)
words
<-­‐
flatMap(lines,
function(line)
{
strsplit(line,
"
")[[1]]
})
wordCount
<-­‐
lapply(words,
function(word)
{
list(word,
1L)
})
counts
<-­‐
reduceByKey(wordCount,
"+",
2L)
output
<-­‐
collect(counts)

14. Initial Prototype
Standalone R package
Install from github

15. Open Source
Development
1. Architecture
2. Usability

16. Architecture
Local Worker
Worker

17. Architecture
Local Worker
Worker
R

18. Architecture
Local Worker
Worker
R
Spark
Context
Java
Spark
Context
R-JVM
bridge

19. Architecture
Local Worker
Worker
R
Spark
Context
Java
Spark
Context
R-JVM
bridge
Spark
Executor
exec
R
Spark
Executor
exec
R

20. Architecture
Local Worker
Worker
R
Spark
Context
Java
Spark
Context
R-JVM
bridge
Spark
Executor
exec
R
Spark
Executor
exec
R

21. R-JVM Bridge
Layer to call JVM
methods directly from R
Automatic argument
serialization
result
<-­‐
callJStatic(
“sparkr.RRDD”,
“someMethod”,
arg1,
arg2)

22. R-JVM Bridge
Use sockets for
communication
Supported across
platforms, languages
R
JVM
Netty
Server

23. Usability
Need for Data Inputs
Read in CSV, JSON, JDBC etc.
High-level API for data manipulation

24. SparkR DataFrames
people
<-­‐
read.df(
“people.json”,
“json”)
avgAge
<-­‐
select(
df,
avg(df$age))
head(avgAge)
DataSources API
Support for schema
dplyr-like syntax

25. SparkR DataFrames
Scala Optimizations
Released in Spark 1.4 !
0
1
2
3
SparkR DataFrame
Scala DataFrame
Python DataFrame
Time (s)
Demo: github.com/cafreeman/SparkR_DataFrame_Demo

26. SparkR Future

27. Big Data & R
Big Data
Small Learning
Partition
Aggregate
Large Scale
Machine Learning

28. Big Data, Small Learning
SparkR DataFrames: Read input, aggregation
Collect results, apply machine learning
Upcoming features:
Support for R transformations
More column functions (e.g. math, strings)

29. Partition Aggregate
Upcoming feature:
Simple, parallel API for SparkR
Ex: Parameter tuning, Model Averaging
Integrated with DataFrames
Use existing R packages

30. Large Scale Machine Learning
Integration with MLLib
Support for GLM, KMeans etc.
model
<-­‐
glm(
a
~
b
+
c,
data
=
df)

31. Large Scale Machine Learning
Key Features
DataFrame inputs
R-like formulas
Model statistics
model
<-­‐
glm(
a
~
b
+
c,
data
=
df)
summary(model)

32. Extensibility
Existing data sources
R package support on
spark-packages.org
Example packages
./bin/sparkR
-­‐-­‐packages
spark-­‐csv

33. Developer Community
>20 contributors including
AMPLab, Databricks, Alteryx, Intel
R and Scala contributions welcome !

34. SparkR
Big data processing from R
DataFrames in Spark 1.4
Future: Large Scale ML & more