R is a hugely popular platform for Data Scientists to create analytic models in many different domains. It is simple and ubiquitous. And a large number of readily available packages make it very powerful for statistical computing. But when these applications should move from the science lab to the production environment of large enterprises a new set of challenges arises. Independently of R, Spark has been very successful as a powerful general-purpose computing platform that leverages clusters of computers and is able to process data at a scale that has not been feasible before. With the introduction of SparkR an exciting new option to productionize Data Science applications has been made available. This talk will give insight into two real-life projects at major enterprises where Data Science applications in R have been migrated to SparkR. Suggested Topics: • Dealing with platform challenges: R was not installed on the cluster. We show how to execute SparkR on a Yarn cluster with a dynamic deployment of R. • Integrating Data Engineering and Data Science: we highlight the technical and cultural challenges that arise from closely integrating these two different areas. • Separation of concerns: we describe how to disentangle ETL and data preparation from analytic computing and statistical methods. • Scaling R with SparkR: we present what options SparkR offers to scale R applications and how we applied them to different areas such as time series forecasting and web analytics. • Performance Improvements: we will show benchmarks for an R applications that took over 20 hours on a single server/single-threaded setup. With moderate effort we have been able to reduce that number to 15 minutes with SparkR. And we will show how we plan to further reduces this to less than a minute in the future. • Mixing SparkR, SparkSQL and MLlib: we show how we combined the three different libraries to maximize efficiency. Summary and Outlook: we describe what we have learnt so far, what the biggest gaps currently are and what challenges we expect to solve in the short- to mid-term.

1. Scaling SparkR in Production.
Lessons from the Field.
Heiko Korndorf
Wireframe, CEO & Founder

2. About me
Heiko Korndorf
• CEO & Founder Wireframe
• MS in Computer Science
• Application Areas: ERP, CRM, BI, EAI
• Serving companies in
• Manufacturing
• Telecommunications
• Financial Services
• Utilities
• Oil & Gas
• Professional Services
Rapid Application Development
for Hadoop/Spark
Test Data Generation/Simulation

3. What we’ll talk about
Classify this talk ….
• Data Science: Scaling your R application with SparkR
• Data Engineering: How to bring Data Science applications into
your production pipelines, i.e. adding R to your toolset.
• Management: Integrating Data Science and Data Engineering with
SparkR

4. Agenda
• SparkR Architecture 1.x/2.x
• Reference Projects I + II
• Approach with Spark 1.5/1.6
• Parallelization via YARN
• Dynamic R Deployment, incl. dependencies/packages
• Approach with Spark 2.0
• Parallelization via SparkR
• R-Graphics: headless environment, concurrency
• Use Spark APIs: SQL, Mllib
• On-Prem vs Cloud (Elasticity/decouple storage and compute)
• Integrating Data Science and Data Engineering
• A Broader Look at the Ecosystem
• Outlook and Next Steps

5. Data Science with R
• Very popular language
• Designed by statisticians
• Large community
• > 10.000 packages
• plus: integrated package management
• But: Limited as Single-Node platform
• Data has to fit in memory
• Limited concurrency for processing

6. SparkR Projects

7. SparkR as seen from R
• Import SparkR-package and initialize SparkSession
• Convert data frames from local R data frames to Spark DataFrame and back
• Read and write data stored in Hadoop HDFS, HBase, Cassandra, and more
• Use Spark Libraries, such as SparkSQL and ML
• User cluster hardware to distribute data frames and parallelize computation

8. SparkR Architecture
• Execute R on cluster
• Data Integration
• Spark DataFrame – R data frame
• Access Big Data File Formats
• Parallelization with UDFs
• Use Spark APIs
• SparkSQL
• Spark MLlib

9. SparkSQL from R
• Execute SQL against
Spark DataFrame
• SELECT
• Specify Projection
• WHERE
• Filter criteria
• GROUPBY
• Group/Aggregate
• JOIN
• Join tables

10. Native Spark ML

11. Time Series Forecasting
• ARIMA(p,d,q)
• AR: p = order of the autoregressive part
• I: d = degree of first differencing involved
• MA: q = order of the moving average part
• Time Series: a series of data points indexed in time order
• Methods:
• Exponential Smoothing
• Neural Networks
• ARIMA:

12. “Pedestrian” Challenges
• Modify some Spark and R (custom-build)
• Submit Spark job with R (incl. packages)
as YARN dependency
• Challenge: R not installed on cluster
• R’s installation location is hard-coded in R
• “R Markdown” produces HTML, PDF,
and more
• Complex objects (.RDS) for metadata,
KPIs, etc.
• Producing additional output during run
• Creating graphics in headless
environments
Installing R (+Pkg’s) on cluster Managing Non-Tabular Output

13. Parallelization with SparkR 1.x
• Sequential computation: > 20 hrs.
• Single-Server, parallelized: > 4.5 hrs

14. Parallelization with SparkR 1.x
• Sequential computation: > 20 hrs.
• Single-Server, parallelized: > 4.5 hrs
• SparkR 1.6.2, 25 nodes, 4 cores: ca. 12 mins.

15. Microsoft R Server for Spark
• Microsoft R Server for HDInsight
integrates Spark and R
• Based on Revolution Analytics
• UDFs via rxExec()
• Data Sources
• RxXdfFile
• RxTextFile
• RxHiveData
• RxParquetData

16. Parallelization with SparkR 2.x
Support for User-Defined Functions
• dapply (dapplyCollect)
• input: DataFrame, func [, Schema]
• output: DataFrame
• gapply (gapplyCollect)
• input: DataFrame¦GroupedData,
groupBy, func [, Schema]
• output: DataFrame
• spark.lapply
• input: parameters, func
• Access to data/HDFS
• output: List

17. Cultural Integration

18. The (Data) Science Process
Public Perception of Science
Source: Birth of a Theorem – with Cedric Villani (https://www.youtube.com/watch?v=yYwydG_aHPE)

19. The (Data) Science Process
Public Perception of Science Science in Reality
Source: Birth of a Theorem – with Cedric Villani (https://www.youtube.com/watch?v=yYwydG_aHPE)

20. Integrating Dev and Prod
• No Need to Re-Write Applications
for Production
• Common Environment for
Development, Test and Production
• “Looks like R to Data Science,
looks like Spark to Data
Engineers”
• Oozie-SparkAction vs ShellAction
• Prepare Dev-/Prod-Environment

21. 2-Level Parallelization
(1) Submit multiple jobs to your cluster:
- Cluster Manager (YARN, Spark, Mesos)
- Spark Job: Driver and Executors
(2) Use GPGPU
- Spark Job: Driver and Executor
- Let Executor use GPGPU
(3) Combine 1 and 2

22. Mix Scala and R
• Call R from Scala
• Add DataScience Module to
your Spark Application
• Use Spark/Scala for ETL, R for
Science code
• Call Spark from R
• Implement high-performance
code in Spark
• More granular control over
cluster resources

23. SparkR: A Dynamic Ecosystem
Hadoop, Spark & R: Many interesting projects and options
• SparkR (Apache, Databricks)
• R Server for Spark (Microsoft)
• Sparklyr (RStudio)
• Oracle R for Analytics, FastR (Oracle)
• SystemML (IBM)
• Renjin (BeDataDriven)

24. Outlook & Misc
• Organizational: Deepen Integration of Data Engineering & Data Science
• Source Code Control & Versioning (git …)
• Continuous Build
• Test Management (RUnit, testthat…?)
• Multi-Output (Rmarkdown)
• Technical: New Approaches
• Simplify/Unify Data Pipelines (SparkSQL)
• Performance Improvement: use MLlib
• Performance Improvement: move calculation to GPU

25. Thank You.
Heiko Korndorf
heiko.korndorf@wireframe.li