The document discusses how Sparklyr allows data scientists to access and work with data stored in Cloudera Enterprise using the popular RStudio IDE. It describes the challenges data scientists face in accessing secured Hadoop clusters and limitations of notebook environments. Sparklyr integration with RStudio provides a familiar environment for data scientists to access Hadoop data and compute using Spark, enabling distributed data science workflows directly in R. The presentation demonstrates how to analyze over a billion records using Spark and R through Sparklyr.

1. 1© Cloudera, Inc. All rights reserved.
Analyzing Hadoop Data with
Sparklyr
Accessing and working with data in Cloudera
Enterprise through the popular RStudio IDE.

2. 2© Cloudera, Inc. All rights reserved.
Your Speakers
Nathan Stephens
Director of Solutions
Engineering (RStudio)
Sean Anderson
Data Science and
Engineering (Cloudera)

3. 3© Cloudera, Inc. All rights reserved.
Data
Preparation
Data
Modeling
Model
Deployment
(maybe)
What does a Data Scientist Do?

4. 4© Cloudera, Inc. All rights reserved.
Challenges in the data science process
Data Engineering Data Science (Exploratory) Production (Operational)
Data Wrangling
Visualization
and Analysis
Model Training
& Testing
Production
Model
Preparation Batch Scoring
Online Scoring
Serving
Dev Tools: IDEs/Notebooks, Collaboration Ops Tools: Versioning, Scheduling, Workflow, Publishing
Data GovernanceGovernance
Processing
Acquisition
Model Quality
& Performance
Experiments
1. Data scientists cannot easily access
Hadoop data or compute using their
favorite languages/frameworks.
“Laptop data science” silos persist.
2. For IT, duplicate environments are
costly and inconsistent, with limited
security and governance. Wants to
move to shared data and infrastructure.

5. 5© Cloudera, Inc. All rights reserved.
Common Limitations
Access
Many times secured clusters are
hard for data science professionals
to connect either because they
don’t have the right permissions or
resources are to scarce to afford
them access. In addition popular
frameworks and libraries don’t read
Hadoop data formats out-of-the-
box.
Scale
Notebook environments seldom
have large enough data storage
for medium, let alone big data.
Data scientists are often
relegated to sample data and
constrained when working on
distributed systems. Popular
frameworks and libraries don’t
easily parallelize across the
cluster.
Developer Experience
Popular notebooks don’t work well
with access engines like Spark
and package deployment and
dependency management across
multiple software versions is often
hard to manage. Then once a
model is built there is no easy
path from model development to
production

6. 6© Cloudera, Inc. All rights reserved.
Cloudera’s Data Science Solution
Familiar tools for
distributed data science
IDE
Integration
Interactive search and
immediate exploration
Search
Audit, lineage,
encryption, key
management, & policy
lifecycles
Navigator
Easy deployment and
flexible scaling
Cloud
Deployment
Modern Real-time
Analytics Engine
Spark
Large-scale ETL & batch
processing engine
Hive-on-
Spark
Multi-Storage, Multi-Environment

7. 7© Cloudera, Inc. All rights reserved.
Hadoop as a Data Science Platform
• Leverage Big Data
• Enable real-time use cases
• Provide sufficient toolset for the Data Analysts
• Provide sufficient toolset for the Data Scientists
+ Data Engineers
• Provide standard data governance capabilities
• Provide standard security across the stack
• Provide flexible deployment options
• Integrate with partner tools
• Provide management tools that make it easy
for IT to deploy/maintain
Hadoop
Kafka, Spark Streaming, Kudu
Spark, Hive, Impala, Hue
Partners
Navigator + Partners
Kerberos, Sentry, Record Service, KMS/KTS
Cloudera Director
Rich Ecosystem
Cloudera Manager/Director

8. 8© Cloudera, Inc. All rights reserved.
Our Goal: Bring more data science users to Hadoop
Help more data scientists
use the power of Hadoop
Use a powerful, familiar
environment with direct access to
Hadoop data and compute
Data Scientist
Data Engineer
Make it easy and secure to
add new users, use cases
Offer secure self-service analytics
and a faster path to production on
common, affordable infrastructure
Enterprise Architect
Hadoop Admin

9. 9© Cloudera, Inc. All rights reserved.
R Studio
• Integrated Development Environment
for R (i.e. like Eclipse or Visual Studio but
for R)
• Runs on the Desktop (Windows, OSX,
Linux) or over the Web as a server to
enable shared resources and
collaboration
• Released in 2012 and now the most
popular front-end for R users with
thousands of downloads per day

10. 10© Cloudera, Inc. All rights reserved.
RStudio
Using R with Spark

11. 11© Cloudera, Inc. All rights reserved.

12. 12© Cloudera, Inc. All rights reserved.

13. 13© Cloudera, Inc. All rights reserved.
What’s new with sparklyr 0.5

14. 14© Cloudera, Inc. All rights reserved.
R for data science toolchain
“You’ll learn how to get your data into R [with Spark], get it into the
most useful structure, transform it, visualize it and model it.”

15. 15© Cloudera, Inc. All rights reserved.
Understand
Transform
dplyr +
sparklyr +
SparkSQL
Visualize
ggplot2
Model
sparklyr +
ML
Spark and R

16. 16© Cloudera, Inc. All rights reserved.
Communicate
R Markdown
Notebooks
R Markdown

17. 17© Cloudera, Inc. All rights reserved.
Spark Deployment Modes
Hadoop Yarn
Driver Node
Livy
Standalone
Cluster
Local
Apache Mesos
EC2
SPARK
EXECUTOR
SPARK
EXECUTOR
YARN
DRIVER LIVY
RStudio
Server
RStudio
Desktop

18. 18© Cloudera, Inc. All rights reserved.
spark.rstudio.com

19. 19© Cloudera, Inc. All rights reserved.
Demo
Analyzing 1 billion records with Spark and R
http://colorado.rstudio.com:3939/content/262/

20. 20© Cloudera, Inc. All rights reserved.
Questions?