This document discusses enabling interactive R and Python environments on Hadoop. It describes installing RStudio Server and RHadoop/rJDBC packages to integrate R with Hadoop, and using Anaconda and Jupyter to integrate Python. The goals are to leverage Hadoop storage and processing, build an easy interface for data scientists, leverage open source projects to minimize costs, and overcome Kerberos challenges. A demo of honeycomb, an R package for interacting with Hadoop, is provided.

1. Enable Interactive R and Python Environments on Hadoop
June 27, 2017 • Room #308
Dave Carlson
Data Science Technical Lead
Zurich North America
Ming Yuan
Architect, Big Data & Cloud
Capital One

2. Disclaimer
The views expressed in this presentation are solely those of
Dave Carlson or Ming Yan and are not those Zurich American
Insurance Company or Capital One.
June 25, 2017 1

3. Goals
 Enables organizations to take advantage of a ecosystem for Big Data
Analytics
• Leverage storage and processing power of Hadoop and Analytical
Power of R
 Build an interactive and easy-to-use interface for data scentists
 Leverage open source projects to minimum the cost
 Overcome technical challenges from Kerberos
+ + -
June 25, 2017 2

4. Why R and Python?
 Analysts have been
transitioning to open
source languages like R
and Python and away
from proprietary
languages like SAS and
SPSS.
 This is true for both
academic and
commercial
communities.

5. June 25, 2017 4
Python integration uses Anaconda and Jupyter

6. June 25, 2017 5
R integration uses RStudio Server and RHadoop

7. R on Hadoop
Install Hadoop
Client Node
Installed
Rhadoop/rJDBC
Installed
RStudio Server
Integrated R
libraries with
RStudio server
June 25, 2017 6

8. RHadoop and RJDBC Packages
 rhdfs
 Provides basic connectivity to HDFS so that R programmers
can browse, read, write, and modify files stored in HDFS
• Installed on the node that will run the R code
 plyrmr
 Enables R users to perform common data manipulation
operations on very large data sets stored on Hadoop
• Installed on every node in Hadoop cluster
 rmr2
 Allows R developers to perform statistical analysis in R via
Hadoop MapReduce functionality on a Hadoop cluster
• Installed on every node in Hadoop cluster
 ravro
 Enable the read and write to avro files from local and HDFS
file system and adds an avro input format for rmr2
• Installed only on the node that will run the R code
 rhbase
 Not use
 rJDBC
• Provides basic connectivity to Hive
with a JDBC driver
• Installed on the node that will run
the R code
• Depends on rJava and DBI packages
June 25, 2017 7

9. R and Hadoop Integration
June 25, 2017 8

10. RStudio Server Installation and Configuration
 Install RStudio Server rpm
 User Authentication against LDAP
 sudo cp /etc/pam.d/login /etc/pam.d/rstudio
 Resource Allocation
 Integrate with R and its libraries in rserver.conf
 Specify R version
 rsession-which-r=/usr/local/bin/R
 Locates shared libraries
 rsession-ld-library-path
=/opt/someapp/lib:/opt/anotherapp/lib
June 25, 2017 9

11. RStudio Server, R and Hadoop Integration
June 25, 2017 10

12. Python -- Anaconda Cluster
 Head Node
 A system configured to act as the intermediary between the cluster and the
outside network
 Can also be referred to as the master or edge node
 Compute node
 The machines managed by the head node that all work together to complete a
single task
Thin Client Head Node
Compute Node
Compute Node
Compute Node
June 25, 2017 11

13. System Architecture
RStudio Server
 Hadoop libraries
 rhdfs
 rmr2
 plyrmr
 rJDBC
 Sparkr
Hadoop Data Node
• Hadoop data node
Anaconda Compute Node
rmr2
plyrmr
Hadoop Management Node
MANAGE
Anaconda Head Node
Anaconda Enterprise Notebook
Thin Client
June 25, 2017 12

14. R Framework
June 25, 2017 13
 R framework requires ‘dplyr 0.5.0’.
 The easiest way to verify that the correct set of dependencies are installed is to use a
fixed snapshot of CRAN using a fixed MRAN (Microsoft) snapshot.
 Add the following lines to your ‘Rprofile.site’ and ‘Renviron.site’ files. If you have a
fresh installation of R you will need to create the files.
$R_HOME/etc/Rprofile.site
options(repos=structure(c(CRAN='https://mran.revolutionanalytics.com/snapshot/2017-
04-01/')))
$R_HOME/etc/Renviron.site
HADOOP_CMD=/usr/bin/hadoop
USE_KERBEROS=0
HIVE_SERVER_HOST=localhost
HIVE_SERVER_PORT=10000
HIVE_JAR_FOLDERS=/usr/lib/hive/lib
 Now install the ‘honeycomb’ package using the ‘devtools’ library within R.
devtools::install_github('ZurichPA/rhdfs', subdir='pkg')
devtools::install_github('ZurichPA/orpheus')
devtools::install_github('ZurichPA/honeycomb')

15. June 25, 2017 14
DEMO
https://zurichpa.github.io/honeycomb

16. Additional References
 R
 RHadoop (https://github.com/RevolutionAnalytics/RHadoop/wiki)
 Zurich Predictive Analytics (https://github.com/zurichpa)
 honeycomb (https://github.com/ZurichPA/honeycomb)
 Python
 Impyla (https://github.com/cloudera/impyla)