Hadoop, being a disruptive data processing framework, has made a large impact in the data ecosystems of today. Enabling business users to translate existing skills to Hadoop is necessary to encourage the adoption and allow businesses to get value out of their Hadoop investment quickly. R, being a prolific and rapidly growing data analysis language, now has a place in the Hadoop ecosystem. With the advent of technologies such as RHadoop, optimizing R workloads for use on Hadoop has become much easier. This session will help you understand how RHadoop projects such as RMR, and RHDFS work with Hadoop, and will show you examples of using these technologies on the Hortonworks Data Platform.

1. © Hortonworks Inc. 2012
Enabling R on Hadoop
July 11, 2013
Page 1

2. © Hortonworks Inc. 2012
Your Presenters
Ravi Mutyala
Systems Architect
Page 2
Paul Codding
Solutions Engineer

3. © Hortonworks Inc. 2012
Agenda
• A Brief History of R
• How R is Typically Used
• How R is Used with Hadoop
• Getting Started
Page 3

4. © Hortonworks Inc. 2012
A Brief History of R
Page 4

5. © Hortonworks Inc. 2012
History of R
Page 5
1976: S
Fortran
John
Chambers
S
1988: S V3
written in C
& statistical
models
included
1998: S V4
1991: R
Created by
Ross Ihaka &
Robert
Gentleman
R
1997: R
Core Group
Formed
2000: R
Version 1.0
released

6. © Hortonworks Inc. 2012
How R is Typically Used
Page 6

7. © Hortonworks Inc. 2012
Main Uses of R
• Statistical Analysis & Modeling
– Classification
– Scoring
– Ranking
– Clustering
– Finding relationships
– Characterization
• Common Uses
– Interactive Data Analysis
– General Purpose Statistics
– Predictive Modeling
Page 7

8. © Hortonworks Inc. 2012
How R is Used with Hadoop
Page 8

9. © Hortonworks Inc. 2012
Hadoop Components
Page 9
OS
Cloud
VM
Appliance
PLATFORM
SERVICES
HADOOP
CORE
DATA
SERVICES
OPERATIONAL
SERVICES
Manage &
Operate at
Scale
Store,
Process and
Access Data
Enterprise Readiness: HA,
DR, Snapshots, Security, …
HORTONWORKS
DATA
PLATFORM
(HDP)
Distributed
Storage & ProcessingHDFS
YARN
(in
2.0)
WEBHDFS
MAP
REDUCE
HCATALOG
HIVE
PIG
HBASE
SQOOP
FLUME
OOZIE
AMBARI

10. © Hortonworks Inc. 2012
Hadoop Components & R
Page 10
OS
Cloud
VM
Appliance
PLATFORM
SERVICES
HADOOP
CORE
DATA
SERVICES
OPERATIONAL
SERVICES
Manage &
Operate at
Scale
Store,
Process and
Access Data
Enterprise Readiness: HA,
DR, Snapshots, Security, …
HORTONWORKS
DATA
PLATFORM
(HDP)
Distributed
Storage & ProcessingHDFS
YARN
(in
2.0)
WEBHDFS
MAP
REDUCE
HCATALOG
HIVE
PIG
HBASE
SQOOP
FLUME
OOZIE
AMBARI
Data Service Components
• Hive
• HBase
Hadoop Core
• Map Reduce
• HDFS

11. © Hortonworks Inc. 2012
Options for R on Hadoop
• Options
– RODBC/RJDBC
– RHive
– RHadoop
• Analysis
– Focus
– Integration Ease
– Benefits
– Limitations
Page 11
RHadoop
RODBC/RJDBC
RHive

12. © Hortonworks Inc. 2012
RODBC/RJDBC
• Focus
– SQL Access from R
• Integration Ease
– Install Hortonworks Hive ODBC Driver
– Install Hive libraries
• Benefits
– Low impact on existing R scripts leveraging other DB packages
– Not required to install Hadoop configuration/binaries on client
machines
• Limitations
– Parallelism limited to Hive
– Result set size
Page 12

13. © Hortonworks Inc. 2012
Deployment Considerations
Page 13
TT , DN
.
.
.
.
.
.
.
TT , DNJTNNHS

14. © Hortonworks Inc. 2012
RHive
• Focus
– Broad access to Hive and HDFS
• Integration Ease
– Requires Hadoop binaries, libraries, and configuration files on
client machines
– Uses Java DFS Client and HiveServer
• Benefits
– Wide range of features expressed through HQL
– rhive-apply R Distributed apply function using HQL
• Limitations
– Requires heavy client deployment
– Dependent on HiveServer, and can’t be used with HiveServer2
Page 14

15. © Hortonworks Inc. 2012
Deployment Considerations
Page 15
TT + DN
.
.
.
.
.
.
.
TT + DN
JT
R Edge
Node
NNHS

16. © Hortonworks Inc. 2012
RHadoop
• Focus
– Tight integration with core Hadoop components
• Benefit
– Ability to run R on a massively distributed system
– Ability to work with full data sets instead of sample sets
• Additional Information
– https://github.com/RevolutionAnalytics/RHadoop/wiki
Page 16

17. © Hortonworks Inc. 2012
RHadoop Architecture
Page 17
R
rhdfs
rhbase
rmr2
HDFS
HBase Thrift
Gateway
Map Reduce
HBase
Streaming
R
R
R
R

18. © Hortonworks Inc. 2012
rhdfs
• Access HDFS from R
• Read from HDFS to R dataframe
• Write from R dataframe to HDFS
• 1.0.6 adds support for Windows (using HDP)
Page 18

19. © Hortonworks Inc. 2012
rhdfs
• Hadoop CLI Commands & rhdfs equivalent
• hadoop fs –ls /
– hdfs.ls(“/”)
• hadoop fs –mkdir /user/rhdfs/ppt
– hdfs.mkdir(“/user/rhdfs/ppt”)
• hadoop fs –put 1.txt /user/rhfds/ppt/
– localData <- system.file(file.path("unitTestData", ”1.txt"), package="rhdfs”)
– hdfs.put(localData, ”/user/rhdfs/ppt/1.txt”)
• hadoop fs –get /user/rhdfs/ppt/1.txt 1.txt
– hdfs.get(”/user/rhdfs/ppt/1.txt”,”test”)
• hadoop fs –rm /user/rhdfs/ppt/1.txt
– hdfs.delete(“/user/rhdfs/ppt/1.txt”)
Page 19

20. © Hortonworks Inc. 2012
rhbase
• Access and change data within HBase
• Uses Thrift API
• Command Examples
– hb.new.table
– hb.insert
– hb.scan.ex
– hb.scan
Page 20

21. © Hortonworks Inc. 2012
rmr2
• Enables writing MapReduce jobs using R
• Ability to parallelize algorithms
• Ability to use big data sets without needing to sample
data
• mapreduce(input, output, map, reduce, …)
• Reduces takes a key and a collection of values which
could be vector, list, data frame or matrix
• 2.2.1 adds support for Windows (using HDP)
Page 21

22. © Hortonworks Inc. 2012
Sample code - wordcount
Page 22
wc.map = !
function(., lines) {!
keyval(!
unlist(!
strsplit(!
x = lines,!
split = pattern)),!
1)}!
wc.reduce =!
function(word, counts ) {!
keyval(word, sum(counts))}!
!
mapreduce(!
input = input ,!
output = output,!
input.format = "text",!
map = wc.map,!
reduce = wc.reduce,!
combine = T)}!

23. © Hortonworks Inc. 2012
More Sample Code
Page 23
groups = rbinom(32, n = 50, prob = 0.4)!
tapply(groups, groups, length)!
groups = to.dfs(groups)!
from.dfs(!
mapreduce(!
input = groups,!
map = function(., v) keyval(v, 1),!
reduce =!
function(k, vv)!
keyval(k, length(vv))))!

24. © Hortonworks Inc. 2012
Deployment Considerations
Page 24
TT , DN,
RS
R
.
.
.
.
.
.
.
TT , DN,
RS
RJT
R Edge
Node
NN
HT
G

25. © Hortonworks Inc. 2012
RHadoop
• Limitations
– Requires installation of R on all TaskTracker nodes
– Does not automatically parallelize algorithms
– Different slot/memory configuration recommended to leave
memory and CPU resources for R
Page 25
OS
Map Reduce
OS
Map Reduce
R

26. © Hortonworks Inc. 2012
Getting Started
Page 26

27. © Hortonworks Inc. 2012
Your Fastest On-ramp to Enterprise Hadoop™!
Page 27
http://hortonworks.com/products/hortonworks-sandbox/
The Sandbox lets you experience Apache Hadoop from the convenience of your own
laptop – no data center, no cloud and no internet connection needed!
The Hortonworks Sandbox is:
• A free download: http://hortonworks.com/products/hortonworks-sandbox/
• A complete, self contained virtual machine with Apache Hadoop pre-configured
• A personal, portable and standalone Hadoop environment
• A set of hands-on, step-by-step tutorials that allow you to learn and explore Hadoop

28. © Hortonworks Inc. 2012
Installation
• Install R on all nodes
• Install dependent
packages
– RJSONIO
– itertools
– digest
– Rcpp
– rJava
– functional
– RCurl
– httr
– plyr
• Download & Install
RHadoop Packages
– rmr2
– rhdfs
– rhbase (requires Thrift)
Page 28

29. © Hortonworks Inc. 2012
Questions & Answers
TRY
Download HDP at hortonworks.com
LEARN
Applying Data Science using Apache
Hadoop Training
FOLLOW
twitter: @hortonworks
Facebook: facebook.com/hortonworks
Page 29
Further questions & comments:
paul@hortonworks.com
ravi@hortonworks.com