1.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.
Sukhendu Chakraborty
DataMesh Team @ {rr}
Big Data Analytics made easy
using Apache Hive to R Connector

2.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.

3.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.
Our cloud-based platform supports both real-time processes
and analytical use cases, utilizing technologies to name a
few: Crunch, Hive, HBase, Avro, Kafka, R
Someone clicks on a {rr} recommendation
every 21 milliseconds
Did You Know?
Our data capacity includes a 1.5 PB Hadoop infrastructure,
which enables us to employ 100+ algorithms in real-time
In the US, we serve 7000 requests per second with an average
response time of 50 ms

4.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.

5.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.
What is R?
• A letter in English alphabet
• An open-source statistical language for
data analytics
– Simple: Easy to install and program
– Popular: One of the most widely used open
sourced statistical tools
– Powerful: Rich set of packages (> 4000) to
perform statistical analysis and plotting
– More info: http://cran.us.r-project.org/

6.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.
But…
• Performance issues
– Typically single threaded
– All the data needs to be in memory
– Not scalable
• Need to know the internals to make it
perform well

7.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.
What’s out there
• Rhadoop/RMR
– Uses Hadoop MR to distribute data in the Hadoop cluster
– No transparency: Limited data preparation support
• RHIPE
– Similar to Rhadoop
– Protobuf dependency
• RHive
– Lets you run HIVE queries from R functions
– Users need to know HQL
– Needs Rserve + rJava

8.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.
R @ {rr} - so far
{rr} cluster R client
HIVE queries
Data access

9.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.
• Transparency Layer
• Pluggable Query generation
• R as an analytical platform
– Data cleanup
– Ad-hoc analytics
– Data preparation
– Distributed analytics using Hadoop
– Result summarization and publishing
R HIVE connector
HIVE (UC 1)
MR (UC 2)

10.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.
OO programming in R
• S4 class system - classes and objects
• Methods and multiple dispatch
• Object validity checking
• Extensible: setGenerics()
• Quick overview: http://www.r-
project.org/conferences/useR-2004/Keynotes/Leisch.pdf

11.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.
Use Case I:
Rollups in HIVE

12.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.

13.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.

14.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.

15.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.

16.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.

17.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.
Use Case II:
Distributed Analytics

18.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.

19.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.

20.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.

21.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.
R @ {rr}
{rr} cluster R client
R HIVE
connector
Data access

22.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.
Future Work
• Extend the connector to handle other data
sources
• Add custom Analytical functions
• Asynchronous execution
• Performance tuning

23.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.
Thank You

24.
© 2014 RichRelevance, Inc. All Rights Reserved. Confidential.
Questions?