• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Enabling R on Hadoop
 

Enabling R on Hadoop

on

  • 13,203 views

Hadoop, being a disruptive data processing framework, has made a large impact in the data ecosystems of today. Enabling business users to translate existing skills to Hadoop is necessary to encourage ...

Hadoop, being a disruptive data processing framework, has made a large impact in the data ecosystems of today. Enabling business users to translate existing skills to Hadoop is necessary to encourage the adoption and allow businesses to get value out of their Hadoop investment quickly. R, being a prolific and rapidly growing data analysis language, now has a place in the Hadoop ecosystem. With the advent of technologies such as RHadoop, optimizing R workloads for use on Hadoop has become much easier. This session will help you understand how RHadoop projects such as RMR, and RHDFS work with Hadoop, and will show you examples of using these technologies on the Hortonworks Data Platform.

Statistics

Views

Total Views
13,203
Views on SlideShare
12,926
Embed Views
277

Actions

Likes
28
Downloads
550
Comments
0

9 Embeds 277

http://www.scoop.it 165
http://dschool.co 55
http://bigdatapress.com 24
http://www.dschool.co 18
https://twitter.com 9
http://loretodoranda.dschool.co 2
http://www.surendranathcentenary.dschool.co 2
http://ramshyam1974.dschool.co 1
http://localhost 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

Enabling R on Hadoop Enabling R on Hadoop Presentation Transcript

  • © Hortonworks Inc. 2012 Enabling R on Hadoop July 11, 2013 Page 1
  • © Hortonworks Inc. 2012 Your Presenters Ravi Mutyala Systems Architect Page 2 Paul Codding Solutions Engineer
  • © Hortonworks Inc. 2012 Agenda • A Brief History of R • How R is Typically Used • How R is Used with Hadoop • Getting Started Page 3
  • © Hortonworks Inc. 2012 A Brief History of R Page 4
  • © Hortonworks Inc. 2012 History of R Page 5 1976: S Fortran John Chambers S 1988: S V3 written in C & statistical models included 1998: S V4 1991: R Created by Ross Ihaka & Robert Gentleman R 1997: R Core Group Formed 2000: R Version 1.0 released
  • © Hortonworks Inc. 2012 How R is Typically Used Page 6
  • © Hortonworks Inc. 2012 Main Uses of R • Statistical Analysis & Modeling – Classification – Scoring – Ranking – Clustering – Finding relationships – Characterization • Common Uses – Interactive Data Analysis – General Purpose Statistics – Predictive Modeling Page 7
  • © Hortonworks Inc. 2012 How R is Used with Hadoop Page 8
  • © Hortonworks Inc. 2012 Hadoop Components Page 9 OS   Cloud   VM   Appliance   PLATFORM  SERVICES   HADOOP  CORE   DATA   SERVICES   OPERATIONAL   SERVICES   Manage & Operate at Scale Store, Process and Access Data Enterprise Readiness: HA, DR, Snapshots, Security, … HORTONWORKS     DATA  PLATFORM  (HDP)   Distributed Storage & ProcessingHDFS   YARN  (in  2.0)   WEBHDFS   MAP  REDUCE   HCATALOG   HIVE  PIG   HBASE   SQOOP   FLUME   OOZIE   AMBARI  
  • © Hortonworks Inc. 2012 Hadoop Components & R Page 10 OS   Cloud   VM   Appliance   PLATFORM  SERVICES   HADOOP  CORE   DATA   SERVICES   OPERATIONAL   SERVICES   Manage & Operate at Scale Store, Process and Access Data Enterprise Readiness: HA, DR, Snapshots, Security, … HORTONWORKS     DATA  PLATFORM  (HDP)   Distributed Storage & ProcessingHDFS   YARN  (in  2.0)   WEBHDFS   MAP  REDUCE   HCATALOG   HIVE  PIG   HBASE   SQOOP   FLUME   OOZIE   AMBARI   Data Service Components •  Hive •  HBase Hadoop Core •  Map Reduce •  HDFS
  • © Hortonworks Inc. 2012 Options for R on Hadoop • Options – RODBC/RJDBC – RHive – RHadoop • Analysis – Focus – Integration Ease – Benefits – Limitations Page 11 RHadoop RODBC/RJDBC RHive
  • © Hortonworks Inc. 2012 RODBC/RJDBC • Focus – SQL Access from R • Integration Ease – Install Hortonworks Hive ODBC Driver – Install Hive libraries • Benefits – Low impact on existing R scripts leveraging other DB packages – Not required to install Hadoop configuration/binaries on client machines • Limitations – Parallelism limited to Hive – Result set size Page 12
  • © Hortonworks Inc. 2012 Deployment Considerations Page 13 TT , DN . . . . . . . TT , DNJTNNHS
  • © Hortonworks Inc. 2012 RHive • Focus – Broad access to Hive and HDFS • Integration Ease – Requires Hadoop binaries, libraries, and configuration files on client machines – Uses Java DFS Client and HiveServer • Benefits – Wide range of features expressed through HQL – rhive-apply R Distributed apply function using HQL • Limitations – Requires heavy client deployment – Dependent on HiveServer, and can’t be used with HiveServer2 Page 14
  • © Hortonworks Inc. 2012 Deployment Considerations Page 15 TT + DN . . . . . . . TT + DN JT R Edge Node NNHS
  • © Hortonworks Inc. 2012 RHadoop • Focus – Tight integration with core Hadoop components • Benefit – Ability to run R on a massively distributed system – Ability to work with full data sets instead of sample sets • Additional Information – https://github.com/RevolutionAnalytics/RHadoop/wiki Page 16
  • © Hortonworks Inc. 2012 RHadoop Architecture Page 17 R rhdfs rhbase rmr2 HDFS HBase Thrift Gateway Map Reduce HBase Streaming R R R R
  • © Hortonworks Inc. 2012 rhdfs • Access HDFS from R • Read from HDFS to R dataframe • Write from R dataframe to HDFS • 1.0.6 adds support for Windows (using HDP) Page 18
  • © Hortonworks Inc. 2012 rhdfs • Hadoop CLI Commands & rhdfs equivalent • hadoop fs –ls / – hdfs.ls(“/”) • hadoop fs –mkdir /user/rhdfs/ppt – hdfs.mkdir(“/user/rhdfs/ppt”) • hadoop fs –put 1.txt /user/rhfds/ppt/ – localData <- system.file(file.path("unitTestData", ”1.txt"), package="rhdfs”) – hdfs.put(localData, ”/user/rhdfs/ppt/1.txt”) • hadoop fs –get /user/rhdfs/ppt/1.txt 1.txt – hdfs.get(”/user/rhdfs/ppt/1.txt”,”test”) • hadoop fs –rm /user/rhdfs/ppt/1.txt – hdfs.delete(“/user/rhdfs/ppt/1.txt”) Page 19
  • © Hortonworks Inc. 2012 rhbase • Access and change data within HBase • Uses Thrift API • Command Examples – hb.new.table – hb.insert – hb.scan.ex – hb.scan Page 20
  • © Hortonworks Inc. 2012 rmr2 • Enables writing MapReduce jobs using R • Ability to parallelize algorithms • Ability to use big data sets without needing to sample data • mapreduce(input, output, map, reduce, …) • Reduces takes a key and a collection of values which could be vector, list, data frame or matrix • 2.2.1 adds support for Windows (using HDP) Page 21
  • © Hortonworks Inc. 2012 Sample code - wordcount Page 22 wc.map = ! function(., lines) {! keyval(! unlist(! strsplit(! x = lines,! split = pattern)),! 1)}! wc.reduce =! function(word, counts ) {! keyval(word, sum(counts))}! ! mapreduce(! input = input ,! output = output,! input.format = "text",! map = wc.map,! reduce = wc.reduce,! combine = T)}!
  • © Hortonworks Inc. 2012 More Sample Code Page 23 groups = rbinom(32, n = 50, prob = 0.4)! tapply(groups, groups, length)! groups = to.dfs(groups)! from.dfs(! mapreduce(! input = groups,! map = function(., v) keyval(v, 1),! reduce =! function(k, vv)! keyval(k, length(vv))))!
  • © Hortonworks Inc. 2012 Deployment Considerations Page 24 TT , DN, RS R . . . . . . . TT , DN, RS RJT R Edge Node NN HT G
  • © Hortonworks Inc. 2012 RHadoop • Limitations – Requires installation of R on all TaskTracker nodes – Does not automatically parallelize algorithms – Different slot/memory configuration recommended to leave memory and CPU resources for R Page 25 OS Map Reduce OS Map Reduce R
  • © Hortonworks Inc. 2012 Getting Started Page 26
  • © Hortonworks Inc. 2012 Your Fastest On-ramp to Enterprise Hadoop™! Page 27 http://hortonworks.com/products/hortonworks-sandbox/ The Sandbox lets you experience Apache Hadoop from the convenience of your own laptop – no data center, no cloud and no internet connection needed! The Hortonworks Sandbox is: •  A free download: http://hortonworks.com/products/hortonworks-sandbox/ •  A complete, self contained virtual machine with Apache Hadoop pre-configured •  A personal, portable and standalone Hadoop environment •  A set of hands-on, step-by-step tutorials that allow you to learn and explore Hadoop
  • © Hortonworks Inc. 2012 Installation • Install R on all nodes • Install dependent packages – RJSONIO – itertools – digest – Rcpp – rJava – functional – RCurl – httr – plyr • Download & Install RHadoop Packages – rmr2 – rhdfs – rhbase (requires Thrift) Page 28
  • © Hortonworks Inc. 2012 Questions & Answers TRY Download HDP at hortonworks.com LEARN Applying Data Science using Apache Hadoop Training FOLLOW twitter: @hortonworks Facebook: facebook.com/hortonworks Page 29 Further questions & comments: paul@hortonworks.com ravi@hortonworks.com