Download to read offline
Uploaded byJean-Baptiste Poullet
Meeting20150109 v1
This document provides an introduction to using RHadoop to interface R with Hadoop. It recommends downloading a Cloudera VM with CentOS, CDH5.3, R 3.x, and Java 1.7 installed. It then recommends downloading RHadoop packages and installing rhdfs, rhbase, rmr2, and plyrmr packages in R. It provides guidance on getting started with RHadoop, including ensuring required packages are installed, enabling HDFS, and potentially configuring the JAVA_HOME environment variable. It also provides pointers for debugging RHadoop programs when run with different Hadoop backends and modes.
More Related Content
Meeting20150109 v1
- 1. 1 RBelgiumStat'Rgy RHadoop: introduction Jean-Baptiste Poullet(RBelgium Founder)
- 2. 2 RBelgiumStat'Rgy Content Intro and docs Getstarted with Rhadoop rhdfs, rhbase, rmr, plyrmr
- 3. 3 RBelgiumStat'Rgy Introduction and docs HadoopFor Dummies - Dirk deRoos Hadoop – The Definitive Guide - Tom White RHadoop: make use of Hadoop framework from R https://github.com/RevolutionAnalytics/rmr2/blob/master/doc s/tutorial.md Big Data Analytics with R and Hadoop - Vignesh Prajapati
- 4. 4 RBelgiumStat'Rgy Get started Download clouderaVM http://www.cloudera.com/content/cloudera/en/documentation/DemoVMs/Cloudera-QuickStart-VM/cloudera_quickstart_vm.html This VM runs CentOS CDH5.3 R 3.x Java v1.7.x Download RHadoop https://github.com/RevolutionAnalytics/RHadoop/wiki/Downloads
- 5. 5 RBelgiumStat'Rgy Get started withRHadoop Make sure the packages are installed systemwise Install Rcpp Install rmr2
- 6. 6 RBelgiumStat'Rgy Get started withRHadoop Install rhdfs Enable HDFS You might also need to configure your JAVA_HOME environment variable
- 7.
- 8.
- 9.
- 10.
- 11. 12 RBelgiumStat'Rgy plyrmr bind.cols , transmuteand where instead of transform , summarize and subset
- 12.
- 13. 14 RBelgiumStat'Rgy Debugging Start with thelocal backend and use debug(). Switch to the hadoop mode in standalone. In standalone, R errors are reported in console, that is in your regular R environment. More info to setup the different hadoop modes is available on http://www.rdatamining.com/big-data/r-hadoop-setup-guide Once your program run with the Hadoop backend with hadoop in standalone, you are ready to switch to pseudo-distributed or distributed modes. No debug() here ! In these two modes, to find R errors you have to dig out the logs, specifically those called "userlogs". See http://blog.cloudera.com/blog/2009/09/apache-hadoop-log-files-where-to-find-the m-in-cdh-and-what-info-they-contain/ In parallel, you should grow your test data set sizes. New bugs can show up with larger files. To print variable values, you can use rmr.str or cat(var1, …, varN, file=stderr()) https://github.com/RevolutionAnalytics/RHadoop/wiki/user-rmr-Debugging-rmr-programs
- 14.
- 15.