Data Hacking with RHadoop


Published on

Rhadoop is an effective platform for doing exploratory data analysis over big data sets. The convenience of an interactive command-line interpreter and the overwhelming number of statistical and machine learning routines implemented in R libraries make a highly effective environment to perform elementary data science.

We'll discuss the basics of RHadoop: what it is, how to install it, and the API fundamentals. Next we'll discuss common use cases that you might want to use RHadoop for. Last, we'll run through an interactive example.

Published in: Technology, Education
  • Examples given in the silde number 24 and 25 does not work!! Could anyone give the working example and explain the same ?? :)
    Are you sure you want to  Yes  No
    Your message goes here
  • may want to check this out to explain application to humans/non-techs - nice write-up
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Data Hacking with RHadoop

  1. 1. Rhadoop  Data  Hacking  Using  R  and  Hadoop  to  do  large-­‐scale   data  science  
  2. 2. Would  You  Like  to…   •  Predict  X?   –  The  outcome  of  a  future  event   –  Who  is  likely  to  do  something   –  Gene?c  factors  leading  to  disease   •  Pre-­‐filter  things  so  humans  can  accomplish   more?   •  Do  all  of  this  faster  and  beCer?  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   2  
  3. 3. Why  R  and  Hadoop?   •  R  is  a  fantas?c  plaHorm  for  data   science   –  Has  a  peer-­‐reviewed  community   and  journal  that  vets  libraries   –  (Mostly)  intui?ve  language   •  Hadoop  is  the  de-­‐facto  plaHorm   for  parallel  processing   •  Today,  we’ll  be  talking  about   rmr,  but  there’s  two  more   packages:  rhbase  and  rhdfs  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   3  
  4. 4. Nothing  Has  Changed.  Everything  Has   Changed.   •  Some  of  the  most  effec?ve  techniques  for  data  mining   are  rela?vely  old   –  Modern  SVM  dates  back  to  ‘92   –  Logis?c  regression  dates  back  to  ‘44   –  Important  elements  of  the  algorithms  date  back  to  Newton   •  Accessibility  and  relevance  have  changed   –  Accessibility  to  data   –  Accessibility  of  computa?onal  power   –  Necessity  of  methods  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   4  
  5. 5. Some  CriBcisms  of  R  &  Rhadoop   •  R  docs  are  wriCen  in  their  own  language  (using  data   frames,  etc.)  that  is  unfamiliar  to  computer   scien?sts   •  R  and  CRAN  documenta?on  are  more  like  old-­‐school   GNU  than  most  Apache  projects   –  Get  used  to  Googling  and  using  R’s  help()  func?on   •  R’s  data  management  facili?es  are  inconsistent   •  Streaming  API  isn’t  super  fast   •  (get  over  it)  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   5  
  6. 6. Comparison  to  Other  R  Parallelism   Frameworks   •  SNOW/SNOWFALL   –  Operates  over  MPI,  Sockets,  or  PVM   –  No  ?e-­‐in  to  a  DFS  (bad  for  data-­‐intensive  compu?ng)   –  Handles  matrix  mul?plica?on  well  (perhaps  beCer)   –  Doesn’t  handle  other  non-­‐trivial  IPC  well  (basically  for  parallel  linear   algebra  and  simula?ons)   •  Rmpi   –  More  code   –  All  synchroniza?on  constructs  are  user-­‐built  (just  like  MPI)  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   6  
  7. 7. Comparison  to  Other  R  Parallelism   Frameworks   •  Others…   –  Only  other  Hadoop  libraries  have  integra?on  with   HDFS/are  appropriate  for  data  intensive   compu?ng   –  Only  Rhadoop  supports  local  and  cluster  based   backends  and  has  an  intui?ve  interface  that   duplicates  closures  in  the  remote  environment   –  Most  environments  are  targeted  towards   modeling  and  simula?on  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   7  
  8. 8. InstallaBon  –  Local  WorkstaBon   •  Install  R   –  Macports  –  sudo port install r-framework! –  Ubuntu  –  sudo apt-get install r-base! –  RHEL  –  sudo yum install R! •  Install  R  dependencies  (inside  R)   –  install.packages(c("Rcpp", "RJSONIO", "itertools", "digest"), repos="”)! •  Install  RMR   –  curl rmr_1.3.1.tar.gz > rmr.tar.gz! –  install.packages("rmr.tar.gz”) # from inside r, in the same directory! •  Configure  the  local  backend  each  ?me  you  run  R   –  rmr.options.set(backend=“local”)!This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   8  
  9. 9. InstallaBon  -­‐  Cluster   •  Install  R  and  all  packages  you  plan  on  using  (rmr,  e1071,  topicmodels,  tm,   etc.)  on  each  node.   •  Use  a  compa?ble  version  of  Hadoop  1  (1.0.3+  or  CDH3+).  Hadoop  2  may   or  may  not  work.   •  The  example  on  the  previous  slide  installs  R  packages  in  your  home   directory,  you  probably  want  to  install  them  to  the  root  install.   •  Configure  environment  variables   export HADOOP_CMD=/usr/bin/hadoop
 export HADOOP_STREAMING=/usr/lib/hadoop/contrib/ streaming/hadoop-streaming-<version>.jar!This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   9  
  10. 10. The  Curse  of   Volume  of  the  Unit  Ball  vs.  Dimensionality   Dimensionality   •  The  volume  of  the  unit  sphere   tends  towards  0  as  the   dimensionality  of  hyperspace   increases   •  Intui?vely  this  means  that  there  is   more  “slop  room”  for  your  dividing   hyperplane  to  fall  into   •  The  amount  of  data  we  need  to   train  a  model  rises  with  the   feature  space,  tending  towards   infinity,  making  the  problem   untenable   •  With  a  small  feature  space,  there   is  no  need  for  lots  of  data   •  Thus,  there  is  liCle  point  in  using   Hadoop  to  implement  many  classic   machine  learning  models  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   10  
  11. 11. The  Hadoop  Data  Science  Flow   •  Join   •  Sample   •  Model   •  Repeat  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   11  
  12. 12. Join   •  Put  two  pieces  of  data  together  using  a   common  key   •  Scenario:   –  Data  is  in  two  flat  files  in  HDFS   –  Turn  rows  into  rows  of  key-­‐value  pairs,  where  the   key  is  the  join  key  and  the  value  is  the  rest  of  the   row  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   12  
  13. 13. Sample   •  Take  a  sample  of  your  (maybe)  joined  data   •  Most  common  method  is  probabilis?cally   •  Numerous  other  techniques  can  leverage  par??ons   and  randomness  of  the  key  hash   •  Scenarios  (a  precursor  for):   –  Supervised  learning/classifica?on   –  Unsupervised  learning/clustering   –  Regression   –  Distribu?on  modeling  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   13  
  14. 14. Model   •  Supervised  learning:  I  want  to  predict  something  and   I  already  know  (some)  of  the  answers.  Also  called   classifica?on  and  binary  classifica?on   •  Unsupervised  learning:  I  want  to  find  natural   groupings  in  the  data  that  I  might  not  have  known   about   •  Regression,  probability  modeling  –  I  want  to  fit  a   curve  to  my  data  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   14  
  15. 15. Repeat   •  Gain  insight  about  the  data   •  Change  your  procedure  (select  only  outliers,   etc.)   •  Gain  more  insight  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   15  
  16. 16. Rhadoop  Impact:  Join,  Sample   •  Work  totally  in  R   •  Execute  large,  complex  joins  such  as  cross   joins  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   16  
  17. 17. Rhadoop  Impact:  Model   •  Most  algorithms  work  perfectly  well  (or   beCer)  over  a  sample  of  the  data   •  Train  and  cross-­‐validate  a  large  number  of   models  in  parallel   •  Perform  model  selec?on  in  the  reduce  phase  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   17  
  18. 18. Rhadoop  API   mapreduce(! input,! output = NULL,! map =,! reduce = NULL,! combine = NULL,! = FALSE,! input.format = "native",! output.format = "native",! vectorized = list(map = FALSE, reduce = FALSE),! structured = list(map = FALSE, reduce = FALSE),! backend.parameters = list(),! verbose = TRUE)!This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   18  
  19. 19. Rhadoop  API   rmr.options.set(backend = c("hadoop", "local"),! profile.nodes = NULL, vectorized.nrows = NULL)
 ! to.dfs(object, output = dfs.tempfile(), ! format = "native")! ! from.dfs(input, format = "native", ! = FALSE, vectorized = FALSE,! structured = FALSE)  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   19  
  20. 20. Doing  Things  the  R  Way   •  Objects   –  my_car = list(color=“green”, model=“volt”)! •  Transforming a vector (list), iterating –  lapply/sapply/tapply – functional programming constructs •  Loops (not preferred) –  for ( i in 1:100) {…}! –  Note this is the same as lapply(1:100, function(i){…})! •  Other control structures – basically as you would expectThis  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   20  
  21. 21. Vectors  in  R   •  R  helps  you!  O_o   •  Every  object  has  a  mode  and  length  and  hence  can  be  interpreted  as  some   sort  of  vector  –  even  primi?ves!   •  Even  primi?ves  such  as  strings  or  integers  are  stored  in  a  vector  of  length   1,  never  free-­‐standing   •  There  are  lots  of  types  of  vectors   –  Lists  (think  linked  list)   –  Atomic  vectors  (think  array)   hCp://cran.r-­‐­‐intro.html#The-­‐intrinsic-­‐aCributes-­‐ mode-­‐and-­‐length   •  Type  coercion  usually  works  the  way  you  would  expect   –  But…  you  may  find  yourself  using  as.list()  or  as.vector()  or  doing  manual  coercion   frequently  depending  on  what  libraries  you’re  using  due  to  mode  not  matching  This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   21  
  22. 22. Example  –  Fake  Data   fakedata = data.frame(x = c(rnorm(100)*.25, rep(. 75,100)+rnorm(100)*.25), y = c(rnorm(100), rep(1,100)+rnorm(100)), z = c(rep(0,100), rep(1,100)) )! ! plot(fakedata[,"x"],fakedata[,"y"],col=sapply(fakedata[,"z"], function(z) ifelse(z>0,"blue","green")))!This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   22  
  23. 23. Examples  –  Simple  Parallelism   rmr.options.set(backend=“local”)! ! ints = to.dfs(1:100)! ! squares = mapreduce(ints, map=function(x) reyval(NULL,x^2))! ! print from.dfs(ints)! ! # notice the result will be ! # keyvals!This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   23  
  24. 24. Examples  –  Trying  Lots  of  SVM  Kernels   kernels = to.dfs(list("linear","polynomial","radial","sigmoid" ))! ! models = from.dfs(mapreduce(kernels,map=function(nothing,kern ) keyval(NULL,svm(factor(z)~.,fakedata,kernel=kern))))! ! plot(models[[1]][["val"]],fakedata)! ! !This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   24  
  25. 25. Examples  –  Different  Models   calls = to.dfs(list(list("glm",z~.,family=binomial("logi t"), fakedata),list("svm",z~.,fakedata)))! ! models = from.dfs(mapreduce(calls, map=function(nothing,callsig) keyval(NULL,[[1]],callsig[2:lengt h(callsig)]))))! ! models[[1]][["val"]]!This  document  is  company  confiden?al  and  is  intended  solely  for  the  use  and  informa?on  of  Booz  Allen  Hamilton   25