SparkR: Enabling Interactive Data Science at Scale on Hadoop
•
15 likes•3,271 views
SparkR enables interactive data science at scale on Hadoop by providing an R interface to Apache Spark. Some key points: - SparkR allows users to manipulate distributed datasets (RDDs) using familiar R operations like map, filter, reduceByKey. - It integrates R and Spark by running R code on Spark executors via JNI, allowing R scripts to process large datasets in parallel. - Examples show how to do tasks like word count and logistic regression on Spark using R code, demonstrating the ability to scale R for data science on big data.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (7)
Similar to SparkR: Enabling Interactive Data Science at Scale on Hadoop
Similar to SparkR: Enabling Interactive Data Science at Scale on Hadoop (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
SparkR: Enabling Interactive Data Science at Scale on Hadoop
- 1. SparkR: Enabling Interactive Data Science at Scale on Hadoop Shivaram Venkataraman Zongheng Yang
- 5. HDFS / HBase / Cassandra Spark SparkR Mesos / YARN Storage Cluster Manager Data Processing
- 6. Outline SparkR API Live Demo Design Details
- 7. RDD Parallel Collection Transformations map filter groupBy … Actions count collect saveAsTextFile …
- 8. Q: How can I use a loop to [...insert task here...] ?! A: Don’t. Use one of the apply functions.! From: http://nsaunders.wordpress.com/2010/08/20/a-brief-introduction-to-apply-in-r/ R
- 9. R + RDD = R2D2
- 10. R + RDD = RRDD lapply lapplyPartition groupByKey reduceByKey sampleRDD collect cache … broadcast includePackage textFile parallelize
- 11. Example: Word Count lines <-‐ textFile(sc, “hdfs://my_text_file”)
- 12. Example: Word Count lines <-‐ textFile(sc, “hdfs://my_text_file”) words <-‐ flatMap(lines, function(line) { strsplit(line, " ")[[1]] }) wordCount <-‐ lapply(words, function(word) { list(word, 1L) })
- 13. Example: Word Count lines <-‐ textFile(sc, “hdfs://my_text_file”) words <-‐ flatMap(lines, function(line) { strsplit(line, " ")[[1]] }) wordCount <-‐ lapply(words, function(word) { list(word, 1L) }) counts <-‐ reduceByKey(wordCount, "+", 2L) output <-‐ collect(counts)
- 14. Demo
- 15. MNIST
- 16. A b || Ax − b ||2Minimize x = (AT A)−1 AT b
- 17. How does this work ?
- 26. Pipelined RDD words <-‐ flatMap(lines,…) wordCount <-‐ lapply(words,…) Spark Executor exec R Spark Executor Rexec
- 27. Pipelined RDD Spark Executor exec R Spark Executor Rexec Spark Executor exec R R Spark Executor
- 28. Alpha developer release One line install ! install_github("amplab-‐extras/SparkR-‐pkg", subdir="pkg")
- 29. SparkR Implementation Very similar to PySpark Spark is easy to extend 292 lines of Scala code 1694 lines of R code 549 lines of test code in R
- 30. EC2 setup scripts All Spark examples MNIST demo Hadoop2, Maven build Also on github
- 31. In the Roadmap High level DataFrame API Integrating Spark’s MLLib from R Reading from SequenceFiles, HBase
- 32. SparkR Combine scalability & utility RDD à distributed lists Serialize closure Re-use R packages
- 33. SparkR https://github.com/amplab-extras/SparkR-pkg Shivaram Venkataraman shivaram@cs.berkeley.edu Spark User mailing list user@spark.apache.org
- 36. Example: Logistic Regression pointsRDD <-‐ textFile(sc, "hdfs://myfile") weights <-‐ runif(n=D, min = -‐1, max = 1) # Logistic gradient gradient <-‐ function(partition) { X <-‐ partition[,1]; Y <-‐ partition[,-‐1] t(X) %*% (1/(1 + exp(-‐Y * (X %*% weights))) -‐ 1) * Y }
- 37. Example: Logistic Regression pointsRDD <-‐ textFile(sc, "hdfs://myfile") weights <-‐ runif(n=D, min = -‐1, max = 1) # Logistic gradient gradient <-‐ function(partition) { X <-‐ partition[,1]; Y <-‐ partition[,-‐1] t(X) %*% (1/(1 + exp(-‐Y * (X %*% weights))) -‐ 1) * Y } #Iterate weights <-‐ weights -‐ reduce( lapplyPartition(pointsRDD, gradient), "+")
- 38. How does it work ? R Shell rJava Spark Context Spark Executor Spark Executor RScript RScript Data: RDD[Array[Byte]] Func: Array[Byte]