Downloaded 189 times
![Example: word_count.R
library(SparkR)
lines
<-‐
textFile(sc,
“hdfs://my_text_file”)
words
<-‐
flatMap(lines,
function(line)
{
strsplit(line,
"
")[[1]]
})
wordCount
<-‐
lapply(words,
function(word)
{
list(word,
1L)
})](https://image.slidesharecdn.com/e4p5z1b4thurensvl2lt-signature-9f1097b540f1c004e6501073d62b3a70af4704a05c67c402ba256f1c4f6d4951-poli-141121182858-conversion-gate02/85/SparkR-Enabling-Interactive-Data-Science-at-Scale-11-320.jpg)
![Example: word_count.R
library(SparkR)
lines
<-‐
textFile(sc,
“hdfs://my_text_file”)
words
<-‐
flatMap(lines,
function(line)
{
strsplit(line,
"
")[[1]]
})
wordCount
<-‐
lapply(words,
function(word)
{
list(word,
1L)
})
counts
<-‐
reduceByKey(wordCount,
"+",
2L)
output
<-‐
collect(counts)](https://image.slidesharecdn.com/e4p5z1b4thurensvl2lt-signature-9f1097b540f1c004e6501073d62b3a70af4704a05c67c402ba256f1c4f6d4951-poli-141121182858-conversion-gate02/85/SparkR-Enabling-Interactive-Data-Science-at-Scale-12-320.jpg)
More Related Content
Similar to SparkR: Enabling Interactive Data Science at Scale
![谷歌留痕技术教程[ 𝙩𝙤𝙥 𝟮𝟯𝟯. 𝙘 𝙤𝙢 ]](https://cdn.slidesharecdn.com/ss_thumbnails/top233-260130173900-2eb784f9-thumbnail.jpg?width=640&height=640&fit=bounds)
SparkR: Enabling Interactive Data Science at Scale
- 1. SparkR: Enabling Interactive Data Science at Scale Shivaram Venkataraman Zongheng Yang
- 2. Fast ! ScalableFlexible
- 3.
- 4. Fast ! Scalable Flexible Statistics ! Plots Packages
- 5. Outline SparkR API Live Demo Design Details
- 6. RDD Parallel Collection Transformations map filter groupBy … Actions count collect saveAsTextFile …
- 7. R + RDD= R2D2
- 8. R + RDD= RRDD lapply lapplyPartition groupByKey reduceByKey sampleRDD collect cache filter … broadcast includePackage textFile parallelize
- 9. SparkR – Rpackage for Spark R RRDD Spark
- 10. Example: word_count.R library(SparkR) lines <-‐ textFile(sc, “hdfs://my_text_file”)
- 11. Example: word_count.R library(SparkR) lines <-‐ textFile(sc, “hdfs://my_text_file”) words <-‐ flatMap(lines, function(line) { strsplit(line, " ")[[1]] }) wordCount <-‐ lapply(words, function(word) { list(word, 1L) })
- 12. Example: word_count.R library(SparkR) lines <-‐ textFile(sc, “hdfs://my_text_file”) words <-‐ flatMap(lines, function(line) { strsplit(line, " ")[[1]] }) wordCount <-‐ lapply(words, function(word) { list(word, 1L) }) counts <-‐ reduceByKey(wordCount, "+", 2L) output <-‐ collect(counts)
- 13.
- 14.
- 15. A b ||Ax − b ||2 Minimize x = (ATA)−1ATb
- 16. How does thiswork ?
- 17. Dataflow Local Worker Worker
- 18. Dataflow Local R Worker Worker
- 19. Dataflow Local RSpark Context Java Spark Context JNI Worker Worker
- 20. Dataflow Local Worker R Spark Context Worker Java Spark Context JNI Spark Executor exec R Spark Executor exec R
- 22.
- 24. Dataflow Local Worker R Spark Context Worker Java Spark Context JNI Spark Executor exec R Spark Executor exec R
- 25. Pipelined RDD words <-‐ flatMap(lines,…) wordCount <-‐ lapply(words,…) Spark Executor exec R Spark Executor R exec
- 26. Pipelined RDD Spark Executor exec R Spark Executor R exec Spark Executor exec R R Spark Executor
- 27. Alpha developer release One line install ! install_github("amplab-‐extras/SparkR-‐pkg", subdir="pkg")
- 28. SparkR Implementation Verysimilar to PySpark Spark is easy to extend 329 lines of Scala code 2079 lines of R code 693 lines of test code in R
- 29. EC2 setup scripts All Spark examples MNIST demo YARN, Windows support Also on github
- 30. Developer Community 13contributors (10 from outside AMPLab) Collaboration with Alteryx
- 31. On the Roadmap High level DataFrame API Integrating Spark’s MLLib from R Merge with Apache Spark
- 32. SparkR RDD àdistributed lists Run R on clusters Re-use existing packages Combine scalability & utility
- 33. SparkR https://github.com/amplab-extras/SparkR-pkg ShivaramVenkataraman shivaram@cs.berkeley.edu Zongheng Yang zongheng.y@gmail.com SparkR mailing list sparkr-dev@googlegroups.com