Download as PDF, PPTX
val messages = KafkaUtils.createStream[String, String, StringDecoder,
StringDecoder]
(
sc,
kafkaParams,
Map(topic -> threadsPerTopic),
storageLevel = StorageLevel.MEMORY_ONLY_SER
)
val revenue = messages.filter(m => m.contains(“[PURCHASE]”))
.map(p => p.split(“t”)(4)).reduceByKey(_ + _)
Mini-batch](https://image.slidesharecdn.com/spark-150605080255-lva1-app6892/85/Introduction-to-Apache-Spark-Ecosystem-11-320.jpg)
![Using the Shell
Launching:
Modes:
MASTER=local ./spark-shell # local, 1 thread
MASTER=local[2] ./spark-shell # local, 2 threads
MASTER=spark://host:port ./spark-shell # cluster
spark-shell
pyspark (IPYTHON=1)](https://image.slidesharecdn.com/spark-150605080255-lva1-app6892/85/Introduction-to-Apache-Spark-Ecosystem-18-320.jpg)
Uploaded byBojan Babic
Introduction to Apache Spark Ecosystem
This document introduces Apache Spark, an open-source cluster computing system that provides fast, general execution engines for large-scale data processing. It summarizes key Spark concepts including resilient distributed datasets (RDDs) that let users spread data across a cluster, transformations that operate on RDDs, and actions that return values to the driver program. Examples demonstrate how to load data from files, filter and transform it using RDDs, and run Spark programs on a local or cluster environment.
More Related Content
Similar to Introduction to Apache Spark Ecosystem
Introduction to Apache Spark Ecosystem
- 1.
- 2.
- 3.
- 4.
- 5. What is Spark? Efficient •General execution graphs • In-memory storage Usable • Rich APIs in Java, Scala, Python • Interactive shell Fast and Expressive Cluster Computing System Compatible with Apache Hadoop
- 6.
- 7. Key Concepts Resilient DistributedDatasets • Collections of objects spread across a cluster, stored in RAM or on Disk • Built through parallel transformations • Automatically rebuilt on failure • in hadoop work RDD corresponds to partition elsewhere dataframe Operations • Transformations (e.g. map, filter, groupBy) • Actions (e.g. count, collect, save)
- 8. Working With RDDs RDD RDD RDD RDD Transfor mations Acti on Value linesWithSpark= textFile.filter(l => l.contains("Spark”)) linesWithSpark.count() 74 linesWithSpark.first() # Apache Spark textFile = sc.textFile(”/var/log/hadoop/somelog”)
- 9. Spark (batch) example Loaderror messages from a log into memory, then interactively search for various patterns val sc = new SparkContext(config) val lines = spark.textFile(“hdfs://...”) val messages = lines.filter(l => l.startswith(“ERROR”)) .map(e => e.split(“t“)(2)) messages.cache() messages.filter(s => s.contains(“mongo”)).count() messages.filter(s => s.contains(“500”)).count() Base RDD Transformed RDD Action
- 10.
- 11. Spark Streaming example valsc = new StreamingContext(conf, Seconds(10)) val kafkaParams = Map[String, String](...) val messages = KafkaUtils.createStream[String, String, StringDecoder, StringDecoder] ( sc, kafkaParams, Map(topic -> threadsPerTopic), storageLevel = StorageLevel.MEMORY_ONLY_SER ) val revenue = messages.filter(m => m.contains(“[PURCHASE]”)) .map(p => p.split(“t”)(4)).reduceByKey(_ + _) Mini-batch
- 12. Fault Recovery RDDs tracklineage information that can be used to efficiently recompute lost data val msgs = textFile.filter(l => l.contains(“ERROR”)) .map(e => e.split(“t”)(2)) HDFS File Filtered RDD Mapped RDD filter (func = startsWith (…)) map (func = split (...))
- 13.
- 14. Software Components • Sparkruns as a library in your program (1 instance per app) • Runs tasks locally or on cluster – Mesos, YARN or standalone mode • Accesses storage systems via Hadoop InputFormat API – Can use HBase, HDFS, S3, … Your application SparkContext Local threads Cluster manager Worker Spark executor Worker Spark executor HDFS or other storage
- 15. Task Scheduler • Generaltask graphs • Automatically pipelines functions • Data locality aware • Partitioning aware to avoid shuffles = cached partition = RDD joi n filte r groupB y Stage 3 Stage 1 Stage 2 A: B : C : D : E : F : ma p
- 16. Advanced Features • Controllablepartitioning – Speed up joins against a dataset • Controllable storage formats – Keep data serialized for efficiency, replicate to multiple nodes, cache on disk • Shared variables: broadcasts, accumulators
- 17.
- 18. Using the Shell Launching: Modes: MASTER=local./spark-shell # local, 1 thread MASTER=local[2] ./spark-shell # local, 2 threads MASTER=spark://host:port ./spark-shell # cluster spark-shell pyspark (IPYTHON=1)
- 19. SparkContext • Main entrypoint to Spark functionality • Available in shell as variable sc • In standalone programs, you’d make your own (see later for details)
- 20. RDD Operators • map •filter • groupBy • sort • union • join • leftOuterJoin • rightOuterJoin • reduce • count • fold • reduceByKey • groupByKey • cogroup • cross • zip sample take first partitionBy mapWith pipe save ...
- 21. Add Spark toYour Project • Scala: add dependency to build.sbt libraryDependencies ++= Seq( ("org.apache.spark" %% "spark-core" % "1.2.0"). exclude("org.mortbay.jetty", "servlet-api") ... ) • make sure you exclude overlapping libs
- 22. and add Spark-Streaming •Scala: add dependency to build.sbt libraryDependencies += "org.apache.spark" % "spark-streaming_2.11" % "1.2.0"
- 23.
- 24. Other Iterative Algorithms Timeper Iteration (s)
- 25.
- 26. Conclusion • Spark offersa rich API to make data analytics fast: both fast to write and fast to run • Achieves 100x speedups in real applications • Growing community with 25+ companies contributing
- 27. Thanks!!! disclosure: some partsof this presentation have been inspired by Matei Zaharia ampmeeting in Berlkey