Apache Spark is a cluster computing platform designed to be fast and general-purpose. It provides a unified analytics engine for large-scale data processing across SQL, streaming, machine learning, and graph processing. Spark programs can be written in Java, Scala, Python and R. It works by building resilient distributed datasets (RDDs) that can be operated on in parallel. RDDs support transformations like map, filter and join and actions like count, collect and save. Spark also provides caching of RDDs in memory for improved performance.

1. Lightning-fast cluster computing
Apache Spark

2. What is Apache Spark?
Cluster computing platform designed to be fast and general-purpose.
Fast
Universal
Highly Accessible

3. Unified stack

4. Comparison with MR
val textFile = spark.textFile("hdfs://...")
val counts = textFile.flatMap(line =>
line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")

5. Examples:Word Count
val sc = new SparkContext(...)
val textFile = sc.textFile("hdfs://...")
val counts = textFile.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")

6. val sc = new SparkContext(...)
val inputRDD = sc.textFile("log.txt")
val errorsRDD = inputRDD.filter(line => line.contains("error"))
val warningsRDD = inputRDD.filter(line => line.contains("warning"))
val badLinesRDD = errorsRDD.union(warningsRDD)
badLinesRDD.persist()
badLinesRDD.count()
badLinesRDD.collect()
Examples:Log Mining

7. How it works?
RDD
Resilient
Distributed
Dataset

8. Example Hadoop RDD
partitions = One per HDFS block
dependencies = none
compute = read corresponding block
preferredLocations = HDFS block locations
partitioner = none
Advanced: RDD as interface

9. Direct Acyclic Graph (DAG)
hadoopRDD
errorsRDD warningsRDD
badLinseRDD
filterfilter
union

10. Function Name Purpose Example
map() Apply a function to each element in the RDD and return an RDD of the result. rdd.map(x => x + 1)
flatMap() Apply a function to each element in the RDD and return an RDD of the contents of
the iterators returned. Often used to extract words.
rdd.flatMap(x => x.to(3))
filter() Return an RDD consisting of only elements that pass the condition passed to
filter().
rdd.filter(x => x != 1)
distinct() Remove duplicates. rdd.distinct()
union() Produce an RDD containing elements from both RDDs. rdd.union(other)
intersection() RDD containing only elements found in both RDDs. rdd.intersection(other)
join() Perform an inner join between two RDDs. rdd.join(other)
groupByKey() Group values with same key rdd.groupByKey(other)
RDD Transformations

11. RDD actions
Function Name Purpose Example
count() Number of elements in RDD rdd.count()
collect() Return all elements from the RDD rdd.collect()
saveAsTextFile() Saves RDD elements to an external
storage system
rdd.saveAsTextFile(“hdfs://...”)
take(num) Return num elements from RDD rdd.take(10)
reduce(func) Combine the elements of the RDD
together in parallel (e.g., sum)
rdd.reduce((x, y) => x + y)
takeOrdered(num)(ordering) Return num elements regarding provided
ordering
rdd.takeOrdered(2)(myOrdering)

12. RDD Caching
Level Space Used CPU Time In Memory On disk Comments
MEMORY_ONLY High Low Y N
MEMORY_ONLY_SER Low High Y N
MEMORY_AND_DISK High Medium Some Some Spills to disk if there is too
much data to fit in memory.
MEMORY_AND_DISK_SER Low High Some Some Spills to disk if there is too
much data to fit in memory.
Stores serialized
representation in memory.
DISK_ONLY Low High N Y

13. How it works?
Main program which controls the flow
Driver Executors
Nodes that execute actions

14. How it works?
DAG
Scheduler
Coordination between
RDDs, driver and
nodes

15. What is Spark Application

16. Advanced Topics: Stages

17. Advanced Topics: Shuffling

18. Spark Stack
SQL
Streaming
Machine Learning
GraphX

19. if not… DEMO everyone?
?