Spark is a big data processing tool built with Scala that runs on the Java Virtual Machine (JVM). It is up to 100 times faster than Hadoop for iterative jobs because it keeps intermediate data in memory rather than writing to disk. Spark uses Resilient Distributed Datasets (RDDs) that allow data to be kept in memory across transformations and actions. RDDs also maintain lineage graphs to allow recovery from failures.

1. A big data processing tool built with Scala and runs on JVM
Introduce to
Spark
ADB 2017
Yen Hao Huang
1

2. Big Data
● 4Vs
○ Volume/Variety/Velocity/Veracity
Due to the rise of Big Data, faster tools are required for
processing data.
2

3. Hadoop
3

4. Hadoop
● A platform to store and process large scale data
● Features
○ Scalable
○ Economical : many cheap servers
○ Flexible : schema-less
○ Reliable : replicas
4

5. Hadoop MapReduce
● Map
- Divide job to multiple tiny tasks and distribute to
servers
● Reduce
- Summary the results from those servers
5

6. Hadoop MapReduce
6
Figure Refence

7. ● File I/O - write the middle process data to disk
Hadoop - Bottleneck
Iteration Iteration
Read Read WriteWrite
7

8. Spark
8

9. RDD
In-memory computation framework
9

10. RDD (Resilient Distributed Dataset)
● Write the middle process data to memory
● 10 - 100 times faster than hadoop
Iteration Iteration
Read Memory Read WriteMemory Write
RDD
10

11. Spark
● Features
○ Speed
○ Ease of use : Scala、Python、Java、R
○ Supports hadoop : HDFS、MapReduce
○ Accessibility : runs on many platforms
11

12. RDD Features
● Computations
○ Transformation - Lazy compute
○ Action - Execute the computations
○ Persistence - Keep RDD in ram/ disk
Transformation
RDD OutputAction
12

13. ● Error Fixing
RDD Lineage
RDD1 RDD2
Transformation Action
[ 7, 10 ]
[ 2, 3 ] [ ?, ? ]f(x) = x2
+1
13
RDD2RDD1
2
[ 7, 10 ]
Fix !
1

14. Spark Functionality
14