This document compares MapReduce and Spark frameworks. It discusses their histories and basic functionalities. MapReduce uses input, map, shuffle, and reduce stages, while Spark uses RDDs (Resilient Distributed Datasets) and transformations and actions. Spark is easier to program than MapReduce due to its interactive mode, but MapReduce has more supporting tools. Performance benchmarks show Spark is faster than MapReduce for sorting. The hardware and developer costs of Spark are also lower than MapReduce.

1. MapReduce
vs/and Spark
Tudor Lapusan
BigData Romanian Tour - Timisoara

2. History

3. MapReduce basic functionalities
● Fault tolerance
● Monitoring &
status updates
● Scalability

4. Hadoop MapReduce
Input Map Reduce Output

5. Hadoop MapReduce
Input Map Shuffle Reduce Output

6. MapReduce DAG
A
D
B
C
E
F

7. Spark
● RDD
● Operations : Transformations and Actions

8. RDD - Resilient Distributed Dataset
RDD is fault-tolerant collection of elements
distributed across many servers on which we can
perform parallel operations.

9. RDD
Scala code
val data = Array(1, 2, 3, 4, 5, 6, 7, 8)
val rddData = sc.parallelize(data)

10. RDD
Scala code
val rddFile = sc.textFile("data.txt")

11. RDD persistence
MEMORY_ONLY
MEMORY_AND_DISK
MEMORY_ONLY_SER
MEMORY_AND_DISK_SER
DISK_ONLY
MEMORY_ONLY_2
MEMORY_AND_DISK_2
OFF_HEAP

12. Transformations
RDD 1
RDD 2
Transformations are operations on RDDs that return new
RDDs

13. Transformations
RDD 1
InputRDD
{1,2,3,4,5,6}
MapRDD
{2,3,4,5,6,7}
FilterRDD
{1,2,3,5,6}
map x => x +1 filter x => x != 4

14. Actions
RDD 1
Actions are the operations on RDD which return a final value
or write the data to an external storage system.
RDD 1

15. Actions
RDD 1
InputRDD
{1,2,3,4,5,6}
MapRDD
{2,3,4,5,6,7}
FilterRDD
{1,2,3,5,6}
map x => x +1 filter x => x != 4
count()=6 take(2)={1,2} saveAsTextFile()

16. Spark DAG
RDD 1
RDD 2
RDD 4
RDD 6
RDD 3
RDD 5
Action
Transformation
Stage

17. Spark DAG vs MapReduce DAG
RDD 1
RDD 2
RDD 4
RDD 6
RDD 3
RDD 5
A
B
D C
E
F

18. Programing languages
MapReduce
Java
Ruby
Perl
Python
PHP
R
C++
Spark
Java
Scala
Python

19. Easy of use
- Spark is easier to program and include an
interactive mode.
- Hadoop MapReduce is harder to program
but many tools are available to make it
easier.

20. Performance : Sort Benchmark 2013

21. Performance : Sort Benchmark 2014

22. Costs

23. Costs : hardware recommendation
Spark MapReduce Hadoop
Cores 8-16 4
Memory 8GB to hundreds of GB 24GB
Disks 4-8 4-6 one-TB disks
Network 10GB or more 1GB Ethernet
Spark recommendation Hortonworks recommendation

24. Costs : developers

25. Questions
tudor.lapusan@gmail.com
@tlapusan