This document provides an overview of Apache Spark, including: - The problems of big data that Spark addresses like large volumes of data from various sources. - A comparison of Spark to existing techniques like Hadoop, noting Spark allows for better developer productivity and performance. - An overview of the Spark ecosystem and how Spark can integrate with an existing enterprise. - Details about Spark's programming model including its RDD abstraction and use of transformations and actions. - A discussion of Spark's execution model involving stages and tasks.

1. Apache Spark Core
Nguyen Trung Thanh

2. Agenda
• Problems of Big Data
• The Hadoop Approach
• Comparison to Existing Techniques
• Apache Spark Ecosystem
• How Spark can be integrated into an existing enterprise
• Programming Model
• Q&A

3. PROBLEMS OF BIG DATA
The flood of data is coming from many
sources:
~ 6 billion messages per day
~ 2 billion page views per day
~ 9 petabytes of storage
~ 90 gigabytes of log data per day
Machine logs, RFID readers, sensor
networks, vehicle GPS traces…

4. PROBLEMS OF BIG DATA

5. SOLUTION:Distributed Processing Engine

6. DISTRIBUTED SYSTEM

7. THE HADOOP APPROACH

8. HADOOP:No Unified Vision

9. MAPREDUCE1

10. MAPREDUCE2

11. YARN

12. YARN

13. WE CAN DO BETTER WITH
Better Developer Productivity
Rich APIs for Scala, Java, and Python
Interactive shell
Better Performance
General execution graphs
In-memory storage

14. SPARK INTEGRATED EXISTING ENTERPRISE

15. SPARK CORE CONCEPT

16. SPARK ECOSYSTEM

17. SPARK INTEGRATED EXISTING ENTERPRISE

18. CLUSTER MANAGER
• Standalone : a simple cluster manager
included with Spark that makes it easy to set
up a cluster.
• Apache Mesos : a general cluster manager
that can also run Hadoop MapReduce and
service applications.
• Hadoop YARN : the resource manager in
Hadoop 2.

19. SPARK INTEGRATED EXISTING ENTERPRISE

20. SPARK INTEGRATED EXISTING ENTERPRISE

21. SPARK INTERNAL COMPONENT

22. PROGRAMMING MODEL
• Native support for
multiple languages with
identical APIs
• Use of closures,
iterations, and other
common language
constructs to minimize
code
• Unified API for batch
and streaming
Python
lines = sc.textFile(...)
lines.filter(lambda s: “ERROR” in s).count()
Scala
val lines = sc.textFile(...)
lines.filter(s => s.contains(“ERROR”)).count()
Java
JavaRDD<String> lines = sc.textFile(...);
lines.filter(new Function<String, Boolean>() {
Boolean call(String s) {
return s.contains(“error”);
}
}).count();
 High-Productivity Language Support

23. PROGRAMMING MODEL

24. RDD ABSTRACTION
Immutable distributed collection of data, which
is partitioned across machines in a cluster.
• Resilient: It can be recreatedwhen data in
memory is lost
• Distributed: stored in memory across the
cluster
• Dataset: data that comes from file or created
programmatically

25. PROGRAMMING MODEL
How Spark Works:
• SRDD: Parallel collection with partitions
• User application create RDDs, transform them, and
run actions.
• This results in a DAG (Directed Acyclic Graph) of
operators.
• DAG is compiled into stages
• Each stage is executed as a series of Task (one Task
for each Partition).

26. RDD ABSTRACTION
 Programming Interface: Programmer can perform 3 types of operations:
Transformations
• Create a new
dataset from and
existing one.
• Lazy in nature. They
are executed only
when some action is
performed.
• Example :
• Map(func)
• Filter(func)
• Distinct()
Actions
• Returns to the
driver program a
value or exports
data to a storage
system after
performing a
computation.
• Example:
• Count()
• Reduce(funct)
• Collect
• Take()
Persistence
• For caching datasets
in-memory for
future operations.
• Option to store on
disk or RAM or
mixed (Storage
Level).
• Example:
• Persist()
• Cache()

27. RDD OPERATION:TRASFORMATION

28. RDD OPERATION:ACTION

29. RDD OPERATION
val inputRDD = sc.textFile("log.txt")
val errorsRDD = inputRDD.filter(line => line.contains("error"))
val warningsRDD = inputRDD.filter(line => line.contains(“warnings"))
val badLineRDD=errorsRDD.union(warningsRDD)

30. RDD OPERATION
val input = sc.parallelize(List(1, 2, 3, 4))
val result = input.map(x => x * x)
println(result.collect().mkString(","))

31. RDD OPERATION

32. RDD OPERATION

33. WORKING WITH Key/Value PAIRS
val pairs = lines.map(x => (x.split(" ")(0), x))
pairs.filter{case (key, value) => value.length < 20}

34. WORKING WITH Key/Value PAIRS
rdd.mapValues(x => (x, 1)).reduceByKey((x, y) => (x._1 + y._1, x._2 + y._2))

35. WORD COUNT

36. WORD COUNT

37. WORD COUNT

38. WORD COUNT

39. WORD COUNT

40. WORD COUNT

41. WORD COUNT

42. EXECUTION COMPONENTS
 Task : The fundamental unit of execution in Spark
 Stage: Set of Tasks that run parallel
 DAG: Logical Graph of RDD operations
 RDD: Parallel dataset with partitions

43. EXECUTION COMPONENTS

44. EXECUTION COMPONENTS

45. EXECUTION COMPONENTS

46. EXECUTION COMPONENTS

47. PHYSICAL EXECUTION PLAN

48. EXECUTION PLAN

49. PHYSICAL EXECUTION PLAN

50. PHYSICAL EXECUTION PLAN

51. STAGES AND TASKS

52. SPARK ON HADOOP CLUSTER

53. SPARK USERS

54. CONCLUSION
• Because of In-memory processing, computations are very
fast. Developers can write iterative algorithms without
writing out a result set after each pass through the data.
• Suitable for scenarios when sufficient memory available in
your cluster.
• It provides an integrated framework for advanced analytics
like Graph processing, Stream Processing, Machine Learning
etc. This simplifies integration.
• It’s community is expanding and development is happening
very aggressively.
• It’s comparatively newer than Hadoop and only few users.

55. Q&A