This document provides an overview of Apache Spark, including its goal of providing a fast and general engine for large-scale data processing. It discusses Spark's programming model, components like RDDs and DAGs, and how to initialize and deploy Spark on a cluster. Key aspects covered include RDDs as the fundamental data structure in Spark, transformations and actions, and storage levels for caching data in memory or disk.

1. Apache Spark 101
June 2016
Abdullah Cetin CAVDAR
@accavdar
#AnkaraSparkDay

2. Apache Spark's Goal

3. Apache Spark
is a fast and general engine for
large-scale data processing

4. Most Active Project in Big Data
Spark Survey 2015

5. Top 10 Industries Using Spark

6. Many Types of Product

7. Spark Engine
unified engine across diverse workloads &
environments

8. Programming Languages

9. Open Source Spark Ecosystem

10. Most Important Aspects

11. Spark
Programming
Model

12. Challenge?
Fast data sharing across parallel
jobs

13. Data Sharing in MapReduce

14. Data Sharing in Apache Spark

15. Components

16. Cluster Managers

17. Initializing Apache Spark
SparkConf and SparkContext

18. Apache Spark Shell
Python and Scala

19. RDD (Resilient Distributed Dataset)
An RDD is a read-only collection of objects
partitioned across a set of machines that
can be rebuilt if a partition is lost

20. RDD
Read-Only = Immutable
Parallelism
Caching

21. RDD
Partitioned = Distributed
More partitions = More parallelism

22. RDD
Rebuilt = Resilient
Recover lost data partitions
By replaying data lineage

23. RDD Operations

24. RDD Operations

25. Partitions
logical division of data / basic unit of
parallelisim

26. RDD Lineage
Lazy Evaluation

27. DAG (Directed Acyclic Graph)

28. Transformation & Action

29. RDD Creation
Parallelizing a collection
into driver application memory
for only prototyping and testing
Loading an external data set
le://, hdfs://, s3n://
sc.textFile()
sc.hadoopFile(), sc.newAPIHadoopFile()
sqlContext.read()

30. Word Count :)

31. Driver & Workers
Main Program is executed on Driver
Transformations are executed on Workers
Actions transfer from Workers to Driver
Driver cannot get data from executors except action and accumulator

32. RDD Dependencies
Minimize shuffle / Wide
Dependencies

33. RDD Persistence / Caching
persist() or
cache()
Without cache, it will restart from the rst RDD
LRU (Least Recently Used)
Default Storega Level: MEMORY_ONLY

34. Storage Levels

35. Shared Variables
Accumulators and Broadcast
Variables

36. Accumulators
Used to implement counters or sums

37. Broadcast Variables
Keep a read-only variable cached on each
machine

38. Spark UI
Default port 4040

39. Deploying to a Cluster
Use spark-submit

40. Data Frames & Performance
Distributed collection of rows organized
into named columns

42. Tips
Avoid groupByKey and wide dependencies
Use enough number of partitions
Use coalesce not to make too many small les
Be cautious on Serialization/Deserialization

43. Major Features in 2.0

44. Thank you

45. #AnkaraSparkDay