The DAGScheduler is responsible for computing the DAG of stages for a Spark job and submitting them to the TaskScheduler. The TaskScheduler then submits individual tasks from each stage for execution and works with the DAGScheduler to handle failures through task and stage retries. Together, the DAGScheduler and TaskScheduler coordinate the execution of jobs by breaking them into independent stages of parallel tasks across executor nodes.

1. Core services behind Spark Job
Execution
DAGScheduler and TaskScheduler

2. Agenda
● Spark Architecture
● RDD
● Job-Stage-Task
● Bird Eye View
● DAGScheduler
● TaskScheduler

3. Spark Architecture

4. Spark Architecture
Driver
Hosts SparkContext
Cockpit of Jobs and Task
Execution
Schedules Tasks to run
on executors
Contains DAGScheduler
and TaskScheduler
Executor
Static allocation vs
Dynamic allocation
Sends Heartbeat and Metrics
Provides In memory storage
for RDD
Communicates directly with
driver to execute task

5. RDD — Resilient Distributed Dataset
● RDD is the primary data abstraction in Spark and the core of Spark
● Motivation for RDD
● Features of RDD

6. RDD — Resilient Distributed Dataset
● RDD creation
● RDD Lineage
● Lazy Execution
● Partitions

7. RDD operations
● Transformations
● Actions

8. A sample Spark program
val conf = new SparkConf().setAppName(appName).setMaster(master)
val sc = new SparkContext(conf)
val file = sc.textFile("hdfs://...") // This is an RDD
val errors = file.filter(_.contains("ERROR")) // This is an RDD
val errorCount = errors.count() // This is an “action”

9. Job-Stage-Task
What is a Job?
Top level work item
Computation job ==
Computation Partition of RDD
Target RDD Lineage

10. Job-Stage-Task
Job divided into stages
Logical Plan → Physical plan (execution unit)
Set of Parallel task
Stage Boundary (Shuffle)
Computation of stage triggers parents stage
execution

11. Types of Stages
ShuffleMapStage:
Intermediate stage in execution DAG
Saves map output → fetched later
Pipelined operations before shuffle
Can be Shared across jobs
ResultStage:
Final stage executing action
Works on one or many partitions

12. Job-Stage-Task
Smallest unit of execution
Comprises of function and placement
preference
Task operate on a single partition
Launched on executor and ran there

13. Types of Task
ShuffleMapTask:
Intermediate stage
Returns MapStatus
ResultStageTask:
Last Stage
Returns output to driver

15. DAGScheduler
● Initialization
● Stage-oriented scheduling (Logical - Physical)
● DAGSchedulerEvent (Job or Stage)
● Stage Submissions

16. DAGScheduler Responsibilities
● Computes an execution DAG, submits to stages
to TaskScheduler

17. DAGScheduler Responsibilities
● Computes an execution DAG
● Determines the preferred locations to run each
task on, keeps track of cached RDD

18. DAGScheduler Responsibilities
● Computes an execution DAG
● Determines the preferred locations to run each
task on
● Handles failures due to shuffle output files being
lost (FetchFailed, ExecutorLost)

19. DAGScheduler Responsibilities
● Computes an execution DAG
● Determines the preferred locations to run each
task on
● Handles failures due to shuffle output files being
lost (FetchFailed, ExecutorLost)
● Stage retry

20. TaskScheduler
● LifeCycle

21. TaskSet and TaskSetManager
● What is a TaskSet?
○ Fully independent sequence of task
● Why TaskSetManager
● Responsibilities of TaskSetManager
○ Scheduling tasks in a TaskSet
○ Completion notification
○ Retry and Abort
○ Locality preference

22. TaskScheduler’s Responsibilities
● responsible for submitting tasks for execution for every stage
● works closely with DAGScheduler for resubmission of stage
● tracks the executors in a Spark application (executorHeartBeat and executorLost)

23. Thank You