Dynamic resource allocation in Apache Spark allows executors to be dynamically added or removed based on the workload of applications. Extra executors are added when applications have pending tasks to help balance workload, and idle executors are removed to free resources for other applications. The dynamic allocation policies control when executors are requested or removed based on factors like pending tasks and executor idle time. An external shuffle service is also used to improve shuffle performance.

1. Dynamic Resource Alloca1on
in Apache Spark
Yuta Imai
@imai_factory

2. 1. RDD Graph
val text = "Hello Spark, this is my first Spark application."
val textArray = text.split(" ").map(_.replaceAll(" ",""))
val result = sc.parallelize(textArray)
.map(item => (item, 1))
.reduceByKey((x,y) => x + y)
.collect()

3. Array Array ParallelCollec1onRDD
Par11on0
Par11on1
Par11on2
Par11on3
MapPar11onsRDD
Par11on0
Par11on1
Par11on2
Par11on3
ShuffledRDD
Par11on0
Par11on1
sc.parallelize() .map(…) .reduceByKey(…) .collect()
2. DAG Scheduler

4. Array Array ParallelCollec1onRDD
Par11on0
Par11on1
Par11on2
Par11on3
MapPar11onsRDD
Par11on0
Par11on1
Par11on2
Par11on3
ShuffledRDD
Par11on0
Par11on1
sc.parallelize() .map(…) .reduceByKey(…) .collect()
2. DAG Scheduler
Narrow Dependency Shuffle Dependency

5. Array Array ParallelCollec1onRDD
Par11on0
Par11on1
Par11on2
Par11on3
MapPar11onsRDD
Par11on0
Par11on1
Par11on2
Par11on3
ShuffledRDD
Par11on0
Par11on1
sc.parallelize() .map(…) .reduceByKey(…) .collect()
2. DAG Scheduler
Narrow Dependency Shuffle Dependency
Stage0 Stage1
Task0
Task1
Task2
Task3
Task4
Task5

6. 3. Task Scheduler
Par11on0
Par11on1
Par11on2
Par11on3
Par11on0
Par11on1
Par11on2
Par11on3
Task0
Task1
Task2
Task3
Executors

7. Shuffle File
iterator.map(…).map(...)...
Executor
Thread
Storage
Worker Node
iterator.map(…).map(...)...
Executor
Thread
Worker Node

8. DYNAMIC RESOURCE ALLOCATION

9. Dynamic Resource Alloca1on
• Adds extra executors to an app which has
pending tasks.
– Offloads challenge for exact resource planning for
an app.
• Removes idle executors from an app.
– Helps a long running app to free idle executors.

10. Overview
Tasks
Executors

11. Overview
Tasks
Executors
Insufficient capacity

12. Overview
Tasks
Executors
Insufficient capacity

13. Overview
Tasks
Executors
Insufficient capacity

14. Overview
Tasks
Executors
Insufficient capacity Op1mal capacity

15. Overview
Tasks
Executors
✔ ✔
Insufficient capacity Op1mal capacity Idle executors

16. Tasks
Executors
✔ ✔
Overview
Insufficient capacity Op1mal capacity Idle executors
Op1mal capacity

17. Request Policy
• An app starts with user specified # of executors.
./bin/spark-submit
--class <main-class>
--master <master-url>
--num-executors <# of executors>
• Ader spark.dynamicAlloca1on.schedulerBacklogTimeout(sec), App
starts reques1ng new executors, if it has pending task(s).
• App requests new executors every
spark.dynamicAlloca1on.sustainedSchedulerBacklogTimeout(sec),
with doubling # of requests like 1, 2, 4, 8, 16…

18. Remove Policy
• An app removes an executor when it has been idle for more
than spark.dynamicAlloca1on.executorIdleTimeout seconds.

19. External Shuffle Service
iterator.map(…).map(...)...
Executor
Thread
Storage
Worker Node
iterator.map(…).map(...)...
Executor
Thread
Worker Node

20. External Shuffle Service
iterator.map(…).map(...)...
Executor
Thread
Storage
Worker Node
iterator.map(…).map(...)...
Executor
Thread
Worker Node

21. External Shuffle Service
iterator.map(…).map(...)...
Executor
Thread
Storage
Worker Node
iterator.map(…).map(...)...
Executor
Thread
Worker Node
Shuffle
Service
Shuffle
Service