Spark Summit EU talk by Luc Bourlier
•
3 likes•1,734 views
Dynamic Resource Allocation, Do More With Your Cluster
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Spark Summit EU talk by Luc Bourlier
Similar to Spark Summit EU talk by Luc Bourlier (20)
More from Spark Summit
More from Spark Summit (20)
Recently uploaded
Recently uploaded (20)
Spark Summit EU talk by Luc Bourlier
- 1. DYNAMIC RESOURCE ALLOCATION, DO MORE WITH YOUR CLUSTER Luc Bourlier Lightbend
- 2. ● Dynamic Resource Allocation ● ^^ in Spark ● External Shuffle Service ● Configuration ● Demo ● Spark Streaming
- 4. Dynamic Resource Allocation Cluster I’d like some resources for a job
- 5. Dynamic Resource Allocation Cluster Oki. Thanks
- 9. Dynamic Resource Allocation Cluster Hmm, actually, I don’t need all this power anymore.
- 11. Why? ● Shared cluster ● Optimization of resource usage When? ● variable load job Dynamic Resource Allocation Cluster
- 14. Spark Dynamic Allocation Cluster Manager Worker Node Executor Worker Node Worker Node Executor Driver Scheduler(s) need 2 executors
- 15. tasks are waiting too long Spark Dynamic Allocation Cluster Manager Worker Node Executor Worker Node Worker Node Executor Driver Scheduler(s) need 1 more executor Executor
- 16. executor has been idle for a while Spark Dynamic Allocation Cluster Manager Worker Node Executor Worker Node Worker Node Executor Driver Scheduler(s) Executor terminate the executor
- 18. External Shuffle Service ● Did we lose any data?
- 19. External Shuffle Service Shuffle write Shuffle fetch Map Task Reduce Task Aggregator bucket bucket bucket Aggregator Map Task Reduce Task Aggregator bucket bucket bucket Aggregator Map Task Reduce Task Aggregator bucket bucket bucket Aggregator
- 20. External Shuffle Service Shuffle write Shuffle fetch Map Task Reduce Task Aggregator bucket bucket bucket Aggregator Map Task Reduce Task Aggregator bucket bucket bucket Aggregator Map Task Reduce Task Aggregator bucket bucket bucket Aggregator
- 21. External Shuffle Service Shuffle write Shuffle fetch Map Task Reduce Task Aggregator bucket bucket bucket Aggregator Map Task Reduce Task Aggregator bucket bucket bucket Aggregator Map Task Reduce Task Aggregator bucket bucket bucket Aggregator
- 22. External Shuffle Service ● Extracted from executor ● Manage the local aggregated data for the shuffle operations ● Maintain the data until the application is done.
- 23. Configuration
- 24. Configuration ● Dynamic Allocation ○ spark.dynamicAllocation.enabled ○ spark.dynamicAllocation.initialExecutors ○ spark.dynamicAllocation.maxExecutors ○ spark.dynamicAllocation.minExecutors
- 25. Configuration ● Dynamic Allocation ○ spark.dynamicAllocation.schedulerBacklogTimeout ○ spark.dynamicAllocation.executorIdleTimeout ○ spark.dynamicAllocation.sustainedSchedulerBacklogTimeout
- 26. Configuration ● External Shuffle Service ○ spark.shuffle.service.enabled ○ spark.shuffle.service.port
- 27. Demo Dynamic Allocation in Action
- 29. Configuration Values? It depends …. No, seriously
- 30. Configuration Values ● spark.dynamicAllocation.initialExecutors ● spark.dynamicAllocation.maxExecutors ● spark.dynamicAllocation.minExecutors Depends on workload and how many resources are potentially available to you.
- 31. Configuration Values ● spark.dynamicAllocation.schedulerBacklogTimeout Too short, might trigger for short burst of tasks. Too long, might be less effective. ● spark.dynamicAllocation.sustainedSchedulerBacklogTimeout Executor start duration. Default set to schedulerBacklogTimeout.
- 32. Configuration Values ● spark.dynamicAllocation.executorIdleTimeout Relative to the duration of the longer task. No big drawback on being too long, except cost.
- 33. Spark Streaming
- 35. • In most case, schedulerBacklogTimeout longer than batch interval. • executorIdleTimeout a portion of batch interval. • Should allow to manage processing delay. • Not compatible with the dynamic rate estimator. Spark Streaming
- 36. More Dynamic?
- 37. More Dynamic? https://github.com/twosigma/Cook ‘Fair’ job scheduler for Spark on top of Mesos ● Not a recommendation, just a suggestion. ● Some assembly required.
- 39. External Shuffle Service Cluster Manager Worker Node Executor Worker Node Worker Node Driver Scheduler(s) Executor External Shuffle Service External Shuffle Service External Shuffle Service