Video: https://www.youtube.com/watch?v=DNWaMR8uKDc&feature=youtu.be

1. Spark Tuning for Enterprise
System Administrators
Anya T. Bida, PhD
Rachel B. Warren

2. Don't worry about missing something...
Video: https://www.youtube.com/watch?v=DNWaMR8uKDc&feature=youtu.be
Presentation: http://www.slideshare.net/anyabida
Cheat-sheet: http://techsuppdiva.github.io/
!
!
Anya: https://www.linkedin.com/in/anyabida
Rachel: https://www.linkedin.com/in/rachelbwarren
!
!
!2

3. About Anya About Rachel
Operations Engineer
!
!
!
Spark & Scala Enthusiast /
Data Engineer
Alpine Data
!
alpinenow.com

4. About You*
Intermittent
Reliable
Optimal
Spark practitioners
mySparkApp Success
*

5. Intermittent
Reliable
Optimal
mySparkApp Success

6. Default != Recommended
Example: By default, spark.executor.memory = 1g
1g allows small jobs to finish out of the box.
Spark assumes you'll increase this parameter.
!6

7. Which parameters are important?
!
How do I configure them?
!7
Default != Recommended

8. Filter* data
before an
expensive reduce
or aggregation
consider*
coalesce(
Use* data
structures that
require less
memory
Serialize*
PySpark
serializing
is built-in
Scala/
Java?
persist(storageLevel.[*]_SER)
Recommended:
kryoserializer *
tuning.html#tuning-
data-structures
See "Optimize partitions."
*
See "GC investigation." *
See "Checkpointing." *
The Spark Tuning Cheat-Sheet

9. Intermittent
Reliable
Optimal
mySparkApp Success
Memory trouble
Initial config

10. Intermittent
Reliable
Optimal
mySparkApp Success
Memory trouble
Initial config

11. !11

12. !12

13. !13
How many in the
audience have their own
cluster?

14. !14

15. Fair Schedulers
!15
YARN
<allocations>
<queue name="sample_queue">
<minResources>4000 mb,0vcores</minResources>
<maxResources>8000 mb,8vcores</maxResources>
<maxRunningApps>10</maxRunningApps>
<weight>2.0</weight>
<schedulingPolicy>fair</schedulingPolicy>
</queue>
</allocations>
SPARK
<allocations>
<pool name="sample_queue">
<schedulingMode>FAIR</sch
<weight>1</weight>
<minShare>2</minShare>
</pool>
</allocations>

16. Fair Schedulers
!16
YARN
<allocations>
<queue name="sample_queue">
<minResources>4000 mb,0vcores</minResources>
<maxResources>8000 mb,8vcores</maxResources>
<maxRunningApps>10</maxRunningApps>
<weight>2.0</weight>
<schedulingPolicy>fair</schedulingPolicy>
</queue>
</allocations>
SPARK
<allocations>
<pool name="sample_queue">
<schedulingMode>FAIR</sch
<weight>1</weight>
<minShare>2</minShare>
</pool>
</allocations>

17. Fair Schedulers
!17
YARN
<allocations>
<queue name="sample_queue">
<minResources>4000 mb,0vcores</minResources>
<maxResources>8000 mb,8vcores</maxResources>
<maxRunningApps>10</maxRunningApps>
<weight>2.0</weight>
<schedulingPolicy>fair</schedulingPolicy>
</queue>
</allocations>
SPARK
<allocations>
<pool name="sample_queue">
<schedulingMode>FAIR</sch
<weight>1</weight>
<minShare>2</minShare>
</pool>
</allocations>

18. Fair Schedulers
!18
YARN
<allocations>
<queue name="sample_queue">
<minResources>4000 mb,0vcores</minResources>
<maxResources>8000 mb,8vcores</maxResources>
<maxRunningApps>10</maxRunningApps>
<weight>2.0</weight>
<schedulingPolicy>fair</schedulingPolicy>
</queue>
</allocations>
SPARK
<allocations>
<pool name="sample_queue">
<schedulingMode>FAIR</sch
<weight>1</weight>
<minShare>2</minShare>
</pool>
</allocations>

19. Fair Schedulers
!19
YARN
<allocations>
<queue name="sample_queue">
<minResources>4000 mb,0vcores</minResources>
<maxResources>8000 mb,8vcores</maxResources>
<maxRunningApps>10</maxRunningApps>
<weight>2.0</weight>
<schedulingPolicy>fair</schedulingPolicy>
</queue>
</allocations>
SPARK
<allocations>
<pool name="sample_queue">
<schedulingMode>FAIR</sch
<weight>1</weight>
<minShare>2</minShare>
</pool>
</allocations>
Use these parameters!

20. Fair Schedulers
!20
YARN
<allocations>
<user name="sample_user">
<maxRunningApps>6</maxRunningApps>
</user>
<userMaxAppsDefault>5</userMaxAppsDefault>
!
</allocations>

21. Fair Schedulers
!21
YARN
<allocations>
<user name="sample_user">
<maxRunningApps>6</maxRunningApps>
</user>
<userMaxAppsDefault>5</userMaxAppsDefault>
!
</allocations>

22. What is the memory limit for
mySparkApp?
!22

23. !23
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
!
!
What is the memory limit for
mySparkApp?

24. !24
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
!
!
What is the memory limit for
mySparkApp?

25. !25
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
!
!
<maxResources>___mb</maxResources>
Limitation
What is the memory limit for
mySparkApp?

26. What is the memory limit for
mySparkApp?
!26
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
!
!
Reserve 25% for overhead

27. !27
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
!
!
What is the memory limit for
mySparkApp?

28. !28

29. !29
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
mySparkApp_mem_limit = driver.memory + (executor.memory
x dynamicAllocation.maxExecutors)
What is the memory limit for
mySparkApp?

30. !30
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
mySparkApp_mem_limit = driver.memory + (executor.memory
x dynamicAllocation.maxExecutors)
What is the memory limit for
mySparkApp?

31. !31
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
mySparkApp_mem_limit = driver.memory + (executor.memory
x dynamicAllocation.maxExecutors)
What is the memory limit for
mySparkApp?
Limitation: Driver must not be
larger than a single node.

32. !32
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
mySparkApp_mem_limit = driver.memory + (executor.memory
x dynamicAllocation.maxExecutors)
What is the memory limit for
mySparkApp?

33. !33
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
mySparkApp_mem_limit = driver.memory + (executor.memory
x dynamicAllocation.maxExecutors)
!
!
!
!
!
!
!

34. !34
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
mySparkApp_mem_limit = driver.memory + (executor.memory
x dynamicAllocation.maxExecutors)
!
!
!
!
!
!
!
Parameter Default Recommended
spark.executor.cores 1(Yarn mode) 5 or less

35. !35
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
mySparkApp_mem_limit = driver.memory + (executor.memory
x dynamicAllocation.maxExecutors)
!
executors per node= (cores per node) / (5cores per executor)
!
!
!
!
!

36. !36
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
mySparkApp_mem_limit = driver.memory + (executor.memory
x dynamicAllocation.maxExecutors)
!
executors per node= (cores per node) / (5cores per executor)
!
executor.memory = (memory per node) / (executors per node)
!
!
!

37. !37
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
mySparkApp_mem_limit = driver.memory + (executor.memory
x dynamicAllocation.maxExecutors)
!
executors per node= (cores per node) / (5cores per executor)
!
executor.memory = (memory per node) / (executors per node)
!
maxExecutors=(executors per node) x (num nodes)
!

38. !38
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
mySparkApp_mem_limit = driver.memory + (executor.memory
x dynamicAllocation.maxExecutors)
What is the memory limit for
mySparkApp?

39. !39
Max Memory in "pool" x 3/4 = mySparkApp_mem_limit
!
mySparkApp_mem_limit = driver.memory + (executor.memory
x dynamicAllocation.maxExecutors)
What is the memory limit for
mySparkApp?
Verify my calculations respect this
limitation.

40. !40

41. Intermittent
Reliable
Optimal
mySparkApp Success
Memory trouble
Initial config

42. Intermittent
Reliable
Optimal
mySparkApp Success
Memory trouble
Initial config

43. mySparkApp memory issues

44. Reduce the memory needed for
mySparkApp. How?
Gracefully handle memory
limitations. How?
mySparkApp memory issues

45. Reduce the memory needed for
mySparkApp. How?
Gracefully handle memory
limitations. How?
mySparkApp memory issues

46. Reduce the memory needed for
mySparkApp. How?
Gracefully handle memory
limitations. How?
mySparkApp memory issues
here let's talk about one scenario

48. Reduce the memory needed for
mySparkApp. How?
Gracefully handle memory
limitations. How?
mySparkApp memory issues
persist(storageLevel.[*]_SER)

49. Reduce the memory needed for
mySparkApp. How?
Gracefully handle memory
limitations. How?
mySparkApp memory issues
persist(storageLevel.[*]_SER)

50. Reduce the memory needed for
mySparkApp. How?
Gracefully handle memory
limitations. How?
mySparkApp memory issues
persist(storageLevel.[*]_SER)
Recommended: kryoserializer *

51. Reduce the memory needed for
mySparkApp. How?
Gracefully handle memory
limitations. How?
mySparkApp memory issues
persist(storageLevel.[*]_SER)
Recommended: kryoserializer *

52. Reduce the memory needed for
mySparkApp. How?
Gracefully handle memory
limitations. How?
mySparkApp memory issues
persist(storageLevel.[*]_SER)
Recommended: kryoserializer *
Spark SQL's
Optimizer

53. Reduce the memory needed for
mySparkApp. How?
Gracefully handle memory
limitations. How?
mySparkApp memory issues

54. Reduce the memory needed for
mySparkApp. How?
Gracefully handle memory
limitations. How?
mySparkApp memory issues
here let's talk about one scenario

55. Symptoms:
!55
• mySparkApp is running for several hours
Container is lost.
• Several Executors are lost.

56. Symptoms:
!56
• mySparkApp is running for several hours
Container is lost.
• Several Executors are lost.
• Behavior is intermittent (sometimes succeeds,
sometimes fails).

57. Potential Solution: RDD.checkpoint()
!57

58. Potential Solution: RDD.checkpoint()
!58
Use in these cases:
!
!
Function:
How-to:
!
!

59. Potential Solution: RDD.checkpoint()
!59
Use in these cases:
!
!
Function:
• saves the RDD to stable
storage (eg hdfs or S3)
How-to:
!
!

60. Potential Solution: RDD.checkpoint()
!60
Use in these cases:
!
Function:
• saves the RDD to stable
storage (eg hdfs or S3)
How-to:
Cache first!
SparkContext.setCheckpointDir(directory: String)
RDD.checkpoint()

61. Potential Solution: RDD.checkpoint()
!61
Use in these cases:
• high-traffic cluster
• network blips
• preemption
• disk space nearly full
!
!
Function:
• saves the RDD to stable
storage (eg hdfs or S3)
How-to:
Cache first!
SparkContext.setCheckpointDir(directory: String)
RDD.checkpoint()

62. Intermittent
Reliable
Optimal
mySparkApp Success
Memory trouble
Initial config

63. Intermittent
Reliable
Optimal
mySparkApp Success
Memory trouble
Initial config
Instead of 2.5 hours, myApp
completes in 1 hour.

64. Cheat-sheet
techsuppdiva.github.io/

65. Intermittent
Reliable
Optimal
mySparkApp Success
Memory trouble
Initial config
HighPerformanceSpark.com

66. Further Reading:
• Spark Tuning Cheat-sheet
techsuppdiva.github.io
• Apache Spark Documentation
https://spark.apache.org/docs/latest
• Checkpointing
http://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing
https://github.com/jaceklaskowski/mastering-apache-spark-book/blob/master/spark-rdd-checkpointing.adoc
• Learning Spark, by H. Karau, A. Konwinski, P. Wendell, M. Zaharia, 2015
!66

67. More Questions?
!67
Video: https://www.youtube.com/watch?v=DNWaMR8uKDc&feature=youtu.be
Presentation: http://www.slideshare.net/anyabida
Cheat-sheet: http://techsuppdiva.github.io/
!
!
Anya: https://www.linkedin.com/in/anyabida
Rachel: https://www.linkedin.com/in/rachelbwarren
!
!
Thanks!

68. Extra slides

69. "Checkpointing"*
Checkpoint* reliably using
RDD.checkpoint()
Need better Driver
failure recovery?*
Metadata Checkpoint*
Using stateful
transformations?*
RDD Checkpoint*
SPARK-9947 Separate Metadata and State Checkpoint