Downloaded 220 times
![Filter* data
before an
expensive reduce
or aggregation
consider*
coalesce(
Use* data
structures that
require less
memory
Serialize*
PySpark
serializing
is built-in
Scala/
Java?
persist(storageLevel.[*]_SER)
Recommended:
kryoserializer *
tuning.html#tuning-
data-structures
See "Optimize partitions."
*
See "GC investigation." *
See "Checkpointing." *
The Spark Tuning Cheat-Sheet](https://image.slidesharecdn.com/9gbanyabida-160224015238/85/Spark-Tuning-for-Enterprise-System-Administrators-By-Anya-Bida-8-320.jpg)
![Reduce the memory needed for
mySparkApp. How?
mySparkApp memory issues
persist(storageLevel.[*]_SER)
Recommended: kryoserializer *](https://image.slidesharecdn.com/9gbanyabida-160224015238/85/Spark-Tuning-for-Enterprise-System-Administrators-By-Anya-Bida-27-320.jpg)
Uploaded bySpark Summit
Spark Tuning for Enterprise System Administrators By Anya Bida
The document discusses tuning Apache Spark applications for enterprise system administrators. It covers identifying important Spark configuration parameters and how to configure them. Specifically, it addresses how to reduce memory usage through techniques like serialization and data structure optimization. It also provides guidance on gracefully handling memory limitations through checkpointing to avoid driver failures.
More Related Content
Similar to Spark Tuning for Enterprise System Administrators By Anya Bida
Spark Tuning for Enterprise System Administrators By Anya Bida
- 1. Spark Tuning forEnterprise System Administrators Anya T. Bida, PhD Rachel B. Warren
- 2. Don't worry aboutmissing something... Presentation: http://www.slideshare.net/anyabida Cheat-sheet: http://techsuppdiva.github.io/ ! ! Anya: https://www.linkedin.com/in/anyabida Rachel: https://www.linkedin.com/in/rachelbwarren ! ! !2
- 3. About Anya AboutRachel Operations Engineer ! ! ! Spark & Scala Enthusiast / Data Engineer About Alpine Data ! alpinenow.com Alpine deploys Spark in Production for our Enterprise Customers
- 4. About You* Intermittent Reliable Optimal Enterprise SystemAdministrators mySparkApp Success *
- 5.
- 6. Default != Recommended Example:By default, spark.executor.memory = 1g 1g allows small jobs to finish out of the box. Spark assumes you'll increase this parameter. !6
- 7. Which parameters areimportant? ! How do I configure them? !7 Default != Recommended
- 8. Filter* data before an expensivereduce or aggregation consider* coalesce( Use* data structures that require less memory Serialize* PySpark serializing is built-in Scala/ Java? persist(storageLevel.[*]_SER) Recommended: kryoserializer * tuning.html#tuning- data-structures See "Optimize partitions." * See "GC investigation." * See "Checkpointing." * The Spark Tuning Cheat-Sheet
- 9. Intermittent Reliable Optimal mySparkApp Success mySparkApp memoryissues Shared Cluster
- 10.
- 11.
- 12. Fair Schedulers !12 YARN <allocations> <queue name="sample_queue"> <minResources>4000mb,0vcores</minResources> <maxResources>8000 mb,8vcores</maxResources> <maxRunningApps>10</maxRunningApps> <weight>2.0</weight> <schedulingPolicy>fair</schedulingPolicy> </queue> </allocations> SPARK <allocations> <pool name="sample_queue"> <schedulingMode>FAIR</sch <weight>1</weight> <minShare>2</minShare> </pool> </allocations>
- 13. Fair Schedulers !13 YARN <allocations> <queue name="sample_queue"> <minResources>4000mb,0vcores</minResources> <maxResources>8000 mb,8vcores</maxResources> <maxRunningApps>10</maxRunningApps> <weight>2.0</weight> <schedulingPolicy>fair</schedulingPolicy> </queue> </allocations> SPARK <allocations> <pool name="sample_queue"> <schedulingMode>FAIR</sch <weight>1</weight> <minShare>2</minShare> </pool> </allocations> Configure these parameters too!
- 14.
- 15. What is thememory limit for mySparkApp? !15
- 16. !16 Max Memory in"pool" x 3/4 = mySparkApp_mem_limit ! ! ! <maxResources>8000 mb</maxResources> Limitation What is the memory limit for mySparkApp? Reserve 25% for overhead.
- 17.
- 18. !18 Max Memory in"pool" x 3/4 = mySparkApp_mem_limit ! mySparkApp_mem_limit = driver.memory + (executor.memory x dynamicAllocation.maxExecutors) What is the memory limit for mySparkApp?
- 19. !19 Max Memory in"pool" x 3/4 = mySparkApp_mem_limit ! mySparkApp_mem_limit = driver.memory + (executor.memory x dynamicAllocation.maxExecutors) What is the memory limit for mySparkApp?Limitation: Each driver and executor must not be larger than a single node. Limitation: Driver and executor memory must not be larger than a single node. ! (yarn.nodemanager.resource.memory-mb - 1Gb) executor.memory ~ # executors per node Limitation
- 20. !20 Max Memory in"pool" x 3/4 = mySparkApp_mem_limit ! mySparkApp_mem_limit = driver.memory + (executor.memory x dynamicAllocation.maxExecutors) Limitation: maxExecutors should not exceed pool allocation. ! Yarn: <maxResources>8vcores</maxResources> Limitation What is the memory limit for mySparkApp?
- 21. !21 I want alittle more information... Top 5 Mistakes When Writing Spark Applications by Mark Grover and Ted Malaska of Cloudera http://www.slideshare.net/hadooparchbook/top-5-mistakes-when-writing-spark-applications How-to: Tune Your Apache Spark Jobs (Part 2) by Sandy Ryza of Cloudera http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/ I want lots more...
- 22.
- 23. Intermittent Reliable Optimal mySparkApp Success mySparkApp memoryissues Shared Cluster
- 24. Reduce the memoryneeded for mySparkApp. How? Gracefully handle memory limitations. How? mySparkApp memory issues
- 25. Reduce the memoryneeded for mySparkApp. How? mySparkApp memory issues here let's talk about one scenario
- 27. Reduce the memoryneeded for mySparkApp. How? mySparkApp memory issues persist(storageLevel.[*]_SER) Recommended: kryoserializer *
- 28. Gracefully handle memory limitations.How? mySparkApp memory issues Reduce the memory needed for mySparkApp. How?
- 29. Gracefully handle memory limitations.How? mySparkApp memory issues here let's talk about one scenario
- 30. Symptoms: !30 • mySparkApp isrunning for several hours Container is lost. • I notice one container fails, then the rest fail one by one • The first container to fail was the driver • Driver is a SPOF
- 31. Investigate: !31 collect unbounded datato the driver • Driver failures are often caused by: • I verified only bounded data is brought to the driver, but still the driver fails intermittently.
- 32. Potential Solution: RDD.checkpoint() !32 Usein these cases: • high-traffic cluster • network blips • preemption • disk space nearly full ! ! Function: • saves the RDD to stable storage (eg hdfs or S3) How-to: SparkContext.setCheckpointDir(directory: String) RDD.checkpoint()
- 33. Intermittent Reliable Optimal mySparkApp Success mySparkApp memoryissues Shared Cluster Instead of 2.5 hours, myApp completes in 1 hour.
- 34.
- 35. Intermittent Reliable Optimal mySparkApp Success mySparkApp memoryissues Shared Cluster HighPerformanceSpark.com
- 36. Further Reading: • LearningSpark, by H. Karau, A. Konwinski, P. Wendell, M. Zaharia, 2015, O'Reilly https://databricks.com/blog/2015/02/09/learning-spark-book-available-from-oreilly.html • Scheduling: https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application • Tuning the Spark Conf: Mark Grover and Ted Malaska from Cloudera http://www.slideshare.net/hadooparchbook/top-5-mistakes-when-writing-spark-applications Sandy Ryza (Cloudera) http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/ • Checkpointing: http://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing • Troubleshooting: Miklos Christine from Databricks https://spark-summit.org/east-2016/events/operational-tips-for-deploying-spark/ • High Performance Spark by R. Warren, H. Karau, coming in 2016, O'Reilly http://highperformancespark.com/ !36
- 37. More Questions? !37 Presentation: http://www.slideshare.net/anyabida Cheat-sheet:http://techsuppdiva.github.io/ ! ! Anya: https://www.linkedin.com/in/anyabida Rachel: https://www.linkedin.com/in/rachelbwarren ! !