The document outlines the major categories of memory in Apache Spark, specifically focusing on Java and Python memory management. It details the different types of Java heap memory, including storage, execution, user, and reserved memory, as well as discussing off-heap memory and Python worker memory. Additionally, it references monitoring and instrumentation in Spark's memory management processes.

1. Spark memory model

2. Major categorization
● Java heap memory
○ Characterised by Garbage collection
● JVM off-heap memory / direct memory
● Python memory

3. Java heap memory

4. Java heap memory
1. Storage Memory — JVM heap space reserved for cached data
2. Execution (or shuffle) Memory — JVM heap space used by data-structures during
shuffle operations (joins, group-by’s and aggregations). Earlier (before Spark 1.6), the
term shuffle memory was also used to describe this section of the memory.
3. User Memory — For storing the data-structures created and managed by the user’s
code
4. Reserved Memory — Reserved by Spark for internal purposes.

5. Java heap memory
(spark.memory.fraction)
(spark.memory.storageFraction)

6. Java heap memory (legacy)
Spark2.x vs Spark3.x

7. Java off-heap memory

8. Java off-heap memory
1. Off heap dataframes
2. VM overheads — Interned strings, etc.

9. Java off-heap memory

10. Java off-heap memory (legacy)

11. Python worker memory

12. 1. Python worker memory — limits the memory in JVM for Python objects
2. Pyspark Executor memory — limits the memory of the actual Python process
Python worker memory

13. Python worker memory

14. References
Monitoring and Instrumentation - Spark 3.3.2 Documentation
Decoding Memory in Spark — Parameters that are often confused | by Sohom Majumdar | Walmart Global Tech Blog | Medium
Apache Spark Memory Management. This blog describes the concepts behind… | by Suhas N M | Analytics Vidhya | Medium