The document discusses performance monitoring in Spark applications and presents tools such as Sparklint and flame graphs for analyzing task composition and event timelines. It highlights memory profiling techniques, including the use of Java Flight Recorder and JVM commands to gather performance data, while also noting missing metrics from Spark regarding input/output operations and disk spilling. Overall, it provides guidelines for optimizing resource usage and identifying bottlenecks in Spark applications.

1. Performance Monitoring
in Spark Applications
ROHITASH JAIN,
SOFTWARE ENGINEER, SIGMOID

2. Hardware Trends

3. Comparison

6. Data Locality Concept

7. Bottleneck factors
▶ CPU
▶ DISK
▶ Memory
▶ Network

8. Phases in Reduce Task

9. Event Timeline in Spark UI

10. Task Composition

11. PROFILING TOOLS
▶ SPARKLINT
▶ FLAME GRAPHS
▶ GC LOGS from JVM

12. ∙Live view of batch & streaming application stats or
Event by event analysis of historical event logs
∙Stats and graphs for:
∙Idle time
∙Core usage
∙Task locality
SPARKLINT

13. Design

14. Screenshot

16. FLAME GRAPHS
▶ Flame Graph Visualization
Recipe:
▶ Gather multiple stack traces
▶ Aggregate them by sorting alphabetically by function/method name
▶ Visualization using stacked colored boxes
▶ Length of the box proportional to time spent there

17. HOW TO - FLAME GRAPHS
▶ Enable Java Flight Recorder eg: ./spark-submit --conf
"spark.driver.extraJavaOptions"="-XX:+UnlockCommercialFeatures -
XX:+FlightRecorder" --conf "spark.executor.extraJavaOptions"="-
XX:+UnlockCommercialFeatures -XX:+FlightRecorder“
▶ Collect information of the process with JMCD eg: jcmd <pid> JFR.start
duration=10s filename=$PWD/myoutput.jfr
▶ Refer to this article https://gist.github.com/kayousterhout/7008a8ebf2bab
eedc7ce6f8723fd1bf4 for converting your JFR file to an SVG file.

18. Memory profiling
▶ --conf ”spark.executor.extraJavaOptions=-XX:SurvivorRatio=16 -
XX:+UseG1GC - XX:+PrintGCDetails -XX:+PrintGCTimeStamps -
XX:+PrintReferenceGC - XX:+PrintAdaptiveSizePolicy”
▶ GC EASY for log analysis.

19. What’s missing from Spark metrics?
1. Time blocked on reading input data and writing output
data (HADOOP-11873)
2. Time spent spilling intermediate data to disk (SPARK-3577)

20. Questions ??