We all dread “Lost task” and “Container killed by YARN for exceeding memory limits” messages in our scaled-up spark yarn applications. Even answering the question “How much memory did my application use?” is surprisingly tricky in the distributed yarn environment. Sqrrl has developed a testing framework for observing vital statistics of spark jobs including executor-by-executor memory and CPU usage over time for both the JDK and python portions of pyspark yarn containers. This talk will detail the methods we use to collect, store, and report spark yarn resource usage. This information has proved to be invaluable for performance and regression testing of the spark jobs in Sqrrl Enterprise.

1. Monitoring the Dynamic Resource
Usage of Scala and Python Spark
Jobs in Yarn
Ed Barnes, Ruslan Vaulin and Chris McCubbin
Sqrrl Data

3. ML Application Workflow
Yarn Cluster
Spark
ETL Prediction
Python
Database/
Graph
Model
Scala
Stats
Models
ETL Training
Database/
Graph
Model

4. Taking Spark Applications into
Production
• Requires execution framework
• Scalable, Robust, Tested
• Test at scale
• Many issues show up only at scale
– Performance
– Memory requirements
– Failures
– Scaling
• Debugging distributed applications is really hard!

5. Spark UI: job level

6. Spark UI: task level

7. Spark UI: task level
OOM
Exception

8. Case Study: Py4J Issue
• Testing engineer - “Your code blows up with OOM
when processing X amount of data!”
• Why?!!!!
• Nothing obvious in Spark UI!

9. Case Study: Py4J Issue
• Testing engineer - “Your code blows up with OOM
when processing X amount of data!”
• Why?!!!!
• Nothing obvious in Spark UI!
OOM
Exception

10. YARN
Tooling Approach Requirements
● Cluster-wide process monitoring
● Per-node, per-process statistics
● Identification of high CPU, memory usage
● Record process hierarchy/timing of Spark jobs under YARN
● Characterize scaling behaviors for production/releases
● Integrate with internal processes/harnesses for general
development and test use
© 2016 Sqrrl | All Rights Reserved | Proprietary and Confidential
Sqrrl Server
Data Node 1
Sqrrl Server
Data Node 2
Sqrrl Server
Data Node n
Sqrrl Server
Name Node

11. YARN
Overall System Design
Django/Apache
MySQL
Sqrrl Server
Data Node 1
process_monitor.py
Sqrrl Server
Data Node 2
process_monitor.py
Sqrrl Server
Data Node n
process_monitor.py
Sqrrl Server
Name Node
REST API
● Centralized reporting/logging system
○ REST API for scripts
○ Browser-based reporting
○ Database for process data and logging
● Main performance script (run_perf.py)
● Process monitoring script
(process_monitor.py)
Performance
Test System
run_perf.py

12. YARN
Scripts/Operation
● Main performance harness
(run_perf.py)
○ Gets run_id through REST API
○ Starts monitoring script
(process_monitor.py) on each cluster
node
○ Initiates Spark jobs and adds log
messages for correlation of process
data and job execution
© 2016 Sqrrl | All Rights Reserved | Proprietary and Confidential
Django/Apache
MySQL
Sqrrl Server
Data Node 1
process_monitor.py
Sqrrl Server
Data Node 2
process_monitor.py
Sqrrl Server
Data Node n
process_monitor.py
Sqrrl Server
Name Node
REST API
Performance
Test System
run_perf.py

13. YARN
Scripts/Operation
● Main performance harness
(run_perf.py)
○ Gets run_id through REST API
○ Starts monitoring script
(process_monitor.py) on each cluster
node
○ Initiates Spark jobs and adds log
messages for correlation of process
data and job execution
© 2016 Sqrrl | All Rights Reserved | Proprietary and Confidential
Django/Apache
MySQL
Sqrrl Server
Data Node 1
process_monitor.py
Sqrrl Server
Data Node 2
process_monitor.py
Sqrrl Server
Data Node n
process_monitor.py
Sqrrl Server
Name Node
REST API
Performance
Test System
run_perf.py

14. YARN
Scripts/Operation
● Main performance harness
(run_perf.py)
○ Gets run_id through REST API
○ Starts monitoring script
(process_monitor.py) on each cluster
node
○ Initiates Spark jobs and adds log
messages for correlation of process
data and job execution
© 2016 Sqrrl | All Rights Reserved | Proprietary and Confidential
Django/Apache
MySQL
Sqrrl Server
Data Node 1
process_monitor.py
Sqrrl Server
Data Node 2
process_monitor.py
Sqrrl Server
Data Node n
process_monitor.py
Sqrrl Server
Name Node
REST API
Performance
Test System
run_perf.py

15. YARN
Scripts/Operation
● Monitoring script (process_monitor.py)
○ 10s interval between collecting
process information
○ Jobs running on YARN
■ Recursively pull process tree from
Node Manager process
○ Collect process data and adds to
database associated with unique
run_id for aggregate reporting
© 2016 Sqrrl | All Rights Reserved | Proprietary and Confidential
Django/Apache
MySQL
Sqrrl Server
Data Node 1
process_monitor.py
Sqrrl Server
Data Node 2
process_monitor.py
Sqrrl Server
Data Node n
process_monitor.py
Sqrrl Server
Name Node
REST API
Performance
Test System
run_perf.py

16. Database Design
ProcessMonitor
id
time_stamp
monitor
run_id
pid
ppid
pcpu
pmem
cpu_time
elapsed_time
resident_size
mapped
private
shared
short_command
long_command
Runs
id
start_time
end_time
...
suite_name
user
overall_status
Log
id
run_id
msg_time
msg_type
message

17. User Experience

18. User Experience

19. User Experience

20. User Experience

21. Py4J Issue: Evidence
• Memory spiked during loading of trained ML models into
python process.
• Only on driver!
Memory spikes
Python Process
Memory

22. Diagnosis and Conclusions
• Loading trained model from HDFS required decryption with
java libraries and Py4J for loading into python.
• Py4J protocol inflated size by almost 100x!
• Solution: implement custom protocol for loading data from
jvm to python via socket.

23. Py4J issue: After Fix
• Memory spikes are gone!
• Ready for production!

24. Recommendations & Lessons
Learned
• Do not take scalability for granted!
• Understand Spark’s architecture
– Python/JVM interaction
• Follow best practices
– Iterators not Lists
– Careful with joins
• Understand your computing demands
• Test at scale
• Invest in tools
• Think distributed and your code will shine!

25. Thank You.
sqrrl.com
@SqrrlData
ebarnes@sqrrl.com
ruslan@sqrrl.com
chris@sqrrl.com