SpotFlow: Tracking Method Calls and States at Runtime, ICSE (Demo), 2024

1. SpotFlow: Tracking Method
Calls and States at Runtime
Andre Hora
andrehora@dcc.ufmg.br
https://github.com/andrehora/spotflow
1
ICSE 2024

2. Motivation
Understanding the runtime behavioral aspects of a software system is
fundamental for software engineering, e.g., testing and code comprehension
Despite the importance of runtime analysis, few tools have been created and
made public to support developers extracting information from software execution
[Rabiser et al.]
To overcome these issues, we propose SpotFlow, a tool to ease runtime
analysis in Python
2

3. SpotFlow in a Nutshell
SpotFlow executes and monitors a Python program, collecting detailed
information on method calls and states
SpotFlow gathers data at the method-level for every method call, such as
executed lines, argument values, return values, and thrown exceptions
SpotFlow can directly detect what classes, methods, test methods, and calls
ran which lines (tracing tools typically work at the file-level and can only detect
what files ran which lines)
SpotFlow can be run from the command line
SpotFlow is publicly available at https://github.com/andrehora/spotflow
3

4. SpotFlow: Domain Model
4
MonitoredProgram
MonitoredMethod
MethodCall
CallState
ArgState
ReturnState
ExceptionState
VarStateHistory
VarState

5. SpotFlow: Domain Model
MonitoredProgram
MonitoredMethod
MethodCall
CallState
ArgState
ReturnState
ExceptionState
VarStateHistory
VarState
5

6. 6
application
tests

7. 7
application
tests SpotFlow
SpotFlow objects

8. 8
SpotFlow objects

9. 9
SpotFlow objects
Method calls
and states

10. 10
SpotFlow objects
Executed lines of
code (for each call,
not for each file)

11. 11
SpotFlow objects
Variable values over time
● counter: 0, 1, and 2
● word: “ABC” and “DEF”

12. SpotFlow: Practical Applications
Support novel empirical studies on runtime analysis
Support the development of novel software testing tools
Support the creation of novel datasets with runtime metrics
12

13. Support Novel Empirical Studies on Runtime Analysis
Monitoring the Execution of 14K Tests: Methods Tend to Have One Path that
Is Significantly More Executed, FSE (IVR), 2024
We monitor the execution of 14,177 tests from 25 real-world Python systems and
assess 11,425 tested paths from 2,357 methods
We show that one tested path is prevalent
and receives most of the calls, while others
are significantly less executed
13

14. Support Novel Empirical Studies on Runtime Analysis
Test Polarity: Detecting Positive and Negative Tests, FSE (IVR), 2024
Test polarity is an automated approach to detect
positive and negative tests
We provide a preliminary study to analyze the test
polarity of 2,054 test methods of the Python
Standard Library
We find that most of the analyzed test methods
are negative (88%) and a minority is positive (12%)
14

15. Support the Development of Novel Software Testing Tools
PathSpotter: Exploring Tested Paths to Discover Missing Tests, FSE (Demo),
2024
PathSpotter is a tool to automatically identify
tested paths and support the detection of
missing tests
It successively guided us in improving the
test suites of relevant systems, including
CPython, Pylint, and Jupyter Client
6 merged PRs, created/updated 32 test
methods, and added 80 novel assertions
15

16. Support the Creation of Novel Datasets
TestDossier: A Dataset of Tested Values Automatically Extracted from Test
Execution, MSR (Data and Tool), 2024
TestDossier is a dataset of tested values automatically extracted from the
execution of Python tests
We monitored the test suites of 15 Python Standard Libraries. It contains 1,234
distinct variables and 133,169 values
16

17. Summary
Tool to ease runtime analysis in Python
Collects detailed information on method calls and states
Can directly detect what classes, methods, test methods, and calls ran
which lines
SpotFlow supports:
● Novel empirical studies on runtime analysis
● The development of novel software testing tools
● The creation of novel datasets with runtime metrics
17

18. SpotFlow: Tracking Method
Calls and States at Runtime
Andre Hora
andrehora@dcc.ufmg.br
https://github.com/andrehora/spotflow
18
ICSE 2024