SlideShare a Scribd company logo
1 of 18
Download to read offline
SpotFlow: Tracking Method
Calls and States at Runtime
Andre Hora
andrehora@dcc.ufmg.br
https://github.com/andrehora/spotflow
1
ICSE 2024
Motivation
Understanding the runtime behavioral aspects of a software system is
fundamental for software engineering, e.g., testing and code comprehension
Despite the importance of runtime analysis, few tools have been created and
made public to support developers extracting information from software execution
[Rabiser et al.]
To overcome these issues, we propose SpotFlow, a tool to ease runtime
analysis in Python
2
SpotFlow in a Nutshell
SpotFlow executes and monitors a Python program, collecting detailed
information on method calls and states
SpotFlow gathers data at the method-level for every method call, such as
executed lines, argument values, return values, and thrown exceptions
SpotFlow can directly detect what classes, methods, test methods, and calls
ran which lines (tracing tools typically work at the file-level and can only detect
what files ran which lines)
SpotFlow can be run from the command line
SpotFlow is publicly available at https://github.com/andrehora/spotflow
3
SpotFlow: Domain Model
4
MonitoredProgram
MonitoredMethod
MethodCall
CallState
ArgState
ReturnState
ExceptionState
VarStateHistory
VarState
SpotFlow: Domain Model
MonitoredProgram
MonitoredMethod
MethodCall
CallState
ArgState
ReturnState
ExceptionState
VarStateHistory
VarState
5
6
application
tests
7
application
tests SpotFlow
SpotFlow objects
8
SpotFlow objects
9
SpotFlow objects
Method calls
and states
10
SpotFlow objects
Executed lines of
code (for each call,
not for each file)
11
SpotFlow objects
Variable values over time
● counter: 0, 1, and 2
● word: “ABC” and “DEF”
SpotFlow: Practical Applications
Support novel empirical studies on runtime analysis
Support the development of novel software testing tools
Support the creation of novel datasets with runtime metrics
12
Support Novel Empirical Studies on Runtime Analysis
Monitoring the Execution of 14K Tests: Methods Tend to Have One Path that
Is Significantly More Executed, FSE (IVR), 2024
We monitor the execution of 14,177 tests from 25 real-world Python systems and
assess 11,425 tested paths from 2,357 methods
We show that one tested path is prevalent
and receives most of the calls, while others
are significantly less executed
13
Support Novel Empirical Studies on Runtime Analysis
Test Polarity: Detecting Positive and Negative Tests, FSE (IVR), 2024
Test polarity is an automated approach to detect
positive and negative tests
We provide a preliminary study to analyze the test
polarity of 2,054 test methods of the Python
Standard Library
We find that most of the analyzed test methods
are negative (88%) and a minority is positive (12%)
14
Support the Development of Novel Software Testing Tools
PathSpotter: Exploring Tested Paths to Discover Missing Tests, FSE (Demo),
2024
PathSpotter is a tool to automatically identify
tested paths and support the detection of
missing tests
It successively guided us in improving the
test suites of relevant systems, including
CPython, Pylint, and Jupyter Client
6 merged PRs, created/updated 32 test
methods, and added 80 novel assertions
15
Support the Creation of Novel Datasets
TestDossier: A Dataset of Tested Values Automatically Extracted from Test
Execution, MSR (Data and Tool), 2024
TestDossier is a dataset of tested values automatically extracted from the
execution of Python tests
We monitored the test suites of 15 Python Standard Libraries. It contains 1,234
distinct variables and 133,169 values
16
Summary
Tool to ease runtime analysis in Python
Collects detailed information on method calls and states
Can directly detect what classes, methods, test methods, and calls ran
which lines
SpotFlow supports:
● Novel empirical studies on runtime analysis
● The development of novel software testing tools
● The creation of novel datasets with runtime metrics
17
SpotFlow: Tracking Method
Calls and States at Runtime
Andre Hora
andrehora@dcc.ufmg.br
https://github.com/andrehora/spotflow
18
ICSE 2024

More Related Content

Similar to SpotFlow: Tracking Method Calls and States at Runtime

OORPT Dynamic Analysis
OORPT Dynamic AnalysisOORPT Dynamic Analysis
OORPT Dynamic Analysis
lienhard
 
Integrating Model Checking and Procedural Languages
Integrating Model Checking and Procedural LanguagesIntegrating Model Checking and Procedural Languages
Integrating Model Checking and Procedural Languages
butest
 
Synthesizing Continuous Deployment Practices in Software Development
Synthesizing Continuous Deployment Practices in Software DevelopmentSynthesizing Continuous Deployment Practices in Software Development
Synthesizing Continuous Deployment Practices in Software Development
Akond Rahman
 

Similar to SpotFlow: Tracking Method Calls and States at Runtime (20)

OORPT Dynamic Analysis
OORPT Dynamic AnalysisOORPT Dynamic Analysis
OORPT Dynamic Analysis
 
Automated Testing: An Edge Over Manual Software Testing
Automated Testing: An Edge Over Manual Software TestingAutomated Testing: An Edge Over Manual Software Testing
Automated Testing: An Edge Over Manual Software Testing
 
How to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated TestingHow to Actually DO High-volume Automated Testing
How to Actually DO High-volume Automated Testing
 
Debug me
Debug meDebug me
Debug me
 
Specification and Detection of SOA Antipatterns
Specification and Detection of SOA AntipatternsSpecification and Detection of SOA Antipatterns
Specification and Detection of SOA Antipatterns
 
CTFL chapter 06
CTFL chapter 06CTFL chapter 06
CTFL chapter 06
 
Dc35579583
Dc35579583Dc35579583
Dc35579583
 
Testing
TestingTesting
Testing
 
The Use of Static Code Analysis When Teaching or Developing Open-Source Software
The Use of Static Code Analysis When Teaching or Developing Open-Source SoftwareThe Use of Static Code Analysis When Teaching or Developing Open-Source Software
The Use of Static Code Analysis When Teaching or Developing Open-Source Software
 
Code coverage based test case selection and prioritization
Code coverage based test case selection and prioritizationCode coverage based test case selection and prioritization
Code coverage based test case selection and prioritization
 
Empirical research methods for software engineering
Empirical research methods for software engineeringEmpirical research methods for software engineering
Empirical research methods for software engineering
 
Integrating Model Checking and Procedural Languages
Integrating Model Checking and Procedural LanguagesIntegrating Model Checking and Procedural Languages
Integrating Model Checking and Procedural Languages
 
OpenTelemetry 101 FTW
OpenTelemetry 101 FTWOpenTelemetry 101 FTW
OpenTelemetry 101 FTW
 
Programming Fundamentals lecture 3
Programming Fundamentals lecture 3Programming Fundamentals lecture 3
Programming Fundamentals lecture 3
 
pertemuan 1 system dev
pertemuan 1 system devpertemuan 1 system dev
pertemuan 1 system dev
 
Fuzz
FuzzFuzz
Fuzz
 
Synthesizing Continuous Deployment Practices in Software Development
Synthesizing Continuous Deployment Practices in Software DevelopmentSynthesizing Continuous Deployment Practices in Software Development
Synthesizing Continuous Deployment Practices in Software Development
 
System Testing.pptx
System Testing.pptxSystem Testing.pptx
System Testing.pptx
 
pertemuan 1.ppt
pertemuan 1.pptpertemuan 1.ppt
pertemuan 1.ppt
 
Systems_Overview.ppt
Systems_Overview.pptSystems_Overview.ppt
Systems_Overview.ppt
 

Recently uploaded

Jax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckJax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined Deck
Marc Lester
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Lisi Hocke
 

Recently uploaded (20)

Jax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckJax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined Deck
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdf
 
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
 
Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?
 
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
 
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
 
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanWorkshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024
 
Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...
Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...
Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...
 
Test Automation Design Patterns_ A Comprehensive Guide.pdf
Test Automation Design Patterns_ A Comprehensive Guide.pdfTest Automation Design Patterns_ A Comprehensive Guide.pdf
Test Automation Design Patterns_ A Comprehensive Guide.pdf
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
 
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
 
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
 
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate  AI Earns First Commission in 3 Hours..pdfAuto Affiliate  AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
 
^Clinic ^%[+27788225528*Abortion Pills For Sale In harare
^Clinic ^%[+27788225528*Abortion Pills For Sale In harare^Clinic ^%[+27788225528*Abortion Pills For Sale In harare
^Clinic ^%[+27788225528*Abortion Pills For Sale In harare
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeCon
 
Lessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfLessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdf
 
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jGraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
 
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with GraphGraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
 

SpotFlow: Tracking Method Calls and States at Runtime

  • 1. SpotFlow: Tracking Method Calls and States at Runtime Andre Hora andrehora@dcc.ufmg.br https://github.com/andrehora/spotflow 1 ICSE 2024
  • 2. Motivation Understanding the runtime behavioral aspects of a software system is fundamental for software engineering, e.g., testing and code comprehension Despite the importance of runtime analysis, few tools have been created and made public to support developers extracting information from software execution [Rabiser et al.] To overcome these issues, we propose SpotFlow, a tool to ease runtime analysis in Python 2
  • 3. SpotFlow in a Nutshell SpotFlow executes and monitors a Python program, collecting detailed information on method calls and states SpotFlow gathers data at the method-level for every method call, such as executed lines, argument values, return values, and thrown exceptions SpotFlow can directly detect what classes, methods, test methods, and calls ran which lines (tracing tools typically work at the file-level and can only detect what files ran which lines) SpotFlow can be run from the command line SpotFlow is publicly available at https://github.com/andrehora/spotflow 3
  • 10. 10 SpotFlow objects Executed lines of code (for each call, not for each file)
  • 11. 11 SpotFlow objects Variable values over time ● counter: 0, 1, and 2 ● word: “ABC” and “DEF”
  • 12. SpotFlow: Practical Applications Support novel empirical studies on runtime analysis Support the development of novel software testing tools Support the creation of novel datasets with runtime metrics 12
  • 13. Support Novel Empirical Studies on Runtime Analysis Monitoring the Execution of 14K Tests: Methods Tend to Have One Path that Is Significantly More Executed, FSE (IVR), 2024 We monitor the execution of 14,177 tests from 25 real-world Python systems and assess 11,425 tested paths from 2,357 methods We show that one tested path is prevalent and receives most of the calls, while others are significantly less executed 13
  • 14. Support Novel Empirical Studies on Runtime Analysis Test Polarity: Detecting Positive and Negative Tests, FSE (IVR), 2024 Test polarity is an automated approach to detect positive and negative tests We provide a preliminary study to analyze the test polarity of 2,054 test methods of the Python Standard Library We find that most of the analyzed test methods are negative (88%) and a minority is positive (12%) 14
  • 15. Support the Development of Novel Software Testing Tools PathSpotter: Exploring Tested Paths to Discover Missing Tests, FSE (Demo), 2024 PathSpotter is a tool to automatically identify tested paths and support the detection of missing tests It successively guided us in improving the test suites of relevant systems, including CPython, Pylint, and Jupyter Client 6 merged PRs, created/updated 32 test methods, and added 80 novel assertions 15
  • 16. Support the Creation of Novel Datasets TestDossier: A Dataset of Tested Values Automatically Extracted from Test Execution, MSR (Data and Tool), 2024 TestDossier is a dataset of tested values automatically extracted from the execution of Python tests We monitored the test suites of 15 Python Standard Libraries. It contains 1,234 distinct variables and 133,169 values 16
  • 17. Summary Tool to ease runtime analysis in Python Collects detailed information on method calls and states Can directly detect what classes, methods, test methods, and calls ran which lines SpotFlow supports: ● Novel empirical studies on runtime analysis ● The development of novel software testing tools ● The creation of novel datasets with runtime metrics 17
  • 18. SpotFlow: Tracking Method Calls and States at Runtime Andre Hora andrehora@dcc.ufmg.br https://github.com/andrehora/spotflow 18 ICSE 2024