SlideShare a Scribd company logo
1 of 17
3/5/15
Challenge the future
Delft
University of
Technology
Understanding Software Performance
Regressions Using Differential Flame
Graphs
Cor-Paul Bezemer, Johan Pouwelse, Brendan Gregg
2Understanding Software Performance Regressions Using DFGs
Regression Testing
3Understanding Software Performance Regressions Using DFGs
Regression Testing Cycle
1)Write code
2)Deploy to continuous integration system
3)Test
4)Deploy to production code or return to 1)
4Understanding Software Performance Regressions Using DFGs
Performance Regression Testing
• Functional testing is difficult, but:
• Functionality is either OK or broken
• Performance testing is even more difficult:
• Depends on requirements of application
• A library may be fast enough for one application, while it's too slow for
another
• How much performance regression is acceptable?
5Understanding Software Performance Regressions Using DFGs
6Understanding Software Performance Regressions Using DFGs
What to Measure
• Built-in performance metrics (or counters)
• Response time, throughput, queue lengths, utilization
• Usually system-wide
• Profilers
• Per-function info on execution time, bytes written, etc.
• Higher granularity == more expensive
• After measuring: tons of data! :) / :(
7Understanding Software Performance Regressions Using DFGs
8Understanding Software Performance Regressions Using DFGs
Detecting I/O Regression
• Previous work:
• “Detecting and analyzing I/O performance regressions”,
C. Bezemer, E. Milon, A. Zaidman, J. Pouwelse (Journal of Software:
Evolution and Process 26 (12), 1193-1212)
• Approach for finding out which function exhibits I/O write
regression over two software versions
• Results promising, but sometimes difficult to interpret
9Understanding Software Performance Regressions Using DFGs
Work in Progress: Visualization
10Understanding Software Performance Regressions Using DFGs
Flame Graphs (1)
def a ( ) :
if some condition :
b()
…
def b ( ) :
if some condition :
c ()
...
def c( ):
...
def d ( ) :
if some condition :
c ()
...
Stack trace, CPU time
a(), 100
a()→b(), 25
a()→b()→c(), 10
d(), 50
d()→c(), 10
11Understanding Software Performance Regressions Using DFGs
Flame Graphs (2)
12Understanding Software Performance Regressions Using DFGs
Flame Graphs (3)
13Understanding Software Performance Regressions Using DFGs
Differential Flame Graphs (1)
14Understanding Software Performance Regressions Using DFGs
Demo
15Understanding Software Performance Regressions Using DFGs
Differential Flame Graphs (2)
• You can do this with performance profiles..
• But also with other data?
• Ideas:
• Website clickpaths
• Parallel/distributed computing
• ...
• Let us know! http://corpaul.github.io/flamegraphdiff/
• (they use this at Netflix, so it must be pretty cool!)
16Understanding Software Performance Regressions Using DFGs
Discussion
• Data collection is difficult
• No out of the box solutions
• Expensive
• Data variation between executions can be considerable
• Especially for 'non-stable' metrics (CPU, memory)
17Understanding Software Performance Regressions Using DFGs
c.bezemer@tudelft.nl
http://corpaul.github.io/flamegraphdiff/

More Related Content

What's hot

Coverage Solutions on Emulators
Coverage Solutions on EmulatorsCoverage Solutions on Emulators
Coverage Solutions on Emulators
DVClub
 
Testing using load runner performance testing
Testing using load runner  performance testingTesting using load runner  performance testing
Testing using load runner performance testing
SivaprasanthRentala1975
 

What's hot (20)

Coverage Solutions on Emulators
Coverage Solutions on EmulatorsCoverage Solutions on Emulators
Coverage Solutions on Emulators
 
Regression Testing with Symfony
Regression Testing with SymfonyRegression Testing with Symfony
Regression Testing with Symfony
 
Test Automation for Embedded Devices
Test Automation for Embedded DevicesTest Automation for Embedded Devices
Test Automation for Embedded Devices
 
Practical unit testing in c & c++
Practical unit testing in c & c++Practical unit testing in c & c++
Practical unit testing in c & c++
 
JavaCro'14 - Profile any environment with Java Flight Recorder – Johan Janssen
JavaCro'14 - Profile any environment with Java Flight Recorder – Johan JanssenJavaCro'14 - Profile any environment with Java Flight Recorder – Johan Janssen
JavaCro'14 - Profile any environment with Java Flight Recorder – Johan Janssen
 
Linux binary analysis and exploitation
Linux binary analysis and exploitationLinux binary analysis and exploitation
Linux binary analysis and exploitation
 
Chris brown ti
Chris brown tiChris brown ti
Chris brown ti
 
.NET Debugging Workshop
.NET Debugging Workshop.NET Debugging Workshop
.NET Debugging Workshop
 
Test execution may_04_2006
Test execution may_04_2006Test execution may_04_2006
Test execution may_04_2006
 
Overview of FreeBSD PMC Tools
Overview of FreeBSD PMC ToolsOverview of FreeBSD PMC Tools
Overview of FreeBSD PMC Tools
 
STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017
 
How We Test Linux
How We Test LinuxHow We Test Linux
How We Test Linux
 
Mesa and Its Debugging
Mesa and Its DebuggingMesa and Its Debugging
Mesa and Its Debugging
 
Benchmarking PyCon AU 2011 v0
Benchmarking PyCon AU 2011 v0Benchmarking PyCon AU 2011 v0
Benchmarking PyCon AU 2011 v0
 
Testing using load runner performance testing
Testing using load runner  performance testingTesting using load runner  performance testing
Testing using load runner performance testing
 
Fantastic performance and where to find it
Fantastic performance and where to find itFantastic performance and where to find it
Fantastic performance and where to find it
 
Mixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting exampleMixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting example
 
Continuous Performance Testing for Microservices
Continuous Performance Testing for MicroservicesContinuous Performance Testing for Microservices
Continuous Performance Testing for Microservices
 
Linux Server Deep Dives (DrupalCon Amsterdam)
Linux Server Deep Dives (DrupalCon Amsterdam)Linux Server Deep Dives (DrupalCon Amsterdam)
Linux Server Deep Dives (DrupalCon Amsterdam)
 
Symfony 2 under control
Symfony 2 under controlSymfony 2 under control
Symfony 2 under control
 

Viewers also liked

Logging library migrations - A case study for the Apache Software Foundation ...
Logging library migrations - A case study for the Apache Software Foundation ...Logging library migrations - A case study for the Apache Software Foundation ...
Logging library migrations - A case study for the Apache Software Foundation ...
corpaulbezemer
 
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
Chakkrit (Kla) Tantithamthavorn
 
Optimizing the Performance-Related Configurations of Object-Relational Mappin...
Optimizing the Performance-Related Configurations of Object-Relational Mappin...Optimizing the Performance-Related Configurations of Object-Relational Mappin...
Optimizing the Performance-Related Configurations of Object-Relational Mappin...
corpaulbezemer
 

Viewers also liked (7)

Logging library migrations - A case study for the Apache Software Foundation ...
Logging library migrations - A case study for the Apache Software Foundation ...Logging library migrations - A case study for the Apache Software Foundation ...
Logging library migrations - A case study for the Apache Software Foundation ...
 
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
 
An exploratory study of the state of practice of performance testing in Java-...
An exploratory study of the state of practice of performance testing in Java-...An exploratory study of the state of practice of performance testing in Java-...
An exploratory study of the state of practice of performance testing in Java-...
 
Optimizing the Performance-Related Configurations of Object-Relational Mappin...
Optimizing the Performance-Related Configurations of Object-Relational Mappin...Optimizing the Performance-Related Configurations of Object-Relational Mappin...
Optimizing the Performance-Related Configurations of Object-Relational Mappin...
 
Blazing Performance with Flame Graphs
Blazing Performance with Flame GraphsBlazing Performance with Flame Graphs
Blazing Performance with Flame Graphs
 
Hype vs. Reality: The AI Explainer
Hype vs. Reality: The AI ExplainerHype vs. Reality: The AI Explainer
Hype vs. Reality: The AI Explainer
 
TEDx Manchester: AI & The Future of Work
TEDx Manchester: AI & The Future of WorkTEDx Manchester: AI & The Future of Work
TEDx Manchester: AI & The Future of Work
 

Similar to SANER 2015 ERA track: Differential Flame Graphs

Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...
Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...
Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...
Databricks
 
Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)
Babul Mirdha
 

Similar to SANER 2015 ERA track: Differential Flame Graphs (20)

PERFORMANCE COMPARISON ON JAVA TECHNOLOGIES - A PRACTICAL APPROACH
PERFORMANCE COMPARISON ON JAVA TECHNOLOGIES - A PRACTICAL APPROACHPERFORMANCE COMPARISON ON JAVA TECHNOLOGIES - A PRACTICAL APPROACH
PERFORMANCE COMPARISON ON JAVA TECHNOLOGIES - A PRACTICAL APPROACH
 
Performance comparison on java technologies a practical approach
Performance comparison on java technologies   a practical approachPerformance comparison on java technologies   a practical approach
Performance comparison on java technologies a practical approach
 
Debuggers in system software
Debuggers in system softwareDebuggers in system software
Debuggers in system software
 
Groovy In the Cloud
Groovy In the CloudGroovy In the Cloud
Groovy In the Cloud
 
ICPE 2022 - Data Challenge
ICPE 2022 - Data ChallengeICPE 2022 - Data Challenge
ICPE 2022 - Data Challenge
 
C# Production Debugging Made Easy
 C# Production Debugging Made Easy C# Production Debugging Made Easy
C# Production Debugging Made Easy
 
Real time debugging: using non-intrusive tracepoints to debug live systems
Real time debugging: using non-intrusive tracepoints to debug live systemsReal time debugging: using non-intrusive tracepoints to debug live systems
Real time debugging: using non-intrusive tracepoints to debug live systems
 
Real time debugging: using non-intrusive tracepoints to debug live systems
Real time debugging: using non-intrusive tracepoints to debug live systemsReal time debugging: using non-intrusive tracepoints to debug live systems
Real time debugging: using non-intrusive tracepoints to debug live systems
 
Debugging Modern C++ Application with Gdb
Debugging Modern C++ Application with GdbDebugging Modern C++ Application with Gdb
Debugging Modern C++ Application with Gdb
 
程序员实践之路
程序员实践之路程序员实践之路
程序员实践之路
 
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
 
Deep into your applications, performance & profiling
Deep into your applications, performance & profilingDeep into your applications, performance & profiling
Deep into your applications, performance & profiling
 
GCC Compiler as a Performance Testing tool for C programs
GCC Compiler as a Performance Testing tool for C programsGCC Compiler as a Performance Testing tool for C programs
GCC Compiler as a Performance Testing tool for C programs
 
Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...
Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...
Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...
 
Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)
 
Apache Spark Performance is too hard. Let's make it easier
Apache Spark Performance is too hard. Let's make it easierApache Spark Performance is too hard. Let's make it easier
Apache Spark Performance is too hard. Let's make it easier
 
STATISTICAL ANALYSIS FOR PERFORMANCE COMPARISON
STATISTICAL ANALYSIS FOR PERFORMANCE COMPARISONSTATISTICAL ANALYSIS FOR PERFORMANCE COMPARISON
STATISTICAL ANALYSIS FOR PERFORMANCE COMPARISON
 
REDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTS
REDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTSREDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTS
REDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTS
 
REDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTS
REDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTSREDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTS
REDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTS
 
Oleksii Moskalenko "Continuous Delivery of ML Pipelines to Production"
Oleksii Moskalenko "Continuous Delivery of ML Pipelines to Production"Oleksii Moskalenko "Continuous Delivery of ML Pipelines to Production"
Oleksii Moskalenko "Continuous Delivery of ML Pipelines to Production"
 

Recently uploaded

Recently uploaded (20)

AI Hackathon.pptx
AI                        Hackathon.pptxAI                        Hackathon.pptx
AI Hackathon.pptx
 
Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...
Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...
Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...
 
how-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfhow-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdf
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning Framework
 
The Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionThe Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion Production
 
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfMicrosoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Kraków
 
OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024
 
INGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by DesignINGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by Design
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
 
A Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationA Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data Migration
 
What need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java DevelopersWhat need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java Developers
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in Michelangelo
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
 
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purityAPVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
 
Crafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM IntegrationCrafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM Integration
 
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
 
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
 
5 Reasons Driving Warehouse Management Systems Demand
5 Reasons Driving Warehouse Management Systems Demand5 Reasons Driving Warehouse Management Systems Demand
5 Reasons Driving Warehouse Management Systems Demand
 

SANER 2015 ERA track: Differential Flame Graphs

  • 1. 3/5/15 Challenge the future Delft University of Technology Understanding Software Performance Regressions Using Differential Flame Graphs Cor-Paul Bezemer, Johan Pouwelse, Brendan Gregg
  • 2. 2Understanding Software Performance Regressions Using DFGs Regression Testing
  • 3. 3Understanding Software Performance Regressions Using DFGs Regression Testing Cycle 1)Write code 2)Deploy to continuous integration system 3)Test 4)Deploy to production code or return to 1)
  • 4. 4Understanding Software Performance Regressions Using DFGs Performance Regression Testing • Functional testing is difficult, but: • Functionality is either OK or broken • Performance testing is even more difficult: • Depends on requirements of application • A library may be fast enough for one application, while it's too slow for another • How much performance regression is acceptable?
  • 5. 5Understanding Software Performance Regressions Using DFGs
  • 6. 6Understanding Software Performance Regressions Using DFGs What to Measure • Built-in performance metrics (or counters) • Response time, throughput, queue lengths, utilization • Usually system-wide • Profilers • Per-function info on execution time, bytes written, etc. • Higher granularity == more expensive • After measuring: tons of data! :) / :(
  • 7. 7Understanding Software Performance Regressions Using DFGs
  • 8. 8Understanding Software Performance Regressions Using DFGs Detecting I/O Regression • Previous work: • “Detecting and analyzing I/O performance regressions”, C. Bezemer, E. Milon, A. Zaidman, J. Pouwelse (Journal of Software: Evolution and Process 26 (12), 1193-1212) • Approach for finding out which function exhibits I/O write regression over two software versions • Results promising, but sometimes difficult to interpret
  • 9. 9Understanding Software Performance Regressions Using DFGs Work in Progress: Visualization
  • 10. 10Understanding Software Performance Regressions Using DFGs Flame Graphs (1) def a ( ) : if some condition : b() … def b ( ) : if some condition : c () ... def c( ): ... def d ( ) : if some condition : c () ... Stack trace, CPU time a(), 100 a()→b(), 25 a()→b()→c(), 10 d(), 50 d()→c(), 10
  • 11. 11Understanding Software Performance Regressions Using DFGs Flame Graphs (2)
  • 12. 12Understanding Software Performance Regressions Using DFGs Flame Graphs (3)
  • 13. 13Understanding Software Performance Regressions Using DFGs Differential Flame Graphs (1)
  • 14. 14Understanding Software Performance Regressions Using DFGs Demo
  • 15. 15Understanding Software Performance Regressions Using DFGs Differential Flame Graphs (2) • You can do this with performance profiles.. • But also with other data? • Ideas: • Website clickpaths • Parallel/distributed computing • ... • Let us know! http://corpaul.github.io/flamegraphdiff/ • (they use this at Netflix, so it must be pretty cool!)
  • 16. 16Understanding Software Performance Regressions Using DFGs Discussion • Data collection is difficult • No out of the box solutions • Expensive • Data variation between executions can be considerable • Especially for 'non-stable' metrics (CPU, memory)
  • 17. 17Understanding Software Performance Regressions Using DFGs c.bezemer@tudelft.nl http://corpaul.github.io/flamegraphdiff/

Editor's Notes

  1. Regression testing is done to make sure that old errors are not reintroduced Functional regression testing is fairly straightforward to interpret: did we break existing functionality after doing an update?