Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Performance Regression Analysis: Accomplishments and Challenges

458 views

Published on

My invited talk at the The Fifth International Workshop on Large-Scale Testing (LT 2016, co-located with ICPE).

Published in: Software
  • Be the first to comment

  • Be the first to like this

Performance Regression Analysis: Accomplishments and Challenges

  1. 1. Performance Regression AnalysisPerformance Regression Analysis Cor-Paul Bezemer Title:QueensLogo_colour.eps Creator:Adobe Illustrator(R) 12 CreationDate:9/10/08 LanguageLevel:2 Accomplishments and ChallengesAccomplishments and Challenges
  2. 2. ““A burnt child dreads fire”A burnt child dreads fire”
  3. 3. Regression testing makes sure we doRegression testing makes sure we do not make the same mistake twicenot make the same mistake twice Write code Execute regression test Deploy or fix changes
  4. 4. Regression testing also helps us to makeRegression testing also helps us to make sure we are not re-introducing more bugssure we are not re-introducing more bugs
  5. 5. Performance regression testing helpsPerformance regression testing helps ensure we are not breaking performanceensure we are not breaking performance Write code Execute performance test Deploy if performance is OK or fix changes
  6. 6. Performance regression testingPerformance regression testing can/should be done on several metricscan/should be done on several metrics
  7. 7. Performance regression testingPerformance regression testing can/should be done on several metricscan/should be done on several metrics
  8. 8. Performance regression testingPerformance regression testing can/should be done on several metricscan/should be done on several metrics
  9. 9. Performance regression testingPerformance regression testing can/should be done on several metricscan/should be done on several metrics
  10. 10. Directions in research onDirections in research on performance regression analysisperformance regression analysis
  11. 11. Directions in performanceDirections in performance regression analysis researchregression analysis research 1. Execution profile comparison 2. Source code analysis 3.Visualization
  12. 12. Execution profile comparisonExecution profile comparison
  13. 13. Execution profile comparisonExecution profile comparison
  14. 14. Disadvantages of executionDisadvantages of execution profile comparisonprofile comparison - Actually needs performance tests - Need to run performance tests several times because of variation - Small changes may be difficult to interpret
  15. 15. Source code analysisSource code analysis Identify risky commits and recommend those for testing (Huang et al. @ ICSE 2014)
  16. 16. Disadvantages of sourceDisadvantages of source code analysiscode analysis - Can not replace 'real' performance measurements (yet)
  17. 17. Visualization ofVisualization of performance regressionperformance regression analysis resultsanalysis results
  18. 18. Control chartsControl charts (Nguyen et al. @ MSR 2014)(Nguyen et al. @ MSR 2014)
  19. 19. Differential flame graphsDifferential flame graphs (Bezemer et al. @ SANER 2015)(Bezemer et al. @ SANER 2015)
  20. 20. Deciding on the baselineDeciding on the baseline performanceperformance
  21. 21. Deciding on the acceptableDeciding on the acceptable performance deviationperformance deviation
  22. 22. Reporting results to theReporting results to the developerdeveloper
  23. 23. Availability ofAvailability of performance testsperformance tests
  24. 24. Execution time of testsExecution time of tests
  25. 25. Performance regression testing helpsPerformance regression testing helps ensure we are not breaking performanceensure we are not breaking performance Write code Execute performance test Deploy if performance is OK or fix changes
  26. 26. Directions in performanceDirections in performance regression analysis researchregression analysis research 1. Execution profile comparison 2. Source code analysis 3.Visualization
  27. 27. Availability ofAvailability of performance testsperformance tests
  28. 28. Execution time of testsExecution time of tests

×