This document discusses using Application Performance Management (APM) tools to detect performance regressions in web applications. It presents a case study where performance regressions were injected into test systems and then evaluated whether commercial and open source APM tools could detect the issues. The study found that APM tools can successfully detect some performance regressions, but they have limitations like producing large reports that require manual exploration and lacking actionable suggestions for fixes. The document concludes that APM tools show promise as a way to deploy performance regression detection research into practice.

1. 1
Studying the Effectiveness of Application
Performance Management (APM) Tools
for Detecting Performance Regressions
for Web Applications
Tse-Hsun Chen Ahmed E. Hassan Weiyi ShangTarek M. Ahmed Cor-Paul Bezemer
An Experience Report

2. 2

3. 3
$1.6 billion loss for a one-second slowdown

4. What is a performance regression?
4
Version 1 Version 2
Does version 2 have worse
performance than the version 1?

5. Performance regression:
changed source code + unchanged workload
change in performance?
5
Version 1
Version 2
requests
requests
requests
Performance
metrics
Performance
metrics
Unchanged
workload Testing environment
Testing environment
Changed
source
code
Change in
performance?

6. MSR techniques are widely used to detect
performance regressions
6

7. MSR techniques are widely used to detect
performance regressions
7
Building statistical models Mining data associationsComparing distributions

8. An example of using MSR techniques to detect
performance regressions
8
Version 1 Version 2
Build
model
Verify
model
Model Prediction
error
< threshold
> threshold
Shang et al. ICPE 2015

9. The extensive performance regression
detection research is hardly adopted in practice
9
Too complex to implement by
practitioners
Not easily accessible

10. Application performance management
(APM) tools are well adopted in practice
10

11. APM tools work with:
unchanged source code + changing workload
11
requests
requests
requests
Real users
Field environment
Slow transactions
Changing
workload
Unchanged
source code

12. 12
An example report from an APM tool
Most time consuming
transactions
Time breakdown for the
transaction

13. 13
Performance
regressions
detection
research
+
unchanged workload
changed source code
Not well adopted
unchanged source code
+
changing workload
Widely adopted

14. Research Questions:
Can APM tools detect performance
regressions?
Can APM tools be used as a vehicle
to deploy performance regression
detection research in practice?
14

15. Case study: subject systems and injected
performance regressions
CPU overhead
Memory overhead
15
Inefficient use
of database
• Unreleased resources
• Inefficient use of data streams
• Unnecessary loop
• Loop with a missing break
condition
• One-by-one processing
• Excessive data

16. We leverage APM tools to detect injected
performance regressions
Slow transactions in new version
Slow transactions in old version
16
Version 1
Version 2
requests
requests
requests
Unchanged
workload Testing environment
Testing environment
Changed
source
code

17. 0
1
2
3
4
5
6
APM tools can detect performance
regressions
17
#ofdetected
issuesdetected
Commercial Open source
Total 6 issues
injected

18. APM tools have rooms of improvement
18
APM tools are a great vehicle to push
performance regression detection
research into practice.
Large amounts of reports
for manual exploration
Basic analysis
Lack of actionable
suggestion

19. 19
$1.6 billion loss for a one-second slowdown

20. 20

21. 21
unchanged source code
+
changing workload
Widely adopted
Performance
regressions
detection
research
changed source code
+
unchanged workload
Not well adopted

22. 22

23. Case study: subject systems and injected
performance regressions
CPU overhead
Memory overhead
23
Inefficient use
of database
• Unreleased resources
• Inefficient use of data streams
• Unnecessary loop
• Loop with a missing break
condition
• One-by-one processing
• Excessive data

24. 24

25. 0
1
2
3
4
5
6
APM tools can detect performance
regressions
25
#ofdetected
issuesdetected
Commercial Open source
Total 6 issues
injected

26. 26
Weiyi Shang shang@.encs.concordia.ca