Presented at the ICST 2012

1. Using Control Charts for Detecting and
Understanding Performance
Regressions in Large Software
Thanh Nguyen
Supervisor: Ahmed E. Hassan
Queen’s University, Kingston, Canada
1

2. 2
Motivation

3. Performance is very important aspect
in modern software engineering
Local
Single user
Asynchronous
interaction
3
Before Now
Network
Massive multi-users
Synchronous
interaction
Facebook
50%
Canadian
190

4. Performance testing is very important
to ensure software performance
Local
Single user
Asynchronous
interaction
4
Network
Massive multi-users
Synchronous
interaction
Higher latency
Require fast servers
Require central storage
Need states synchronization
Need guarantee response time
Need redundancy

5. I focus on one kind of performance
testing: regression load test
5
Version 1 Version 1.1

Baseline Target

6. How to detect performance
regression?
6
Applying load
Version 1.1
Version 1
CPU %,
Memory usage
CPU %,
Memory usage
Detect
regression

7. Challenge in Performance Regression
Testing
7
Layer 1 –
Agent 1
Layer 1 –
Agent 2
Layer 2 –
Agent 1
Layer 2 –
Agent 2
Layer 2 –
Agent 3
Layer 2 –
Agent 4
Layer 3 –
Agent 1
Layer 4 –
Agent 1
56 counters x 8 agents = 448 counters
56 counters x 2 agents = 112 counters
Layer 1 Layer 2
Lots of data
Not enough
automation
Limited research

8. 8
Approach

9. Data mining -> R&R
9
Reduce Relate

10. Performance regression detection is
similar to monitoring and forecasting
production servers load. 10

11. Disk Subsystem Capacity
Management [Trubin and Merritt 2003]
11
eader/Outsider servers detector and detector of
unaway processes; and
eaders/Outsiders bar charts generator.
To get detailed information of the servers’ behav
for the previous day, the system publishes the SP
chart on the Intranet web site for each exception,
shown in Figure 11, where Saturday is the example

12. Proposed approach to use control charts to
find performance regression
12
Baseline
Performance counters
Target
Performance counters
Determine the
LCL, CL, UCL
730
735
740
745
750
755
760
765
770
775
0 5 10 15 20 25
Performance counter

13. Using control charts to verify load test
results
13
Baseline
Performance counters
Target
Performance counters
Determine the
LCL, CL, UCL
730
735
740
745
750
755
760
765
770
775
0 5 10 15 20 25
Performance counter
Violation
ratio
Reduce

14. 14
Baseline
Performance counters
Target
Performance counters
Target
Performance counters
730
740
750
760
770
780
0 10 20 30
Performance counter
Baseline
Performance counters
720
730
740
750
760
770
780
0 10 20 30
Performance counter
Low
violation
ratio
High
violation
ratio
We can use violation ratio to detect
regression
Relate

15. 15
Progress

16. My three research questions
Is there a performance regression?
Fairly confident that it works [ICPE 12]
16
Where does performance regression
occur?
 Need more validation [APSEC 11]
What causes the regression?
 In progress

17. Experiment set up
17
Baseline
Performance counters
Target
Performance counters
Target
Performance counters
Number of out-of-control counter is small
Average violation ratio should be low

18. Experiment set up
18
Baseline
Performance counters
Target
Performance counters
Target
Performance counters
Number of out-of-control counter is large
Average violation ratio should be high

19. 19
Normal
Problem
0
2
4
6
8
10
12
14
16
18
Number of out-of-control counters

20. 20
Normal
Problem0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Average violation ratio

21. Where does performance regression
occur?
21
Backend
Other
backend
Front End

22. Backend 50% stress
22
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
B B B B B O
Violation ratio

23. Frontend 50% stress
23
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
F F F F F F O
Violation ratio

24. 24
Obstacles

25. Obstacles #1: Inputs are unstable
25
0
5
10
15
20
25
30
35
40
45
1 2 3 4 5 6
CPU%
Time
Version 1.0
Version 1.1Is there a
performance
regression?

26. It is very difficult to maintain stable
input across test runs
26
Applying load
Version 1.1
Version 1
CPU %,
Memory usage
CPU %,
Memory usage
Detect
regression
Randomization Cache
Warm up
Background tasks

27. Solution #1: Scale the counter according to
the input
• Step 1: Determine α and β
• Step 2:
27
CPU% Request/s
May not
work?
c = a *l + b
¢ct = ct *
a *lt + b
a *lb + b

28. Solution #1: Example of the effectiveness of
scaling
28

29. Obstacles #2: Multiple inputs
29
0
5
10
15
20
25
30
35
40
45
10 20 30 40 50 60 70 80 90 100
Density%
CPU Usage
Density plot of two test runs
Version 1.0
Version 1.1
IF … THEN
…
ELSE
…

30. Solution #2: Isolating the counters
30
0
5
10
15
20
25
30
35
40
45
10 20 30 40 50 60 70 80 90 100
Density%
CPU Usage
Density plot of two test runs
Version 1.0
Version 1.1
Local minima

31. 31