Download to read offline
More Related Content
![[HCMC STC Jan 2015] Making IT Count – Agile Test Metrics](https://cdn.slidesharecdn.com/ss_thumbnails/agiletestmetricsv1-150209010817-conversion-gate01-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Automated Detection of Performance Regressions Using Statistical Process Control Techniques
Automated Detection of Performance Regressions Using Statistical Process Control Techniques
- 1. Automated Detection ofPerformance Regressions Using Statistical Process Control Techniques Thanh Nguyen, Bram Adams, ZhenMing Jiang, Ahmed E. Hassan Queen’s University, Kingston, Canada Mohamed Nasser, Parminder Flora Research in Motion, Waterloo, Canada 1
- 2.
- 3. What is aperformance regression? 3 Version 1 Version 1.1 Baseline Target
- 4. How to detectperformance regression? 4 Applying load Version 1.1 Version 1 CPU %, Memory usage CPU %, Memory usage Detect regression
- 5. Challenge in PerformanceRegression Testing 5 Layer 1 – Agent 1 Layer 1 – Agent 2 Layer 2 – Agent 1 Layer 2 – Agent 2 Layer 2 – Agent 3 Layer 2 – Agent 4 Layer 3 – Agent 1 Layer 4 – Agent 1 56 counters x 8 agents = 448 counters 56 counters x 2 agents = 112 counters Layer 1 Layer 2 Lots of data
- 6.
- 7. Data mining ->Reduce and Relate 7 Reduce Relate
- 8. Proposed approach touse control charts to find performance regression 8 Baseline Performance counters Target Performance counters Determine the LCL, CL, UCL 730 735 740 745 750 755 760 765 770 775 0 5 10 15 20 25 Performance counter
- 9. Using control chartsto verify load test results 9 Baseline Performance counters Target Performance counters Determine the LCL, CL, UCL 730 735 740 745 750 755 760 765 770 775 0 5 10 15 20 25 Performance counter Violation ratio Reduce
- 10. 10 Baseline Performance counters Target Performance counters Target Performancecounters 730 740 750 760 770 780 0 10 20 30 Performance counter Baseline Performance counters 720 730 740 750 760 770 780 0 10 20 30 Performance counter Low violation ratio High violation ratio We can use violation ratio to detect regression Relate
- 11.
- 12. Obstacles #1: Inputsare unstable 12 0 5 10 15 20 25 30 35 40 45 1 2 3 4 5 6 CPU% Time Version 1.0 Version 1.1Is there a performance regression?
- 13. It is verydifficult to maintain stable input across test runs 13 Applying load Version 1.1 Version 1 CPU %, Memory usage CPU %, Memory usage Detect regression Randomization Cache Warm up Background tasks
- 14. Solution #1: Scalethe counter according to the input • Step 1: Determine α and β • Step 2: 14 CPU% Request/s c = a *l + b ¢ct = ct * a *lt + b a *lb + b
- 15. Solution #1: Exampleof the effectiveness of scaling 15
- 16. Obstacles #2: Multipleinputs 16 0 5 10 15 20 25 30 35 10 20 30 40 50 60 70 80 90 100 Density% CPU Usage Density plot of two test runs IF … THEN … ELSE …
- 17. 0 5 10 15 20 25 30 35 10 20 3040 50 60 70 80 90 100 Density% CPU Usage Density plot of two test runs Solution #2: Isolating the counters 17 Local minima
- 18. Scale and filter 18 Applyingload Version 1.1 Version 1 CPU %, Memory usage CPU %, Memory usage Detect regression Scale Scale Filter Filter
- 19.
- 20. Experiment set up 20 Baseline Performancecounters Target Performance counters Target Performance counters Average violation ratio should be low
- 21. 21 Baseline Performance counters Target Performance counters Target Performancecounters Average violation ratio should be high Experiment set up
- 22.
- 23.
- 24. 24 Experiment set up V.S. Precisionis high Recall should be high
- 25.
- 26.