Neotys organized its first Performance Advisory Council in Scotland, the 14th & 15th of November. With 15 Load Testing experts from several countries (UK, France, New-Zeland, Germany, USA, Australia, India…) we explored several theme around Load Testing such as DevOps, Shift Right, AI etc. By discussing around their experience, the methods they used, their data analysis and their interpretation, we created a lot of high-value added content that you can use to discover what will be the future of Load Testing. You want to know more about this event ? https://www.neotys.com/performance-advisory-council

1. The Myth of Continuous Performance Testing
Stephen Townshend

2. Introduction

4. 27/11/2017

5. What is CPT?

7. Fail Pass

8. The concerns

9. The concerns
• The limited value of component tests
• Losing sight of the big picture
• Test asset maintenance
• Automated pass/fail outcomes
• Production-like environments
27/11/2017

10. The limitedvalue of component testing

11. Performancerisk

12. What component testingtells us
What component testing
tells us
What it does NOT tell us
Response Time • Component specific
response time
• User/customer experience
in the real world
Capacity • Component capacity
• Component resource usage
in isolation
• Capacity of integrated
solution under real
conditions
• Issues relating to the way
components integrate
Stability • Nothing • Stability of the solution
over prolonged periods of
use

15. Credit card example

16. Losingsight of thebig picture
• Keeping sight of the business purpose is incredibly important. It helps us
make better decisions about what and how to performance test
appropriately for each situation.
• Performance is not a problem we can necessarily break into smaller parts.
The sum of the parts is not the same as the whole.

20. Test asset maintenance

21. Test asset maintenance

22. Fail Pass
• Is meaningful to the business
• Makes mathematical sense
• Is not overly complicated
• Is maintainable/configurable
Automatedpass/failoutcome

23. Automatedpass/failoutcome

24. Automatedpass/failoutcome

25. Production-likeenvironments
• In an ideal world we would always test in prod (or prod-like) environments
• Principle of DevOps
• But in reality – this is almost never the case

26. Production
Environment
Test
Environment

27. Production-likeenvironments
• How non-production environments differ from production:
• Lower spec (physical) or different hardware configuration (virtual, cloud)
• Different numbers of nodes or servers
• Missing integration points
• Different application or database configuration
• The absence of realistic volumes of data
• Background jobs which run in production but not test
• All of this eats into the confidence we can have in the results

28. Production-likeenvironments
Difference from production Risk/impact
Lower spec hardware • Environment may not scale to meet production peak
load
Different number of nodes • Environment may not scale to meet production peak
load
• Load balancing related issues may not be picked up
Missing integrations • End user experience not measured accurately
• Performance issues relating to external integrations
missed
Different app or DB configuration • Any test results are unlikely to reflect real world
performance (response time, capacity, or stability)
Absence of realistic data volumes • Artificially quick response times (databases are faster
without data)
Background jobs not running • Conflicts between online users and background jobs
not assessed (user experience, server resources, stability)

29. When is it appropriate?

30. When is continuous performancetestingmost appropriate?

31. Moving in the right direction

32. Movingin the rightdirection

33. FIN