- Automating performance tests through continuous integration can provide direct feedback on performance changes after code releases and infrastructure changes. It allows performance issues to be detected and addressed earlier. - Key best practices include starting with a single important test scenario, focusing on robustness over realism, visualizing trend data over time, and analyzing results to update thresholds and catch regressions. - The goal is to continuously monitor performance through the pipeline and in production to better understand impacts of changes and flag any performance issues for further investigation. Automated tests complement but do not replace thorough acceptance testing.

1. Best Practices of performance tests in
Continuous Integration
By Joerek van Gaalen

2. Joerek van Gaalen
• Performance specialist since 2005
• Independent Performance Specialist since
2018
• 100+ performance test projects

3. Relevant Experience

4. YOLT
Money manager app
Aggregation platform for 2rd parties
PSD2 ready
Millions of requests per day on open banking

5. YOLT
• Owner of performance aspect
• Agile way of working with a glance of DevOps
• Microservices architecture (45 services, 7 teams)
• CI/CD implemented with 1 to 4 releases per day
• Setting up performance tests in Continuous Integration
• Help the teams with their performance challenges

6. Goal ofthe talk
• Improve grip by automating tests
• Share best practices and guidelines
• Other activities you should do to improve grip on performance

7. WHY?

8. Why do wewant it
• Acceptance tests are late in the process
• It can run independent from the performance engineer
• Direct feedback – Fix or continue
• Trending – The more the better

9. Problems & Risks
• It’s NOT easy to do
• Too many false positives or negatives
• Scripts and thresholds need too many rework
• People don’t know how to interpret the results
• People start to ignore the tests

10. Best Practices

11. Acceptancetest vs Automatedtest
Traditional Acceptance tests
• Has the goal to prove the application meets the requirements
• Realistic simulation of production
• Different types of tests

12. Acceptancetest vs Automatedtest
Automated tests
• Should show difference with prior tests
• Runs automatically after builds, in release pipelines or scheduled
• Usually a load test scenario
• Shouldn’t necessarily be a realistic simulation (but preferred!)

13. Approach –Start small
• Start with a single script
• Your most important and meaningful script
• Then later add new pages, transactions, variations and scripts
• If possible, clone the acceptance default load test

14. Approach –Robustness

15. Approach –Realism
• Being realistic is ideal
The more realistic, the more meaningful your results are
• Deviate to be less realistic if necessary to improve robustness
• Realistic load model is nice, but having sufficient measurements is better
• Spend time on your test data and environment

16. Approach –Other robustness factors
• Self healing test data
• Avoid randomization on ‘many’ things
- Test data
- Opening random URLs
- Random iterations
- Random think times are still good

17. The process –The release pipeline
• Add load test(s) into the pipeline – after functional tests
• Block release pipeline of test fails
• Distribute results to people who are involved

18. The process –Run scheduled too
• Generally: The more the better
But it should not frustrate the process
• Also run tests scheduled because not all changed are due to releases
- Infra changes
- Sneaking changes outside the pipeline
- Network changes (routes change too)

19. Visualise the data
• Detailed results per test <link>
• Graphical trend lines of your data <link>
• Tagging of releases or changes
• Errors per transaction and requests
• Make data easily available for everyone

20. Which scenario to automate?
• I would say a load test is the most proper scenario
• Duration is a balance of faster pipelines and more robust measurements
• Duration depends on:
- duration of the user flows
- deviation of the response times and errors
- number of measurements you need for stable result
• What about an automated stress test?

21. Analysis - 1
• End result show give PASS or FAIL
• Set thresholds on average response times + 10-25% of baseline
• Set thresholds on max error percentage (0-3%)
• Set thresholds on All requests and individual requests & transactions

22. Analysis - 2
• Manual analysis on failure and sometimes on pass
• Keep updating thresholds

23. Analysis –My experience
• It’s a learning journey
• Stopped using percentiles – too voilatile
• Changed load model to increase number of measurements
• Didn’t set too strict thresholds

24. Synthetic Monitoring
• Monitor your production environment with your scripts
• Monitor your acceptance environment too
• Annotate releases

25. Issues along myjourney- 1
• Too many false negatives
• Unstable environments and services
• Overview of results and comparisons
• I’m the single point of contact
• Hard to test micro services isolated

26. Issues along myjourney- 2
• It’s hard to transfer true performance testing skills
• Automated tests is a real benefit, but doesn’t replace through acceptance
tests

27. Ultimategoal
• Continuous load test
• Automatic recognition of anomalities – reducing false positives/negatives

28. Endthoughts
• Automated tests will not replace traditional acceptance tests
• Setting up automated tests can be a long journey

29. Q&A
• What about automated stress tests?
• What about automated self adjusting thresholds?
• Do you find isolated tests useful or too time consuming with no benefits?