https://www.neotys.com/performance-advisory-council/federico_toledo If you consider that performance testing is a must for your CI/CD pipeline to detect degradations as soon as possible, this talk will be useful for you. How do you manage to run a performance test against each service of your system? And what about running almost every day, all the tests? It’s essential to take into consideration important aspects from the beginning to get the most out of it. Yes, it’s a big effort, but it’s worth it. I want to share our experience and lessons learned from working with different teams in charge of maintaining tests and the infrastructure for continuous integration and delivery focused on the performance testing tasks.

1. Adding Performance Verifications
in Continuous Delivery
by Federico Toledo
(COO at Abstracta)
federico@abstracta.us
@fltoledo

2. WHO IS ABSTRACTA?
Abstracta is a world leader in
software testing focused on
harnessing automation to improve
performance and reduce time to
market. Our mission is to make the
world a better place by co-creating
winning technologies of the
highest quality.
4 Offices Worldwide
80 Software Quality Engineers
10 Years in Business
95% Client Retention Rate
+300 Projects
Thought Leaders
Testing Educators

3. www.abstracta.us
Agenda
Introduction to the
typical approach to
performance
testing
Why it does
not work for
continuous
delivery
A different
approach with a
different goal
21 3

4. Introduction
1

5. www.abstracta.us
vs
Response Times
Resource Consumption
Performance

6. What is
performance
testing
about?

7. www.abstracta.us
Source:http://www.miniatur-wunderland.de/
Simulation

8. www.abstracta.us
Measurement

9. www.abstracta.us
Finding the Bottlenecks and the Breaking Point

10. www.abstracta.us
Types of Performance Testing

11. www.abstracta.us
Load Simulation Tools

12. www.abstracta.us
}
Simulation

13. www.abstracta.us
Simulation

14. www.abstracta.us
HTTP
HTTP
Server
Simulation

15. www.abstracta.us
Summary of the typical approach
• Define bunch of test cases (user flows) and load scenario.
• Automate them with the tool (http level).
• Run tests, analyze, work with devs and ops to improve.
• Continue until we reach the SLAs or good enough
performance (response times and resource consumption).

16. www.abstracta.us
What’s special about Continuous Delivery?
Continuous
Integration:
merge code at least
once a day.
Continuous
Deployment:
deliver continuously
to users.
Continuous
Delivery:
deliver continuously to
business people,
frequently to users
(production).
1 2 3
We need an automated pipeline with different checks.
Let’s add performance validations to this pipeline!

17. Do you think that the typical approach
for performance testing would work?
2

18. www.abstracta.us
Issues with the typical approach for CI/CD
The pipeline will run at least once a day
(probably more frequently)
End-to-End scripts are
fragile (need lots of
maintenance)
If there is an issue or
degradation, it’s hard to
find the cause
1 2

19. What can we do differently?
Different approach with different goals
3

20. www.abstracta.us
1 2
A different approach with a different goal
Frequent releases to production implies:
• Smaller difference between versions.
• Reduced risk (less uncertainty).
Detect any degradation in the
system as soon as possible.
Typical goal:
Know if the system will
support the expected load.
Goal for performance tests in
Continuous Delivery:

21. www.abstracta.us
Definition of
the test cases
to automate
Detecting degradations as soon as possible
Selection of
the tool
Design of the
load scenarios
1 2 3

22. www.abstracta.us
We love JMeter, but...

23. www.abstracta.us
We love JMeter, but...

24. www.abstracta.us
We love JMeter, but...

25. www.abstracta.us
Understanding what changed

26. www.abstracta.us
Takeaways
A Choose a CI-friendly tool.
www.abstracta.us

27. www.abstracta.us
Let’s talk about the test
What is the simplest test we can
run in order to detect degradations?
www.abstracta.us

28. www.abstracta.us
What is the simplest test we can run in
order to detect degradations?
Hint:
We want to avoid false
negatives.
Specially because we will
run these tests on a daily
basis.
End-to-End tests (http
level) are fragile.
We want to test early
and frequently.
Another hint:Remember:

29. www.abstracta.us
Cohn’s Pyramid on Test Automation
Equivalent to:
HTTP traffic between user and server.
Combining different endpoints.
Test endpoints in isolation.

30. www.abstracta.us
Cohn’s Pyramid on Test Automation
Measure user experience
User centered design
More costly (scripting, infrastructure)
Cheaper (easier)
Early results
More Frequent / faster tests
No idea about the user experience

31. www.abstracta.us
What we want to do:
Let’s add unit performance tests to our pipeline
This way we have the simplest test to discover
automatically degradations of the system as soon
as possible.
• Independent tests.
• Each one running against one endpoint at a time.

32. www.abstracta.us
Detect degradations right after they are introduced
ResponseTimes
0
100
200
300
400
500
600
09/12 09/16 09/20 09/24 09/28 10/02 10/06
Average response
times (ms)
Performance
assert threshold
Release branch
created
Percent KOs
0
25
50
75
100
PercentageKOs

33. www.abstracta.us
Detect degradations right after they are introduced
Requestspersecond
0
50.0
100.0
150.0
200.0
250.0
300.0
09/12 09/16 09/20 09/24 09/28 10/02 10/06
Means requests
per second
Performance
assert threshold
Release branch
created
Percent KOs
0
25
50
75
100
PercentageKOs

34. www.abstracta.us
Define scenario and assertions

35. www.abstracta.us
Define scenario and assertions

36. www.abstracta.us
Define scenario and assertions

37. www.abstracta.us
Define scenario and assertions

38. www.abstracta.us
Define scenario and assertions

39. www.abstracta.us
Define scenario and assertions

40. www.abstracta.us
Define scenario and assertions
Load
● 350 threads
Assertions
Given
● Exclusive environment
● < 1% error
● P95 Response Times < 130ms + 10%
● Throughput >= 150 TPS – 10%

41. www.abstracta.us
You should also check Keptn and Pitometer

42. www.abstracta.us
Takeaways
A Choose a CI-friendly tool.
www.abstracta.us
B Use Unit performance tests to detect
degradations as soon as possible.

43. Is that enough?
3

44. www.abstracta.us
Fallacy of the composition
Testing the parts is not enough to know how the whole will behave.
You need to test the whole to see how the parts interact.
Parts Parts
Parts Parts
Whole

45. www.abstracta.us
Testing Strategy
Every month or Q?
Every week?
Every day?

46. www.abstracta.us
Testing Strategy
Similar to production
Scaled down

47. www.abstracta.uswww.abstracta.us
Takeaways
B
Choose a CI-friendly tool.
Use Unit performance tests to detect
degradations as soon as possible.
C Complement with integration and load tests.
A

48. Is that enough?
3

49. www.abstracta.us
What about the client side?
Web
- Google page speed
- Website.io
- Yslow
- APM
- Etc!
What about mobile?

50. www.abstracta.us
Apptim (www.apptim.com)

51. www.abstracta.us
Summary of my learnings
www.abstracta.us
B
C
Choose a CI-friendly tool.
Use Unit performance tests to detect
degradations as soon as possible.
Complement with integration and load tests.
Don’t forget the client side.
A
D

52. Adding Performance Verifications in
Continuous Delivery
by Federico Toledo (COO at Abstracta)
federico@abstracta.us
@fltoledo
¡Muchas
gracias!