The document discusses the need for continuous and comprehensive performance testing. It provides examples of performance issues that arose when testing was not rigorous enough. Performance can degrade with new releases or branches that are not properly tested. Testing also needs to simulate complex scenarios like node failures to ensure the system works as expected under different conditions. Rigorous performance testing helps avoid production issues and ensure the system maintains acceptable throughput.

1. Continuous Performance
Testing
Mairbek Khadikov

2. Story #1
Performance, we are
losing it

3. Story #1
New version goes live soon
Let's check performance
Delivery
Manager

4. Story #1
Performance degraded
=
Performance
Tester

5. Story #1
Let's profile our app
Developer

6. Story #1
Performance increased
on 10%
Performance But it's still worse than
Tester
the previous release

7. Story #1
We can't wait much longer
Delivery
Manager CPU utilization is 50%

8. Story #1
Alert is rased
too late

9. Story #2
It's all tested, but...

10. Story #2
We deployed new release
It passed all smoke tests
Works incorrect with load
Operations balancer

11. Story #2
We tested all this features
QA

12. Story #2
System has acceptable
throughput
Performance
Tester

13. Story #2
Doh, one singleton unit is
not thread safe
Developer

14. Story #2
Performance test
does not check that
system works
correctly

15. Story #3
Forgotten branches

16. Story #3
Performance degraded in
the last release candidate
Performance
Tester

17. Story #3
What branch exactly impacted
performance?
Developer

18. Story #3
I've never heard about the half
of this branches
Performance
Tester

19. Story #3
Large projects need a
comprehensive
performance testing
process

20. Story #4
Did you test that?

21. Story #4
Production system failed
this morning
VP of
Engineering

22. Story #4
Two nodes were restarted
simultaneously
Developer

23. Story #4
Those nodes consumed to
much memory
Operations

24. Story #4
This is a complex scenario
we can't do it continuously
QA Lead

25. Story #4
Simulation of
maintenance operations
and failures was not a
part of testing

26. Performance Testing
Revised

27. Performance Testing
Continuous
Automatically test builds

28. Performance Testing
Continuous
Rise alert flag in case of
performance degradation

29. Performance Testing
Traceable
Store all testing results

30. Performance Testing
Validated
Guaranty that system works
correctly under workload

31. Performance Testing
Transparent
Result should include monitoring
and profiling information

32. Performance Testing
Transparent
Modules should be tested in
isolation where possible

33. Performance Testing
Pessimistic
Simulate potential maintenance
operations and failure

34. Thank you
@mairbek
mkhadikov@gmail.com
mairbek.github.com