The document discusses how traditional performance testing does not align well with DevOps approaches and proposes continuous performance testing as a better alternative. Continuous performance testing involves evaluating performance at each stage of the development pipeline as appropriate, providing more frequent and visible performance feedback across builds. The document outlines best practices for implementing continuous performance testing such as incorporating performance requirements early, integrating performance skills into teams, and starting with smaller tests that are expanded over time with feedback.

1. @USI_LeeBarnes
Continuous Performance
Testing in DevOps
1

2. @USI_LeeBarnes
About Me…
• CQO of Forte Group
• Involved in software quality and testing for over 25 years
• Most of that time focused on automation in testing and performance
testing
• Passionate about helping organizations build quality into their
applications
2
Email: lee.barnes@fortegrp.com
Twitter: @USI_LeeBarnes
LinkedIn: linkedin.com/in/leebarnes
Blog: utopiasolutions.com/blog
COPYRIGHT FORTE GROUP

3. @USI_LeeBarnes
3
Performance Testing is Dead!
@USI_LeeBarnes COPYRIGHT UTOPIA SOLUTIONS, INC.
As the ONLY evaluation of a system’s performance
as the ONLY evaluation of system performance

4. @USI_LeeBarnes
“Traditional” Performance Testing
4
Plan Design Build Test Deploy
Performance Testing “Rules”
• Application is functionally complete
• Code is frozen for duration of the test
• Executed in a production-like environment
Performance Test
Many Weeks
Many Months
COPYRIGHT FORTE GROUP

5. @USI_LeeBarnes
“Traditional” Performance Testing
5
What if we find issues?
Delay release or….
COPYRIGHT FORTE GROUP

6. @USI_LeeBarnes
Traditional Performance Testing and DevOps
6
Performance Test Performance Test
Performance Test Performance Test
Short / Agile
Development Cycles
Lengthy / Rigid
Performance Tests
+ =
COPYRIGHT FORTE GROUP

7. @USI_LeeBarnes
What is Continuous Performance?
• Continuous Performance is not…
─ More frequent traditional performance tests
─ Moving traditional performance testing earlier in the delivery cycle
─ Faster or completely automated performance testing
• But it is…
─ Evaluation of performance at each stage (as appropriate) of the delivery pipeline
─ More frequent and visible performance feedback
─ Trending of performance data across builds
7 COPYRIGHT FORTE GROUP

8. @USI_LeeBarnes
How are we doing?
8
Low Medium High Elite
Automated Unit Tests 57% 66% 84% 87%
Automated Acceptance Tests 28% 38% 48% 58%
Automated Performance Tests 18% 23% 18% 28%
Automated Security Tests 15% 28% 25% 31%
- Data from 2019 Accelerate State of DevOps Report
Test Automation By DevOps Performance Profile
COPYRIGHT FORTE GROUP

9. @USI_LeeBarnes
What’s Missing?
9
• Performance requirements not part of
user stories
• Lack of ownership and accountability
for performance in delivery teams
• Delivery teams lack performance
engineering expertise
• APM tools / skills not present in the
delivery teams or pipeline
Common Obstacles to
Continuous Performance
Testing and Analysis
COPYRIGHT FORTE GROUP

10. @USI_LeeBarnes
Think About Performance Early
10
As a dealer I want to create
product configurations on
a mobile device so I can
generate quotes at the
customer site
Performance Factors
• Peak of 1200 quotes per
hour
• Varying network conditions
• Avg screen to screen
response time < 2 seconds
COPYRIGHT FORTE GROUP

11. @USI_LeeBarnes
Incorporating Performance Requirements
11
… as constraints The system will respond in less than 2 seconds for all steps
in the product configuration process while under a peak
load of 1,200 quotes per hour
… as acceptance criteria for existing user stories
As a dealer I want to create a
product configuration for a quote
so that my customer can purchase
the equipment they need
• All steps in the product config.
process will have a response
time of less than 2 seconds
• Peak expected throughput for
quote configuration is 1,200 per
hour
COPYRIGHT FORTE GROUP

12. @USI_LeeBarnes
Building Organizational Accountability
12
Accountability Awareness
=
Discuss performance factors early
Incorporate performance related
requirements
Visibility
+
Include documenting performance
factors and acceptance criteria in
definition of done
COPYRIGHT FORTE GROUP

13. @USI_LeeBarnes
Integrate Performance Skills and Tools
• Integrate existing performance testing / engineering skills into teams
─ Start small and expand expertise
─ Create a virtual CoE to provide guidance and oversight
• Bring APM tooling into delivery cycle
─ Performance diagnostics and monitoring
13 COPYRIGHT FORTE GROUP

14. @USI_LeeBarnes
Transition from Centralized to Integrated
14 COPYRIGHT FORTE GROUP
Performance Testing
Integrated with
Delivery Teams
Centralized
Performance
Testing

15. @USI_LeeBarnes
Where Can We Learn About Performance?
15
Integration
User
Acceptance
Performance /
Load
Production
Commit
COPYRIGHT FORTE GROUP

16. @USI_LeeBarnes
Performance at the Commit Stage
• Unit Performance Testing (Microbenchmarks)
─ Should be true unit tests (i.e. all dependencies mocked) but use multiple threads and iterations
─ Including as part of CI is not trivial – there are solutions to help (JMH is most popular)
• Pipeline Considerations
─ Executed on every commit (if functional unit tests pass)
─ Executed on dedicated machine (JMH can help)
─ Shouldn’t automatically break the build
─ Balance length of tests against need for fast feedback
─ Trend results across builds and feed back to developers
(SPL can assist with automated evaluation)
16
Project Type
SPL Performance unit testing research project
JMH Standalone microbenchmarking framework
Caliper Standalone microbenchmarking framework
JUnitPerf* Extension of Junit testing framework
ContiPerf* Extension of Junit testing framework
Japex* Standalone microbenchmarking framework
* project is available, but no longer maintained
Stefan, Petr, et al. “Unit Testing Performance in Java Projects.” Proceedings of the
8th ACM/SPEC on International Conference on Performance Engineering - ICPE '17,
2017, doi:10.1145/3030207.3030226.
COPYRIGHT FORTE GROUP

17. @USI_LeeBarnes
Performance at the Integration Stage
• Integration Stage Performance Testing and Measurement
─ Focus on the most critical components at first (scale as appropriate)
─ Small/short tests  Individual calls/requests (small number of threads executing a low number of iterations)
─ Dependencies mocked or virtualized to reduce setup complexity
─ Components tested individually
─ Measure system metrics via APM tools
• Pipeline Considerations
─ Executed on every deployment
─ Shouldn’t automatically break the build
─ Trend results across builds and publish
17 COPYRIGHT FORTE GROUP

18. @USI_LeeBarnes
Testing Monoliths vs. Microservices
18
Microservices Architecture
Service A Service B Service C
…
• Easier to map tests to components
• Mocking dependencies and data can be straight forward
COPYRIGHT FORTE GROUP
Monolithic Architecture
• Lines between “components” may
not be clear
• Can be more difficult to mock
dependencies and data
UI
Business Logic
Persistence

19. @USI_LeeBarnes
Performance at the Acceptance Stage
• Acceptance Stage Performance Testing and Measurement
─ Execute additional (less critical) component tests
─ Capture performance metrics during automated regression test execution
─ “Traditional” system-level performance test as necessary based on risk
• Pipeline Considerations
─ Component tests executed on every deployment
─ Component dependencies mocked or virtualized to reduce setup complexity
─ Larger system-level tests executed “out of flow” in dedicated environment with
no mocking
19
Component
performance tests and
automated regression
executed here
System-level
performance tests
executed here
COPYRIGHT FORTE GROUP

20. @USI_LeeBarnes
Putting it all together
20
Test Environment
Entity Under Test
(App, Component, etc.)
Results
Collection
Broadcast
Results
/
Alerts
APM
Continuous Integration
COPYRIGHT FORTE GROUP
Performance Test Tool

21. @USI_LeeBarnes
Implementation Approach
• Start slow and small – then speed up and scale
• Make continuous performance part of your technical backlog
• Avoid information overload
─ Is your team ready to consume the information provided?
─ Do they understand it?
─ Are they ready to act on it?
• Take care of supporting factors
─ Think critically about test scope
─ Test environments and data ready to support on-demand tests
• Solicit feedback and continuously improve!
21 COPYRIGHT FORTE GROUP

22. @USI_LeeBarnes
Performance in Production
Testing in Production
─ Confirm environment is capable of supporting
expected load
─ Test the entire system (CDN, etc.) vs. just what’s
behind the firewall
─ Understand the user experience based on
location / network conditions
22
Testing & Monitoring in Production
─ Monitor synthetic transactions in production
─ Feedback loop between upstream testing and
production monitoring data to continuously
improve monitoring and test design
Key Considerations
─ Solicit broad IT input for production testing and monitoring strategy
─ Coordinate tests with infrastructure providers
─ Identify ideal test windows to eliminate impact on customers and results (or
cordon off part of the environment for live users during execution)
─ Ensure the system “knows” it’s being tested
─ Eliminate (or mock) requests to 3rd party services
COPYRIGHT FORTE GROUP

23. @USI_LeeBarnes
Testing, Monitoring & Observability
23
Testing
Known-knowns
Ability to confirm the answer to
questions for which we know
what the answer should be…
Is page-to-page response time less
than 2 seconds under a load of
5,000 concurrent users?
Monitoring
Known-unknowns
Ability to answer questions
that we don’t yet know the
answer to…
Will the page-to-page response
time in production always be
less than 2 seconds?
Observability
Unknown-unknowns
Ability to answer questions that
we didn’t know we would have to
ask…
Show me all UK users running
Chrome 80.0.3987.149 on
Windows 10 Pro build 18363 that
uploaded files on January 3rd
COPYRIGHT FORTE GROUP

24. @USI_LeeBarnes
Key Takeaways
• Traditional end-of-cycle performance tests don’t jive well with modern development
approaches
• Continuous performance testing is evaluating performance as appropriate at every
stage of your pipeline
• Continuous performance testing should be part of the way your organization delivers
software
• Start small, get feedback and continuously improve!
24 COPYRIGHT FORTE GROUP

25. @USI_LeeBarnes
Continuing the Conversation
25
Lee Barnes
Chief Quality Officer
Forte Group
Email: lee.barnes@fortegrp.com
Twitter: @USI_LeeBarnes
LinkedIn: linkedin.com/in/leebarnes
Blog: utopiasolutions.com/blog
COPYRIGHT FORTE GROUP