Presentation briefly summarizes typical performance tests and their applicability in software development life-cycle

1. Lightweight performance testing
and benchmarking reference
model
Intrum Global Technologies
Eduards Salnikovs

2. Typical performance engineering responsibility domains
1. UI, API, Application – application development and
maintenance organization
2. Application server – vendor for COTS, application
maintenance and operations organization for service
3. Database – vendor for COTS, maintenance and operations
organization for service
4. OS, Virtualization, Network, IO, HW – vendor for COTS,
infrastructure operations or cloud provider for service
2
Database
Application server
UI
OS and Virtualization
NW IO HW
API, Application w
o
r
k
l
o
a
d

3. Workload types
Each workload might endure two types of tests:
Sunny day tests – top speeds, non disruptive state of environment
Cloudy day tests – with workload variance, contentions, background processes
3
ActualArtificial Workload

4. Applicability and purpose of tests
4
Project
planning
and sizing
Project
execution and
development
Production
operations and
capacity forecasting
Observational tests
Stack technical audits
Experimentation tests
Observational tests
Performance
benchmarking and
pre-post release
performance
non-regression
testing
Performance
characterization of
prototype HW/SW
stack
Performance
analisys,
configuration
optimization

5. Test type by workload (1)
5
Type of test Micro-benchmark
Objectives of test
Validate ad hoc results and on a high level HW/SW stack, performance endurance in as short period of time as possible and
with minimum resources
Test inputs &
requirements
 Test DoD defined
 Approved timeframes and specialst availability reservations by project or service manager
Time Frame Few days to several weeks to setup, run and report on results
Advantages
 Small effort required from project owner regarding test objective definition
 Narrower number of components and stack layers tested
 Simple execution and easily repeatable
 Quick to start-up and execute
Disadvantages
 Artificial workload not resembling production workload profile
 Rough results and findings with implicit and explicit assumptions

6. Test type by workload (2)
6
Type of test Simulation or macro-benchmark
Objectives of test Validate hardware/software stack against non-functional requirements with as equivalent as possible workload to production
Test inputs &
requirements
 Test DoD defined
 Workload simulation profile defined
 Production equivalent HW/SW stack environment
 Approved timeframes and specialst availability reservations by project or service manager
Time Frame Weeks to months to define workload, run tests and report on results
Advantages
 Simulated workload as close as possible to production behaviour
 Less effort for setup than production replay scenario
Disadvantages
 More effort required than micro-benchmark to setup and run
 Significant effort required in workload definition phase and test data preparation
 Production equivalent HW/SW stack environment setup and runnning costs

7. Test type by workload (3)
7
Type of test Production workload replay
Objectives of test Validate hardware/software stack against non-functional requirements with production workload
Test inputs &
requirements
 Test DoD defined
 Workload simulation profile defined
 Production equivalent HW/SW stack environment
 Approved timeframes and specialst availability reservations by project or service manager
Time Frame One or more calendar months to define workload, run tests and report on results
Advantages  Production equivalent environment and result precision
Disadvantages
 Heavy effort required in workload definition phase and test data preparation
 Production equivalent HW required
 Production workload capture functionality development and/or capture tool adaptation effort

8. Test type by workload (4)
8
Type of test Production workload or technical audit
Objectives of test Validate existing production workload trends, forecast capacity and identify important observation/monitoring values
Test inputs &
requirements
 Scope of observation defined and approved by project or service manager
 Diagnostics tooling access to production approved
 Approved timeframes and specialst availability reservations by project or service manager
 Approved by system owner impact and changes in production configuration (if neccesarry)
Time Frame Days up to weeks to setup, run and report on results
Advantages  Result precision based on production environment measurements
Disadvantages
 Production operational risks during test exection
 Production performance degradation during tests

9. 9
Anti-Methods or Anti-Patterns
Blame-Someone-Else Anti-Method
1. Find a system or environment component
you are not responsible for
2. Hypothesize that the issue is with that
component
3. Redirect the issue to the responsible team
4. When proven wrong, go to 1
Streetlight Anti-Method
1. Pick observability tools that are:
o familiar
o found on the Internet
o found at random
2. Run tools
3. Look for obvious issues
Drunk Man Anti-Method
• Change things at random until the problem
goes away
Random Change Anti-Method
1. Measure a performance baseline
2. Pick a random attribute to change (eg, a tunable)
3. Change it in one direction
4. Measure performance
5. Change it in the other direction
6. Measure performance
7. Were the step 4 or 6 results better than the
baseline? If so, keep the change; of not, revert
8. Goto step 1
Passive Benchmarking Anti-Method
1. Pick a benchmark tool
2. Run it with a variety of options
3. Make a slide deck of the results
4. Hand the slides to management
Traffic Light Anti-Method
1. Open dashboard
2. All green? Assume everything is good.
3. Something red? Assume that's a problem.

10. Suggested test summary report
Minimum item set
• Sequence of functional and operation attributes executed (files, APIs, features, ..)
• Duration of the test
• Resource utilization measurements during test
• Average and peak load description i.e. TPS@ResponseTime graph
• Error rate and error types during execution
• Operational workload success/failure rate
• What was the limiting factor for the benchmark
• Summary and recommendations for further steps
• Appendix: Tech stack details incl. configuration and binary build version(s)

11. 11
Additional reading
• Methodology in details - http://www.brendangregg.com/methodology.html
• Methodology ppt deck - https://www.slideshare.net/brendangregg/linux-performance-analysis-and-tools
• Measuring Transaction Processing Power - https://www.hpl.hp.com/techreports/tandem/TR-85.2.pdf
• Byzantine general problem - https://en.wikipedia.org/wiki/Byzantine_fault
• Byzantine vs Transaction commit problem - https://hplabs.itcs.hp.com/techreports/tandem/TR-88.6.pdf