Good performance testing is really important, but so many companies are doing it really badly. There is a false belief that the quality of performance testing can be best improved by introducing advanced techniques and practices, but most companies are struggling to get the basics right first. Is your team making any of the top 7 mistakes? Learn what you can do to avoid them, and how you can help your company improve the quality of their performance testing. This presentation was delivered in November 2017 at the first Neotys Performance Advisory Council, held at Borthwick Castle in Scotland.

1. © Copyright 2017 Stuart Moncrieff, www.myloadtest.com
The Top 7 Mistakes in
Performance Testing
…and the importance of getting the basics right first

2. 3 © Copyright 2017 Stuart Moncrieff, www.myloadtest.com
Good performance testing is really important…
…but so many people do it really badly…
…and that really bothers me.

3. © Copyright 2017 Stuart Moncrieff, www.myloadtest.com2
Stuart Moncrieff: Web Performance Evangelist
How am I qualified to be here?
• Web performance specialist since 2002
• 15 years of consulting experience
• I have seen almost all the ways that people mess up
when they are doing performance testing.

4. © Copyright 2017 Stuart Moncrieff, www.myloadtest.com4
What I think I look like when I’m talking
about Performance Testing:
What I actually look like when I’m talking
about Performance Testing:

5. © Copyright 2017 Stuart Moncrieff, www.myloadtest.com5
The “we want the advanced training” paradox
Companies usually ask to include “expert
level” content in training courses for their
staff, even when their staff don’t
understand the basics.
But their staff are all searching Google for entry-level
training material.
See also:
• The Dunning-Kruger effect

6. © Copyright 2017 Stuart Moncrieff, www.myloadtest.com6
The structural problem: Most companies can’t
differentiate between good and bad performance testing
Performance Testing treated as an empty
ritual to be performed before deployment.
Patterns of dysfunction:
• Not even noticing problems in Production because
performance monitoring is inadequate.
• Problems discovered in Production, even though they
could have been discovered in Test.
• Testing team can blame anything missed on “differences
between Test and Production environments”.
• Ops team tries to solve performance problems in
Production without involving performance testers.
Tuning in Production. Throwing hardware at the
problem.

7. © Copyright 2017 Stuart Moncrieff, www.myloadtest.com7
Mistake 1: Not adding validation checks in your scripts
Load testing tools will automatically detect
HTTP error codes, but error pages are often
served with an HTTP 200 response code.
How do you know your application is not throwing errors
under load?
What you should do:
• Add verification checks for each page request.
• Check for something that indicates success and is
unique to the page.
• Bonus points: static analysis of scripts as they are
checked into version control.

8. © Copyright 2017 Stuart Moncrieff, www.myloadtest.com8
Mistake 2: Not monitoring the Test environment
“Setting up monitoring will take too long,
so we won’t include infrastructure metrics
like CPU utilisation in our report.”
How do you diagnose the root cause of load-related
problems found during testing?
What you should do:
• Make the effort to set up infrastructure monitoring in
the Test environment.
• Bonus points: have exactly the same monitoring in Test
and Production environments.

9. © Copyright 2017 Stuart Moncrieff, www.myloadtest.com9
Mistake 3: Bad Workload Modelling (or none at all)
“We are going to generate 1000 concurrent
users worth of load”
What does that even mean? Web apps care more about
requests per minute than they do about how many virtual
users are configured in your testing tool.
What you should do:
• Use real-world usage data as an input
• Define a Peak Hour usage model that includes a
transaction rate (e.g. orders per hour).
• Include network conditions (Network Virtualization)
• Bonus points: what % of your traffic is generated by
bots?

10. © Copyright 2017 Stuart Moncrieff, www.myloadtest.com10
Mistake 4: “We only do testing” (silos)
“Our job is all about finding defects in the
Test environment.”
Performance testers can have a stake in system
architecture, capacity planning, monitoring, non-functional
requirements, contracts with vendors (SLAs), and incidents
in Production.
What you should do:
• Don’t ignore the app after go-live.
• Keep looking for defects in the Production environment
(using monitoring tools).
• Be an advocate for application performance at every
stage of the software lifecycle.

11. © Copyright 2017 Stuart Moncrieff, www.myloadtest.com11
Mistake 5: “We’re response time testers” (tunnel vision)
“Our job is to measure response times with
our load testing tool.”
Your job is to find load and performance-related
problems…preferably before they reach Production.
What you should do:
• Check for errors under load.
• Test Failover under load.
• Test system behaviour when there are interface
outages under load.
• Ensure that performance problems and system metrics
will be visible to the Ops team in Production.

12. © Copyright 2017 Stuart Moncrieff, www.myloadtest.com12
Mistake 6: Ignoring errors and other defects
“We tried to run our test last night, but the
web server crashed. It’s okay though;
they’ve restarted everything and deleted
the logs, so we can re-run the test.”
So it sounds like you actually found a few defects last night.
What you should do:
• Constantly be asking yourself “if this happened in
Production, would it be a problem?”
• If you see something, say something.
• Some of your most interesting days as a testers will
start out with “hmmm…that’s odd.”

13. © Copyright 2017 Stuart Moncrieff, www.myloadtest.com13
Mistake 7: Miscalculating error rates
If you are reporting your error rate as:
!""#" %&'( =
*&++(, -"&.+&/'0#.+
-#'&1 -"&.+&/'0#.+
…then you are doing it wrong.
What you should do:
• Report on the probability of the user completing the
entire business process, not just individual steps.
• Bonus points: Under controlled conditions, anything
above an 0% error rate indicates a problem that
should be investigated.
Transaction Passed Failed Total
Front page 99,041 959 100,000
Search catalogue 99,002 998 100,000
Browse catalogue 99,033 967 100,000
View item 98,966 1,034 100,000
Checkout 51 49 100
TOTAL 396,093 4,007 400,100

14. © Copyright 2017 Stuart Moncrieff, www.myloadtest.com14
Summary of Mistakes
What have we covered?
1. Scripts without enough verification checks
2. Not monitoring the Test environment
3. Bad Workload Modelling
4. Focusing on “performance testing”, rather than
“performance”
5. Thinking your responsibility starts and ends with
response times
6. Ignoring defects that are right in front of you
7. Miscalculating error rates

15. 15 © Copyright 2017 Stuart Moncrieff, www.myloadtest.com
Thank you for watching!
…and thank you to Neotys for hosting this event.
Contact:
• Twitter: @StuartMoncrieff
• Email: stuart@myloadtest.com
• Web: http://www.myloadtest.com
• LinkedIn: https://www.linkedin.com/in/stuartmoncrieff/