This document summarizes a talk by Evgeny Rahman about testing in production. The talk discusses what is meant by "production" and different levels of being in production. It also covers what testing is, different types of testing like unit and integration testing, challenges of environments and data between dev and prod, techniques for testing in production like canary releases and A/B testing, dealing with chaos in production through fault injection and observability, and managing risks to security, compliance and customers. The talk advocates for testing techniques that can increase confidence in a system's resilience without replacing other types of testing.
1. Testing in Production - Evgeny Rahman
Testing in Production
A talk about testing, production, what these terms mean, and a look at what people do in order to
keep their businesses running without totally giving up on the former or completely breaking the
latter.
More information about this talk may be available in the server log.
Evgeny Rahman
2. Testing in Production - Evgeny Rahman
What is “production?”
• An app on a server
• A spreadsheet on a desktop
“The one that matters”
3. Testing in Production - Evgeny Rahman
How “in production” are you?
• UAT
• Alpha/Beta
• First Release
• Internal/External
• Rollout
• Unknown
4. Testing in Production - Evgeny Rahman
How “in production” are you?
Deployment != Release
5. Testing in Production - Evgeny Rahman
Testing
• What is testing?
• Why do we test?
• Who does the testing?
• When do we test?
• Where do we test?
6. Testing in Production - Evgeny Rahman
The “Real” Testing Pyramid
The Real World
User Acceptance
End to End
Integration
Unit
7. Testing in Production - Evgeny Rahman
Environments & Data
• Classic
• Dev/Test/Perf-UAT/Prod
• Parity
• Drift
• Maintenance
• *SECRET* The best test data is in Production *SECRET*
9. Testing in Production - Evgeny Rahman
Testing in Production!
“ Program testing can be used to show the presence of bugs,
but never to show their absence! “
– Edsger W Dijkstra
20. Testing in Production - Evgeny Rahman
Danger
• Security
• Compliance
• Observability
• Unintended consequences
• Think of the customers!
• Doesn’t replace all other testing
21. Testing in Production - Evgeny Rahman
Danger
“While any increase in confidence in the system’s resiliency is
positive, it’s still just that: an increase, not a completion of
perfect confidence. Any complex system can (and will) fail in
surprising ways.”
- John Allspaw, Etsy
22. Testing in Production - Evgeny Rahman
Resources
• https://medium.com/netflix-techblog/fit-failure-injection-testing-
35d8e2a9bb2
• https://medium.com/@copyconstruct/testing-in-production-the-safe-
way-18ca102d0ef1
• https://opensource.com/article/17/8/testing-production
• https://martinfowler.com/articles/practical-test-pyramid.html
• https://featureflags.io/canary-testing/
23. TESTING_IN_PRODUCTION - Evgeny Rahman
This is the Q&A Section
Please:
• Ask me questions
• Comments are also welcome
• My LinkedIn profile is at
https://www.linkedin.com/in/evgenyrahman/
I'm Evgeny, I work on the MVP team. I work on MVPs.
Alpha/Beta
Alpha
First testable version. Prone to being buggy.
Beta
More stable than alpha. Some software perpetually in Beta (Gmail)
Unknown
Low traffic site
"Brochureware" site
Next slide - Deployment != Release
Alpha/Beta
Alpha
First testable version. Prone to being buggy.
Beta
More stable than alpha. Some software perpetually in Beta (Gmail)
Unknown
Low traffic site
"Brochureware" site
Deployment != Release
Measuring against an expected outcome
Do you do anything different if a test fails?
Should you integrate with a dev environment or with a production environment?
Licensing/logistical issues for data and integrations in lower environments
Unpredictability of behavior and combination of variables in Production is tough to consistently replicate in lower environments
User data copied from Production needs to be scrubbed and things like emails get tricky
Is your application or service designed with supportability in mind?
Does it know what environment it is in?
Is it easy to tell what is going on?
“Cell Phone Plan” support - unlimited nights and weekends
Don’t be this guy
Real User Testing
See how users are doing with your software
Don’t “track” users if you don’t need to – this is about your software, not private individuals
The user might not be on a fancy machine with a high speed connection
The most important slide of all
Who cares if your animations work if your bank account is empty
A little release to see how it does
Microsoft
“Drinking our own champagne”
Be your own customer
Use for
UAT
Usability
Production Support
Anything!
Careful
Test your hypothesis
Collect real data
There is a whole suite of tools called the Simian Army
Why let hackers have all the fun?