Testing in-production
•Download as PPTX, PDF•
0 likes•72 views
This document summarizes a talk by Evgeny Rahman about testing in production. The talk discusses what is meant by "production" and different levels of being in production. It also covers what testing is, different types of testing like unit and integration testing, challenges of environments and data between dev and prod, techniques for testing in production like canary releases and A/B testing, dealing with chaos in production through fault injection and observability, and managing risks to security, compliance and customers. The talk advocates for testing techniques that can increase confidence in a system's resilience without replacing other types of testing.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Testing in-production
Similar to Testing in-production (20)
Recently uploaded
Recently uploaded (20)
Testing in-production
- 1. Testing in Production - Evgeny Rahman Testing in Production A talk about testing, production, what these terms mean, and a look at what people do in order to keep their businesses running without totally giving up on the former or completely breaking the latter. More information about this talk may be available in the server log. Evgeny Rahman
- 2. Testing in Production - Evgeny Rahman What is “production?” • An app on a server • A spreadsheet on a desktop “The one that matters”
- 3. Testing in Production - Evgeny Rahman How “in production” are you? • UAT • Alpha/Beta • First Release • Internal/External • Rollout • Unknown
- 4. Testing in Production - Evgeny Rahman How “in production” are you? Deployment != Release
- 5. Testing in Production - Evgeny Rahman Testing • What is testing? • Why do we test? • Who does the testing? • When do we test? • Where do we test?
- 6. Testing in Production - Evgeny Rahman The “Real” Testing Pyramid The Real World User Acceptance End to End Integration Unit
- 7. Testing in Production - Evgeny Rahman Environments & Data • Classic • Dev/Test/Perf-UAT/Prod • Parity • Drift • Maintenance • *SECRET* The best test data is in Production *SECRET*
- 8. Testing in Production - Evgeny Rahman
- 9. Testing in Production - Evgeny Rahman Testing in Production! “ Program testing can be used to show the presence of bugs, but never to show their absence! “ – Edsger W Dijkstra
- 10. Testing in Production - Evgeny Rahman
- 11. Testing in Production - Evgeny Rahman Real User Monitoring
- 12. Testing in Production - Evgeny Rahman What does the User see? Expectations Reality
- 13. Testing in Production - Evgeny Rahman Money • Do you accept payments? • Can you accept payments? • Are orders making it through your system?
- 14. Testing in Production - Evgeny Rahman Canary Releases Feature Flags Phased Rollouts
- 15. Testing in Production - Evgeny Rahman Dogfooding
- 16. Testing in Production - Evgeny Rahman A/B Testing
- 17. Testing in Production - Evgeny Rahman Chaos • People and computers out on the Internet • Users don't follow scripts • Random failure • Malice
- 18. Testing in Production - Evgeny Rahman Chaos • Fault Injection Testing • Resiliency • Security
- 19. Testing in Production - Evgeny Rahman Chaos
- 20. Testing in Production - Evgeny Rahman Danger • Security • Compliance • Observability • Unintended consequences • Think of the customers! • Doesn’t replace all other testing
- 21. Testing in Production - Evgeny Rahman Danger “While any increase in confidence in the system’s resiliency is positive, it’s still just that: an increase, not a completion of perfect confidence. Any complex system can (and will) fail in surprising ways.” - John Allspaw, Etsy
- 22. Testing in Production - Evgeny Rahman Resources • https://medium.com/netflix-techblog/fit-failure-injection-testing- 35d8e2a9bb2 • https://medium.com/@copyconstruct/testing-in-production-the-safe- way-18ca102d0ef1 • https://opensource.com/article/17/8/testing-production • https://martinfowler.com/articles/practical-test-pyramid.html • https://featureflags.io/canary-testing/
- 23. TESTING_IN_PRODUCTION - Evgeny Rahman This is the Q&A Section Please: • Ask me questions • Comments are also welcome • My LinkedIn profile is at https://www.linkedin.com/in/evgenyrahman/
- 24. Testing in Production - Evgeny Rahman Thank you!
Editor's Notes
- I'm Evgeny, I work on the MVP team. I work on MVPs.
- Alpha/Beta Alpha First testable version. Prone to being buggy. Beta More stable than alpha. Some software perpetually in Beta (Gmail) Unknown Low traffic site "Brochureware" site Next slide - Deployment != Release
- Alpha/Beta Alpha First testable version. Prone to being buggy. Beta More stable than alpha. Some software perpetually in Beta (Gmail) Unknown Low traffic site "Brochureware" site Deployment != Release
- Measuring against an expected outcome Do you do anything different if a test fails?
- Should you integrate with a dev environment or with a production environment? Licensing/logistical issues for data and integrations in lower environments Unpredictability of behavior and combination of variables in Production is tough to consistently replicate in lower environments User data copied from Production needs to be scrubbed and things like emails get tricky
- Is your application or service designed with supportability in mind? Does it know what environment it is in? Is it easy to tell what is going on? “Cell Phone Plan” support - unlimited nights and weekends
- Don’t be this guy
- Real User Testing See how users are doing with your software Don’t “track” users if you don’t need to – this is about your software, not private individuals
- The user might not be on a fancy machine with a high speed connection
- The most important slide of all Who cares if your animations work if your bank account is empty
- A little release to see how it does
- Microsoft “Drinking our own champagne” Be your own customer Use for UAT Usability Production Support Anything! Careful
- Test your hypothesis Collect real data
- There is a whole suite of tools called the Simian Army Why let hackers have all the fun?