https://www.meetup.com/DomCode/events/266491910/

1. Testing on
Production
deep back-end edition
Mykola V. Gurov

2. bol.com
webshop & platform
ecommerce

3. 80+ autonomous teams
microservices x 100s

4. Deep Back-End
Logistics
Purchasing

5. Testing Environments
• The place where the value is generated. The
rest is an overhead.

6. Testing Environments
• Isolation (unit, integration test)
• Exclusive. Controlled. Fast.
• Dev workstations / CI servers.
• Best for functional testing.

7. Testing Environments
• E2E. Staging. Acceptance. Shared.
• Integrated testing

8. 💧
@Ngurov

9. https://www.bol.com/nl/p/jap-sounds-ap07m-draadloze-oordopjes-wit/9200000117994164/

10. Shadow Transactions

11. Shadow - new algo

12. Functionally tested
• Unit and Integration testing (BDD-style)
• Knowns

13. Why not to test
on Production?
It just sits there waiting…

16. –Kaner, Cem (November 17, 2006). Exploratory Testing (PDF).
Quality Assurance Institute Worldwide Annual Software Testing
Conference. Orlando, FL. Retrieved November 22, 2014.
via https://en.wikipedia.org/wiki/Software_testing
“Software testing is an investigation
conducted to provide stakeholders with
information about the quality of the
software product or service under test."
What is testing?

18. Separate release from
deployment

19. 💧
@Ngurov

20. Database Change

21. Shadow across systems
🚚

22. Shadow across systems
🚚

23. Gradually out of shadow
🚚

24. 💧
@Ngurov

25. Out of the
shadow

26. Gradually out of shadow
🚚

27. Golden Order
a.k.a. “Lucky” customer

28. White-listing

29. 💧@Ngurov Fast Forward >>

31. Dry Run/Preview
Perform calculation without side-effects

33. Survival of the fittest

34. 💧@Ngurov

35. Performance testing
• Collect, visualize and alert on metrics
• Shut some nodes down
• Pause a queue consumer

36. Chaos
engineering
Disruption on purpose

37. Best used together
• Feature Toggles
• (Continuous Delivery + Trunk Based
Development)
• Microservices
• Team Autonomy

38. 💧@Ngurov

39. Testing Environments
The code merged to master is expected to
be 100% production-ready
"knowns"

40. Testing Environments
• Data & Behavior
• Always second class -> Delays
• Test tooling

41. Testing Environments
Investment into the most valuable
environment:
Resilience
Monitoring
Observability

43. 💧
Conclusions

44. Testing on Production is
not a sin
but rather an opportunity
done responsibly

45. Maintaining useful
staging environments
isn’t trivial or cheap

46. Youtube materials
• What do you know about testing in
production? https://youtu.be/z-ATZTUgaAo
Michael Bryzek QCon New York 2017
• GTAC 2007: Ed Keyes - Sufficiently Advanced
Monitoring is Indistinguishable from
Testing https://www.youtube.com/watch?
v=uSo8i1N18oc

47. @Ngurov
https://mgurov.github.io/talks/testing-on-production/