http://aadays.pl/speakers/mykola-gurov/

2. Testing on
Production
deep back-end edition
Mykola V. Gurov

3. !"
T

4. Plan
• Context
• Shadow I II III
• Other techniques
• Performance and resilience
• Reflection
• Conclusion

5. bol.com
webshop & platform
ecommerce

6. source: https://www.emerce.nl/nieuws/bolcom-
twee-miljard-euro-omzet-2018

7. 60+ autonomous teams
microservices x 100

8. One PRO

9. Deep Back-End
Logistics
Purchasing

10. #
@Ngurov

12. Shadow Transactions

13. Shadow - new algo

14. Functionally tested
• Unit and Integration testing (BDD-style)
• Knowns

15. Why not to test
on Production?
It just sits there waiting…

17. #
@Ngurov

18. Database Change

19. ID Generation

20. State

21. #@Ngurov

22. Shadow across systems
$

23. Shadow across systems
$

24. Gradually out of shadow
$

25. One toggle to rule them all

26. #@Ngurov

27. Out of the
shadow

28. Golden Order
a.k.a. “Lucky” customer

29. White-listing

30. Test a.k.a.
Sandbox accounts

31. Dry Run/Preview
Perform calculation without side-effects

32. Buddy service connectivity

33. Survival of the fittest

34. #@Ngurov

35. Performance testing
• Collect, visualize and alert on metrics
• Shut some nodes down
• Pause a queue consumer

36. Chaos
engineering
Disruption on purpose

37. #@Ngurov

38. Separate release from
deployment

39. Best used together
• Feature Toggles
• (Trunk Based Development + Continuous
Delivery)
• Microservices
• Dynamic Infrastructure

40. Testing Environments

41. Testing Environments
• Isolation (unit, integration test). Exclusive.
Controlled. Fast. Dev workstations / CI
servers. Best for functional testing.
• Here we look for knowns

42. Testing Environments
• E2E. Staging. Acceptance. Shared.

43. Testing Environments
• Data & Behavior
• Always second class -> Delays
• Test tooling

44. Testing Environments
Investment into the most valuable
environment:
Observability
Monitoring
Resilience

46. Testing Environments
NB: code delivered should be production-
ready
e.g. no security breaches or ignored errors

47. #
Conclusions

48. Testing on Production
isn't a sin but rather an
opportunity
done responsibly

49. Maintaining useful
staging environments
isn’t trivial or cheap

50. Youtube materials
• What do you know about testing in
production? https://youtu.be/z-ATZTUgaAo
Michael Bryzek QCon San Francisco 2016
• GTAC 2007: Ed Keyes - Sufficiently Advanced
Monitoring is Indistinguishable from
Testing https://www.youtube.com/watch?
v=uSo8i1N18oc

51. @Ngurov