The art and joy of testing in production
•
0 likes•164 views
Testing in production, software development, devops, deploy, release, canary releasing, ring based releases, feature flags, staging environments
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The art and joy of testing in production
Similar to The art and joy of testing in production (20)
More from Geert van der Cruijsen
More from Geert van der Cruijsen (20)
Recently uploaded
Recently uploaded (20)
The art and joy of testing in production
- 1. THE ART & JOY OF TESTING IN PRODUCTION 2020 GEERT VAN DER CRUIJSEN @GEERTVDC
- 2. GEERT VAN DER CRUIJSEN CLOUDNATIVEARCHITECT/DEVOPSCOACH/FULLCYCLEDEVELOPER @GEERTVDC
- 3. WHERE DOES YOUR SOFTWARE DELIVER VALUE?
- 4. PRODUCTION WHERE DOES YOUR SOFTWARE DELIVER VALUE?
- 6. IS YOUR PRODUCTION ENVIRONMENT WORKING RIGHT NOW? DO YOU KNOW?
- 7. IS YOUR PRODUCTION ENVIRONMENT WORKING RIGHT NOW? DO YOU KNOW? GREEN DASHBOARD?
- 8. IS YOUR PRODUCTION ENVIRONMENT WORKING RIGHT NOW? DO YOU KNOW? NO ALERTS?
- 9. IS YOUR PRODUCTION ENVIRONMENT WORKING RIGHT NOW? DO YOU KNOW? NO ANGRY USERS?
- 10. ARE THESE YOUR USERS? YOU DON’T KNOW?
- 11. WHEN WAS THE LAST TIME YOU RELEASED TO PRODUCTION?
- 12. WHEN WAS THE LAST TIME YOU RELEASED TO PRODUCTION? HOW DID YOU CHECK IT WAS SAFE?
- 13. WHEN WAS THE LAST TIME YOU RELEASED TO PRODUCTION? HOW DID YOU CHECK IT WAS SAFE? STAGING?
- 14. THE MANY PROBLEMS OF STAGING ENVIRONMENTS
- 15. STAGING GIVES A FALSE SENSE OF TRUST
- 16. STAGING ENVIRONMENTS FALSE SENSE OF TRUST
- 17. STAGING ENVIRONMENTS FALSE SENSE OF TRUST IS STAGING EQUAL TO PRODUCTION?
- 18. STAGING ENVIRONMENTS FALSE SENSE OF TRUST IS STAGING EQUAL TO PRODUCTION? REALLY?
- 19. STAGING ENVIRONMENTS FALSE SENSE OF TRUST REAL DATA? REAL COMPLEXITY? REAL USERS? REAL LOAD? SAME INFRASTRUCTURE? REAL MONITORING?
- 20. “IN A COMPLEX LANDSCAPE YOUR APPLICATION IS NEVER FULLY UP”
- 21. FOCUS ON TRUSTING PRODUCTION FOCUS ON MTTR INSTEAD OF BEING CERTAIN ABOUT STAGING STAGING ENVIRONMENTS – FALSE SENSE OF TRUST
- 23. STAGING ENVIRONMENTS EXPENSIVE 30% UP TO OF TOTAL COSTS
- 24. STAGING ENVIRONMENTS EXPENSIVE PRODUCTION LIKE INFRASTRUCTURE SYNCING PRODUCTION DATA CREATING TEST DATA DATA ANONYMIZATIONMULTIPLE TEST ENVIRONMENTS RESETTING ENVIRONMENTS SYNC TEAMS
- 25. STAGING ENVIRONMENTS EXPENSIVE YOUR TIME IS A ZERO SUM GAME
- 27. STAGING ENVIRONMENTS NATURAL HANDOVER POINT YOU BUILD IT YOU RUN IT WERNER VOGELS CTO AMAZON
- 28. STAGING IS A BOTTLENECK IN THE RELEASE PROCESS
- 29. STAGING ENVIRONMENTS BOTTLENECK DON’T YOU LOVE CONTEXT SWITCHES?
- 30. NO MORE STAGING? NOW WHAT?
- 31. NO MORE STAGING? NOW WHAT?
- 32. TESTING IN PRODUCTION WHERE TO START?
- 33. TESTING IN PRODUCTION WHERE TO START?
- 34. TESTING IN PRODUCTION WHERE TO START?
- 35. TESTING IN PRODUCTION VERIFY IN PRODUCTION YOU WANT TO DO THIS ANYWAY
- 36. TESTING IN PRODUCTION VERIFY IN PRODUCTION MEASURE USER IMPACT VERIFY BUSINESS PROCESSES
- 37. VERIFY IN PRODUCTION - EXAMPLES USER SYNC SERVICE 1 2 ALERTS
- 38. VERIFY IN PRODUCTION - EXAMPLES USER SYNC SERVICE 1 2 SLO ALERTS
- 39. VERIFY IN PRODUCTION - EXAMPLES E-COMMERCE 1
- 40. VERIFY IN PRODUCTION - EXAMPLES E-COMMERCE 1 CAN IMPACT USERS DO NOT SHIP CAN IMPACT KPIS
- 41. TESTING WE HAVE TO CHANGE OUR VIEW ON HOW WE LOOK AT
- 42. TESTING WE HAVE TO CHANGE OUR VIEW ON HOW WE LOOK AT UNIT TESTS INTEGRATION TESTS UI TESTS MANUAL TESTS
- 43. NEW VIEW ON TESTING PRE PROD TESTING TESTING IN PROD DEPLOY
- 44. NEW VIEW ON TESTING DEPLOY https://medium.com/@copyconstruct/testing-in-production-the-safe-way-18ca102d0ef1 Credits - Cindy Sridharan VERIFYING TEEING PROFILING CHAOS TESTING MONITORING A/B TESTING REAL USER MONITOR AUDITING CANARYING MONITORING TRAFFIC SHAPING RING BASED RELEASE FEATURE FLAGGING EXCEPTION TRACKING INTEGRATION TESTS TAP COMPARE LOAD TESTS SHADOW TESTS CONFIG TESTS UNIT TESTS COMPONENT TESTS FUZZ TESTS STATIC ANALYSIS REGRESSION TESTS CONTRACT TESTS LINT TESTS ACCEPTANCE TESTS MUTATION TESTS UI TESTS PENETRATION TESTS PRE-PROD TESTING IN PRODUCTION RELEASE POST RELEASE
- 45. TESTING IN PRODUCTION SPLIT DEPLOY & RELEASE DEPLOY RELEASE A NEW VERION RUNNING ON PRODUCTION INFRASTRUCTURE MOVING PRODUCTION TRAFFIC TO NEWLY DEPLOYED VERSION
- 46. DEPLOY HOW CAN WE DEPLOY WITHOUT IMPACTING USERS?
- 47. DEPLOY PHASE TESTS INTEGRATION TESTING NOT AND END TO END TEST FOCUS ON INTEGRATION BETWEEN COMPONENTS THINK SMALLER NOT BIGGER CONTRACT TESTING
- 48. DEPLOY PHASE TESTS TAP COMPARE 1 2 V1 2 V2
- 49. DEPLOY PHASE TESTS TAP COMPARE 1 2 V1 2 V2
- 50. DEPLOY PHASE TESTS TAP COMPARE TOOLS DIFFERENCIA DIFFY 2 V1 2 V2 COMPARISON
- 51. DEPLOY PHASE TESTS LOAD TESTING FOCUS ON DIFFERENCES SINCE LAST VERSION
- 52. DEPLOY PHASE TESTS CONFIG TESTING CONFIGURATION CAUSES MORE OUTAGES THAN CODE DOES
- 53. RELEASE MOVE TRAFFIC TO USERS IN A CONTROLLED WAY SO IT’S EASY TO ROLL BACK
- 54. SAFE RELEASING CANARY RELEASES SO MUCH EASIER WITH SERVICE MESHES V1 V2 99% 1%
- 55. SAFE RELEASING FOCUSSED CANARIES DON’T JUST FOCUS ON THE 1% V1 V2
- 56. SAFE RELEASING RING BASED RELEASES RELEASE 1 RING AT A TIME
- 57. SAFE RELEASING RING BASED RELEASES HOW TO DEFINE YOUR RINGS
- 58. SAFE RELEASING RING BASED RELEASES GOOD IDEA?
- 59. SAFE RELEASING RING BASED RELEASES BETTER IDEA
- 60. SAFE RELEASING FEATURE FLAGS MORE CONTROL MORE TECHNICAL DEPT
- 61. POST-RELEASE OUR APPLICATIONS IS NEVER FULLY UP, ARE USERS IMPACTED?
- 62. j MONITORING KNOWN UNKNOWNS OBSERVABILITY UNKNOWN UNKNOWNS POST RELEASE IMPACTED USERS?
- 63. j POST RELEASE FOCUS ON RESILIENCY CHAOS ENGINEERING
- 64. RECAP INVEST IN PRODUCTION OVER STAGING OUR APPLICATION WILL FAIL ON PRODUCTION 100% VERIFY ON PRODUCTION SPLIT DEPLOY & RELEASE WORK WITH RESILIENCE IN MIND ALL THIS AINT EASY!
- 65. GEERT VAN DER CRUIJSEN @GEERTVDC THANK YOU! NDC OSLO