Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

End-to-End test environments, a dead End road


Published on

With the rise of Distributed Architecture, independent DevOps teams and automated CI/CD the End-to-End test environments needs to be reconsidered.
They become flaky, shaky, untrustworthy and hard to maintain. Why are End-to-End test environments a dead End road and what are the alternatives.

Published in: Technology
  • @Davy Kenis Sorry for the late reply, I missed this one.... I don't have any experience in the mainframe area at the Rabobank. But the online banking environment that is using API's is in the end also using the mainframes. I think the general idea is that you should scope your tests. A system starts and ends somewhere, so is its responsibility. How are the systems communicating with each other? There should the scope start and end.
    Are you sure you want to  Yes  No
    Your message goes here
  • Completely agree! However i do see challenges whenever we need some testing for applications who are not API based yet (eg. Mainframe applications). How did you took care of those within rabobank?
    Are you sure you want to  Yes  No
    Your message goes here

End-to-End test environments, a dead End road

  1. 1. A dead End road 2/8/2019 Roy Braam @rbraam
  2. 2. Roy Braam @rbraam2/8/2019 End-to-End environments
  3. 3. Agenda • Definition • Problem • @ the Rabobank • Why doing it • Alternatives • Questions
  4. 4. The End-to-End test environment My definition
  5. 5. “A fully integrated environment with all systems that are part of some flow or functionality, with the single purpose of testing”
  6. 6. Aliases • Chain Environment • User Acceptance Environment • Production Acceptance Environment • Broad-stack Environment • NGA Environment • Acceptance Environment • Pre Production Environment • Enterprise-wide Integration Test Environment • Full-stack Environment • Production-like Environment
  7. 7. Full blown setup of multiple dependent systems that communicate to each other to achieve something
  8. 8. The dead end road What is the problem?
  9. 9. Problems with E2E • Rise of DevOps,CI/CD and Microservices • Test data hell
  10. 10. Test Data hell • Asking for test-data • When consuming data, test- data needs to be prepared for every run • Not 'your' test-data, others may change it • Test-data needs to be correlated • Bye-bye independent DevOps Transaction Timeline User Accounts User Transactions
  11. 11. Problems with E2E • Rise of DevOps,CI/CD and Microservices • Test data hell • Not so “production like”
  12. 12. Not so “production like” • Simultaneously testing new version • Multiple updates before deploying in production • Not deploying in the End- to-End environment Version: P Version: P Version: P Version: PVersion: P Version: P Version: P+1 Version: P+1 Version: P+1 Version: P-1 Version: P+1 Version: P-2Version: P-3Version: P-4 Version: P+2 E2E-environment
  13. 13. Problems with E2E • Rise of DevOps,CI/CD and Microservices • Test data hell • Not so “production like” • Test scope • False positives • Slow Feedback time • Reduce time-to-market • Costs
  14. 14. @ the Rabobank We love End-2-End environments
  15. 15. On-premises
  16. 16. On-premises
  17. 17. On-premises
  18. 18. On-premises
  19. 19. Dev: Hey, can you help me out. Flow x is not working on the environment 3, can you have a look if the platform is the cause Dev: Yes.... Hey, now it's working. O, wait. You are looking at the chain environment 3. Me: Sure. I don't see any problem with the platform. Can you show me the broken flow on my device? Support conversation to a team
  20. 20. Me: Yes, that is what you said, environment 3... Dev: No, I meant acceptance environment 3 Me: But that's the same, it's environment 3. Dev: No, it is an acceptance environment 3 Me: But that's the same platform environment Dev: But not to us, in acceptance environment 3 we connect to services in environment 5 Me: What !!?!?
  21. 21. Migrating to the public Cloud End-to-End Environment
  22. 22. Why? What are the reasons teams rely on End-to-End environments
  23. 23. Test data
  24. 24. “Tests with semantic correct data can only be run in a End-2-End test environment.”
  25. 25. Improve by • Pull that data out of the e2e environment • Tests in lower level of the Pyramid • Build for failure
  26. 26. Trust
  27. 27. “I want to see it work before I go to production”
  28. 28. Improve by • Move tests down the pyramid • Proof no new issues are found • For every incident, create an automated test case • Invest in tests to gain trust • Proof it will not break in production
  29. 29. Immaturity
  30. 30. “System X had a new release that broke our system, even though the contract is the same.”
  31. 31. Improve by • Stricter and clearer contracts • Consumer Driven Contract testing • Automate an 1-on-1 integration system if needed • Resilience
  32. 32. Compensation
  33. 33. “When they release a new version we first want to test our system, they often break things.”
  34. 34. • Put the responsibility where it belongs • Resilience • Run your automated expectations against their test environement Improve by
  35. 35. Requirement
  36. 36. “Our stakeholders demand that we first show it in an End-to-End environment, with that we guarantee our quality”
  37. 37. • Are stakeholders telling us how to guarantee quality? • Gain their trust • Always try to improve your process Improve by
  38. 38. Alternatives Ways to minimize usage of E2E environments
  39. 39. CDC Tests
  40. 40. CDC Tests • Consumer Driven Contract Tests • Testing pattern • 2006 (by Martin Fowler??) • Consumer offers expectations to producer
  41. 41. CDC Tests • Expectations provided as tests by consumer • Producer: Trust to not break consumers • Consumer: Guarantee to not break contracts • API design • API usage ProducerConsumer Repo
  42. 42. CDC tools • Spring Cloud Contract (SCC) • Pact • Postman/Newman
  43. 43. Reveal Unused Interfaces Well-Fittedness Costs Stability Feedback time - - - - - - + - - - + + + + + + + Maintenance Effort +- + Complexity Test Data Setup Testing Data Semantics +- - + + + E2E Tests CDC TestsMock Tests Isolation +- + +
  44. 44. Minimize Risks
  45. 45. • Separate deployment from release • Canary releases • Blue/Green deployments • Feature toggle • Test groups • Monitor • Focus on mean time to recover
  46. 46. Sources • Practical test pyramid: test-pyramid.html • 7 reasons for CDC: • Thoughtworks tech radar integration-test-environments