Download to read offline
Uploaded byArtjoker
QA Evening Максим Колотилкин - Test State Pattern
Wix has 20 software engineers across 4 cities testing 3 different projects. They developed infrastructure for writing, executing, analyzing and managing Selenium-based tests. To address unstable tests, they implemented a test-state pattern where each test begins in a "Beta" state, is run multiple times, and potentially transitions to "Stable", "Flaky", or "Failed" states based on pass/fail criteria over time. This provides a way to continuously monitor test stability without separate annotation groups.
More Related Content
Similar to QA Evening Максим Колотилкин - Test State Pattern
QA Evening Максим Колотилкин - Test State Pattern
- 1. Stable automation testing Test-statepattern maximk@wix.com
- 2. Wix automation ▪ 20software engineers in 4 different cities ▪ 3 teams, each responsible for different projects Lithuania Ukraine Vilnius Kyiv Dnipro Wix Engineering Locations Israel Tel-Aviv Be’er Sheva
- 3. Wix automation ▪ 20software engineers in 4 different cities ▪ 3 teams, each responsible for different projects ▪ Infrastructure for writing, executing, analyzing and managing Selenium-based tests ▪ Internal services and tools for the R&D group Lithuania Ukraine Vilnius Kyiv Dnipro Wix Engineering Locations Israel Tel-Aviv Be’er Sheva
- 5.
- 6. #1 The problem- more breaking points #2 The solution #3 A better solution E2E tests
- 7. E2E tests Writing astable test is difficult and writing a new stable test is even more fragile.
- 8. E2E tests Writing astable test is difficult and writing a new test is even more fragile. Why?
- 9. E2E tests Writing astable test is difficult and writing a new test is even more fragile. They can pass locally, but fail remotely.
- 10. We try toavoid... noise (analysis becomes difficult)
- 11. and that couldlead to unstable tests running in production Release @Test @Test @Test @Test
- 12. #1 The problem- more breaking points #2 The solution - @BetaTest #3 A better solution E2E tests
- 13. Split the testsinto 2 “run” groups Release No Beta tests = less coverage @Test @BetaTest @Test @BetaTestBeta
- 14. @BetaTest (owner ="tzipi") public void newTest() { … }
- 15. Beta run groupreports results to server Results @Test @BetaTestBeta Release
- 16. Run groups indifferent configurations @BetaTest Results Beta Release @Test
- 17. @Override public void beforeTest(Descriptiondescription) { if (isBetaTestInReleaseRunMode(description) || isNonBetaTestInBetaRunMode(description)) { TestUtils.ignoreTest(); } }
- 18. @Test Results “Approve” stable betatests @BetaTestBeta Release
- 19. How do wefind the stable ones?
- 20. How do wefind the stable ones? Tests that pass X times in a row, with maximum Y failures Results
- 21. So, for eachstable test notify the owner to replace @BetaTest with @Test.
- 22. SERIOUSLY ??? So, for eachstable test notify the owner to replace @BetaTest with @Test.
- 23. #1 The problem- more breaking points #2 The solution - @BetaTest #3 A better solution - test state pattern E2E tests
- 24. No need fora special @BetaTest annotation @Test
- 25. Each test beginswith a beta state Beta
- 26. remains in betastate for some time Beta
- 27. and (hopefully) changesto stable Beta Stable pass X times
- 28. @Override public void beforeTest(Descriptiondescription) { TestState state = getTestState(description); if (isBetaTestInReleaseRunMode(description, state) || isNonBetaTestInBetaRunMode(description, state)) { TestUtils.ignoreTest(); } }
- 29. what happens ifa test keeps failing? fail X times Beta
- 30. hall of shame- you manually fix those fail X times Beta Failed Beta manual fix
- 31. what happens ifa stable test starts to “act weird”? Beta Stable pass X times
- 32. we call itflaky Beta pass X times Become unstable Stable Flaky pass X times
- 33. what if it’sreally stable? Beta Stable pass X times
- 34. we call thatsuper stable Beta pass X times always passpass X times Stable Super Stable
- 35. full state pattern Beta passX times fail X times manual fix Failed Beta Stable Become unstable pass X times Flaky always pass Super Stable
- 36.
- 37.
- 38. Implementing it ina project Test Observer - testStarted - testIgnored - testFailed - testPassed Java Client API Core Web Web View
- 39. Object Modeling Test - Name* -State** - Creation date - Update date - Annotations Run Result - Test Name - Passed/Failed - Creation date - Configuration*** 1 n
- 40.