The test state pattern allows Wix to automatically detect, retry and quarantine flaky tests to minimize false positives. In this presentations we describe how Wix manages to run thousands of Selenium tests every day at a very high level of stability by utilizing the test state pattern including the architecture of the solution, code examples and implementation tips. This presentation was given by Roi Ashkenazi as part of Selenium Israel Meetup #17 which was held in Wix at 11.9.16.

1. Test State Pattern
Roi Ashkenazi
Elad Benitah
roia@wix.com
eladb@wix.com
WixAutomation

3. 01 / The Problem

4. E2E tests have more
breaking points.

5. New E2E tests are
even more fragile.

6. Pass locally, fail remotely
● Timing issues
● Remote connection issues
● Bad luck

7. We try to avoid …
False positives
Causes lack of trust
Noise
Analysis becomes difficult
Shame

8. @Test
@Test
@Test
Release
Unstable tests
running in
production

9. 02 / Solution: @BetaTest

10. @BetaTest
@BetaTest
@BetaTest
Beta
@Test
@Test
@Test
Release
Split the tests into 2 “run groups”

11. @BetaTest(owner = "tzipi")
public void newTest() {
…
}

12. @BetaTest
@BetaTest
@BetaTest
Beta
@Test
@Test
@Test
Release
Results
Beta run group reports results to server

13. @BetaTest
@BetaTest
@BetaTest
Beta
@Test
@Test
@Test
Release
Run groups in different configurations
Results

14. @Override
public void beforeTest(Description description) {
if (isBetaTestInReleaseRunMode(description) ||
isNonBetaTestInBetaRunMode(description)) {
TestUtils.ignoreTest();
}
}

15. @BetaTest
@BetaTest
@BetaTest
Beta
@Test
@Test
@Test
Release
“Approve” stable BetaTests
Results

16. How do we find
the stable ones?

17. Results
Tests that pass X times in a row,
with maximum Y failures.
How do we find
the stable ones?

18. So, for each stable test
notify the owner to replace
@BetaTest with @Test

20. 03 / We Can Do Better

21. @Test
@Test
@Test
Beta
@Test
@Test
@Test
Release
No need for special @BetaTest
annotation
Results

22. Beta
Each test begins with a Beta state

23. Beta
Remains in Beta state for some time

24. Beta
And (hopefully) changes to stable
Stable
pass X times

25. @Override
public void beforeTest(Description description) {
TestState state = getTestState(description);
if (isBetaTestInReleaseRunMode(description, state) ||
isNonBetaTestInBetaRunMode(description, state)) {
TestUtils.ignoreTest();
}
}
Run tests according to their state

26. What happens if a test keeps failing?
Beta
fail X times

27. What happens if a test keeps failing?
Beta
Failed
Beta
fail X times

28. Failed beta - manual fix
Beta
Failed
Beta
fail X times
manual fix

29. Beta
What happens if a stable test
starts to “act weird”?
Stable
pass X times

30. Beta Stable
FlakyFlak
y
Become
unstable
pass X times
pass X times
What happens if a stable test
starts to “act weird”?

31. Beta
Full pattern
Failed
Beta
fail X times
Stable
FlakyFlak
y
Become
unstable
pass X times
pass X times
manual fix

32. Stable automation is possible!
Results
@Test
@Test
@Test
Beta
@Test
@Test
@Test
Release
@Test
@Test
@Test
Flak
y
Failed Beta

33. 04 / Architecture

34. API Core Web
DB
Web View
Java Client
Test Observer
- testStarted
- testIgnored
- testFailed
- testPassed

35. Test
- Name*
- State**
- Creation date
- Update date
- Annotations
Object modeling
Run Result
- Test Name
- Passed / failed
- Creation date
- Configuration***
1 n

36. 05 / Live Demo

38. This is where you are going to present your final words.
This slide is not meant to have a lot of text.Thank You!