The document discusses the issue of false positives in testing automation, detailing causes such as flaky and outdated tests. It outlines strategies to minimize these false positives, including the implementation of annotations like @ignoreonbug and @betatest for handling test cases effectively. The document emphasizes the importance of adapting tests to new behaviors to prevent unnecessary failures during the refactoring process.

1. 1
Wix Automation
The False Positive Paradox
Efrat Attas

2. 2
What?!
What is “False Positive”?

3. 3
What?!
Release cycle
• Daily /Weekly deployments of multiple
products
• QA verifies each version using manual and
automation tests (accompanied by
automation engineers)

4. 4
What?!
Test failure  not necessarily a bug
False positives – What is it?

5. 5
What?! Understand!
False positives – What are the consequences?
• Time consuming (in case of many failures)
• Bottleneck

6. 6
Why?!
Why is it happening?

7. 7
Why?
• Unstable test – “flaky”
• Outdated test
False positives – Why is it happening?

8. 8
Why? Invent!
Reduce false positives to a minimum
• Identify
• Isolate
• Analyze
• Fix the flakiness

9. 9
How?!
How to reduce outdated tests?

10. 10
How?!
Bonus! Can also be used for “real” issues in product – true positives
Solution
• Ignore On Bug annotation
Outdated test
• Ignore test on bug with a link to Jira issue
Motivation
• Outdated test must be adapted to new
behavior without causing unnecessary failures
during the refactor process

11. 11
Solution
@IgnoreOnBug(issue = ”SE-5541”)
@Test
public void verifyButtonText() { … }
@IgnoreOnBug

12. 12
Solution
@IgnoreOnBug
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface IgnoreOnBug {
String issue();
}

13. 13
Solution
@IgnoreOnBug
public interface TestObserver {
default void starting(Description description) {
}
default void before(Description description) {
}
default void ending() {
}
default void skipped(AssumptionViolatedException e, Description description) {
}
default void succeeded(Description description) {
}
default void failed(Throwable e, Description description) {
}
default void finished(Description description) {
}
}

14. 14
Solution
@IgnoreOnBug
@Override
public void before(Description description) {
if(hasAnnotation(description, IgnoreOnBug.class)) {
String issue = getJiraIssueFromMethodAnnotation(description);
if(!isJiraIssueSolved(issue)) {
ignoreTest();
}
}
}

15. 15
How?!
How to reduce unstable tests?

16. 16
How?!
Unstable test
*What is stable enough?
• New tests are executed independently, without
affecting existing test results
Motivation
• New test has to prove it is stable enough*
before entering the entire test collection
Solution
• Beta test mechanism

17. 17
Solution
@BetaTest
• Beta tests will run in a specific configuration, X times every night
• Each test will report its run result to a dedicated server
• Beta tests which passed more thanY times will be “approved”

18. 18
Solution
@BetaTest
@BetaTest(owner = "someone")
public void verifyButtonText() { … }

19. 19
Solution
@BetaTest
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface BetaTest {
String owner();
}

20. 20
Solution
@BetaTest
Collection<Class<? extends Annotation>> testMethodAnnotations =
Arrays.asList(Test.class, BetaTest.class);
@Override
protected List<FrameworkMethod> computeTestMethods() {
return getAnnotatedMethods(getTestClass(), testMethodAnnotations);
}

21. 21
Solution
@BetaTest
public interface TestObserver {
default void starting(Description description) {
}
default void before(Description description) {
}
default void ending() {
}
default void skipped(AssumptionViolatedException e, Description description) {
}
default void succeeded(Description description) {
}
default void failed(Throwable e, Description description) {
}
default void finished(Description description) {
}
}

22. 22
Solution
@BetaTest
@Override
public void before(Description description) {
if (shouldIgnoreBetaTest(description)) {
ignoreTest("Test in beta mode - ignored");
}
}

23. 23
Solution
@BetaTest
@Override
public void succeeded(Description description) {
sendStatus(description, TestStatus.TEST_PASSED);
}
@Override
public void failed(Throwable e, Description description) {
sendStatus(description, TestStatus.TEST_FAILED);
}

24. 24
Solution
@BetaTest

25. 25
Thank you!
Questions?