The document discusses the prevalence of flaky tests in software testing, noting that around 16% of tests exhibit flakiness and re-running tests is a common but ineffective solution. It emphasizes that flakiness should be seen as an opportunity for improvement through better test design and disciplined practices. The author advocates for the codification of best practices into automated feedback loops to enhance the effectiveness of automated testing.

1. The
Death of Flaky Tests
By Dave Haeffer,
@TourDeDave

3. http://bit.ly/saucecon18-1
~1.5% of runs report a flaky result
~16% of tests have some level of flakiness
Re-runs cited as a common mitigation strategy

4. http://bit.ly/saucecon18-2

5. “a re-run culture is toxic”
“Flaky” tests that caught a bug that was missed
Your application is flaky

6. http://bit.ly/saucecon18-3

7. “take flake back” - #hooray4flake
Tracked stats for individual test runs using standard deviation
At a glance way to spot tests which require attention
http://testarmada.io/

8. http://bit.ly/saucecon18-4

9. You can avoid flaky tests with good test design practices.

10. http://bit.ly/saucecon18-5

11. Flakiness is something we design into our tests.
Flakiness is an opportunity to improve, don’t ignore it.
Acronym and questions.

12. http://bit.ly/saucecon18-6
If you do not treat test automation well,
why are you expecting miracles?

13. http://bit.ly/saucecon18-7

14. http://bit.ly/saucecon18-8
http://bit.ly/saucecon18-7

15. http://bit.ly/saucecon18-9
http://bit.ly/saucecon18-8
http://bit.ly/saucecon18-7

16. http://bit.ly/saucecon18-10
http://bit.ly/saucecon18-9
http://bit.ly/saucecon18-8
http://bit.ly/saucecon18-7

17. 100
0 100
Code Score (%)
RunResult(Pass%)
Individual Test Matrix
Reliable
External Issues
Likely To Fail
Broken

18. http://bit.ly/saucecon18-11

19. Available tooling doesn’t cater to people with existing
Selenium code.
Best practices are all around us, but we’re not all using them.
Let’s codify these practices into automated feedback loops to
make the path to successful automated testing simpler for
everyone.

20. Image sources:
https://thenounproject.com/icon/1532282/
https://thenounproject.com/icon/947849/
Created by dDara
from the Noun Project
Created by Adrien Coquet
from the Noun Project
Test Code In Flight Test Code At Rest

21. NOTE: You Need Test Run Data Too
Created by dDara
from the Noun Project
Created by Adrien Coquet
from the Noun Project
Abstract Event Listener
Tapping The Wire Protocol
Log Parsing
Static Code Analysis

22. 100
0 100
Code Score (%)
RunResult(Pass%)
Individual Test Matrix
Reliable
External Issues
Likely To Fail
Broken

23. Abstract Event Listener

24. Abstract Event Listener

25. Abstract Event Listener

26. Abstract Event Listener

27. Abstract Event Listener
C# - http://bit.ly/saucecon2018-12
Java - http://bit.ly/saucecon2018-13
Python- http://bit.ly/saucecon2018-14
Ruby - http://bit.ly/saucecon2018-15
JS - http://bit.ly/saucecon2018-16

28. Static code analysis
a.k.a. “linting”
“A linter or lint refers to tools that analyze source
code to flag programming errors, bugs, stylistic
errors, and suspicious constructs. The term is
originated from a Unix utility that examined C
language source code.”
https://en.wikipedia.org/wiki/Lint_(software)

29. Static code analysis
a.k.a. “linting”
“A linter or lint refers to tools that analyze source
code to flag programming errors, bugs, stylistic
errors, and suspicious constructs. The term is
originated from a Unix utility that examined C
language source code.”
https://en.wikipedia.org/wiki/Lint_(software)

30. Outro
●
Zoom if (just a little bit)
●
Remove “faayy from our vocabulary
●
Let’s truly treat test code liae app code by
oferif equivaleft toolif
http://davehaeffner.com/contact/