Leveraging data visualization to improve the efficiency of large-scale test automation infrastructure. Watch the full talk at: http://youtube.com/watch?v=oRIci6n566w

1. Big Data
Makes The Flake Go Away
Leveraging data visualization to
improve the efficiency of large-scale
test automation infrastructure
Dave Cadwallader
Automation Infrastructure

2. What are we going to learn?
1. What is test flake?
2. What problems does it cause?
3. Why is it so hard to prevent?
4. What can we do to stop it?

3. Who am I
and why should you listen to me?
Dave Cadwallader
Sr. Engineering Manager
Automation Infrastructure

4. Who am I
and why should you listen to me?
60,000 minutes (41 days) of
testing per day
Dave Cadwallader
Sr. Engineering Manager
Automation Infrastructure

5. Who am I
and why should you listen to me?
Dave Cadwallader
Co-Creator of TestArmada
We don’t make the test libraries you use.
We make the test libraries you use better.

6. Takeaways You’ll Get
1. Understanding of the various types of test flake
2. How to use statistics and data visualization to help
measure your own flake levels
3. How to squash test flake once you’ve found it
4. How to get involved in a community-driven effort to
end test flake

7. What does Automation mean at
…and how does flake get in the way?

16. When Tests Fail in CI
Wow, thanks!
My app has a bug!
CI is flakey!

17. flake

18. flake
problems
caused by

19. confidence erosion
a growing mistrust of the
systems designed to keep
us safe

20. flake
why is
such a pain?

21. non-deterministic
“…given the same input,
exhibits different behaviors
on different runs”
https://en.wikipedia.org/wiki/Nondeterministic_algorithm

22. pass/fail flake

23. performance flake

24. flake
take
back

25. #hooray4flake

26. High Concurrency
will make test suites fast!
Bright Idea We Had:

31. Smoosh the Sandcastle!

34. What we Wanted

35. What we Got

40. Retry Tail
Extra rounds of
testing tacked onto
the end of a test
suite when one or
more failing tests
are retried.

41. High Concurrency
will make test suites fast!
is weakened by test flake.

42. Magellan
by
a test orchestrator
(runs your existing test library)
Massively Parallel
Fault Tolerant

54. 1% flake might not cause failures,
But it does cause perceived slowness
Every blip counts.

55. BLOOP
by

57. the sound nicknamed Bloop is the most
likely to come from some sort of animal…

58. Icequakes!

59. Anatomy of Bloop
UDP

60. https://microchip.wdfiles.com/local--files/tcpip:tcp-vs-udp/TCP_vs_UDP.JPG

61. Let’s Measure Some Stuff!

62. Flake Rate
Tests that eventually passed
after requiring one or two retries

67. Suite Runtime
Total Duration of All Tests in a Suite

72. Per-Team Operational Health

74. Individual Test Timings

77. Individual Test Timings
By Test

79. Keep on Slicing

81. Interpreting Lines is Tough…
We Need Statistics!

83. Standard Deviation

84. Standard Deviation
“…a measure of how
spread out numbers are.”
http://www.mathsisfun.com/data/standard-deviation.html

85. http://www.mathsisfun.com/data/standard-deviation.html
The heights (at the shoulders) are: 600mm, 470mm, 170mm, 430mm and 300mm.
…the mean (average) height is 394 mm. The Standard Deviation is 147 mm.

91. Individual Test Timings
By Browser

94. Individual Test Timings
By Test and Browser

98. Pass/Fail Rate
By Test and Browser

102. What have we learned?
1. When trends appear chaotic, find
another dimension to slice
2. Keep slicing until a difference is
found between slices
3. Use those differences to narrow
down root causes

103. Common Causes of Flake
1. Long-Running Tests
2. Live Network Calls
3. Non-Deterministic
Application Bugs

105. Bloop Roadmap
1. Open Source it!
2. More Stats!
3. Determine test run order
based on flake/timing history

109. @TestArmada
@geek_dave
testarmada.github.io
Join the Community
#hooray4flake