James Farrier, a software engineer with 12 years of experience in test automation, discusses improving testing efficiency through strategies like 'shift left' and the challenges of unreliable tests. He highlights the impact of flaky tests on productivity and suggests using classification algorithms to detect and quarantine them. The document emphasizes the need for prioritizing test strategies, understanding team dynamics, and refining testing efforts to reduce bugs and enhance overall effectiveness.

1. James Farrier
From - Auckland New Zealand
Studied - Software Engineering
Been working in Test Automation for 12 years
Recently founded Appsurify
About Me

2. With less time
(Thanks Agile)
we need to
improve our
testing efficiency

3. What does ‘Shift
Left’ mean?

4. Testing Manifesto

5. How our CI should
look
10 sec 10 sec 30 sec 2 min
Commit Build
Unit
Tests
API
Tests
UI Tests
10 sec
Deploy
1 day
Manual
Tests
30 sec
Release
No bugs
found
PARTY!

6. Context switching
is terrible for your
teams
productivity

7. How our CI should
look
10 sec 10 sec 30 sec 2 min
Commit Build
Unit
Tests
API
Tests
UI Tests
10 sec
Deploy
1 day
Manual
Tests
30 sec
Release
No bugs
found
PARTY!

8. To rely on
automation, we
need a reliable
signal

9. How our CI should
look
10 sec 10 sec 30 sec 2 min
Commit Build
Unit
Tests
API
Tests
UI Tests
10 sec
Deploy
1 day
Manual
Tests
30 sec
Release
No bugs
found
PARTY!

10. What really
happens - Dev 1
9 am
1 min 1 min After 10 min
1000 tests
fail, WTF?!
10 min
Commit Build
Unit
Tests
API
Tests
Investig
ation
Let’s rerun!
Fix
10:30 am
Ready

11. What really
happens - Dev 2
10 am
1 min 1 min After 10 min
1000 tests
fail, WTF?!
10 min
Commit Build
Unit
Tests
API
Tests
Investig
ation
10:30 am
Meeting
11:30 am
Queue Lunch

12. What really
happens - Dev 1
10:30 am
1 min 15 min 1 min 30 min all
pass after
reruns
Commit Build Coffee
Unit
Tests
API
Tests
2 hours, 100
failures
UI Tests

13. What really
happens - Dev 1
11 am

14. What really
happens - Dev 1
1 pm
1 min 15 min 1 min 30 min all
pass after
reruns
Commit Build Coffee
Unit
Tests
API
Tests
2 hours, 100
failures
UI Tests
40 min
Investig
ation
Create
Fix

15. What really
happens - Dev 2
1 pm
1 min 15 min 1 min 30 min all
pass after
reruns
Commit Build Coffee
Unit
Tests
API
Tests
2 hours, 100
failures
UI Tests
30 min
Investig
ation

16. What really
happens - Dev 2
3:30 pm

17. What really
happens - Dev 2
3:30 pm
1 min 15 min 1 min 30 min all
pass after
reruns
Commit Build Coffee
Unit
Tests
API
Tests
2 hours, 100
failures
UI Tests
30 min
Investig
ation
At 4pm
Re-
queue
Dev 1’s
change
labeled
critical
Build
cancelled
Go
Home

18. What really
happens - Dev 1
4:30 pm
1 min 15 min 1 min 30 min all
pass after
reruns
Commit Build Coffee
Unit
Tests
API
Tests
Removed
UI Tests
1 hour
Deploy
3 hours.
Testers leave at
9 pm
Manual
Tests
Bug
Missed
Release
2 am
Bug
Found

19. Developers hate
flaky and slow
CI/build pipelines

20. Developers hate
flaky and slow
CI/build pipelines
Dev 1 Dev 2
● 70 min waiting on API
tests
● 10 min investigating
API tests
● 2 hours waiting on UI
tests
● 40 min investigating
UI tests
● Loses 2+ hours
context switching
● 1 hour trying to
understand new code
● Productivity < 2 hours
● 10 min investigating
same API failures as
Dev 1
● 30 min investigating
same UI failures as
Dev 1
● 2 hours waiting on UI
tests
● 2 hours waiting on
Dev 1’s changes
● Productivity < 2 hours

21. Shifting Left Challenges
Tests Are
Unreliable
Not
Enough
Time To
Test
Tests Take
Too Long
To Run

22. Elite Companies Test Automation
1 Written in Code
2 Shared
Responsibility
3Visible Results
4Gated CI/CD

23. What can we do
about long
running test suites

24. Run less tests?
But then we might
miss something!

25. What about code
coverage? That’s
difficult to setup
and still runs too
many tests!

26. The best
companies run a
prioritized set of
tests

27. To rely on
automation, we
need a reliable
signal

28. 74% of test
failures at Google
are caused by
Flaky Tests

29. Almost 16% of our tests have some level of flakiness
associated with them!
John Micco, Google
“

30. Manually remove
flaky tests from
test runs

31. Delete flaky tests

32. Fix flaky tests

33. Manually dealing
with flaky tests
Advantages Disadvantages
● Stops flaky tests
breaking the build
● Misses defects
● Eventually recreate
the same tests
● Disheartening
● May not be possible
(3rd party)
● Doesn’t scale

34. Rerun flaky tests,
great in theory -
can be bad in
practice

35. Using a
classification
algorithm to
detect flaky tests
and then
quarantine the
unreliable results

36. Which is likely to
be Flaky? Failure
early in the test or
at late?

37. Some of the most
common failures
with Selenium
occur early i.e.
Browser setup

38. Which is likely to
be Flaky? Error or
Assert Failure?

39. Which is likely to
be Flaky? If we
change Transfers
section a
Transfers test that
fails or a Payment
test that fails?

40. Which is likely to
be Flaky? New
Error or an error
that’s caused
flaky failures
before?

41. Which is likely to
be Flaky? A test
that has never
flaked before or a
test that looks like
Xmas lights?

42. Now we have
useful automation

43. Manual Testing Challenges
Unsure
What To
Test
Use Your
Gut To
Decide
Not
Enough
Time

44. Identify risky
commits and the
areas that are
likely to have
defects

45. Analyze the
commit and
defect history

46. Who is more likely
to create a bug?
Someone who
created bugs in
the past or
someone who
hasn’t created a
bug before?

47. Who is more likely
to create a bug?
Someone
experienced with
this area or
someone new to
this area of the
code?

48. Who is more likely
to create a bug?
Someone
experienced with
this type of task or
someone new to
this task type?

49. Where is more
likely to have a
bug? An area
which 10 devs
have changed or
one?

50. Where is more
likely to have a
bug? An area
which 10 devs
have changed or
one?

51. Which is more
likely to have a
bug? A small
commit or a
really big
commit?

52. Which is more
likely to have a
bug? A Java file
change or a C++
file change?

53. Which is more
likely to have a
bug? A sane
commit message
or something like
this?

54. Who is more likely
to create a bug in
the morning?
Someone you
know got no
sleep or someone
well rested?

55. Now we can
reduce our
testing scope
whilst finding all
the bugs

56. Testing Manifesto

57. Why are we
creating so many
bugs?

58. Assign tasks
based on who is
available

59. Not everyone has
the same skill set

60. jamesfarrier@appsurify.com
Thanks for Listening