Learn what shift left is, why it usually doesn't work and how to solve those problems. We will focus on test automation and the struggles of having it incorporated into the continuous integration (CI) or continuous deployment (cd) strategy.
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
Shifting Testing Left - The Pain Points and Solutions
1. James Farrier
From - Auckland New Zealand
Studied - Software Engineering
Been working in Test Automation for 12 years
Recently founded Appsurify
About Me
5. How our CI should
look
10 sec 10 sec 30 sec 2 min
Commit Build
Unit
Tests
API
Tests
UI Tests
10 sec
Deploy
1 day
Manual
Tests
30 sec
Release
No bugs
found
PARTY!
7. How our CI should
look
10 sec 10 sec 30 sec 2 min
Commit Build
Unit
Tests
API
Tests
UI Tests
10 sec
Deploy
1 day
Manual
Tests
30 sec
Release
No bugs
found
PARTY!
9. How our CI should
look
10 sec 10 sec 30 sec 2 min
Commit Build
Unit
Tests
API
Tests
UI Tests
10 sec
Deploy
1 day
Manual
Tests
30 sec
Release
No bugs
found
PARTY!
10. What really
happens - Dev 1
9 am
1 min 1 min After 10 min
1000 tests
fail, WTF?!
10 min
Commit Build
Unit
Tests
API
Tests
Investig
ation
Let’s rerun!
Fix
10:30 am
Ready
11. What really
happens - Dev 2
10 am
1 min 1 min After 10 min
1000 tests
fail, WTF?!
10 min
Commit Build
Unit
Tests
API
Tests
Investig
ation
10:30 am
Meeting
11:30 am
Queue Lunch
12. What really
happens - Dev 1
10:30 am
1 min 15 min 1 min 30 min all
pass after
reruns
Commit Build Coffee
Unit
Tests
API
Tests
2 hours, 100
failures
UI Tests
14. What really
happens - Dev 1
1 pm
1 min 15 min 1 min 30 min all
pass after
reruns
Commit Build Coffee
Unit
Tests
API
Tests
2 hours, 100
failures
UI Tests
40 min
Investig
ation
Create
Fix
15. What really
happens - Dev 2
1 pm
1 min 15 min 1 min 30 min all
pass after
reruns
Commit Build Coffee
Unit
Tests
API
Tests
2 hours, 100
failures
UI Tests
30 min
Investig
ation
17. What really
happens - Dev 2
3:30 pm
1 min 15 min 1 min 30 min all
pass after
reruns
Commit Build Coffee
Unit
Tests
API
Tests
2 hours, 100
failures
UI Tests
30 min
Investig
ation
At 4pm
Re-
queue
Dev 1’s
change
labeled
critical
Build
cancelled
Go
Home
18. What really
happens - Dev 1
4:30 pm
1 min 15 min 1 min 30 min all
pass after
reruns
Commit Build Coffee
Unit
Tests
API
Tests
Removed
UI Tests
1 hour
Deploy
3 hours.
Testers leave at
9 pm
Manual
Tests
Bug
Missed
Release
2 am
Bug
Found
20. Developers hate
flaky and slow
CI/build pipelines
Dev 1 Dev 2
● 70 min waiting on API
tests
● 10 min investigating
API tests
● 2 hours waiting on UI
tests
● 40 min investigating
UI tests
● Loses 2+ hours
context switching
● 1 hour trying to
understand new code
● Productivity < 2 hours
● 10 min investigating
same API failures as
Dev 1
● 30 min investigating
same UI failures as
Dev 1
● 2 hours waiting on UI
tests
● 2 hours waiting on
Dev 1’s changes
● Productivity < 2 hours
33. Manually dealing
with flaky tests
Advantages Disadvantages
● Stops flaky tests
breaking the build
● Misses defects
● Eventually recreate
the same tests
● Disheartening
● May not be possible
(3rd party)
● Doesn’t scale