1. # L F M F
(and others
who shall r
anonymous emain
even if they
deserve it)! don’t
Learn
from
My
Fail:
Tales
of
Test
Automa6on
Gone
Wrong
(Plus
a
Li*le
Bit
about
How
to
Do
It
Right)
Elisabeth
Hendrickson
Quality
Tree
So7ware,
Inc.
@testobsessed

2. Fail
Pa;ern
#0:
“No
6me
to
automate”
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
No
automaFon
means
rapid
growth
in
the
manual
tesFng
burden.
Or
untested
so7ware.

3. Velocity!
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.

4. Fail
Pa;ern
#1:
Automa6on
for
the
sake
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
of
automa6on
If
you
really
want
a
mess,
use
a
count
of
automated
tests
as
a
measure
of
progress.

5. Image
of
prinFng
office
from
hJp://www.shorpy.com/node/12652
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.

6. Fail
Pa;ern
#2:
All
automa6on
is
end-­‐to-­‐
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
end
through
the
GUI
End-­‐to-­‐end
tests
are
slow
and
fragile.
Some
are
good.
More
are
not
beJer.

7. Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
Flooded
basement
image
courtesy
of
carlpenergy
Showerhead
image
courtesy
of
deusx
Images
from
Flickr
licensed
through
CreaFve
Commons
AJribuFon

8. Fail
Pa;ern
#3:
Dispropor6onate
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
Amount
of
Time
Spent
Fixing
Broken
Tests
Just
when
you
get
one
working
again,
another
breaks.

9. Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
Image
courtesy
of
cdrussorusso
From
hJp://www.flickr.com/photos/23516192@N08/2631474033/
Licensed
Under
CreaFve
Commons
AJribuFon

10. Fail
Pa;ern
#4:
Fragile
Tests
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
They
pass,
they
fail,
but
there
seems
to
be
no
correlaFon
between
the
test
results
and
the
actual
so7ware
behavior.

11. Hey, the
build is red.!
year.!
full moon this
Tuesday with a
Yeah, it’s the 2nd
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.

12. Fail
Pa;ern
#5:
Inter-­‐dependent
Tests
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
When
one
fails,
they
all
fail
together.

13. Run #1!
Run #3!
Run #2!
Run #4!
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.

14. Fail
Pa;ern
#6:
Useless
Tests
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
The
tests
can
never
fail,
so
they
yield
no
informaFon.

15. Yes, I have
actually
… seen this in
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
tests.!
assert True
But rare ly. More
… common: tes rets that
ify the s oftwa
ver
does w hat the
softwar e does.!

16. Fail
Pa;ern
#7:
Bad
Tests
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
Obfuscated
intent,
massive
duplicaFon,
and
Fght
coupling
between
the
interface
and
the
tests
makes
for
an
unmaintainable
mess.

17. Test Login
Input Text first_name fred
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
Input Text last_name flintstone
Input Text user_name fflint
Input Text password p@ssw0rd
Click Submit
Click Logout
Click Login
Input Text username fflint
WAT?!?!
Input Text password badpassword
Click Submit
Input Text ...

18. Fail
Pa;ern
#8:
Bad
Code
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
Smelly
test
automaFon
code
that
violates
SOLID
principles
is
as
much
of
a
problem
as
smelly
producFon
code.

19. Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.

20. Fail
Pa;ern
#9:
Only
specialists
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
automate
They
might
be
good
at
automaFon,
but
they’re
fighFng
an
uphill
baJle.

21. Fail
Pa;ern
#10:
Forcing
devout
non-­‐
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
coders
to
write
automa6on
If
they
lack
either
interest
or
apFtude
to
automate,
the
results
will
be
bad.

22. Fail
Pa;ern
#11:
Tests
That
Don’t
Get
Run
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
Someone
writes
them,
no
one
runs
them.
So
when
they
suffer
bitrot,
no
one
can
hear
them
scream.

23. Unrun !
tests!
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
become !
creaky,!
dusty,!
sad.!
Photo
by
crabchick
hJp://www.flickr.com/photos/crabchick/2627370039/
Licensed
under
CreaFve
Commons
AJribuFon

24. Fail
Pa;ern
#12:
Failing
Tests
Ignored
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
They
run.
They
fail.
No
one
cares.

25. Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
Image
of
the
1906
San
Francisco
Earthquake
a7ermath
from
hJp://www.shorpy.com/node/9866

26. #LFMF
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.
Automate important expectations!
Treat test code with the same level of
care as production code!
Run all the automated tests in the CI
build (no manual kickoff)!
Fix failing tests immediately!
Foster a “whole team tests” culture!

27. Teema Frisk!
@hackyourtv!
Juha Rantanen!
Jyrki Pulliainen!
Jarkko Kailanto!
Many thanks to …!
Copyright
©
2012
Elisabeth
Hendrickson,
Quality
Tree
So7ware,
Inc.