Blunders in Test Automation

W3

Test
Automation

10/4/17
11:30

Blunders
in
Test
Automation

Presented
by:

Dorothy

Graham

Software
Test
Consultant

Brought
to
you
by:

350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073

888-‐-‐-‐268-‐-‐-‐8770
·∙·∙
904-‐-‐-‐278-‐-‐-‐0524
-‐
info@techwell.com
-‐
http://www.starwest.techwell.com/

Dorothy

Graham

Software
Test
Consultant

In
software
testing
for
more
than
forty
years,
Dorothy
Graham
is
coauthor
of
four

books—Software
Inspection,
Software
Test
Automation,
Foundations
of
Software

Testing,
and
Experiences
of
Test
Automation—and
is
currently
working
with

Seretta
Gamba
on
a
test
automation
patterns
wiki.
A
popular
and
entertaining

speaker
at
conferences
and
seminars
worldwide,
Dot
has
attended
STAR

conferences
since
the
first
one
in
1992.
She
was
a
founding
member
of
the
ISEB

Software
Testing
Board
and
a
member
of
the
working
party
that
developed
the

ISTQB
Foundation
Syllabus.
Dot
was
awarded
the
European
Excellence
Award
in

Software
Testing
in
1999
and
the
first
ISTQB
Excellence
Award
in
2012.
Learn

more
about
Dot
at
DorothyGraham.co.uk.

info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
1
Test Automation Blunders
Prepared and presented by
Dorothy Graham
email: info@dorothygraham.co.uk
Twitter: @DorothyGraham
2
Blunder
• from old Norse word “blundra”
– meaning “to shut one’s eyes”
• now means
– mistake caused by ignorance, carelessness
– or not thinking things through
• people blunder when they don’t see or
understand

3
Contents
• Test automation blunders
– Testing-Tools-Test
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– How hard can it be?
– Stable Application Myth
– Project / Not Project
– Inside the Box
– Isolationism
• Conclusion
4
Testing-Tools-Test
• blunder: thinking that tools actually do testing
– most important, at the root of other blunders
• shouldn’t have been called “testing tools”
• better names:
– “tester assistance tools”
– “check-running tools”
– “test execution tools”
– “test running tools”
– “partial-support-for-some-aspects-of-test-
execution-and-comparison tools”

5
People (testers) vs tools
• what do people do?
– think, evaluate, assess, decide, observe, interpret
– recognize patterns, have new ideas, find bugs
– make mistakes
• what do tools do?
– they just run stuff - whatever they’ve been
programmed to execute (including bad tests)
– intelligence level: zero
Get tools to do what computers do best,
get testers to do what people do best
6
Contents
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– Inside the Box
– Isolationism
• Conclusion

7
Who needs GPS?
• if you don’t know where you are going, any
road will do (Lewis Caroll)
• where are you going with your automation?
– testing and automation are different activities
– different activities require different objectives
• good objectives for testing?
– find bugs, gain confidence, investigate
• good objectives for automation?
– Hint: they shouldn’t be the same!
8
What finds most bugs?
regression tests exploratory testing
likelihood of
finding bugs
most often
automated
What is usually automated?

9
Automation success = find lots of bugs?
• tests find bugs, not automation
• automation is a mechanism for running tests
• the bug-finding ability of a single test is not
affected by the manner in which it is executed
• “find bugs” can be a dangerous objective
– especially for regression automation!
Automated tests Manual Scripted Exploratory Fix Verification
Experiences of Test Automation, Ch 27, p 503, Ed Allen & Brian Newman
10
fast
testing
slow
testing
Effectiveness
Low
High
EfficiencyManual testing Automated
Efficiency and effectiveness
poor
fast
testing
poor
slow
testing
goodgood
greatest
benefit
not good but
common
worst
better

11
Contents
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– Inside the Box
– Isolationism
• Conclusion
12
Silver bullet solution: get the right tool
• no such thing as “the right tool” or “best tool”
– what’s “the best car”?
poor benefits
low cost
good benefits
high cost
good benefits
low cost
poor benefits
high cost
benefits
cost budget
investment in
good automation
good benefits
moderate cost
commercial tools?
open source tools?

13
Silver bullet: success is automatic
• automation is (much) more than just a tool
• it takes time and effort to succeed
– building good automation is a learning process
• management support is critical
– high level managers need to understand
automation capability & limitations, and have
realistic expectations and budget
– “people issues” – people use the automation,
people develop the automation
14
Contents
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– Inside the Box
– Isolationism
• Conclusion

15
Automate x% of tests?
manual
tests automated
tests
new approaches,
e.g. monkey
testing, HiVAT*
manual tests
automated
(% manual)
tests (&
verification)
not possible to
do manually
tests not
automated
yet
*High Volume Automated Testing See http://kaner.com
tests that shouldn’t
be automated
(users, colours,
captcha, too long)
16
Testware architecture
– poor architecture gives
high maintenance cost
• most frequent cause of
abandoned automation /
shelfware
– two layers of abstraction
• technical: for long life
• human: for wide use
– using the tool’s
architecture ties you to
that tool (version)
Testers
Test Execution Tool
runs scripts
HL Keywords
Structured
Scripts
testware
architecture
write tests (in DSTL)
Ch 5

17
Contents
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– Inside the Box
– Isolationism
• Conclusion
18
How hard can it be?
• different activities require different skills
• classic blunder: let the testers automate
– automating without automation skills?
• newer blunder: let the automators write tests
– testing without testing skills?
• if testers are automators è a conflict of interest
– do you run tests or do you automate tests?
– automation is better long-term, BUT
– deadline pressure pushes you back into manual
testing

19
Tools will replace testers?
• “we can reduce the number of testers once we
have the tool”
– what are your testers like?
• mindless morons, or
• intelligent investigators?
– need more skills, not fewer
– automation can free testers to do more test
design, exploratory testing
• and find more bugs
– tools don’t replace testers, they support them
20
Contents
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– Inside the Box
– Isolationism
• Conclusion

21
Stable application myth
• can’t start automating until the application (or
the GUI) is stable
– throw-back to testing attitudes 30 years ago?
• testing comes at the end?
• testing is (only) execution?
• testing is more than testing [execution]
• automation is more than execution
• design your automation early, ready to run
when anything is ready to test
22
• test automation pyramid
– Mike Cohn, Lisa Crispin (Ch 1)
– more unit/component tests
– fewer GUI tests
• instability an opportunity rather than a problem
– build flexibility & robustness in your automation
against common types of changes
• which aspects are stable? (GUI often late)
– execute tests for stable parts when ready
– build (GUI) tests now but with detail abstracted out
When to write automated tests

23
Contents
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– Inside the Box
– Isolationism
• Conclusion
24
Project / not project
• not thinking of automation as a project
– doesn’t need funding, resourcing
– just do it in your “spare time”
• thinking that automation is (just) a project
– when will automation be finished? (wrong question!)
– needs on-going continuous improvement
– refactoring at regular intervals
• project to start automation, then continued
support

25
Contents
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– Inside the Box
– Isolationism
• Conclusion
26
Automated tests/automated testing
Select / identify test cases to run
Set-up test environment:
• create test environment
• load test data
Repeat for each test case:
• set-up test pre-requisites
• execute
• compare results
• log results
• analyse test failures
• report defect(s)
• clear-up after test case
Clear-up test environment:
• delete unwanted data
• save important data
Summarise results
Automated tests
Select / identify test cases to run
Set-up test environment:
• create test environment
• load test data
Repeat for each test case:
• set-up test pre-requisites
• execute
• compare results
• log results
• clear-up after test case
Clear-up test environment:
• delete unwanted data
• save important data
Summarise results
Analyse test failures
Report defects
Automated testing
Automated processManual process

27
Contents
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– Inside the Box
– Isolationism
• Conclusion
28
Isolationism: isolated from
• change: encasing your first efforts in stone
– automated tests benefit from review and refactoring
• realism: optimism as a strategy for automation
– automated tests need to be tested and have bugs
– realistic expectations
• managers: not making automation benefits visible
– to the people who matter, in a way that communicates
• developers: not collaborating
– design for automated testability, help developers
• the wider world: not seeking existing knowledge
– books, wiki*, articles, blogs, discussions, events
*TestAutomationPatterns.wikispaces.com

29
Conclusion: see how these are wrong
• Summary of test automation blunders
– Testing-Tools- DON’T –Test they just run stuff
– Who needs GPS? automation needs direction
– NO Silver Bullet needs time and effort
– The Wrong Thing no testware architecture, just manual tests
– How hard can it be? needs different skills
– Stable Application Myth opportunity for flexibility
– Project at times / Not just a Project ongoing, refactor
– Inside the Box automation is more than automation
– Isolationism collaborate, experiment, read, learn
30
Thank you!
• More information:
• downloads www.DorothyGraham.co.uk
– articles and papers
• email info@DorothyGraham.co.uk
• blog http://dorothygraham.blogspot.com
– including automation, certification
• twitter
– @DorothyGraham
• TestAutomationPatterns.org
– free wiki of automation advice,
with Seretta Gamba
eBook available at
informit.com/swtest
ing
Save 35% with
discount code
SWTESTING

Blunders in Test Automation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Blunders in Test Automation

Similar to Blunders in Test Automation (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

Blunders in Test Automation