In chess, the word blunder means a very bad move by someone who should know better. Even though functional test automation has been around for a long time, people still make some very bad moves and serious blunders. The most common misconception in automation is thinking that manual testing is the same as automated testing. And this misguided thinking accounts for most of the blunders in system level test automation. Dorothy Graham takes you on a tour of these blunders, including the Stable-Application Myth (you can’t start automating until the application is stable), Inside-the-Box Thinking (automating only the obvious test execution), and the Project/Non-Project Dilemma (failing to treat automation like a project by not funding or resourcing it and treating automation as only a project). Other blunders include Testing–Tools–Test, Silver Bullet, Automating the Wrong Thing, Who Needs GPS, How Hard Can It Be, and Isolationism. New skills, approaches, and objectives are needed or you’ll end up with inefficient automation, high-maintenance costs, and wasted effort. Join Dot to discover how you can avoid these common blunders and achieve valuable test automation.

1. K1
Keynote
10/18/2017 8:30:00 AM
Blunders in Test Automation
Presented by:
Dorothy Graham
Software Test Consultant
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888-­‐268-­‐8770 ·∙ 904-­‐278-­‐0524 - info@techwell.com - https://www.techwell.com/

2. Dorothy Graham
Software Test Consultant
In software testing for more than forty years, Dorothy Graham is coauthor of four
books – Software Inspection, Software Test Automation, Foundations of Software
Testing, and Experiences of Test Automation – and is currently working with
Seretta Gamba on a test automation patterns wiki. A popular and entertaining
speaker at conferences and seminars worldwide, Dot has attended STAR
conferences since the first one in 1992. She was a founding member of the ISEB
Software Testing Board and a member of the working party that developed the
ISTQB Foundation Syllabus. Dot was awarded the European Excellence Award in
Software Testing in 1999 and the first ISTQB Excellence Award in 2012. Learn
more about Dot at DorothyGraham.co.uk.

3. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
1
Test Automation Blunders
Prepared and presented by
Dorothy Graham
email: info@dorothygraham.co.uk
Twitter: @DorothyGraham
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
© Dorothy Graham 2017
2
Blunder
• from old Norse word “blundra”
– meaning “to shut one’s eyes”
• now means
– mistake caused by ignorance, carelessness
– or not thinking things through
• people blunder when they don’t see or
understand

4. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
3
Contents
• Test automation blunders
– Testing-Tools-Test
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– How hard can it be?
– Stable Application Myth
– Project / Not Project
– Inside the Box
– Isolationism
• Conclusion
Twitter: @DorothyGraham
4
Testing-Tools-Test
• blunder: thinking that tools actually do testing
– most important, at the root of other blunders
• shouldn’t have been called “testing tools”
• better names:
– “tester assistance tools”
– “check-running tools”
– “test execution tools”
– “test running tools”
– “partial-support-for-some-aspects-of-test-
execution-and-comparison tools”

5. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
5
People (testers) vs tools
• what do people do?
– think, evaluate, assess, decide, observe, interpret
– recognize patterns, have new ideas, find bugs
– make mistakes
• what do tools do?
– they just run stuff - whatever they’ve been
programmed to execute (including bad tests)
– intelligence level: zero
Get tools to do what computers do best,
get testers to do what people do best
6
Contents
• Test automation blunders
– Testing-Tools-Test
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– How hard can it be?
– Stable Application Myth
– Project / Not Project
– Inside the Box
– Isolationism
• Conclusion
Twitter: @DorothyGraham

6. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
7
Who needs GPS?
• if you don’t know where you are going, any
road will do (Lewis Caroll)
• where are you going with your automation?
– testing and automation are different activities
– different activities require different objectives
• good objectives for testing?
– find bugs, gain confidence, investigate
• good objectives for automation?
– Hint: they shouldn’t be the same!
8
What finds most bugs?
regression tests exploratory testing
likelihood of
finding bugs
most often
automated
What is usually automated?

7. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
9
Automation success = find lots of bugs?
• tests find bugs, not automation
• automation is a mechanism for running tests
• the bug-finding ability of a single test is not
affected by the manner in which it is executed
• “find bugs” can be a dangerous objective
– especially for regression automation!
Automated tests Manual Scripted Exploratory Fix Verification
Experiences of Test Automation, Ch 27, p 503, Ed Allen & Brian Newman
10
fast
testing
slow
testing
Effectiveness
Low
High
EfficiencyManual testing Automated
Efficiency and effectiveness
poor
fast
testing
poor
slow
testing
goodgood
greatest
benefit
not good but
common
worst
better

8. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
11
Contents
• Test automation blunders
– Testing-Tools-Test
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– How hard can it be?
– Stable Application Myth
– Project / Not Project
– Inside the Box
– Isolationism
• Conclusion
Twitter: @DorothyGraham
12
Silver bullet solution: get the right tool
• no such thing as “the right tool” or “best tool”
– what’s “the best car”?
poor benefits
low cost
good benefits
high cost
good benefits
low cost
poor benefits
high cost
benefits
cost budget
investment in
good automation
good benefits
moderate cost
commercial tools?
open source tools?

9. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
13
Silver bullet: success is automatic
• automation is (much) more than just a tool
• it takes time and effort to succeed
– building good automation is a learning process
• management support is critical
– high level managers need to understand
automation capability & limitations, and have
realistic expectations and budget
– “people issues” – people use the automation,
people develop the automation
14
Contents
• Test automation blunders
– Testing-Tools-Test
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– How hard can it be?
– Stable Application Myth
– Project / Not Project
– Inside the Box
– Isolationism
• Conclusion
Twitter: @DorothyGraham

10. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
15
Automate x% of tests?
manual
tests automated
tests
new approaches,
e.g. monkey
testing, HiVAT*
manual tests
automated
(% manual)
tests (&
verification)
not possible to
do manually
tests not
automated
yet
*High Volume Automated Testing See http://kaner.com
tests that shouldn’t
be automated
(users, colours,
captcha, too long)
16
Testware architecture
– poor architecture gives
high maintenance cost
• most frequent cause of
abandoned automation /
shelfware
– two layers of abstraction
• technical: for long life
• human: for wide use
– using the tool’s
architecture ties you to
that tool (version)
Testers
Test Execution Tool
runs scripts
HL Keywords
Structured
Scripts
testware
architecture
write tests (in DSTL)
Ch 5

11. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
17
Contents
• Test automation blunders
– Testing-Tools-Test
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– How hard can it be?
– Stable Application Myth
– Project / Not Project
– Inside the Box
– Isolationism
• Conclusion
Twitter: @DorothyGraham
18
How hard can it be?
• different activities require different skills
• classic blunder: let the testers automate
– automating without automation skills?
• newer blunder: let the automators write tests
– testing without testing skills?
• if testers are automators è a conflict of interest
– do you run tests or do you automate tests?
– automation is better long-term, BUT
– deadline pressure pushes you back into manual
testing

12. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
19
Tools will replace testers?
• “we can reduce the number of testers once we
have the tool”
– what are your testers like?
• mindless morons, or
• intelligent investigators?
– need more skills, not fewer
– automation can free testers to do more test
design, exploratory testing
• and find more bugs
– tools don’t replace testers, they support them
20
Contents
• Test automation blunders
– Testing-Tools-Test
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– How hard can it be?
– Stable Application Myth
– Project / Not Project
– Inside the Box
– Isolationism
• Conclusion
Twitter: @DorothyGraham

13. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
21
Stable application myth
• can’t start automating until the application (or
the GUI) is stable
– throw-back to testing attitudes 30 years ago?
• testing comes at the end?
• testing is (only) execution?
• testing is more than testing [execution]
• automation is more than execution
• design your automation early, ready to run
when anything is ready to test
22
• test automation pyramid
– Mike Cohn, Lisa Crispin (Ch 1)
– more unit/component tests
– fewer GUI tests
• instability an opportunity rather than a problem
– build flexibility & robustness in your automation
against common types of changes
• which aspects are stable? (GUI often late)
– execute tests for stable parts when ready
– build (GUI) tests now but with detail abstracted out
When to write automated tests

14. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
23
Contents
• Test automation blunders
– Testing-Tools-Test
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– How hard can it be?
– Stable Application Myth
– Project / Not Project
– Inside the Box
– Isolationism
• Conclusion
Twitter: @DorothyGraham
24
Project / not project
• not thinking of automation as a project
– doesn’t need funding, resourcing
– just do it in your “spare time”
• thinking that automation is (just) a project
– when will automation be finished? (wrong question!)
– needs on-going continuous improvement
– refactoring at regular intervals
• project to start automation, then continued
support

15. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
25
Contents
• Test automation blunders
– Testing-Tools-Test
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– How hard can it be?
– Stable Application Myth
– Project / Not Project
– Inside the Box
– Isolationism
• Conclusion
Twitter: @DorothyGraham
26
Automated tests/automated testing
Select / identify test cases to run
Set-up test environment:
• create test environment
• load test data
Repeat for each test case:
• set-up test pre-requisites
• execute
• compare results
• log results
• analyse test failures
• report defect(s)
• clear-up after test case
Clear-up test environment:
• delete unwanted data
• save important data
Summarise results
Automated tests
Select / identify test cases to run
Set-up test environment:
• create test environment
• load test data
Repeat for each test case:
• set-up test pre-requisites
• execute
• compare results
• log results
• clear-up after test case
Clear-up test environment:
• delete unwanted data
• save important data
Summarise results
Analyse test failures
Report defects
Automated testing
Automated processManual process

16. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
27
Contents
• Test automation blunders
– Testing-Tools-Test
– Who needs GPS?
– Silver Bullet
– The Wrong Thing
– How hard can it be?
– Stable Application Myth
– Project / Not Project
– Inside the Box
– Isolationism
• Conclusion
Twitter: @DorothyGraham
28
Isolationism: isolated from
• change: encasing your first efforts in stone
– automated tests benefit from review and refactoring
• realism: optimism as a strategy for automation
– automated tests need to be tested and have bugs
– realistic expectations
• managers: not making automation benefits visible
– to the people who matter, in a way that communicates
• developers: not collaborating
– design for automated testability, help developers
• the wider world: not seeking existing knowledge
– books, wiki*, articles, blogs, discussions, events
*TestAutomationPatterns.wikispaces.com

17. info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
29
Conclusion: see how these are wrong
• Summary of test automation blunders
– Testing-Tools- DON’T –Test they just run stuff
– Who needs GPS? automation needs direction
– NO Silver Bullet needs time and effort
– The Wrong Thing no testware architecture, just manual tests
– How hard can it be? needs different skills
– Stable Application Myth opportunity for flexibility
– Project at times / Not just a Project ongoing, refactor
– Inside the Box automation is more than automation
– Isolationism collaborate, experiment, read, learn
30
Thank you!
• More information:
• downloads www.DorothyGraham.co.uk
– articles and papers
• email info@DorothyGraham.co.uk
• blog http://dorothygraham.blogspot.com
– including automation, certification
• twitter
– @DorothyGraham
• TestAutomationPatterns.org
– free wiki of automation advice,
with Seretta Gamba
eBook available at
informit.com/swtest
ing
Save 35% with
discount code
SWTESTING