1.
Deadly Traps of Test
Automation
STC JULY 2014
THAO VO

2.
A landscape
2
1. Automation idol
2. UI addiction
3. Over misusing on commercial
tools
4. Poor collaboration
5. Lazy maintenance
6. Intensive ROI
Source: http://www.lifehack.org/articles/productivity/how-to-manage-
common-productivity-traps-for-improved-productivity.html

3.
3
Trap #1: Adore Automation
Source: http://www.theguardian.com/media/shortcuts/2014/mar/16/could-robots-be-journalist-of-future

4.
Trap #1: Adore Automation
Symptoms
• Depending on automation for all testing
activities
• Absence of code reviews
• Absence of exploratory testing
• Absence of UX testing
• All tests are built by developers
4
Source: http://www.calgolftech.com/1-problem-with-buying-golf-clubs/

5.
Trap #1: Adore Automation
5
Treatments
Prevent comparison between
manual and automation
Use automation in conjunction
with manual techniques
Use automation to provide a
baseline – a sanity check
Source: http://cureforhives.org/chronic-
idiopathic-urticaria-treatments

6.
6
Trap #2: Love UI so much

7.
A Pyramid
7
GUI
Tests
Acceptance
Tests
Integration Tests
Unit Tests

8.
Trap #2: Love UI so much
Symptoms
• Limited technical skills
• Cannot understand the
application or system
architecture
• Huge proportion of tests are
being run via the UI
• No collaboration with
developers
• Difficult to setup test context
• Test results are slow & fragile
8
Source: http://www.bostonsearchgroup.com/blog/ceo-
search-break-through-medical-devices-for-wound-closure/

9.
Trap #2: Love UI so much
Treatments
• Limit the investment in UI
automation
• Invest more at lowest possible
level with clear test purposes
• Collaborate with developers
• Make sure automation gives a
fast feedback
9

10.
10
Trap #3: Over misusing on
commercial testing tools
Source: http://www.fastcoexist.com/1679718/your-bad-eating-habits-are-contagious

11.
Trap #3: Over misusing on
commercial testing tools
Symptoms
11
A commercial tool forms the basis
a testing strategy
Only certain teams or individuals
can access the tool or run tests
Developers and testers have not
been consulted in tool selections Source: http://www.sfgate.com/news/article/Apple-
exploring-cars-medical-devices-to-reignite-5239850.php

12.
Trap #3: Over misusing on
commercial testing tools
Treatments
• Use open source tools wherever
possible
• Use tools that can be extended
or integrated into existing
development tool chain
• Use tools that match specified
test intent
• Use clear matrix tool evaluation
• Keep educating!!!
12
Source: http://stugalactic.co.uk/student-shopping-
medicine.aspx

13.
13
Trap #4: Too pride to
collaborate when creating
tests
Source: http://www.jonathankettleborough.com/tag/collaboration-2/

14.
Trap #4: Poor collaboration
Symptoms
• Automated testers are isolated
from team: duplicated test
scripts, too much tests but
inefficiency
• Poor automation design
• Maintainability or compatibility
issues
• No definition of quality: unclear,
not match the desired system
quality 14
Source: http://thaineuro.org/category/pharma-and-
healthcare/healthcare/

15.
Trap #4: Poor collaboration
Treatments
• Collaborate to create good tests
and prevent duplication
• Ensure good technical practices
and design
• Collaborate to improve quality
definition and verification
• Limit investment in UI based
automated tests
15
Source: http://medicine1.info/medicine/

16.
16
Trap #5: Too lazy
maintenance
Source: http://www.tastyhuman.com/the-illusion-of-8-hour-workday-are-you-hard-working-or-just-lazy/

17.
Trap #5: Too lazy
maintenance
Symptoms
• Test suite has not been run
recently – state is unknown
• Test suite requires manual
intervention
• Duplication within automation
code
• Small changes trigger a cascade
of failures
17
Source: http://orangedocfamilymedicine.com/

18.
Trap #5: Too lazy
maintenance
Treatments
• Automated tests should be
executed using a CI
environment
• Ensure tests are always
executable – even if the system
is not being actively developed
• Ensure collaboration between
developers and testers
18
Source: http://www.psmag.com/blogs/the-101/the-
spectrum-of-color-response-take-your-medicine-51195/

19.
19
Trap #6: Cut costs through
automation
Source: http://insuranceknock.com/questions/155/saving-money-with-2-phrases

20.
Trap #6: Cut costs through
automation
Symptoms
• Using a predicted ROI to
reduce testing budget
• Hidden investments are not
counted
20
Source: http://bloomintegrativehealth.com/news-updates/

21.
Trap #6: Cut costs through
automation
Treatments
• Clear objectives and reasons
for automation
• Cost-benefit analysis should
include ongoing maintenance
• Quality is the key
21
Source: https://today.duke.edu/2013/03/operationmedicinedrop

22.
Summary
22

23.
23
Q&A

24.
References
• [1]. Testing Experience – published No. #21,
#23 - http://www.testingexperience.com/
• [2]. 7 Deadly Sins of Agile Software Test
Automation – Adrian Smith
http://www.slideshare.net/adrianlsmith/7-
deadly-sins-of-software-test-automation
• [3]. Manual Test cannot be automated
http://www.satisfice.com/blog/archives/58
• [4]. Testing Club Vietnam 2014 Trends in SW
Testing – Paul Holland
24

25.
Profile
• Thao Vo – QA Architect at KMS Technology Inc.
(http://www.kms-technology.com/); Board
member of HCMC Software Testing Club
(http://www.hcmc-stc.org/).
• Email: thaovo@kms-technology.com or
thaovo@hcmc-stc.org or vvthao@yahoo.com
• LinkedIn: http://vn.linkedin.com/pub/thao-
van-vo/24/31a/b35
25

26.
© 2014 HCMC Software Testing Club
THANK YOU