1. 7 DEADLY SINS
OF
AUTOMATED
TESTING
Dr Adrian Smith
September 2012
Engineering Innovation.
Thursday, 20 September 12

2. Adrian Smith
• Background in Engineering
• Software development using Agile and Lean
Diverse Ex
Diverse Ex
• Technical and Organisational Coach
Aerospace Engineering
• Founded a startup product development and
Commercial and military engineering
Aerospace Engineering
analysis and manufacturing experienc
consulting business Diverse Experienc
Commercial and military engineering
programs including A380 and F35.
analysis and manufacturing experienc
programs including A380 and F35.
Agile Software Developm
Aerospace Engineering
Software development, architecture a
Agile Software Developm
Commercial and military engineering design,
management for engineering CAE, au
Software development, architecture a
analysis and manufacturing experience on major
scientific and digital media.
programs including A380 and F35. engineering CAE, au
management for
scientific and digital media.
Systems Integration
Thursday, 20 September 12
Agile Software Development
Integration of logistics, financial, engi

3. Geeks hate repetition
Thursday, 20 September 12

4. Airbus A380 Wing
Thursday, 20 September 12

5. Thursday, 20 September 12

6. Envy
Flawed comparison
of manual testing
and automation
Thursday, 20 September 12

7. How management
see testing
Thursday, 20 September 12

8. How management
would like to see testing
Thursday, 20 September 12

9. Manual vs Automation
• A flawed comparison
• Assumes that automation can replace manual
testing effort
• Automation generally doesn’t find new defects
• Testing is not merely a sequence of
repeatable actions
• Testing requires thought and learning
Thursday, 20 September 12

10. Ideal automation targets
• Regression testing - assessing current state
• Automation of test support activities
• Data generation or sub-setting
• Load generation
• Non-functional testing
• Deterministic problems
• Big data problems
Thursday, 20 September 12

11. Common symptoms
• Relying on automation as the basis
for all testing activities
• All tests are built by developers
• Absence of code reviews
• Absence of exploratory testing
• Absence of user testing
Thursday, 20 September 12

12. Suggested approach
• Avoid comparison between manual and
automated testing - both are needed
• Distinguish between the automation and the
process that is being automated
• Use automation to provide a baseline
• Use automation in conjunction with manual
techniques
Thursday, 20 September 12

13. GLUTTONY
Over indulging on commercial test tools
Thursday, 20 September 12

14. Promise of automation
• Software vendors
have sold automation
as the capture-replay
of manual testing
processes
• Miracle tools that solve
all testing problems
Thursday, 20 September 12

15. License barrier
• Commercial licenses restrict usage
• Not everyone can run the tests
• Typically, organisations create special
groups or privileged individuals
Thursday, 20 September 12

16. Incompatible technology
• Underlying technology of commercial tools is
often not compatible with the development
toolchain
• Special file formats or databases
• Lack of version control for tests
• Tests cannot be versioned within the software
• Continuous integration problems
• Can’t be adapted or extended by the developers
Thursday, 20 September 12

17. Justifying the expense
• Financial commitments
distort judgement
• Difficult to make objective
decisions
• Tendency to use the tool
for every testing problem
• People define their role by
the tools they use
Thursday, 20 September 12

18. Common symptoms
• A commercial tools form the basis
of a testing strategy
• Only certain teams or individuals can
access a tool or run tests
• Developers have not be consulted in the
selection of a testing tools
• “We always use <insert tool-name> for testing!”
Thursday, 20 September 12

19. Suggested approach
• Use Open Source software tools where ever
possible
• Use tools that can easily be supported by the
development team and play nicely with existing
development tool chain
• Ensure any commercial tools can be executed
in a command-line mode
Thursday, 20 September 12

20. Lust
User interface forms the basis for all testing
Thursday, 20 September 12

21. Testing through the GUI
• Non-technical testers often approach testing
through the user interface
• Ignores the underlying system and application
architecture
• Resulting tests are slow and brittle
• Difficult to setup test context - resulting in
sequence dependent scripts
Thursday, 20 September 12

22. Investment profile
Manual
Exploratory
Interface
Confidence
Speed / Feedback
Collaboratively
built around
system behaviour Acceptance
Exercises
components and Integration
systems
Unit/Component
Investment / Importance
Developer built
optimised for fast
feedback
Thursday, 20 September 12

23. Architecture
• Understanding
application and
system architecture
improves test design
• Creates opportunities
to verify functionality
at the right level
Thursday, 20 September 12

24. Test design
Test Intent
(Clearly identifies what the Test Data
test is trying to verify)
Test
Implementation System Under
(Implementation of the test Test
including usage of test data)
Thursday, 20 September 12

25. F.I.R.S.T. class tests
F Fast
I Independent
R Reliable
S Small
T Transparent
Thursday, 20 September 12

26. Common symptoms
• Testers cannot draw the
application or system architecture
• Large proportion of tests are
being run through the user interface
• Testers have limited technical skills
• No collaboration with developers
• Intent of tests is unclear
Thursday, 20 September 12

27. Suggested approach
• Limit the investment in automated tests that are
executed through the user interface
• Collaborate with developers
• Focus investment in automation at lowest
possible level with clear test intent
• Ensure automation give fast feedback
Thursday, 20 September 12

28. Pride
Too proud to
collaborate when
creating tests
Thursday, 20 September 12

29. Poor collaboration
• Organisations often create
specialisations of roles
and skills
• Layers of management and
control then develop
• Collaboration becomes difficult
• Poor collaboration = poor tests
Thursday, 20 September 12

30. Automating too much
• Delegating test automation to a special group
inhibits collaboration
• Poor collaboration can results in duplicate test
cases / coverage
• Duplication wastes effort and creates
maintenance issues
• Setting performance goals based around test-
cases automated leads to problems
Thursday, 20 September 12

31. No definition of quality
• Automated testing effort should match the
desired system quality
• Risk that too-much, too-little or not the right
things will be tested
• Defining quality creates a shared
understanding and can only be achieved
through collaboration
Thursday, 20 September 12

32. Good collaboration
• Cross-functional
Acceptance
teams built Criteria
better software Analyst
• Collaboration Specification and
improves Elaboration
Collaboration Tester
definition and
verification Developer
Automation
Thursday, 20 September 12

33. Specification by Example
• Recognises the value of
collaboration in testing
• More general than
ATDD and/or BDD
• Based around building a
suite of Living Documentation
that can be executed
Thursday, 20 September 12

34. Common symptoms
• Automated tests are being built
in isolation from team
• Intent of tests is unclear
or not matched to quality
• Poor automation design (abstraction,
encapsulation, ...)
• Maintainability or compatibility issues
Thursday, 20 September 12

35. Suggested approach
• Collaborate to create good tests and avoid
duplication
• Limit the investment in UI based automated
tests
• Collaborate with developers to ensure good
technical practices (encapsulation, abstraction,
reuse, ... )
• Test code = Production code
Thursday, 20 September 12

36. SLOTH
Too lazy to properly
maintain automated tests
Engineering Innovation.
Thursday, 20 September 12

37. Automated Test Failures
• Many potential causes of failure
• Unless maintained - value is slowly eroded
System Reference
Interface Data
Change Changes
Time
New OS
Feature Patch
Thursday, 20 September 12

38. Importance of maintenance
Manual test execution Value of
Unmaintained
Maintained automation Automated Test
Suite
Unmaintained automation
Cost / Effort
Potential
Value of Maintained
Automated Test
Suite
Time
Thursday, 20 September 12

39. Continuous integration
Thursday, 20 September 12

40. Common symptoms
• Test suite has not been recently
run - state is unknown
• Continuous Integration history
shows consistent failures following
development changes / release
• Test suite requires manual intervention
• Duplication within automation code
• Small changes triggers a cascade of failures
Thursday, 20 September 12

41. Suggested Approach
• Ensure automated tests are executed
using a Continuous Integration
environment
• Ensure test are always running - even if
system in not being actively developed
• Make test results visible - create
transparency of system health
• Ensure collaboration between developers
and testers
Thursday, 20 September 12

42. Rage
Frustration with slow, brittle
or unreliable automated tests
Thursday, 20 September 12

43. Slow automation
• Large datasets
• Unnecessary integrations
• Inadequate hardware/environments
• Too many tests
• Reliance on GUI based tests
• Manual intervention
• ... many others
Thursday, 20 September 12

44. Fast Feedback
Thursday, 20 September 12

45. Brittle Tests
• Contain time-bound data
• Have external dependencies
• Rely on UI layout/style
• Rely on sequence of
execution
• Based on production data or
environments
Thursday, 20 September 12

46. Frustration
Thursday, 20 September 12

47. Unreliable Tests
• False positives
• Wastes time investigating
• Failures start being ignored
• Creates uncertainty of
system health
• Workarounds and alternate
tests are created
Thursday, 20 September 12

48. Suggested approach
• Treat automated tests with the same
importance as production code
• Review, refactor, improve ...
• Apply a “Stop the line” approach to test failure
• Eliminate (quarantine) unreliable tests
• Ensure collaboration with developers
• Up-skill / pair testers
Thursday, 20 September 12

49. Avarice Trying to cut
costs through
(Greed) automation
Thursday, 20 September 12

50. Lure of cheap testing
• Testing tool vendors
often try to calculate
ROI based on saving
labour
• Analysis is unreliable and under values the
importance of testing
Thursday, 20 September 12

51. Automation is not cheap
• Adopting test automation tools and techniques
requires significant investment
• Investment in new ways of working
• Investment in skills
• Investment in collaboration
• Ongoing investment in maintenance
Thursday, 20 September 12

52. Common symptoms
• Investment in commercial tools
using a business-case based on
reducing headcount
• Using a predicted ROI as a way of
reducing budget for Testing
• Consolidating automated testing
within a special group
Thursday, 20 September 12

53. Suggested approach
• Ensure the reasons for automation are
clear and are NOT based purely on saving
money/headcount
• Ensure business case for automation
includes costs for ongoing maintenance
Thursday, 20 September 12

54. 7 Deadly Sins
Envy Flawed comparison of manual testing and automation
Gluttony Over indulging on commercial test tools
Lust User interface forms the basis for all testing
Pride Too proud to collaborate when creating tests
Sloth Too lazy to maintain automated tests
Rage Frustration with slow, brittle or unreliable tests
Greed Trying to cut costs through automation
Thursday, 20 September 12

55. Why automate testing ?
Thursday, 20 September 12

56. How geeks really work
Thursday, 20 September 12

57. Thank you
Dr Adrian Smith
September 2012
Engineering Innovation.
Thursday, 20 September 12