Practical Lessons Learned Automating in Testing. Let us forget theory for a moment, and concentrate on the practice of automation. Alan will describe lessons learned from both success and failure; as a tester, an automator, and a manager. But more importantly, you will discover how to apply these lessons and improve your automation. Learn how to stay focused, how to experiment and still add value, how to manage even if you cannot code, and more...

1. Practical Test Automation
Lessons Learned
Test Automation Day 2015
Alan Richardson
CompendiumDev.co.uk
@eviltester

2. Expect controversy when
we automate in software
testing. Automation has
long been controversial.
@eviltester2

3. A Little History...
@eviltester3

4. 1957
@eviltester4

5. “What we need is more automation.”
Ford Motor company VP,
Delmar S. Harder,
Coined “automation” in 1948
1948
@eviltester5

6. Automation is not a ‘thing’
@eviltester6

7. Automation is a ‘process’
(automatization)
@eviltester7

8. 1952
@eviltester8

9. Automatization was hard to spell
"...the author found automatization both
awkward and - from the standpoint of his
weak spelling - hazardous ... it was the ease
of spelling that finally overcame the author's
reticence to coin a new word"
John Diebold,
“Automation”, 1952
@eviltester9

10. The word “Automation” is...
Norbert Wiener
“hideous”
Sir Anthony Eden
“barbarous”
@eviltester10

11. “Science”, May 6th, 1960
@eviltester11

12. “Science”, May 6th, 1960
'automatization',
'automata', 'strategy',
'machine', 'automatic',
'automated',
'programming'
@eviltester12

13. Watch your language
@eviltester13

14. “Testers don’t need to learn
to program to create
automation”
Controversy. Argument. Conflict.
Leads
To
@eviltester14

15. “Testers don’t need to
learn to program, to
program automata”
You do need to learn to program to program
Leads
To
@eviltester15

16. “Testers don’t need to learn
to program, to define test
data” (etc.)
A clear strategy for automating against
Leads
To
@eviltester16

17. Automating requires a
change to your test
process.
Describe your process
with the correct words.
@eviltester17

18. When we tagged ‘automation’ on
● No time to automate
● Less ‘testing’ done
● No Maintenance of automata
● ‘automation’ justified by ROI
● ‘automation’ didn’t fit the metrics and tools
@eviltester18

19. Natural to tag it on when...
● Requirements (xref to)
● Test Cases (lead to)
● Test Scripts (automated by)
● Automation
@eviltester19

20. “Automate all
the test scripts”
“But we don’t
have any test
scripts”
20

21. 1964
@eviltester21

22. "...automation makes it
possible to do many
things that could not be
done without it..." John Diebold,
Beyond Automation,
1964, pg 191
@eviltester22

23. How to start automating?
@eviltester23

24. Start by figuring out
‘Why?’, then ‘Doing’
and ‘Learning’
@eviltester24

25. 1956
@eviltester25

26. How to start automating?
A structured approach for vendor
procurement?
@eviltester26

27. How not to start automating?
A structured approach for vendor
procurement
@eviltester27

28. How not to start automating?
● Identify a set of tools
● Create objective feature assessment lists
● Add weightings to each objective
● Create RFx for tool vendors
● Await RFx Replies
● Review replies
● Choose Tool
● Evaluate and figure out how to use the tool@eviltester28

29. “Many managements genuinely
seem to be looking at this subject as
if they were walking around the edge
of an ice cold swimming pool...”
John Diebold,
Beyond Automation,
1964, pg 66
@eviltester29

30. “...They realize that sooner or later
they are going to have to jump in,
but they try to postpone the leap
with as much rationalization and
fact-gathering as possible”
John Diebold,
Beyond Automation,
1964, pg 66
@eviltester30

31. Evaluate everything
for a long time and
end up with nothing
@eviltester31

32. Evaluate everything
for a long time and
end up with nothing
@eviltester32

33. How to start automating
● Why?
○ Because?
■ How?
● Experiment
@eviltester33

34. Avoid the Danger of
Ratholes. Understand
your objectives
@eviltester34

35. As a:
● Tester:
○ identify the areas to improve
● Manager:
○ prioritise and approve,
○ have a vision,
○ check in frequently
● Automator:
○ build small, solve fast, try multiple approaches
@eviltester35

36. My First Automation...
@eviltester36

37. Ignore the Tools & Demos
● SSADM Test Tool
● Generic Test Case Management Tool
● Test Case Xrefer
● Metrics Collator
● Test Data Generator * 2
● Keyword driven terminal emu automator
● Demos to ‘sell’ automation to clients
@eviltester37

38. My First Automation was
tools and frameworks
@eviltester38

39. My First Automation was
tools and frameworks
@eviltester39

40. My first ‘good’ automation
Automate the generation of ‘Scripts’ from
‘Paths’ to support interactive testing
@eviltester40

41. Do start by solving
problems
@eviltester41

42. Don’t start by building
tools or frameworks
@eviltester42

43. Refactor to Flexible
Abstraction Layers
@eviltester43

44. Automate the easy stuff first
● start small
● the better you get, the easier things
become, so start simple
● 80/20 : automate the 20%, gain 80%
value
@eviltester44

45. Do start with proof of
concepts
@eviltester45

46. Can we automate this system?
● can you execute some basic paths?
● check results where?
● individual control mechanisms?
● synchronisation methods?
46

47. Automate that which
can be automated.
Prioritize by value and
ease.
@eviltester47

48. How do you manage
that which you do not
know how to do?
@eviltester48

49. @eviltester
1860
1947
49

50. “He must, as in the past, be so
complete a master of the technical
aspects of his profession that his
subordinates will recognise his
mastery, and be prepared to learn
from him.”
The Art of Leadership,
Captain S.W. Roskill,
1964, pg 30

51. “The capability of a superintendent
is eminently conspicuous in the
selection of his attendants…”
The Philosophy of Insanity,
1860
51

52. Lessons learned in recruitment
● Phone screen
○ cv, experience & philosophy
○ portfolio
● Face to face
○ pairing
○ review, fix, create, speculate
Technical
Knowledge
Required
@eviltester52

53. “Automation has turned out to
be a much more complex and
difficult problem than was
originally thought.”
John Diebold,
Beyond Automation,
1964, pg 51
53

54. Lessons Learned: Do Stuff
● Automation is not a thing, it is a process
● Watch your language
● Solve problems quickly and maximize value
● Build abstractions, not tools and frameworks
● Experiment small and fast
● Hire well, based on a portfolio and demo
@eviltester54

55. Blogs and Websites
● CompendiumDev.co.uk
● SeleniumSimplified.com
● EvilTester.com
● JavaForTesters.com
● Twitter: @eviltester
Online Training
● Technical Web Testing 101
○ Unow.be/at/techwebtest101
● Selenium 2 WebDriver API
○ Unow.be/at/webdriverapi
Books
Selenium Simplified
Unow.be/rc/selsimp
Java For Testers
leanpub.com/javaForTesters
Alan Richardson
uk.linkedin.com/in/eviltester
http://compendiumdev.co.uk/contact