The document discusses a workshop on test automation. It includes an agenda for the workshop covering topics such as what kind of test automation system participants would like to have, where testing time is spent, how to choose a test automation tool, and what is the right level of test automation for an organization. The workshop uses a format of presenting questions, having group discussions, and then presenting prepared material to promote learning.

1. Test Automation
Pondering Workshop for FYI on Winter 2011
towo@iki.fi
Towo Toivola
Attribution (Tekijä mainittava)
http://creativecommons.org/licenses/by/3.0/

2. Welcome to the Workshop
I hope we all have an appetite for arguing and
learning about test automation
The slides are in English just to be sure, but I will present in
Finnish if the audience prefers it.

3. Many Thanks
Much of this material is heavily indebted
to Maaret Pyhäjärvi, Erkki Pöyhönen,
Petri Kuikka, Risto Kumpulainen, Dean
Leffingwell..
Thanks also go to my friends at work
and in the software engineering
community
Towo Toivola 2009

4. About Me
Experiences define viewpoints
MSc (eng) from HUT
In F-Secure Corp for 11 years
◦ COTS software, custom systems
◦ Many many projects, large and small
A year in Kosovo
◦ IT administration, system building
Seminars, workshops
Interactive speaker
Towo Toivola 2009

5. About You
How many are test engineers?
How many managers?
Any programmers?
How many have some experience on
testing tools?
What is your industry?
What are your most important
requirements for this workshop?
Towo Toivola 2009

6. Disclaimer
As with everything, nothing is absolute
I shall not discuss load-testing much, or test
process automation
Not an expert of the particular tools
Your terminology may differ
Attending this course may not be enough
It may be that you should not do test
automation
There are two worlds, I will try to balance
in between them
Towo Toivola 2009

7. Agenda for Today
0900 Welcome (this) 1255 How would you
0920 What kind of choose your TA tool?
TA system would you 1345 What is the right
like? level for doing TA in
1000 What is the your organization?
goal of your TA 1430 How to take TA
Break
initiative? into use? Break
1105 Where does all 1530 How will you
the testing time go? change your products to
ease TA?
1600 Resourcing TA in
Lunch the long run?
Towo Toivola 2009

8. About the Schedule
There is a lot of ground to cover
This is a new format, bear with me
We can change things
You can check what I am planning to leave out..
There should be enough breaks, we
should be able to concentrate well on
work
◦ What rules should we have?
Towo Toivola 2009

9. Format for this Workshop
We will use this format to deal with the
questions in the agenda:
Presenting a question (5min)
Pondering in small groups (15min)
Discussion with all together (10min)
Presenting of prepared material (10min)
This should make
it into about 30-50 mins
per problem if we don't
get carried away.

10. Why this Format?
Traditional teaching (providing answers) does
not promote thought
Thought and discovery promotes learning
Struggling with a problem first and receiving
guidance after that promotes understanding and
application (I hope)
Interaction promotes understanding and
information dissipation
I am trying to give you the most valuable things
well rather than a lot of things
Technical details are not as important as people
think

11. Question #1
What kind of TA system would you like to have?
(Outline your vision or what you want to have in
one years time)
0920

12. Please take 15 to think about this
First quickly by yourself
Then within your group
Document ideas, keywords and major
agreements on provided office supplies
Be prepared to present your views and
findings, even if you are not so sure

13. Results and Discussion for
Question #1
What kind of TA system would you like to have?
(Outline your vision or what you want to have in
one years time)

14. Why do You Want What You
Want?
Tools and technical solutions often dominate
unduly in the decisions
Organizational boundaries are often taken as
granted and TA may lead to local optimization
You have probably made a lot of implicit
assumptions before even starting to actively
think about the problem
People tend to focus on fixing what they
understand best instead of that which is most
important.

15. Now that you have already discovered what
should be the end result of today, we can
start working in light of that assumption ;)

16. Question #2
What is the goal of your TA initiative?
(Why are you here, why do you want to do TA,
what do you expect to gain?)
1000

17. Group Work Time
Firstquickly by yourself
Then within your group
Document
Be prepared to present

18. Results and Discussion for
Question #2
What is the goal of your TA initiative?

19. What’s the Use of Test Automation
Cost savings
◦ Ability to run the same tests cheaper
Faster development
◦ Ability to implement more features in the
same amount of time
Faster delivery
◦ Ability to do snap changes
Better quality
◦ Ability to reach and maintain higher quality
Towo Toivola 2009

20. Differences to Manual Testing
Some tests that are manually possible
are not possible to automate
Some tests that are possible to automate
are not possible to run manually
Development takes much more time,
execution much less
Much more attention to detail, much less
common sense
Requires more maintenance
Towo Toivola 2009

21. Enabling the Fast Feedback Loop
A common and useful use of test
automation
Minimize the time from programming to
quality information
◦ Code – test – fix – retest in hours VISIBILITY!
◦ If not in minutes!
Continuous Integration
Fully automatic tests and reporting
”The most important limiting factor for
the velocity of an agile team” -DL
Towo Toivola 2009

22. Regression and Automation
VISIBILITY!
Is the software still OK?
Manual regression will get rid of your
good test engineers
Automation does not get bored
The courage to do changes
The ability to do quick releases
Towo Toivola 2009

23. The Machine is Superior to Man
Automation is faster
Automation has more attention to detail
Automation can run through more data
Automation can execute more
combinations
Automation can measure more exactly
Automation does not get tired
Automation works through the night
Automation provides exact logs
Towo Toivola 2009

24. Man is Superior to the Machine
Works around problems
Common sense
Creativity
Understanding of the user
Understanding of priorities
Understands change
Looking outside the box
The most important bugs are found in
the requirements and assumptions
Towo Toivola 2009

25. What Do You Want?
You Need to Prioritize, seriously
If you don’t set clear goals, don’t expect
clear benefits, or much results of any kind
Test faster?
Test more often?
Test better?
Test in new ways?
Test cheaper?
Towo Toivola 2009

26. Common Mistakes
(that cost a lot of money)
Saving a project
Aiming for X% automated
Working with untestable software
Running a lot of tests, producing little
value
VISIBILITY!
Organizational issues
Skills of people working with automation
#1: Unrealistic assumptions
◦ Automation will find loads of new bugs
◦ No real resourcing needed
Towo Toivola 2009

27. Real Benefit of TA
TA is only useful It does this..
when it provides More if you start
information that.. early in the project
Gets a bug fixed More if you run it
Helps you make a constantly
decision Only if you react to
the results
These Only if you believe
require the results
(re)action!

28. It's hard, but it's
worth it.
Test automation, when done sensibly, is
hugely beneficial. It can give you decisive
competitive advantage over your
competitors both financially and in terms of
employee motivation and satisfaction.

29. Break Time
1050
Towo Toivola 2009

30. Question #3
Where does all the testing time go?
(Testing software is an expensive business, why
does it cost so much? What do all those people
spend all their time on?)
1105

31. Group Work Time
Firstquickly by yourself
Then within your group
Document
Be prepared to present

32. Results and Discussion for
Question #3
Where does all the testing time go?

33. The Nature of a Test
Which part can the automation
handle? Towo Toivola 2009

34. Where Does Your Testing Time
Go?
Do you use too much time testing?
Not enough time?
Test planning?
Test environment building?
Regression testing?
Testing new features?
Testing large data collections?
Testing with numerous environments?
Towo Toivola 2009

35. Things You Should Automate
Things that are run very often
◦ Core regression tests
Things that are easy to automate and bring
value
Things that are tough to test manually
◦ Performance
◦ Race conditions
◦ Linear combinations
◦ Large amounts of data
◦ Reliability
Things that have to be run quickly
ROI !!!!11!!!!
Towo Toivola 2009

36. Things You Should Not Automate
Tests that are not important
Tests that are easy to do manually and
hard to automate
Tests that are run seldom
◦ Unless they are critical or very hard to run
manually
Does-it-feel-right
Tests where automation results are not
reliable
Towo Toivola 2009

37. But that is all the Sensible Stuff
It is good to think about how to balance
between explorative and regression
testing, but I bet your testers also use
time on..
Meetings, overhead, bureacracy
Waiting (for fix, hardware, decision, info..)
Arguing, pleading, fuming..
Testing wrong things
Testing wrong builds..

38. In the Insensible Reality..
How much will making execution faster
actually help?
Should you look at the whole process?
Maybe you should fix something else (as
well)?
Could you use TA implementation as a
driver for new development culture?

39. Lunchtime!
I have reserved one hour for lunch.
1155
Towo Toivola 2009

40. Question #4
How would you choose your TA tool?
(Since you want to do TA, you need to somehow
decide which tooling to use. How do you decide?
How do you make the decision an informed one?)
1255

41. Group Work Time
Firstquickly by yourself
Then within your group
Document
Be prepared to present

42. Results and Discussion for
Question #4
How would you choose your TA tool?

43. Finding a Tool
Remember that a tool is not the same as
test automation
“A fool with a tool is a more dangerous
fool”
There are many vendors
There are many free alternatives
Tool finding must be considered real
work
◦ With goals
◦ With resources Towo Toivola 2009

44. Cost, What is it?
Many costs
◦ Cost of the tool (in the receipt)
◦ Cost of taking the tool into use (mainly
hidden)
 Including maintenance
◦ Cost of not doing something else with the
time
◦ Cost of maintenance
You should consider all costs
Cost of tool is usually overrated
Towo Toivola 2009

45. Impact of Tool on Work
Now, what is expensive?
Big tool project
Small tool project Cheap
unsuitable tool
Time
Project end
Towo Toivola 2009

46. My Advice
Always take the one that is most suitable
for you, taking into account:
Cost of tool
Intended use
Personnel skills and attitudes
Goals of the tool project
Future plans for test automation
Don't standardize for the sake of it
Towo Toivola 2009

47. Plan the Tool Finding
Make it a proper project
 If you still do projects
Plan – set goals
Resource explicitly
Have many people join in
Scope out the tools out there
Select a couple for trials
◦ Do real work with them
◦ Not important, busy projects!
Towo Toivola 2009

48. Remember
You will need support in using the tool
You will need changes to the tool
Your software will change
The platform your software will run on
will change as well
You may need changes to your software
Or your development process..
Towo Toivola 2009

49. What Should it be Like?
Relatively easy to take into use
Enables automatic testing, not just
playing with automation
◦ Including reporting
Supports your technology well
Does not seriously restrict your testers
Can be run by a single person and as a
central system
Fits your organization
Towo Toivola 2009

50. Or do You Need a Tool?
You could make one, afterall you make
software
 Just right for you
 Intergrates nicely with your product and
development process
You could just use common utilities to
create a mash-up tool
 Build server
 SSH, remote exec
 Common scripting languages (with TA libraries..)

51. In a Modern, Efficient, Agile Organization
In a galaxy far, far away..
The best people to make the trade-off decisions
between tools are seasoned software
development engineers who work in agile teams
that have holistic responsibility for developing
the software
Trust your professionals to make the right
choice and support them
Their knowledge, interest and motivation are
priceless in the implementation work of the TA
Success will truly inspire others

52. Question #5
What is the right level for doing TA in your
organization?
(Should the SWE:s do UT/TDD? Should there be
paired test engineers perfoming them? Or module
tests? Should there be multi-tier integration tests?
Can you do acceptance-level TA? Should you?)
1345

53. Group Work Time
Firstquickly by yourself
Then within your group
Document
Be prepared to present

54. Results and Discussion for
Question #5
What is the right level for doing TA in your
organization?

55. What is Module- and Unit testing?
Module testing
GUI Testing Tool environment
Unit Testing
Framework
Graphical User Application
Interface (GUI) Programming
Interface (API)
When significant When functionality is
part of When each unit
available from below can be used to
functionality GUI or from side
available in user catch the problems
(components / modules
interface. and their interfaces)
“tester territory” “developer territory”
Towo Toivola 2009

56. Can you
Testing Levels find the
silver
bullet?
Cost /
System
found
bug
Integration
Module
Unit
Bug finding
potential
0% 100%
Towo Toivola 2009

57. Why Test Automation on Many
Levels
Faster automation
Faster feedback loop
◦ Ability to do snap changes
Better quality
◦ Ability to reach and maintain higher quality
Faster (= cheaper) implementation
Less maintenance work
◦ Again, = cheaper
End result: Cost efficiency
Towo Toivola 2009

58. Differences to GUI Automation
API:s are more straightforward to use
◦ Implementation is faster and cheaper
◦ Tools are often free of charge
API:s change much more seldom than a GUI
◦ Requires less maintenance
Some tests that are possible through API:s are not possible
to run using the GUI
Some tests that are possible through the GUI are not
possible to run using the API
◦ You will test a bit less of your application
◦ You can, probably, also module-test your GUI
Running takes typically less time
 Code – test – fix – retest in minutes
Detected bugs are usually easier to fix
Towo Toivola 2009

59. Question #6
How to take TA into use?
(Who should do it? Where do we get the
resources? Do we have the skills? Which project?
Which product? How to create the culture? How
to ensure the long-term commitment of the
organization?)
1430

60. Group Work Time
Firstquickly by yourself
Then within your group
Document
Be prepared to present

61. Results and Discussion for
Question #6
How to take TA into use?

62. Slide of Truth About Everything
Seriously, did you expect that I would have
answers to that question for you?
Some thoughts will follow, but remember
that every case is unique.

63. Test Automation IS Software
System Development
Good test automation requires that you
understand testing well,
And software development
And system
implementation/administration
And preferably test automation too.
Most people don't
reeeally get this
Towo Toivola 2009

64. What Does That Mean?
Source control and versioning
Bug tracking
Requirements definition
Project management
◦ Iterative and incremental if you want to be
successful
Resourcing
Testing!
Fast feedback loop!
Long-term commitment
ETC.
Towo Toivola 2009

65. Don’t Leave the 20% Undone
◦ Usability
◦ Reliability
◦ User base
◦ Significance
◦ Reporting
◦ Analysing problems
◦ Maintenance
Towo Toivola 2009

66. Make sure you generate value ( = $$$$ )
not just run tests
My serious main recommendation:
One test case. Full chain. Beginning of project.
VISIBILITY!
Remember:
TA is only useful when it
provides information that..
Ugly slide.. Gets a bug fixed
Helps you make a decision

67. Question #7
How will you change your products to ease TA?
(Should you? How? Should you change your
process? Why?)
1530

68. Group Work Time
Firstquickly by yourself
Then within your group
Document
Be prepared to present

69. Results and Discussion for
Question #7
How will you change your products to ease TA?

70. A Word of Warning
Automating an untestable software can
be ruinously expensive
Prepare for organization struggles when
you need the software changed
And the process
Prepare for the organization to ignore
any bugs found by automation unless they
are reproduced by hand
Towo Toivola 2009

71. Furthermore
Developing untestable software is
ruinous anyway, stop doing it
There are N people changing the
software, how will n TA people keep up?
Who investigates the problems found in
TA?
Who fixes?
 The bugs in software?
 The changes needed for TA?

72. Question #8
Resourcing TA in the long run?
(Who will make sure TA keep generating value
next year and the one after? How? Changing
software! Maintenance hell!)
1600

73. Group Work Time
Firstquickly by yourself
Then within your group
Document
Be prepared to present

74. Results and Discussion for
Question #8
Resourcing TA in the long run?

75. Maintenance
If you get your tests running fine, don’t
think that’s most of the work done
The rest of your R&D works constantly
– to change your software
Maintenance is in top 2 killers of test
automation
◦ The other is organizational issues
You must be able to maintain your test
code, constantly and easily
Towo Toivola 2009

76. Which Bit Would You Like to
Maintain?
Towo Toivola 2009

77. How the Math Goes..
You invest 3 people for automation
You invest 2 months for finding the tool
You invest 6 months for developing a good
suite of tests
Changes in the application make the tests
unreliable, so they will not be run, the
organization will not believe in bugs found
by the system
You just lost more than 3x(2+6)x5000e =
120 000e worth of investment
I did not include the price of the tool..
Towo Toivola 2009

78. What Should We Learn
Implementing a test automation system
is implementing a constantly supported
service system based on software
Good system building and programming
practises must be followed
◦ Duplicate code is your arch enemy
Resources must be allocated to maintain
the system at all times
Towo Toivola 2009

79. Something I Have Learned Since
the 90’s
Model-based test automation technique
Importance of visual controls
Social mass is more important than
technical mass
Just do it
08/03/10 Towo Toivola 2009

80. Things we (probably) didn't have
time for..
Technicaldetails
Data-driven vs keyword-driven vs model-
based
Discussing individual tools
Discussing how to make software in
general
Much else..

81. Debrief of the Day
What do we remember from today?
Was this day useful?
◦ What needs to be improved?
◦ What was best?
What will you do differently in the future?
BRoA #53
If you learn something and
change nothing, you didn't learn
anything.
The topic is free, let’s have a discussion!
Towo Toivola 2009

82. Thank You for Your Time
We made it to the finish!
Towo Toivola 2009

83. Next is a bunch of reserve slides that may
come in handy at some point...

84. Robustness
Automation scripts must run reliably
When not testing something you should
perform operations in the most robust
way possible
Error handling logic
Duplicate code causes unreliability and
maintenance hell
Common operations belong to libraries
◦ Maintained, high-quality
Towo Toivola 2009

85. Libraries
Required
quality
Utility
libraries
(handful) Use-case
libraries
Amount (handful)
maintenanc Test drivers
e Test case
(handful) scripts
(dozens or
hundreds)
Testing
value Towo Toivola 2009

86. Robustness example
Towo Toivola 2009

87. Data-Driven
If automation can do something, it makes
sense to do it a lot
Loop over data, run combinations
◦ Example: Log in with 10 incorrect accounts
and many correct accounts
Towo Toivola 2009

88. Keyword-Driven
Basically a new language, just for your
testing
Reading in commands and data
instructions from a simple, human
readable format
Enables non-technical people to
understand and generate test cases
◦ Bring in the business knowledge
Requires quite a bit of work
Towo Toivola 2009

89. (a very simple) Example
Go: EM_mainscreen
Do: Enter message: “Hello work”
Go: Logout
Go: EM_mainscreen
Do: Verify message: “Hello work”
Towo Toivola 2009

90. ATDD
Acceptance/automated test driven
development
Agree on the test first, on a level
understandable for all
Keywords implement the test
Software is developed to meet the test
Highly advanced development /
automation method
Towo Toivola 2009

91. (a very simple) Example
Given at effort manager login
screen
When login with incorrect
account
Then incorrect login reported
Towo Toivola 2009

92. Model-Based Testing
Creating a state-machine that models
(some parts) of your software
Verifying the software behaves according
to the state-machine
Combining transitions and data in many
ways
◦ Different algorithms exist
◦ Random is pretty effective
Challenging, very advanced
Towo Toivola 2009

93. (a very simple) Example
State: emanager-login-screen
Input: login-incorrect
Resultstate: badlogin-complaint
Input: login-correct
Resultstate: emanager-mainscreen
State: emanager-mainscreen
Input: logout
Resultstate: emanager-login-
screen
…
Towo Toivola 2009

94. Automatic Testing
Big difference between computer aided
testing and automatic testing
Automatic is automatic, human
intervention is not needed
◦ Starts automatically
◦ Runs automatically
◦ Analyses results automatically
◦ Reports automatically
Could your tests do that?
Towo Toivola 2009

95. Exercise: Automatic
Make your tests run so that they can be executed
automatically
• Cmdline usage of TestPartner
• Driver Script
We should have about a half an hour.
Towo Toivola 2009

96. Reporting
How would you like to get the test results?
Opening the tool?
Going to the DB?
By email?
On a web page?
In a file?
As SMS?
In public view?
Towo Toivola 2009