The document summarizes the author's experience with test automation over the past 15+ years, from 1999 to present. It describes the evolution of test automation from (1) record and playback tools, which had maintenance issues, to (2) data-driven test automation and reuse of generic functions to reduce maintenance, but challenges remained. The author then developed (3) a generic, technology-independent test automation framework at a large insurance company project to further improve maintenance and reuse across systems.

1. ISSN 1866-5705 www.testingexperience.com free digital version print version 8,00 € printed in Germany
16
© iStockphoto.com/pidjoe
The Magazine for Professional Testers
The Future of Testing
December 2011

2. The Future of
Automated
Testing
by Bernd Beersma
A Silver Bullet?
The future of testing? Where will we go with testing? Since my
main focus is on test automation, this article will describe my vi- In 2001 I started at the Computer Management Group (CMG) in
sion on the future of automated testing. the Netherlands as a software tester, adopting the TestFrame™
method as the method for designing test cases. During my first
To understand where the future in automated testing will be, we weeks at CMG I followed courses on test case design and, auto-
first have to understand where it was. So let’s go back to the fu- mated testing with a record and playback tool from a well-known
ture… tool vendor. I learned how to use the record and playback tool to
automate test cases. After that first encounter I rapidly got ac-
We will take a walk down memory lane, and we will do so follow- quainted with other automated testing tools in the field of record
ing my experiences with test automation from 1999 until now. and playback. All of you that were interested can figure out which
tools I used at that moment. Those tools were similar to each oth-
First Contact
er and could create test cases in a quick and easy to understand
I first came in contact with testing and test automation during an way, generating a large number of test cases. But was this the
internship at a hard and software production company in 1999. I silver bullet? No it wasn’t, and the future was not even close at
was on this project for creating a web based interface for config- this moment.
uring a Sun Solaris system which was the beating heart of a com-
The First Generation, Record and Playback
plex hardware solution. Developed in Perl I created this interface,
but being a software engineer I believed that everything I created The so called first generation of automated test tools consisted of
was working fine. One day during lunch I talked to two of the Java record and playback tools. This basically worked like a VCR (ok, for
developers and they told me I should test my just developed soft- our younger professionals a DVD or HDD recorder). A tester needs
ware, so I could verify if it worked as requested. They introduced to record his or her actions on the system under test (SUT). The
me to a software testing method from The American Department recording tool will record this and create an automated test script
of Defense (DOD) and showed me how I could use this to test my of those actions. After this session, the tester can play back this
software. Ok, now, I wouldn’t think about testing my own soft- just created script and the tool will interact with the SUT simulat-
ware, but at that moment it was the best I could do, because there ing the tester and the actions of this tester.
were no testers in the company.
The biggest problem was the maintenance issue. Because of the
So I tested and tested my software, setting up a test plan, creating static scripts created with the record and playback tools, changes
test cases etc. and of course found some defects. During another in the system under test or even test data challenged the test
lunch I asked the Java developers how they use testing as part of automation engineers. This meant changing the automated test
their development process. They said they had to run thousands scripts which is okay for one script, but what if the login sequence
and thousands of tests for their embedded software, but they did for your thousand plus test cases needed changing, or new ob-
it automatically. They programmed their test cases within the jects in your SUT are created and they are mandatory? At that mo-
embedded software and executed these test cases time and time ment you needed to adapt your automated test scripts to these
again, guaranteeing the software worked correct and as request- new situations, creating a lot of work for the test automation
ed. This was my first contact with automated testing, automated engineers.
unit testing.
So, what was the future for us, the test automation engineers?
After that internship I decided that testing would be a part of my Because this time consuming test automation certainly was not
life. Finishing my BA in Software Engineering I always used the the best solution for test automation.
DOD standard to test my software. Although it still was manually,
it was more testing than most of my fellow students did.
116 The Magazine for Professional Testers www.testingexperience.com

3. The Second Generation, Data Driven Test Automation The Second Generation, a Generic Technology Independent Test
Automation Framework
As I learned in my next assignment for CMG at a government or-
ganization in the Netherlands, the solution was data driven test After leaving this project I was convinced this was the approach
automation. This basically meant that you take your test data for test automation. And so I used this framework and approach
out of your automated test scripts and place it outside your tool. for many years at several projects and customers until I started a
This could be an Excel, a CSV or even a plain text file. Storing your project at a big insurance company.
test data outside of your tool meant easier maintenance on your
test data and faster creation of additional test cases. For the test Becoming the lead automation engineer I was responsible for
automation engineers this meant less maintenance on test data, setting up test automation on a large scale and I was convicted
but still the maintenance problem was the biggest challenge for we (me and my colleagues) could improve the framework we had
successful test automation. That’s why the data driven approach been using for years and years. This led me to introducing an ap-
wasn’t the future of test automation, we were getting ahead, but proach which I described in my previous article in Testing Experi-
still had a long way to go. ence 14, Improving automated regression testing.
The Second Generation, the Reuse of Generic Functions
This approach is based on Generic Test Automation and Generic
In addition to this data driven approach, the logical next step was Test Case Design and is technology independent. This basically
to create functions of repeatable or generic parts of your test au- means that it doesn’t matter which system or application you
tomation software. The project I worked on did both and it was want to test. Interacting with the SUT is always the same, because
based on the action or keyword principle of TestFrame. it doesn’t matter if the screen you need to fill out is an HTML ap-
plication or an SAP screen. If we look at the business logic, the
Let’s take a closer look at creating generic functions as part of path we need to test is described as:
test automation. I mentioned earlier the login sequence needed
A. Search for the book “Implementing Automated Software
in a lot of test scripts. This login function is a good example of a Testing” by Elfriede Dustin (Screen A)
test step which can easily be transformed to a generic function.
If you transform this recorded login sequence into this function B. Place the book in your shopping cart (Screen B)
(Function Login(User, Password)), it means that you can use this C. Go to the check out (Screen C)
newly created function everywhere a login is needed, just refer to
D. Review your order (Screen D)
Login(Bernd, Bernd77) and it works with the credentials you send
to the function. Slashing maintenance, because if something E. Confirm your order ( Screen D)
changes in this function, we only need to change the function
and the login will work again for all the test steps where it is used. So these are five steps on a business level, but only four steps on a
technical level, screen A until screen D, but more importantly, they
Again, one step closer to the future, but we aren’t there yet. are all screens; screens with different objects, different checks
and different actions, but still screens.
The Second Generation, the Use of Action Words
Back to the action words: An action word describes an action on As we have been building generic functions for objects like an HT-
your system like “Enter field Name” in combination with the val- MLEditbox or an SAPTextField and putting them into libraries for
ue “Bernd” or “Press button” with the value ”Enter”. These action years, the next step in test automation took shape. Because the
words correspond with a function in your automated test script only difference between HTMLEditBox “Name” and HTMLEditBox
library and are executed on your SUT. “Address” is the technical information needed to steer this object
in the SUT. So our libraries where full of functions to steer differ-
So these generic functions of action words soon led to a test au- ent applications and environments. The only thing that was left
tomation framework which was used to create automated test to do was to identify the technical information and put this in
scripts more easily. The testers would create their test cases in an external file, in our case an XML File. This could easily be done
Excel and the test automation engineers registered those ac- by the automated testing tools we used at that time, commercial
tion words in their framework; this resulted in a big decrease in tools from big vendors or open source tools. For a further detailed
maintenance. However, this still wasn’t ideal and we needed to description of this framework, I would like to refer to my previous
improve our framework. This resulted in so called functional ac- article.
tion words. Again one step closer to the future, but still not there.
I believe this Generic Test Automation framework is a good ap-
The Second Generation, the Use of Functional Action Words
proach for test automation, and stands between the second gen-
The functional action words were action words which consisted eration of test automation and the future. It brings back mainte-
of a static part “Fill Screen Personal Information” combined with nance to a minimum and lets testers create test cases in a logical
all arguments used to fill this screen like “name” and “address” and easy to understand way, but still, you need test automation
with the corresponding input values and often an action to per- engineers with technical knowledge and programming skills.
form “press next”. So now a tester did not have to use multiple And these are not skills every tester possesses of even wants to
action words to fill in a screen, but could use the action word “Fill possess.
Screen XXX”. Again this led to a decrease in maintenance and test
The Next Generation
cases were also easier to create. If we speak of generations of test
automation, I believe this was the second generation of test au- This leads us to the future of test automation and what does it
tomation, record and develop test automation. However, we still look like? So I have been around test automation for several years,
use the same tools as before, only we use them in another way, so I worked for a large Dutch test company, a worldwide tool vendor
this is still not the future of test automation. and now a smaller Dutch test company. I had my share of tools
and still come across new tools very often. So where is the future?
www.testingexperience.com The Magazine for Professional Testers 117

4. I think it’s in different areas like scriptless automation, user
friendliness, standardized integration between tools and the > biography
mindset of combining different tools to fit your needs.
Bernd Beersma
First of all, I see new, upcoming and promising tool vendors with is competence leader test au-
tomation and senior test con-
an innovative approach on test automation. The new approach
sultant with Squerist. He has
is scriptless automation, so functional testers can set up auto- over 9 years experience with
mated test cases without the technical knowledge or the need for different forms of test auto-
programming skills. Tools that are born from the need to create mation and performance test-
an approach by which testers can setup test automation in a sim- ing for different companies.
ple and logical way. I like to use the phrase “Created by testers for Bernd holds a bachelor de-
testers”. And I believe there will be more and more vendors who gree in software engineering
will try to go this way. and became acquainted with
software testing during this
period.
Another thing I hear very often is that automated test tools are
During his numerous customer assignments Bernd creat-
too complex. The majority of testers would like a test tool that is ed different frameworks and solutions for test automation
easy to use, adaptive and comes with a low learning curve, so they with a broad variety of tools. These different approaches
won’t need months of tool training to get started using this tool. led to creating a generic approach for test automation.
So in the future we need tools that have these characteristics. As a result of his knowledge about test automation, he also
gives advice on how to implement test automation and
Furthermore, what the future of test automation needs is stand- does workshops and trainings.
ardized tool integration. Why is it that every vendor creates its Bernd is a member of the TestNet Test Automation work-
own interface for test cases, defects, requirements etc.? It would group and co-initiator of the Test Automation Day.
His goal is to keep on learning and thus improving his skills
be easier if we could use tool A for requirement management,
on test automation.
tool B for automated test execution and tool C for defect man-
agement because these are the best tools for the job. Also when
we decide to use another tool, it’s a pity we need to migrate our
automated test scripts, because they cannot be used by this new
tool. Ok, I must admit a lot of closed source products do interact
on a minor scale with each other. One vendor has developed a
platform on which we can use closed source, open source or even
homebrew. But still, there is a long way to go.
Finally, the future of test automation needs the mindset to use
multiple test tools and not wait for the silver bullet, because it
won’t come. I came across a lot of projects and customers who
still think there is one single test automation solution to solve
all their problems. I am sorry, but there simply isn’t. I believe you
need to choose the tool which best fits your problem or needs. If
you are a web developer and need to run an automated test each
daily build, maybe an open source webtest tool is the tool for you.
If you are a functional tester on an SAP system, you may want to
set up your automated regression test with a commercial tool.
This is not only applicable for test automation tools, but also for
other tools which support your test process like defect manage-
ment or requirement management.
Looking back, we came far from the early days of test automation,
but there is still a long way to go to use test automation on a wide
scale. And we as testers take part in creating this way, together
with tool vendors, open source communities, and automation
enthusiasts.
So, my final conclusion on the future of test automation:
The future of test automation is now, and you’re in it…
118 The Magazine for Professional Testers www.testingexperience.com