An AutomAted testing institute Publication - www.automatedtestinginstitute.com
. AuguST 2009 $8.95
of AutomAtEd SoftwArE
evolutionary 5 StepS to Framework
inTroduCing your ProjeCT How To CreATe A FrAMework
To new TeCHnology For MAinTAinAbiliTy
August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 1
Give me the key...worD: Building A Keyword driver Script in 4 StepS
Still Testing SAP Manually?
Worksoft Certify® Get ahead of the curve and maximize the value of
your SAP investments with Worksoft Certify®, the
for SAP® only code-free approach to end-to-end,
cross-platform business process validation and
Automate automated testing.
Accelerate Worksoft Certify® for SAP® is a specialized
lifecycle management solution for SAP that
Optimize automates and optimizes SAP lifecycle tasks from
change impact analysis through functional testing,
performance testing and audit documentation.
Certify for SAP is unmatched in the industry for
accelerating time to value and reducing overall
project costs by up 40-80%.
Are you ready to test smarter and deploy faster?
Contact us at 1.866.836.1773 or visit
www.worksoft.com to learn more.
August 2009, Volume 1, Issue 2
Evolution of AutomAtEd SoftwArE tESting 14
Reviewing how test automation and its tools have evolved is useful only if it helps to understand where we go from here.
To do that, you must appreciate the technology and market forces that have shaped them and why, and what that portends
for the future. By Linda G. Hayes
EvolutionAry roAd 22
Software technology changes in existing applications often present roadblocks on the road to successful test automation.
Learn what steps will help you navigate your test automation over and around these barriers. By J.L. Perlin
5 StEpS to frAmEwork dEvElopmEnt 28
Developing an automated test framework can greatly increase the ROI of your test automation efforts, but it can also be a
daunting task. Read this article for a step-by-step framework development process. By Dion Johnson
ColumnS & DepartmentS
EditoriAl 4 hot topicS in AutomAtion 38
evolve, adapt and Build microblogging is hot!
Keeping automation on pace with three modes of change. Test automation meets microblogging.
AuthorS And EvEntS 6 i ‘B’log to u 40
Learn about AST authors and upcoming events. Read featured blog posts from the web.
SwEAt thE tEchniquE 8 up-to-dAtE with Ati 42
Find out what’s happening on the Automated Testing
hand me the key...word
Institute’s Online Reference.
Learn how to build a keyword driver script.
Automated Software Testing (AST) Magazine is an Automated
opEn SourcEry 12 Testing Institute (ATI) publication that serves as a companion
Contribute to open Source projects in 4 Steps to the ATI Online Reference.
Learn how to contribute to your favorite open source For more information regarding the magazine visit:
August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 3
Evolve, Adapt and Build
by Dion Johnson
For anything to survive, it must be able
to change itself by three different ways:
evolving, adapting and building. This has
been evident in the natural world, and it also
holds true in the world of software. This
reality should be especially clear for software
testers and test automators, because the very
mission and reason for their existence is to
ensure systems are effectively evolving,
adapting and being built. This conceptual
trilogy is not confined, however, to only the
systems tested using automation, but also
applies to test automation itself.
Evolution is defined as a gradual process to aid in this understanding, this issue of the frameworks. In addition, Hayes describes
in which something changes into a different Automated Software Testing Magazine is where the automation evolutionary approach
and usually more complex or better form. dedicated to exploring each of these modes has come up wanting, and proposes what the
This change is normally an innate response of change. next test automation evolutionary response
to some measured environmental stimulus In the cover story, Linda Hayes delivers should be.
that occurs over time. Adaptation is the act of what is possibly one of the greatest articles Next, J.L. Perlin provides a featured
adjusting oneself to perspective on a
or a shifting ...possibly one of the greatest modern day test
type of change is articles ever on the topic of the test with a peek into a
typically marked with contemporary
by a deliberate and changes to the
modification in the environment. Building ever on the topic of the test automation In this article, an existing software application
is the act of increasing or strengthening, and evolution. Titled “The Evolution of with a substantial suite of automated tests is
may be a response to some outside stimulus, Automated Software Testing”, this article faced with changes to existing application
but it could also be based on a self-imposed looks at software test automation from a components. These changes threaten to render
decision to improve one’s situation. macroscopic point of view and explores the all existing automated tests useless as the
The discipline of test automation has gradual changes in software systems over automated test tool contends with problems
experienced and been shaped by each of the years. It takes us through a trip down communicating with the new components.
these modes of change in the past, and software computing memory lane, moving us This is a problem that many automators have
continue to be defined by them even today. from mainframe to Internet domination, and come across, and if you haven’t, there is
Understanding these three modes of change explaining how the test automation discipline a good chance that you will. So J.L. Perlin
will help to better understand the current state has responded to each evolutionary shift by offers an approach for effectively navigating
of test automation, and will therefore help moving from text capture/playback tools your test automation through such modern
us all to chart a course forward. Therefore, to the broad use of commercial automation day technological changes.
(Continued on page 37)
4 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009
How Do You Network?
Whatever your choice, the Automated Testing Institute is there!
Stay up-to-date with test automation by following the Automated Testing Institute on your so-
cial networking site of choice.
Facebook Myspace Twitter YouTube LinkedIn Blogger
For more information, visit http://www.networking.automatedtestinginstitute.com
August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 5
authors and events
Who’s In This Issue?
linda g. hayes, in the lead feature, offers an AutomAted
eyewitness account of test automation evolution. With over
20 years of experience in software development and testing, SoftwAre teSting
Linda is the founder of three software companies including
AutoTester, the first PC-based test automation tool, and Managing Editor
Worksoft, Inc., pioneer of next generation code-free test Dion Johnson
solutions. Linda holds degrees in accounting, tax and law
and is a frequent industry speaker and award-winning author Contributing Editor
on software quality. Named one of Fortune Magazine’s People to Watch and one of the Edward Torrie
Top 40 Under 40 by Dallas Business Journal, she is a regular columnist and contributor
to numerous publications, including the Automated Software Testing Magazine, Director of Marketing and Events
StickyMinds.com and Better Software magazine. She is the author of the Automated Christine Johnson
Testing Handbook, contributing author to Testing SAP R/3:A Manager’s Guide, and co-
editor of Dare to be Excellent with Alka Jarvis on best practices in the software industry. A publICATIon of ThE AuTomATED
Her article “Quality is Everyone’s Business” won a Most Significant Contribution
award from the Quality Assurance Institute and was published as part of the Auerbach
Systems Development Handbook. You can contact Linda at Lhayes@worksoft.com.
J.l. perlin offers a featured article that provides an
approach for handling modern day technology changes.
Perlin has over 20 years of manual & automated testing ConTACT us
experience and has managed QA teams in both the AST Magazine
public and private sector for organizations including email@example.com
Lockheed Martin, Social Security Administration
(SSA), General Electric, and the Centers for Medicare & ATI Online Reference
Medicaid Services (CMS). Mr. Perlin is currently a lead
test automation engineer working on critical care systems for Philips Patient
Monitoring Informatics. You can email Mr. Perlin at firstname.lastname@example.org.
ATI and Partner Events
dion Johnson provides step-by-step instructions
for building an automated test framework in which you August 17
may develop and maintain your automated tests. Dion has ATI honors finalists Announced
several years of experience in providing IT services to http://www.atihonors.automatedtestinginsti-
both government and private industry in a manner that has tute.com
demonstrated expertise in multiple areas of the software
development lifecycle. With a Bachelor of Science August 17 - 31, 2009
degree in Electrical Engineering, he has spent much of ATI honors Voting period
his professional career as a consultant in the areas of quality assurance (QA), quality http://www.atihonors.automatedtestinginsti-
control (QC), software process improvements, requirements analysis and software test tute.com
automation. As a regular conference speaker and presenter, Dion has delivered award
winning and highly acclaimed presentations at many of the most prestigious industry
conferences, including the StarEast and StarWest International Conference on Software
Testing, Analysis and Review, the Quality Assurance Institute Conference and Better
Software Conference. He also teaches college and business level classes relative to
testing and test automation, and has several published articles in various IT publications.
6 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009
August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 7
Sweat the technique
hand me the key...word
Building A Keyword Driver Script In 4 Steps
Often called “Table Driven”, a Keyword advantage is that it is easier to read keyword
framework interprets automated tests that files than traditional scripts. Keyword files
are developed in a tabular format using typically mirror manual test procedures, and
a vocabulary of keywords. A keyword is the natural language of the verb-like phrases
typically a verb or verb-like phrase that is makes reading a keyword file similar to
associated with an external library function reading a collection of sentences. Finally,
responsible for executing the application while the Keyword framework itself may
commands necessary to satisfy the objective be more technical, the keyword files created
of the test. during everyday application automation are
Therefore, in a keyword framework, a a lot less technical than files with statements
script that may normally appear as illustrated written in code. This allows less technical
in Figure 1 will instead appear as illustrated individuals to effectively contribute to
in Figure 2. automated test creation.
These advantages don’t mean that we’ve
escaped scripting altogether, however. We
haven’t evaded scripting; we’ve merely
changed the nature of the scripting that
needs to be performed. There are at least
two forms of scripting that remain: function
because many of the functions are created development and driver script creation.
in such a way to not only be reusable for If you purchase a commercial keyword
multiple tests within a single application, but framework or obtain an open source keyword
also for tests across multiple applications. framework, you may eliminate the need for
Redundancy may therefore be decreased custom development of the driver script and
Figure 1: Code-based interface
across all applications that an organization may also reduce the function development to
may be responsible for automating. Earlier only those functions that are specific to your
Figure 2 illustrates a tabular automated script development also results from using application and necessary for more efficient
test known as a keyword file, with the this type of framework, due to an increased automation. If you choose to create your own
keywords that dictate the actions to be ability for automation to begin before the framework however, the script development
performed shown in the Action column. application is delivered. Using information is solely up to you.
There are several advantages to using gathered from corresponding manual test This article will focus on the development
keyword files for automated test development cases, requirements or other documentation, of the driver script that interprets a keyword
including increased reusability. Reusability is keyword files can be created well before the file. Following are steps for successfully
increased with a Keyword Driven framework, application is ready for automation. Another creating and implementing a keyword driver
Figure 2: keyword Driven interface
8 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009
script: about the step to anyone reading the application under test
1. Create the keyword file structure keyword file.
2. Identify potential keywords Step 3. IdentIfy Keyword utIlIty
3. Identify keyword utility functions Step 2. IdentIfy potentIal KeywordS funCtIonS, argumentS and return ValueS
4. Create driver script The keywords created will largely depend Function definitions are a product of the
on the functions deemed necessary for framework and of the automator’s personal
Step 1. IdentIfy ColumnS In Keyword fIle implementation of the associated framework. preferences for how the function is to be
The keyword file columns will vary from Not all of the keywords will be identified implemented. One of the keys in developing
framework to framework. The only constant during framework design and development, an effective automation framework utility
The Driver ScripT
callS funcTionS baSeD
on The KeyworD in The
Figure 3: Driver Script pseudocode and keyword File interaction
is that a column will almost certainly exist but will rather continuously become apparent function, however, is to make sure it is
for keywords to be identified. In addition, as tests are automated. If developed properly, independent, reusable, and able to be
the keyword file may have one or more of the keyword framework will be seamlessly implemented as intuitively as possible.
the columns illustrated in Figure 2. These scalable with respect to the addition of new Function arguments play a big role in
columns are defined as follows: keywords over time, but there are some ensuring your functions met these criteria.
• Action – Contains the step’s main generic keywords that may be identified For example, a function associated with the
keyword, and identifies the action that upfront, including: INPUT keyword might need the following
will be executed • INPUT – Inputs a single data item arguments:
• Screen – Contains the name of the screen • PRESS – Presses or clicks a button or • Screen – This argument contains the
on which the action is to be performed link screen on which the input action is to
• Object – Contains the name of the object • VERIFY_SCREEN – Verifies a screen occur
on which the action is to be performed has appeared • Object – This argument contains the
• Value – Contains values that may be • VERIFY_FIELD_VALUE – Verifies object on which the input action is to
entered into the application entered of selected field data • Value – This argument contains the value
• Recovery – Contains Exception • VERIFY_TEXT – Verifies dynamic to be entered into an object
Handling keywords. screen text These arguments correspond to columns in
• Comment – Contains helpful information • INVOKE_APPLICATION – Opens the the keyword file shown in Figure 2, therefore,
August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 9
The ATI Newsletter keeps
you abreast of the latest
information relative to our
Online Reference. Sent to
registered members of the
site on a bi-weekly basis,
this newsletter keeps you
informed on the follow:
ATI’s featured Content
• Featured Videos
• Featured Tutorials
• Featured Articles
• Featured Tools
• Featured Publication
• Featured Poll
• Featured Forum Post
• Popular Tools
• Popular Articles
• Interesting News
• Current Events
Figure 4: Sample Driver Script
the arguments will be fed values from these If the driver script is developed using Ruby, it
three keyword file columns via a Driver may appear as shown in Figure 4. Register on the ATI site today to
Script. Line 1 loads the UtilityFunctions.rb file. receive this email newsletter. To
This is a file you develop that contains your register visit:
Step 4. deSIgn and deVelop drIVer SCrIpt user defined utility functions.
As illustrated in Figure 3, the driver script Lines 3 through 7 are responsible for
is responsible for calling the keyword file, setting up and initializing the keyword file
reading all of the steps, and executing utility which is developed in an Excel spreadsheet.
functions based on the keywords in each step Line 9 begins the loop that reads all of
of the keyword file. the rows in the keyword file.
(Continued on page 37)
10 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009
Tired of searches that
return irrelevant information?
google - AutomAted teSting SeArch engine
The Automated Testing institute has partnered with google to create a software test
automation search engine. if you’re looking for software test automation information, you
will get the best results by using this search engine, because it only searches the sites
that matter most to automators. you can also restrict searches to blogs and forum sites.
August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 11
Ask not what your tool can
do for you
Learn to Contribute to Open Source Projects in 4 Steps
Ask not what your tool can do for you. interested in automated software
Ask what you can do for your tool. testing or interested in one of the
My fellow automators, how often authors writing about automated
have you pointed out the shortcomings software testing. Either way, this is
of tools with which you have worked? a good indicator that a project that
Now, how often have you sought to maintains an automated test tool may
take an active role in fixing these be a good fit for you.
shortcomings? Aside from simply picking a project
I’m sure many of you are thinking, because you like it, you may also
“Fixing tool shortcomings is not my want to pick a project that is modest in
job! I use these tools to help me better size; particularly if you are a beginner
perform my job. If I spend time fixing to open source development. Smaller
tools, then I won’t have any time to do projects typically have smaller code
my real job!” bases, provide easier access to other
I hear all loud and clear, and you developers, and have simpler bug
make good points, but while fixing you on how this may be accomplished. In our submission processes. Therefore,
tool shortcomings is not your responsibility, research we came up with 4 steps. smaller projects may make it easier for you to
it does have several benefits. Fixing tool quickly become productive.
shortcomings is normally not an option with Step 1 – ChooSIng a projeCt Another option is to start your own open
commercial tools, but the beauty of open There are numerous open source projects source project. Maybe there is a browser
source tools, is that it is very much a viable extension or automated test tool plug-in that
ready and willing to receive contributions
option. you’d like to see created to support automation
from eager scripters, and the project list
The May 2009 issue of the Automated of a particular technology; if so, you can lead
grows with each passing day. Following are
Software Testing Magazine contained an the effort in developing the desired extension
links that identify and provide access to some
editorial by Dion Johnson called, “Scripting or plug-in. Just search around first and make
of these projects:
Calisthenics”, and this article presented sure a project doesn’t already exist for what
various options for keeping automation you intend on doing. Not sure how to create
skills sharp during periods of inactivity. • www.sourceforge.net a project? Some pretty good instructions are
The article was referenced in various • www.opensourcetesting.org laid out at ehow.com (http://www.ehow.com/
automation Internet forums with a question • www.code.google.com/soc/ how_2091738_start-open-source-project-
posed to the forum’s community that sourceforge.html) for creating an open source
asked, “How do you keep your skills The challenge is now in determining project with SourceForge.
sharp during periods of inactivity?” One which project you should contribute to. The
of the community’s users suggested that simplest way to pick an open source project Step 2 – pICKIng your ContrIbutIon
contributing to open source projects is a good is to look for one that maintains a tool that Now that you’ve picked a project, the next
way to keep scripting skills sharp. We here you use and/or are interested in. For example, step is determining what you’re going to
at the Automated Testing Institute thought given that you are reading the Automated contribute to that project? You may begin by
this was an enterprising answer and decided Software Testing magazine right now, it asking yourself what shortcoming you’d like
to research and provide information for all of may be safe to assume that you are either to strengthen, and determining if it’s going
12 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009
to be a ‘fix’ or a ‘feature’? In other words, important to do it with a productive tone. or it may be as complicated as modifying
do you want to fix an existing bug, or do you One way that you’ll be able to communicate several code modules. However extensive
want to add a new feature? At this point in with people on the project is by joining the your scripting task, be sure to use the code
the process you may have already decided project’s chat sessions and/or mailing list if conventions defined by the project, including
what you want your contribution to be, and they have one. Mailing lists are often used conventions for indenting, variable naming,
that decision may have played a big part in for discussing recent code changes or to and commenting.
the project you chose to join. If not, however, propose new code changes, so they offer a Upon completion you need to post
there are some tips you your changes to the
can use for picking a
contribution. ...if key players on the project don’t project, a task that
The first and most
obvious thing to do is like the ‘new guy’, there is a good differently
chance that your proposals will get
to actually use open Some projects
source tools. There are submit changes via
a bunch of them that the mailing list,
perform a wide range rejected and sometimes the
of functions. If you project uses a version
work on a computer control system.
at all, you should have no problem finding at great avenue for proposing your changes and Whatever vehicle you use for sharing your
least one open source tool that can help you soliciting feedback. Making your intentions update, that update is normally presented in
with some of your tasks. Pick one, and then known to the key group members early will the form of a patch. A patch is a small piece
identify some of the features you’d like to see improve the chances of your change being of software that contains your fix or new
changed. What you come up with, could be accepted, or it may allow you to receive feature for a computer application. One way
your contribution to the project. feedback explaining why your proposed to create a patch is to run what’s called a diff
In addition, some projects maintain change should be reconsidered. program. The diff program compares original
readily available “to-do” lists that identify Using the bug system to study existing files or directories against modified files
items that need to be fixed. Also, the project’s bug reports is also a great way to learn the or directories, and produces an output file
bug reporting system can be used for culture of the project, because the reports called a patch that contains the differences.
gathering ideas for what to fix. indicate who the key developers are and what Upon submitting the patch, be prepared to
types of contributions normally get accepted. make changes as suggested by the project
Step 3 – learn the projeCt Culture Aside from the bug system, you’ll want to maintainers. For information on creating and
Much like countries and commercial appropriately utilize other project tools, such submitting a patch, visit ehow.com (http://
businesses, each open source project has as the version control system. Developers www.ehow.com/how_2091741_make-patch-
its own unique culture, and your success in on these projects often have little patience open-source-project.html).
a project largely depends on your ability to for individuals that can’t seem to follow Before submitting your patch to the
learn and fit into that project’s culture. As rules, standards and directions. To obtain open source project, be sure you understand
much as you may like to believe success is information on tools and conventions used the open source licensing used by the project
only about technical abilities, it unfortunately by the project, locate and read the developer to ensure the patch is not rejected for legal
goes beyond that. No matter how good your docs. reasons, and to ensure any copyright you may
coding skills are, if key players on the project wish to impose on your developed code is
don’t like the ‘new guy’, there is a good Step 4 – Code and poSt preserved.
chance that your proposals will get rejected. At this point, it’s time to get down to the
It is, therefore, important for you to be business of developing your fix or feature, ConCluSIon
aware of who the core maintainers are, who and it’s imperative to begin by making sure Therefore, my fellow automators, if you
the key contributors are, and who the more you have the most up-to-date project code. find yourself complaining about what your
vocal members of the project are. You’ll also Next, search the code for the location in test automation tool is not doing for you,
want to study how they communicate with which the change is to be applied – a task ask yourself, “What am I doing for my test
one another. This will provide insight into that is often simplified by using an Integrated automation tool?” If more people had this
who you may want to reach out to, and how Development Environment (IDE). Now you mentality, we’d all be a little more skilled,
best to reach out to them. It will be important can begin scripting. Your scripting may be and the discipline of test automation would
to make yourself known, but it’s equally as simple as changing a line or function, probably be a lot better off.
August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 13
of AutomAtEd SoftwArE tESt
An Eye Witness Account
by linda Hayes
14 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009
Test automation has a history of failure. Despite the explosion of
software into every aspect of our lives, only a small percentage of tests
are actually automated today. Not that there aren’t pockets of success or
flashes of promise, but after more than a quarter century of investment by
vendors, customers and consultants alike, manual testing predominates.
So reviewing how test automation tools have evolved is useful only if it helps to
understand where we go from here. To do that, you need to appreciate the technology
ting and market forces that have shaped them and why, and what that portends for the
future. For that reason, this article sets the context of each evolutionary period
within the state of software development and the applications market at the time.
Finally, test automation is a very broad topic, so when I say test automation I am
referring specifically to the automation of an otherwise manual test against a user
interface (UI). This does not mean that test tools cannot be used for non-UI testing,
or that there are not many other types of testing that are automated by a wide
array of approaches and tools, only that they are outside the scope of this article.
August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 15
Text Capture/Playback arose around dynamic content, like date and time stamps or frequently
changing data fields. Timing synchronization was also problematic,
especially as the user community expanded and performance
Mainframes dominated the early enterprise landscape. The mainframe
fluctuated. Whenever the system changed or ran slower than the
was a monolithic environment, closely guarded behind glass walls
original recording, context was lost and errors occurred.
and accessible only to a few. Computing resources were scarce
and expensive, so development processes were formal and highly
But the killer issue turned out to be maintenance. If you recorded
structured, requiring flow charts and extensive documentation before
100 transactions, there were 100 sets of the keystrokes and screens,
being committed to code. The majority of software was custom
so any change had a wide ripple effect. It was usually easier to re-
developed for back office applications such as accounting and
record the test than it was to fix it, which defeated the value premise
payroll. Annual release cycles were the norm, usually punctuated by
of automation and encouraged reversion to manual testing.
The first commercial test tools emerged as a result of the transition
So early on, maintenance loomed as a primary obstacle to test
from batch to online applications, when multi-user terminal access
automation success, with dynamic content and timing close behind.
first became available and manual testing became a necessity. These
tools recorded the keystroke inputs and character-based screen outputs
of a terminal session, then replayed the keystrokes and compared the
Computing soon spread to mid-range systems, but the real
breakthrough came with the announcement of the IBM PC. Personal
This approach was seductive. It sounded so simple: just do the test as
computers had been around for a few years, but these 8-bit platforms
usual but record it into a script file, then play the test back as needed
were generally perceived as truly “personal” computers used strictly
and compare the results. Unfortunately it was also deceptive. Issues
by hobbyists and a few propeller heads (the early term for geeks). But
with the imprimatur of Big Blue, PCs with 16-bit power entered the
corporate computing world.
Application development continued to expand on the larger platforms,
but software for PCs was initially limited either to productivity tools
such as spreadsheets or word processors, or to terminal emulators that
turned them into smart terminals. The DOS operating system sporting
a character-based interface, was single-threaded and severely limited
on memory and storage; the first machines didn’t even offer hard
drives, so functionality was constrained.
The increasing popularity of spreadsheets and word processors
propelled PCs into the corporate community, and the use of terminal
The emulators allowed them to replace “dumb” terminals. This drove
manual testing from the mainframe to the PC. Soon automation tools
Mainframe domination ibM PC Announced
Text Capture/Playback emulator drivers
16 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009
appeared for the PC, capable of driving these emulators and other PC- could use our tool because she was not a programmer.
based applications, and these began to replace the mainframe-based
capture/playback tools because they were cheaper and more powerful. Until she uttered those words it had never occurred to me that all
testers weren’t skilled at scripting. After all, Petroware had been a
One of these tools was from my company, AutoTester. Our tool was software company and we wrote code for a living. Although we sold
originally developed by my first company, Petroware, when we ported AutoTester as enhanced capture/playback/scripting, this was just a
our oil and gas accounting software from an early IBM desktop to euphemism for programming. And the stark fact of the matter is that
the PC. Designed as a “session recorder” by my brother to automate most manual testers are subject matter experts, not programmers.
his programming tasks, we quickly realized we could use the tool
for many purposes including demos, training, task automation and So in the end we traded one set of problems for another: we addressed
testing. But the more we used it the more we found we had to enhance the maintenance, dynamic content and timing issues but introduced a
it with additional commands to perform logic and branching as well level of skill and complexity that excluded the majority of our target
as to handle variable data. users.
Before long, we had developed a proprietary language around the I was devastated. Secretly convinced we had developed an essentially
recording that enabled the scripts to handle timing and synchronization, defective product that tormented our own customers, we started
make decisions based on results, and externalize data into files. Now a project to try to reduce or eliminate the programming aspect by
a single script could loop through 100 records from a file and execute generating scripts, and while we made progress it soon became
reliably regardless of system speed or dynamic content. Because it academic. Windows was born.
also provided more power and control through its scripting language,
increasing reliability while reducing maintenance, we consistently
replaced older capture/playback tools and handily won competitions GUI Capture/Playback/Script
for new customers.
The release of Windows brought even more opportunity but also more
Yet in spite of our early success, nagging concerns persisted. Even challenges. As a multi-threaded operating system it resolved many
though we thought we had addressed the maintenance, dynamic low level issues; the maximum memory limit exploded and resources
content and timing issues caused by capture/playback, we still were managed across sessions. Windows also employed an API layer
observed customers eventually reverting to manual testing. I was first that converted mouse events into operating system messages that were
puzzled then increasingly alarmed. What could be causing this? intelligible and returned screen contents as a mix of objects that could
be queried to return text and images. Recorded scripts became more
I got my answer from a customer who sent their test manager to our legible and reliable and the extra memory headroom enabled even
offices for training. She was ideal for her job: she had over 20 years of more advanced functionality.
experience with the application that was being rewritten and knew the
functionality inside and out. But after only a few hours of training, she Another dynamic entered the mix as well. Personal computers were
began to show signs of discomfort, and on the second day she openly becoming not only ubiquitous but operating platforms in their own
burst into tears. Horrified, I asked her what was wrong. She explained right. Instead of just smart terminals for mainframes or personal
that she was afraid she would lose her job since there was no way she productivity tools, they offered powerful standalone applications.
Client/Server becomes the internet Takes Center
gui Capture/replay Custom Frameworks Commercial Frameworks
August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 17
This transition from mainframes to personal computers caused a slow lane as scripters took over automation.
corresponding shift from a formal and ponderous development process
to a more flexible, free-wheeling approach. Mainframes were so Custom Frameworks
costly that few had access to them; personal computers were so cheap
and plentiful that the masses could afford them. This led to software The PC continued to gain computing power, moving to a 32-bit
companies being born in garages by kids barely out of high school or architecture, and Windows matured into a more stable and capable
operating system that was a virtual monopoly on desktop machines.
Client/server application development became the new norm,
...initial apparent success with re- allowing PCs to offload tasks from backend mainframes and servers
and provide users with ever-easier yet more powerful interfaces. The
cording messages instead of key- battle cry began to replace expensive mainframes with networks of
strokes was soon replaced by disap- PCs and servers.
pointment and failure... But the early days of client/server application development introduced
even more issues for test automation. Commercial development
toolkits became available, offering libraries of sexy new graphical
controls that augmented the standard Win 32 object class library.
college with no patience or need for the constraints of formality. Other component libraries also appeared offering everything from
calculators to calendars to spell checkers, allowing programmers to
The type of applications changed as well. Whereas previously most quickly assemble rich user interfaces with less and less effort.
applications performed back office tasks that were limited to a few
internal users, newer applications expanded their reach to front line The problem was that many of these new controls were non-standard,
operations such as order entry and customer service. Many of the so the standard Windows messaging layer was no longer applicable. A
original custom applications were replaced by commercial, off-the- custom control did not necessarily implement the same messages that
shelf systems and new development moved into areas that interacted the standard classes did. Testing tools that relied on the Windows API
more directly with core business processes. began to fail, diverting more and more automation time and effort to
dealing with custom controls and other non-standard behaviors.
As a result, development cycles began to collapse. Market pressure
began to dictate functionality and annual delivery cycles gave way to This new complexity raised the stakes. Now automation engineers
quarterly releases. More capability was needed faster and faster. This were thrust into the development environment, forcing them to
added even more demand for test automation, because manual testing learn and deal with the underlying implementation of the interface.
was less and less capable of keeping pace with changes and the risk Whereas previously one had only to be comfortable with using logical
profile headed north. constructs and data file processing, now it was necessary to delve
into object methods and properties and grapple with installing source
This new, more hospitable technical environment and market also hooks or writing test APIs. This effort siphoned time and resources
brought new competitors. Frankly, I was hoping that the latest market away from actually automating tests and into a struggle just to interact
entrants would know something we didn’t and show us how to assure with the application’s user interface.
that our customers were successful over the long term. But the newest
tools presented the same problems. Scripting became even more It could not have come at a worse time, as pressure on development
sophisticated to deal with ever more robust applications, and initial continued to increase. The new client/server applications penetrated
apparent success with recording messages instead of keystrokes was mainstream operations whose delivery schedules were driven by
soon replaced by disappointment and failure due to the skills and user demand and stability became even more important. Greater and
effort required to write and maintain ever more complex script code. greater functionality was needed and it had to be delivered faster and
better. Quarterly cycles soon gave way to monthly.
Windows didn’t save test automation.
More and more companies invested in automation to meet this demand
Presented with this reality, the most successful tool vendors and automation engineers began looking for ways to reduce the time
aggressively enlisted partners and consultants who could offload the and effort to code scripts. It became clear that test scripts were no
scripting effort from the manual testers. A new class of tester was different than source code and adopting reusable components made
created: the automation engineer. Whereas previously manual testers sense as a strategy to reduce development and maintenance. Thus
were courted with capture/playback, now they were relegated to the frameworks were born.
18 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009
Large companies soon found
evolution in thought themselves with multiple
Secretly tools and frameworks, some
convinced we had of which had been rewritten
developed an essentially more than once.
defective product that tormented Complicating this was the fact
our own customers, we started a that shrinking development
project to try to reduce or schedules and expanding
functionality meant more and
eliminate the more tests were needed. There
programming was simply not enough time
aspect... and people available to write,
maintain and support one
or more frameworks while
also developing, executing
and maintaining the tests
themselves. Eventually the
overhead became so heavy
that companies – again –
Originally frameworks provided a basic infrastructure for test reverted to manual testing just to keep up with delivery schedules.
execution, handling common tasks such as data file processing, error
and test logging and reporting. But they gradually became more Commercial Frameworks
robust, providing simplified interfaces that employed spreadsheets
and flat files to allow non-technical testers to construct automated By now the Web was taking center stage as a ubiquitous user interface.
test cases using pre-built script components. This introduced a role- This was both good and bad. It was good because it provided a basic
based approach that allocated tasks based on skill sets and enabled set of standards for access through browser APIs and HTML classes,
cooperation between automation engineers and manual testers. but bad because the ability to instantly deploy changes meant delivery
cycles could be daily. And, as static pages and controls gave way to
A number of framework types emerged, differentiated primarily by dynamically scripted ones, object names deteriorated into gibberish
how the functionality was decomposed into components. The most and behavior became unpredictable.
prevalent was referred to as “keyword” or “action word” based and
it entailed writing script code components around business tasks, Pressure to deliver automation faster soared. Several enterprising
such as “Create Order” or “Invoice Customer”, then allowing users automation engineers realized that they were constantly re-inventing
to invoke these from a spreadsheet and pass in data values. Others the same framework over and over and decided to commercialize
organized their components around screens or windows, such as “Add their solution and market it as a library of pre-built components. This
Customer Information” or “Delete Customer Information”. helped to accelerate the implementation curve and reduce the overall
costs. But depending on the type of framework, there was still a fair
These frameworks succeeded in reducing overall development amount of application-specific code required to be developed, usually
and maintenance while allowing non-coders to design and execute complicated by the custom control or dynamic content challenge.
automated tests more easily, but still required custom code. Test
designers could write linear test cases but could not make decisions One of these framework types – my favorite - was designed around
and control the flow of the test based on real-time execution results. object classes. Commonly referred to as “table-driven” or “class-
That still required code. action”, this approach sought to reduce or remove application-specific
code by using components at the object class level: For example,
But frameworks fell victim to their own success. When the architect “Press Button” or “Input Text”. A key advantage was that the same
and chief (often only) automation engineer left the company or moved object classes and actions could be used across applications that used
to another area, his or her replacement almost inevitably believed the same class library, allowing greater reusability with less code. It
it would be easier and better to rewrite the code left behind than it was even possible to share libraries across companies.
would be to maintain it. With no de facto standards, almost every
automation engineer adopted their own pet approaches and designs. As the amount of code that needed to be written continued to
August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 19
decline, the battlefield moved to the objects themselves. Virtually all • Test automation architectures will continue to shift from code
modern test tools rely on an inventory of the application’s testable to data. The only way to keep up with the blistering pace of
objects. Variously referred to as an “application map”, “GUI map” development will be to minimize or remove altogether the need
or “object repository”, this inventory contains a list of each object in for custom code and accelerate or automate the acquisition of the
the application’s interface including its name, class and the attributes application map.
used to identify it. This map enables an object to be added to a test, • Test automation penetration will continue to decline - unless
located and acted upon during execution, and provides a central point something changes. The accepted rule of thumb says that 30-40%
of maintenance in the event of changes. or even 50% of a project should be budgeted for testing, but that
assumed old-fashioned coding, not snap-together systems. There
But as previously noted, object inventories were becoming is no way that hand-crafted test scripts can keep up with high-
increasingly dynamic and unpredictable. Building the map was a slow speed application production.
Nonsense names, closed controls and unpredictable context will disap-
pear when developers are both accountable and rewarded for success-
ful test automation.
and tedious process, often requiring the engineer to navigate to each So what has to change? In my opinion, our only hope is for test
page and objects. But with dynamic pages creating scripted objects automation to be integrated into the development process. No
on the fly, using names that weren’t just incomprehensible but also application should be deemed complete without a test harness. Let’s
changing depending on user choices, session parameters or a host of make developers responsible for making their software testable instead
other factors, this was often an impossible task. Without a predictable of engaging in a constant struggle to compensate for their latest bad
and reusable object inventory, automated tests are unusable. Trying behavior. Nonsense names, closed controls and unpredictable context
to maintain them with daily releases was crazy enough, but if the will disappear when developers are both accountable and rewarded
attributes changed within a release without any predictability, it was for successful test automation.
Such a harness would provide a test object inventory and interface
So history shows us that every time test tools and implementation with defined methods and properties to be shared by developers and
approaches are enhanced to meet the latest challenge, new ones are testers alike. It would be kept current with all software changes and be
introduced. Vendors and customers both find themselves in a perpetual part of a holistic test process that includes automated unit, integration,
state of catch-up, seemingly doomed to repeat the past by continually system and user acceptance.
reverting to manual testing.
Test tool vendors needn’t become extinct; just shift their focus to
What’s Next managing data instead of code. Automation engineers needn’t become
an endangered species; just shift their focus to managing the object
inventory and interface metadata. But don’t react just yet; history also
The only way to break this cycle seems to be to anticipate the future
tells us that changes take decades. Even today there are companies
and be ready for it, instead of reacting to it. The harder question is to
who believe capture/playback is a viable automation strategy, though
know what’s coming. Since I don’t own a crystal ball, all I can do is we have known better for almost 30 years.
look at history and current trends to project what’s coming.
But until management sees the light and directs development to play
History says that: their part, we all need to continue finding ways to reduce the amount
of test code that must be written and maintained, work closely with
• Development will continue to accelerate. Software will development to correct testability issues and manage the impact
continue to insinuate itself into every aspect of our lives and of changes, and educate management that the historical ratio of
jobs, driving demand for more functionality faster. Companies development to test effort is outdated but that the extra investment in
will adopt packaged applications when possible. Developers automation is worth it. And keep trying.
will take advantage of increasing levels of abstraction to keep
The Automated Testing institute (ATi) conducted a
up, using code libraries, code generators, web services and
other middleware technologies to quickly assemble composite podcast interview with linda Hayes. To listen in, visit
20 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009
real adventures in automation
Submit your Stories!
Now’s Your Chance to be Heard
ATI i s here t
The Automated Testing institute believes that
one of the best ways to improve upon test
automation is to listen to the lessons learned by
others. So, if you have an interesting story that
you’d like to share with the world, contact us at
email@example.com to find
August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 21
by j.l. Perlin
22 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009
it is a natural occurrence for products to evolve over time. Advancements in technology and
consumer demands spur radical changes in design that make products more efficient, easier to use,
and less costly to produce. How can you prepare test automation to best handle these changes? it
is becoming increasingly essential to plan for change when deciding upon an automation approach.
in order to do this it helps to understand why these changes are occurring so we as test automation
engineers can understand the catalysts behind these changes and be more prepared not only for
these new technologies and their effects on automation, but also for the interactions we will have
with the organizational stakeholders who have introduced these new improvements.
Evolution of a
So, why do products evolve?
let’s use the automobile as an
example. The first self-propelled
road vehicle was a steam-powered
military tractor invented by the
French engineer Cugnot. it was
a ground-breaking invention, but,
alas, due to low speeds (2.5 miles
per hour), operating dangers, and
the need to stop every ten to fifteen
minutes to build up steam, it was
not an improvement over the horse speed of the vehicle was still slower to the introduction of hybrid vehicles
and buggy. than the horse and buggy, the and all-electric cars, with steps
The next attempt at making range of the vehicle was very short made to bring us closer to the day
a successful automobile was in (about 15 miles before a recharge when vehicles we drive every day
the early 1800’s when a dutch was required), and the cost was can run on hydrogen. but, why
engineer used an engine that ran on exorbitant and out of reach for most did the motor vehicle evolve in this
rechargeable batteries to design the people of that era. manner? what was the catalyst for
first electric vehicle. The vehicle was Fast forward to today, and new improvements and changes
initially successful due to a smaller we see that over the years cars to the overall design? it is the
and lighter weight engine that have become much more fuel same underlying factor that causes
allowed the vehicle to travel faster efficient, safer, and easier to drive. other products to evolve, including
and further than its predecessor. improvements to electric and computer systems and software,
despite the improvements, the top- combustion technologies have led and that factor is market demand.
August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 23
How technology Evolution efficiently improves upon the proverbial forward and the number of products that have
Affects Test Automation horse and buggy. Changing major or even been made available to development teams to
Technologies of the Information minor pieces of an application will most help them create large, complex applications
Age are evolving at a rate of speed that likely require changes to other components of in less time. Automated test tools often have
THE TRUE CHALLENGE FOR SOFTWARE TEST ENGINEERS IS
DETERMINING WHETHER OR NOT THE CHANGING TECHNOLOGY
AFFECTS THE WAY IN WHICH THE PRODUCT IS TESTED.
increases exponentially with each passing the system. And all of these new and updated issues when interacting with newly
day. Applications are growing in size and components need to be tested, along with released third-party custom
complexity. One major software component the legacy components that aren’t changing controls and components,
that has undergone massive changes is the to ensure that they still perform properly and these have become
operating system. Much like the automobile under the new design. The true challenge very popular in recent
engine, operating systems have undergone for software test engineers is determining years. Although very useful
numerous changes since their inception in whether or not the changing technology to developers, they can be a major
order to meet consumer demands. These affects the way in which the product is tested. problem for the
demands have led to changes that In addition, we must determine if we can
have made operating systems use the same tools we used prior to the
increasingly efficient, faster, technology change to test these new
intuitive, scalable, and more components, or if we need to look at
secure while providing powerful new tools and/or a new approach.
features and useful functionality Just like the products themselves,
previously unavailable to users. these tools need to be more efficient,
It seems pretty clear cut, no? Changes easier to use, and affordable.
in market demands spur innovations in But most
technology. What, however, does
this mean for software
t e s t automation
engineers. There may also be
issues with development technologies
of all the like .NET, Windows Presentation
automated tools need to Foundation (WPF), ActiveX, etc., that are
be able to access the new components being used by your developers to design the
of the application and have the ability to bulk of your application. The majority of the
engineers perform all of the methods against the test time we cannot find a tool that can address all
that test objects that our tests require to fulfill their of our needs, but at times we do find one that
these evolving objectives. As technology moves forward, so can handle the majority of the tasks at hand.
applications? It means must our testing approaches, processes, and
tools. A Modern-Day
that we have to lift the
hood, kick the In today’s environment finding a tool
tires and take this new that meets your project requirements can be Imagine that you’re working at an
technology for a test a difficult task, especially when you consider organization and have been there for a couple
drive to ensure it effectively and how quickly technology has been moving of years, and during this time you and your
24 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009
team have been building and maintaining a your concern was centered on testability. The posed similar questions on how to automate
suite of automated tests to execute against the first few controls presented by development tests for these controls but, unfortunately
product that your company markets. When were proposed for use in portions of the lacked any useful responses. Some of these
automation was first introduced to the project application that were minimally accessed by threads were several years old, so it was
new test automation resources were hired users, and that required only a small set of becoming apparent that the tool vendor
(including you) and the first assignment was tests for validation. Worst case scenario: If probably had no plan to release any add-ins
to evaluate the AUT to identify the test tool your automated tool couldn’t handle these for this technology. You next contacted the
requirements. Most importantly you wanted controls you may have only had a handful of manufacturer of these 3rd-party controls
to find a tool that could interact with all of additional tests to execute manually. Then to see if it was possible to further explore
the different components of the application, the other shoe dropped. A new grid control automation possibilities with your tool. The
allowing you to automate more tests and was presented that would be accessed by all manufacturers’ response was that this was
exercise more of the AUT functionality. You of the product’s users and a large portion theoretically possible, but that the controls
looked at a number of different tools and of your automated tests. In other words, if do not officially support your automated
compared them using a set of pre-determined you’re automated test tool couldn’t handle testing tool. When you asked how to make
criteria/requirements. this control; the majority of your automated the controls accessible by the automated tool,
Upon completing the evaluation you tests would become useless. You gently you were only informed that a few of their
selected the test tool that would provide the but firmly communicated your concerns customers had attempted automating tests
most value to your project, identified an regarding the affect that these new controls against these controls but their results weren’t
automation approach that best suited your would have on the existing automation readily unavailable. Hmmm… someone was
project, and then began automating the smoke suite so the developers agreed to send you trying to sell something here.
and regression tests. Eventually your team a sample grid allowing you to evaluate how It was time to find other options for
IT WoULD STILL BE UP To THE TEST TEAM, HoWEVER, To
DETERMINE WHETHER OR NOT THESE NEW CONTROLS WERE AN
IMPRoVEMENT oVER THE PRoVERBIAL ‘HoRSE AND BUggY’
built a large suite of tests that you designed your automated test tool would react to the making your tool adapt to the new technology.
with great deference given to maintainability, newly proposed grid controls.
and you felt pretty comfortable knowing Option 1: Extensibility – Some automated
Researching Solutions testing tools offer extensibility, which means
that most updates and fixes made to the
application could be addressed easily using that you’re able to integrate the tool with the
After receiving the sample grids as
your well-planned framework. new controls by executing small snippets of
promised you began trying to access the
custom code via calls from the test script.
After several new versions of your controls using your existing automated
This solution is often the last resort, because
application were released the development testing tool, and that’s when you learned the
it could require extensive setup, coding, and
team started looking at new 3rd-party bad news; the tool didn’t fully recognize the
almost always requires assistance from the
application controls that could be used grid components. It recognized that a new
to improve upon the current design. In a control object existed, but couldn’t access
meeting, the development team presented any of the rows, columns, menus, or buttons Option 2: Workaround – Another possible
the proposed controls, and these controls contained within this new grid control. Was solution might be to identify a workaround
appeared to provide required application there a way to solve this issue? The first step in the application. Was there another screen
functionality and also looked much nicer than you took was to contact the automated test in the application that you could be used to
the previous, less powerful controls that were tool vendor to see if they supported these accomplish the same goals? Was there a
presently in use. It would still be up to the test controls. To this they responded “No, but way to navigate around this control while
team, however, to determine whether or not you can search our knowledge base to see accomplishing the same test objectives? If
these new controls were an improvement over if anyone has done this”. Searching their so, this would allow the automated tests to
the proverbial ‘horse and buggy’, therefore knowledge base turned up several threads that execute, but you and the developers would
August 2009 www.automatedtestinginstitute.com Automated Software Testing Magazine 25
shoes. You’ve done your research and have
which way to Maximum roi? discovered this cool 3rd-party control that
will fulfill the new and existing system
requirements and/or is a major improvement
over the previous control. You’ve performed
your evaluation, probably of proof-of-
concept, and have presented this idea to the
rest of the development team. Now some
automation engineer is walking in saying
it shouldn’t be implemented because it will
break the majority of the automated tests.
That isn’t going to happen without a fight.
Since it is good policy to avoid fighting in
the workplace, the best thing to do would
be to walk into the developer’s office not
only with a list of constraints, but also a list
of options. Keeping a positive demeanor
is of utmost importance. If possible, the
most effective way to communicate this
information to development would be via
the Quality Assurance (QA) Manager. Inter-
team communication is one of the major
roles of a manager, so as an automation
engineer you might as well take advantage
have to remain aware of the fact that the verify its functionality is worth the effort. of it. Additionally, program management
bypassed controls would need to be tested For example, if a new control is being added may be involved and have input regarding
manually. but it only affects a small number of tests, you the final decision, so it is best for them to
Option 3: Programmatic Solution – Another might want to see if the effort of automating communicate with someone that they are
option would be to design a programmatic the tests against the control is cost-effective. familiar and comfortable with.
solution; a coded solution offered by It may make more sense to allow the test to
remain manual. Conversely, if the new control It is important to include caveats when
developers providing a custom way for communicating these options. For example,
your tests to bypass this screen while still is going to be used frequently and is touched
by a large number of your automated scripts, you might have a programmatic workaround
accomplishing the test objectives. For in mind, but this would require developer
example, if the new controls were being it might be worth the effort. Ultimately, it
boils down to how easy or difficult it will be involvement, add time to the project schedule,
implemented on an ‘Add User’ screen, was remove focus from other current automation
there a way that the developers could allow for automation to adapt, and if the effort is
worth the result. activities, and forego any testing of the new
you to bypass this screen using pre-built user control itself since it is being bypassed. You
data stored in the user information database Presenting Automation’s Case might have found an add-in that could resolve
that your test could access and use? This So you’ve discovered that your automated most of the issues you had documented
would again result in the tests for the new testing tool has problems accessing and during the evaluation, but the add-in was
controls being reserved for manual execution, manipulating these new 3rd-party controls. expensive, code needed to be re-written to
but it would also allow all of your tests that Should you just run to the development team replace the previous code used to manipulate
need to access information processed on this and tell them that the new control can’t be the previous controls, and the add-in still
screen to continue running properly. automated? Before doing this you need to wouldn’t provide all of the functionality
Analyzing Impact keep several things in mind. The developer(s) required to meet the test objectives for this
When identifying different options it also who discovered these new controls and are application component. Or you might explain
makes sense to look at the impact that pushing for their implementation have not that your automated test tool extensibility
the new control has on the application to only a technical stake in this but an emotional features would allow you to handle this new
determine if automating tests to exercise and stake, as well. Put yourself in the developer’s control via automation, but it would also
26 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2009