An A U TO M AT E D T ESTING I NSTITUTE Publication - www.automatedtestinginstitute.com
August 2013 $10.95
Rethinking the Role of
Agility Applied to
Selecting the Right
The Pyramid Approach to
selecting a tool to meet
Good Things Come: Syncronization in HP Functional Tester (QTP)
Approach to Test Tool
By Bernd Beersma
22 Automated Software Testing Magazine
A test tool is one of the best tools
to help us throughout the test cycle
because test tools can support almost
all activities we do in testing: test
management; test design; etc. But if we
do a Google search, we get hundreds of
thousands of hits for test tools. So how
do you as a tester, test manager or test
specialist know how to select the best
tool for your day-to-day business – one
that fits your purpose?
electing a tool, that eventually
fails to meet your requirements or
expectations, can lead to:
1. Long List
Lack of commitment
Phase 1: Long List
Bad business case
The outcome of the long-list phase is, as
the name states, a long list of test tools. But
getting there requires a few steps:
The tool selection pyramid is a structured
selection approach that will help to avoid
The Tool Selection
The Tool Selection Pyramid is composed of
four different phases:
2. Short List
3. Proof of Concept
Identify your stakeholders.
If we want to implement a test tool in our
organizations, this will influence the test
team and other departments, IT and the
maintenance departments, for example. We
want to identify all those who are involved
2. Define measurable, feasible goals for
Test Automation (TA).
Automated Software Testing Magazine
Be sure to involve all the stakeholders.
This again creates commitment, but the
other departments can also add goals of
their own. For instance, business wants
to decrease the time to market when you
select a tool for automated regression
regression test or need to support a
defect-management process. These are
on a high level, and mind mapping is a
good way to gather these requirements.
This approach helps to get the high-level
requirements for the tool in a fast and
Create a business case for TA.
We need to set up a business case to be
sure the project has enough support in
the organization, and can be brought to
Gather general tool information.
After defining the general requirements,
explore the tool market. Look at toolrelated websites and magazines, and
general requirements. In this phase,
we need to go a little deeper into these
requirements. For example, we needed
a tool for automated regression testing.
Now, with a mind map, we find the need
for tools to support the use of external
data, or tools to support cross-browser
2. Gather the detailed requirements.
Add them to a scoring card and determine
weight and priority. The scoring card is
the basis for the next step, the Request
After drafting the requirements we invite
the vendor(s) to set up a plan for the pilot
together with members of the project
team. This needs to be a solid plan, with
fixed timelines and dates, because this
pilot phase needs to have an end, after
which the project team can start the
Set up a project team.
The next thing we need to do is set up
a project team for the tool selection
project. Again, be sure that people from
the different identified departments
are on this project team, although the
majority of the team can be from the
test department. Someone from IT and
maintenance – but also a developer –
can be of real value to your team not
only to create the support within the
organization, but they need to help
identifying requirements that are not
test-related but are an essential part of
the selection process.
Define general requirements.
In this step, focus on defining general
requirements like the need to automate
24 Automated Software Testing Magazine
Create a long list.
This list contains the tools and vendors
we think should enter the next phase in
our tool selection pyramid, the short-list
Before entering the next phase, there is
always a moment of consideration about
whether or not we are ready to enter the
next phase. This go/no-go moment gives
us the possibility to restart the previous
phase or start the selection process over.
Phase 2: Shor t List
This phase also consists of several steps:
Define detailed requirements.
In the long-list phase we came up with
for Information (RFI).
3. An RFI goes to all the vendors on
the long list.
Sometimes this is not possible if there is
no vendor or party to whom to send the
RFI. But if you still want to score, you can
always see if you know an independent
consultant who has experience with that
tool to help you to fill out the form, or
find a third party experienced with the
tool and ask them to fill in the RFI. In
the worst case, the project team has to
fill it out. After getting back the RFI,
you need to determine overall score.
As you can see, this is only an example,
but already 20 different main categories
are identified here. So this shows tool
selection is not that easy.
time. Be sure that all systems are ready
when starting with the next step in this
phase, the actual execution of the POC.
4. A demo.
Invite at max the top five scored tools
to do the demo. The demo is like first
contact, and you never get a second
chance to make a first impression. Be
sure to invite all the project members,
stakeholders to let them see the demo.
They can help you to decide afterwards
which tools to put on your shortlist.
Preferably three vendors will end up
on the short list and this completes our
Again, before entering the next phase,
there always is a moment of consideration
if we are really ready to enter the next
phase. This go/no-go moment will
always give us the possibility to restart
the previous phase or even start all over
again with the selection process.
Phase 3: Proof of
Now we entered our Proof of Concept
or POC phase, during which several
vendors or consultants are invited to do a
proof of concept. This is like a test drive
for the tool, to see how the tool performs
in our situation and environment, and to
see if the tool meets our requirements.
Again, this phase is completed by
executing several steps.
First of all, we draft the requirements
for the successful completion of the
POC. These requirements are the same
for all participants of course, to create
a fair competition. The tool must
support all technologies that are part of
a certain business process chain. After
completion, the requirements drafted
will be also part of the acceptance
criteria for the POC.
Second we invite the vendors to do
the POC, but this is a combined effort
between us and the vendor. We need to
be clear on what we expect of the vendor.
This not only counts for the availability
of the Software Under Test (SUT) but
also of internal resources who will be
executing the POC with the vendor. We
need to setup a plan so we are sure the
POC will be executed and finished on
Keep in mind that user accounts have to
be set up, as does a test date, etc. If you
are sure that everything is good to go,
start executing the POC. This is also a
combined effort. The more you involve
the team by the selection process, the
better the results of the selection will be.
After you and the vendors finished all the
POCs, an evaluation of the POCs needs
to be conducted. We will do a session
with the project team and will see if the
POC meets our requirements. The goal
of this step is to choose the best tool,
preferably one tool; you have selected
two tools that meet all the requirements.
If this is the case, no worries, we will
come up with the definitive selection in
the pilot phase.
But, we haven’t completed this phase yet,
the next thing to do is to send a Request
for Proposal. Even though I strongly
believe that money is a bad driver,
we still want to know what licenses,
trainings, maintenance cost. We need
this input to validate our business case.
In the case of an open-source tool, the
same procedure as for an RFI is started.
Now with all this information gathered
we are ready to update the business case
and confirm we are still on track with
our tool selection.
This completes this phase and we will be
entering the pilot phase, but not before
we do a go/no-go, but again, we are
always allowed to restart the phase or
even the whole selection project.
Phase 4: The Pilot
During this phase, one or two tools
will be used in the pilot project for a
longer period of time. It’s best to setup
a separate project for this phase in a test
Like in all of the other phases, we have
to follow a few steps to complete this
phase. But, after completing these final
steps we have selected the best tool to
help us with our testing, so completing
this phase is worth the wait.
Again we start with the requirements
for completion of this phase. We want
to evaluate the outcome of course
against our acceptance requirements.
An example of a pilot requirement can
be, “testers must be able to automate the
regression test by themselves.”
After drafting the requirements we
invite the vendor(s) to set up a plan for
the pilot together with members of the
project team. This needs to be a solid
plan, with fixed timelines and dates,
because this pilot phase needs to have
an end, after which the project team can
start the evaluation. After setting up this
plan, we start the actual pilot with a kick
off, so all participants in the pilot know
what is expected of them, but also to
create a feeling of collaboration between
the involved parties.
No we are ready to actually start with the
execution of the pilot. This is a combined
effort with shared responsibility between
the vendor and the project team. The
goal is to prove that the tool can be
implemented within the project and the
Now that we have executed the pilot,
final step is evaluation of the pilot.
Gather the project team and see if the
selected tool meets the requirements
and acceptance criteria. And if all
requirements are met and all systems
are go, we have completed our selection
project and the best tool for the job is
ready to be implemented.
That leaves me with some final words of
advice. Whenever you start selecting a
tool, be sure to create awareness within
your organization. Make it a combined
effort and a shared responsibility
between you and all the possible tool
vendors or consultants, and always keep
in mind that if you don’t feel comfortable
after completing a phase, start again.
Wait! There’s More!
Learn more at TestKIT
Automated Software Testing Magazine