1. 12 • Software Test & Performance
How to Keep Quality
Assurance From Being
A Juggling Act

2. the standard answer is, “Why, they do QA, of course!” The
tone invariably implies a “duh.”
But in reality, most companies don’t recognize that
for software quality assurance to be truly effective, it should
be just one component of a three-part strategy: quality
planning (QP), quality assurance (QA) and quality control
(QC).
What’s the difference among the three? These important
concepts govern the whole structure of your QA team as well
as your company’s approach to software quality. You may also
find that once you’re aware of the differences, your approach
to software quality will change, quality will improve, and you
can help save your company money by being more efficient.
The Quality Triumvirate
Quality planning is a QA group’s general approach toward the
management of product quality. QP includes a set of plans
that are specific to each project. Quality assurance is just
that—assurance that the software will have the specified
functionality and will be up to a specified set of quality stan-
dards. Quality control is a set of rules that ascertain whether
successive software builds continue to meet the specified
functionality and quality.
I recall a relevant example from my days working in a civil
engineering laboratory. Our main task was to provide QA/QC
for large construction projects. The QP was a set of docu-
ments based on generic templates that were customized for
each project. The plan laid down the overall framework for
quality testing that we would use on a given project.
QA was the approach we took to evaluate, for example, a
steel bar that the project was going to use, and which we were
seeing for the first time. It involved a lot of manual handling
of the sample, determining which tests would apply, running
these tests, and discussions with the design engineers about
the results as compared to the desired values specified in the
design documents.
When this initial QA appraisal was over, the next task
would be to determine the tests necessary to run as part of
QC: the subset of QA tests to run on every new batch of
steel bars to ensure that they were of the same quality as the
sample.
Overall, we used the QA process with its associated manu-
al handling and extensive testing to obtain a feel for the item
in question. Then, based on this feel, we developed the QC,
the subset of tests to be run on every single such item that the
project would use, to make sure the items had the specified
functionality and quality.
Two sub-teams were part of the quality testing: a QA team
that was mainly based at the lab, and a QC team that was
mainly based at the project site.
Software, Steel,Whatever
Conceptually, software QA is much the same as materials QA.
Here too, the approach consists of quality planning, assur-
ance and control. Quality planning is generally done for the
project by the QA manager. The QA team consists of two sub-
groups: a QA group and a QC group, each with its own lead,
and tasked with doing quality assurance and quality control,
respectively. Figure 1 (see page 14) shows the breakdown of a
quality team and its activities and responsibilities.
The Quality Plan
Quality planning involves the creation of project plans that
specify how to test the software to determine if it meets the
required functionality, performance, scalability and reliabili-
ty, and that it integrates correctly with other applications and
any legacy software. In companies that have a project man-
By Paul Joseph
Most companies that develop software have a
QA team. When asked what this team does,
Paul Joseph is a developer and IT consultant living in Massachusetts.
Some Tasty New
Steps for Your QA
Team to Improve
Quality Right Now
www.stpmag.com • 13

3. 14 • Software Test & Performance SEPTEMBER 2007
QUALITY ACT
agement office, historical plans or
generic templates of plans are main-
tained, and can be used by QA man-
agers as a starting point.
Such quality plans can set the tone
with upper management and develop-
ment and QA/QC teams about the
approach that they can expect.
Sometimes called artifacts, the plans
also can serve as a (sometimes invisi-
ble) backbone of the QA/QC process.
Personally, I wince at the thought that
my spanking-new plans immediately
qualify as an archeological specimen.
Typically, my quality planning
involves creating the following:
• Test strategy plan
• System functionality test plan
• Performance, scalability and relia-
bility test plan
• Integration test plan
• User acceptance test plan
• Project risk analysis plan written
from a QA perspective
These plans don’t get into specifics
about the testing, but rather are plans
about plans, and tend to make for pret-
ty dull reading. Their purpose is to
clearly describe the procedures that will
be followed during the detailed testing
of the product. Let’s look at each in
more detail.
The test strategy plan provides an
overview of how the testing process will
work for the newly proposed project. It
describes the reference documents rel-
evant to the QA process (such as the
formal business requirements), the QA
team, and the tools, techniques and
methods to be used. Additionally, it
specifies the deliverables to QA, smoke
testing, defect tracking, bug triage
(how priorities are assigned to bugs),
new staff straining, handoffs from con-
sultants, change control, senior man-
agement’s responsibilities and so on.
The system functionality plan con-
tinues the test strategy plan by address-
ing generic system issues such as plat-
forms and environments to be used for
the testing and for deployment.
The performance scalability and
durability plan describes how to test the
application against performance, scala-
bility and durability requirements.
The integration test plan describes
integration of the new product with
existing products and systems.
The user acceptance test plan de-
scribes acceptance testing and criteria.
The project risks analysis plan iden-
tifies issues that could threaten, delay
or derail the quality testing.
Each plan should have a sign-off
sheet on which agreed-upon represen-
tatives from senior management
approve each plan. In my last decade
of software development, I’ve seen the
sign-off sheet actually used only once.
However, the very presence of such a
form helps set the tone and expecta-
tions for a project, resulting in senior
management reading the plans in
detail, even if they don’t physically
sign off on them.
You Have My Assurance
Quality assurance confirms that the
application can meet the desired func-
tionality and standards put in place in
advance as specified by the business
requirements and the quality plan.
The QA group’s skill set should
include the ability to comfortably
meet with and talk to people face-to-
face. Its members interface with the
business requirements group and the
development team to understand the
requirements.
The QA team also writes test cases
that address the requirements. They
manually test functionality as it
FIG. 1:THE QUALITY CREW
Project Quality
Management
Quality
Planning
(QA Manager)
Quality Plans
1.Test Strategy Plan
2. System Functionality
Plan
3. Performance, Scala-
bility and Reliability
Plan
4. Integration Test Plan
5.User Acceptance Test
Plan
Quality
Assurance
Quality
Control
QA Team
1. QA Lead+
Individual
Contributors
2. Collocated with
developers
QC Team
1. QC Lead+
Individual
Contributors
2. Collocation
optional
1. Manual inspection of
new features as listed in
the Release Notes for
the build
2. Create new test cases for
this new functionality
3. Manual validation of
defects fixed in the build
4. Interface with develop-
ers, Product Manage-
ment and Quality Control
1. Create and execute test
scripts for selected use
cases written by the QA
team
2. Create and execute
“deep QA” test programs
3. Performance, scaling and
reliability testing
4. Interface with Release
Engineering and Quality
Assurance
TABLE 1: SAMPLE PROJECT FRAMEWORK
Rumba Release 1.5
Builds 63, 64, 65, 66
Scripts Manual Bugs
John NA url test, Release Notes Queue
Rita NA LDAP, Release Notes Queue
Lana NA Release Notes Queue
Joel Client Script NA NA
Monique ETL Program NA NA
Mark Administrator Script NA NA

4. SEPTEMBER 2007 www.stpmag.com • 15
QUALITY ACT
becomes progressively available in the
builds. In my experience, when builds
are done on a weekly basis, the new
build often has the same amount of
new functionality and defects to vali-
date as that of the previous build, so
the time required to cover the new fea-
tures manually and to validate the
bugs remains roughly constant from
build to build.
Hence, one strong requirement of
this process is that once the new appli-
cation can be compiled and run,
release engineering must provide a for-
mal build to QA no less often than
once a week. The first builds may have
only five percent of the total anticipat-
ed functionality, but it starts the rou-
tine of examining new builds and turn-
ing them around within the week.
It’s beneficial to the project to have
a team member (preferably, the QA
lead) sit in on product-management
team meetings and listen as the team
fleshes out the requirements. There is
often pushback to this idea, with some
fiscally conservative managers viewing
this as premature and a waste of time
and money.
In my own experience,
the reverse is often true,
and a QA representative’s
understanding of the
requirements and their
basis pays significant divi-
dends down the road.
Often, requirements get
lost in translation, and
what the development
team delivers to QA does-
n’t match what the QA
lead knows that product
management had in
mind. Being present at
the conceptual and
requirements develop-
ment meetings now gives
the QA lead the confi-
dence to bubble this up
the chain of command
and to quickly prevent
the development team
from wasting time and
effort developing the
wrong feature(s).
I also find it useful for
the QA team to have access to the code
repository and the development tool
used by the development group. At a
minimum, I train the QA team lead to
build the application and deploy it on a
local desktop, using the head of the
checked-in code. This is useful for
identifying new features in advance and
clarifying any discrepancies between
these features and the requirements.
Another technique I find useful,
especially in the early stages of devel-
opment, is to have the lead developer
and the developers who own modules
of the code do periodic presentations
to both the QA group and the product
management group.
This offers several benefits. First, the
developer in question is usually pleased
to be in the limelight and show off his
work. Second, it gives the quality team
a preview of the new functionality and
access to its developer. Finally and
importantly, it gives product manage-
ment an opportunity to see how their
requirements appear in reality.
Controlling Quality
Quality control ensures that successive
builds delivered to the QA team con-
tinue to meet the functionality require-
ments and standards. The QC group’s
skills lean toward scripting, automation
and coding. They implement those test
cases that lend themselves to scripting.
They also work with the
development team to dis-
cover opportunities for
unit testing and release
engineering to determine
how to couple test scripts
with the build process so
that they run automatical-
ly after a build.
Builds occur every
night. If the build is suc-
cessful, an automated
build-management pro-
cess can deploy it auto-
matically. If that deploy-
ment is successful, the
build management pro-
cess can fire off the test
scripts to test the new
deployment.
On completion of the
test scripts, the build man-
agement process collects
the results, including
error details, if any, for-
mats them into an e-mail,
and distributes it to a list
defined in the test strategy plan.
Successful builds are then available to
the QA team for manual testing.
It’s hard to find a tool that creates
scripts that you can set up to be auto-
matically triggered to run, so the scripts
often must be run manually. Given this
manual constraint, the QC team fre-
quently runs its scripts only on the for-
mal weekly builds, not in an every-night
build. In this situation, the QC team
first runs a subset of the scripts on a
build that is deployed to a development
environment—a smoke test. If (and
only if) this build passes this smoke test,
it’s then promoted to the formal QA
testing environment.
The QC group, given its facility with
scripting and coding, is often also
responsible for testing application per-
formance, scalability and reliability.
Performance testing involves working
with the QA group and product man-
agement to identify common use cases,
the frequency with which they will
occur, and the growth of use anticipat-
ed over time. That time is usually
dependent on the company’s hardware
upgrade cycles.
Another responsibility of the QC
group is to do what I call “deep
QA/QC.” This involves validating parts
of the program that aren’t testable by
scripts, but which can be tested pro-
grammatically. It validates not just func-
tionality, but also data.
In one example where deep QA/QC
If you search the Web for QA termi-
nology, you’ll find various definitions of
QP, QA and QC.These terms are dry as
dust and somewhat hard to keep
straight. For the purposes of this arti-
cle, I’ve adopted definitions taken from
the Project Management Body of
Knowledge, the bible of the Project
Management Institute:
Quality planning – Identifying which
quality standards are relevant to the proj-
ect and determining how to satisfy them.
Perform quality assurance – Applying
the planned,systematic quality activities
to ensure that the project employs all
processes needed to meet requirements.
Perform quality control – Monitoring
specific project results to determine
whether they comply with relevant
quality standards and identifying ways
to eliminate causes of unsatisfactory
performance.
QABODY OF
KNOWLEDGE
•
The QA team
should have
access to the code
repository and
development tool
used by the
development
group.
•

5. 16 • Software Test Performance SEPTEMBER 2007
QUALITY ACT
is required, ETL is part of creating
and maintaining the application. To
perform deep QA/QC, the QC sub-
group includes (or has access to) a
developer who will write the necessary
test programs.
Outsourcing Quality
The QC role sometimes lends itself to
outsourcing. But while the practice can
often work well from a technical point
of view, the process has the potential to
roil the internal organization unless
handled correctly.
With the division of quality-checking
activities into two groups, it’s possible
to outsource or offshore QC with a jus-
tification that employees will
find reasonable. This justifica-
tion rests on the premise that
what’s good for the company is
also good for its existing
employees, and that none of
the existing employees will be
let go because of outsourcing.
It’s also a good idea to keep
QA activities that involve a lot
of “face time” internal to the
company as much as possible.
This permits the company to
retain an independent meas-
ure of quality and minimize
disruption to the team should
the need arise to switch outsourcing
companies.
With your QP, QA and QC teams in
place, you’re more than halfway
toward an efficient testing process.
The rest of this article suggests tech-
niques that I’ve found useful—even
necessary—to create an efficient team
and testing process.
Self-Managing Teams
A good manager will mentor a team in
the art of self-management. This isn’t
rocket science—there are some simple
tasks that can help create a “cut and
dried” system suitable to your needs
that will clearly allocate responsibilities
and provide a good project framework.
One technique that works for me is
a simple whiteboard, about 6 feet by 4
feet. Table 1 (see page 14) shows an
example of what I draw on this white-
board. I create one row per team mem-
ber and one set of columns per prod-
uct release being qualified. At the top
of the columns is the release number
and a list of builds that QA has
received for testing to date. Each
release column is further subdivided
into three smaller columns headed
Scripts, Manual and Bugs.
Each team member’s row shows the
scripts they own, their manual tasks
including writing of (release notes) and
any additional manual testing tasks.
The Queue entry listed under the Bugs
column is a placeholder to remind
team members to check their queue in
the bug-tracking system to make sure
that they’ve validated the bugs assigned
to them for that release.
The team lead works with the devel-
opment lead to make sure that every
build has a set of release notes accom-
panying it. The lead then allocates the
release notes to the appropriate mem-
bers of the QA team.
Each member of the team is now
aware of what he’s responsible for.
When they accomplish their tasks, I
have my team members go to the white-
board and check off with a red marker
what they’ve done.
At any time of the day, by simply
glancing at the white board, I have a
good idea of how far the build has been
qualified. When I find that every item
in the matrix has been checked or
marked as NA (not applicable), the
build has been qualified, and I put a
check mark next to the build number. I
then carefully erase the check marks by
the team members next to
their individual items as we pre-
pare for the next build.
This process is repeated
over and over with every
build, including the final can-
didate build. In no time at all,
team members become con-
versant with the system and
use it with minimal monitor-
ing or supervision. Such
autonomy empowers people
to create, succeed, to make
mistakes, to learn from their
mistakes and to grow. It helps
create a mature team that
knows what it needs to do.
One-on-Ones
Another simple tool I find useful is the
“one-on-one.” I meet privately with
each team member at least once a week
for about an hour. This allows me to
really get to know each member, to find
out interesting things about them and
their personalities, and to help manage
and mentor them.
Key pieces of information I give and
take include areas of interest, chal-
lenges, current workload and schedule,
and bottlenecks. While all of this could
come in status reports, such reports
leave out the key information that’s
available only in face-to-face meetings
and which is sometimes conveyed only
by body language.
Weekly Meetings
Weekly team meetings should be devot-
ed to company and project updates,
and identifying and analyzing project
risks. Drive all weekly meetings with a
printed agenda written in advance and
made available at the meeting.
Never present project status reports
TABLE 2:TRACKING OPEN AND CLOSED BUGS
Open Assigned to Dev./ Assigned to QA Total Open Total Closed
Marketing or Testers
Critical 0 (0) 0 (0) 0 (0) 21
Major 3 (0) 1 (0) 4 (0) 263
Average 201 (7) 6 (0) 207 (7) 838
Minor/Cosmetic/ 44 (1) 0 (0) 44 (1) 132
Enhancement
Total 248 (8) 7 (0) 255 (8) 1,254
Total closed bugs (since start of Rumba) as of 05/07: 1,229
Total closed bugs (since start of Rumba) as of 04/30: 1,217
Total bugs since start of Rumba: 1,517
•
Devote weekly team meetings to
project updates, and identifying
and analyzing project risks.
•

6. SEPTEMBER 2007 www.stpmag.com • 17
QUALITY ACT
at a meeting. Instead, e-mail status
reports beforehand, and at the meet-
ing, ask only for clarification ques-
tions. Meetings should always begin on
time, ideally at the same time of day,
day of the week, and in the same meet-
ing room. This is of particular impor-
tance if the meeting is with an offshore
team, where ideas of punctuality and
regularity can differ markedly from
those in the U.S. Meetings should fin-
ish early or on time.
Status Reports
The status report is an important tool,
and should be among every team mem-
ber’s weekly task list. The report
should list accomplishments of the pre-
vious week, status of their planned
activities for the current week, and
what they plan to do next week. Also
included should be any risks or bottle-
necks, and any time off required over
the next 30 days.
I consolidate individual status
reports into a weekly status report for
the whole group, keeping the con-
tributing team members’ names next
to their corresponding items to
ensure traceability, transparency and
ownership. I then add a summary of
the bug status.
For example, Table 2 lists open and
closed bugs, and the status of open
bugs along with their priority. I distrib-
ute the status report to the list defined
in the test strategy plan.
Bug M*A*S*H
It’s important to perform regu-
lar bug triages. I like to have at
least one triage session per
week. Typically, a bug triage
team will include a representa-
tive each from product man-
agement, development and
QA/QC teams, and looks only
at new bugs and assigns priori-
ties. Gradually becoming
familiar with the needs of
product management, the QA
lead is often able to prioritize
defects in line with product manage-
ment and can sometimes take over the
triage role.
Location, Location, Location
Ideally, the QA team should be collo-
cated with the developers. Collocation
is a tried-and-true technique much
recommended by seasoned project
managers.
The benefits are numerous, but the
main advantages are that the QA team
members quickly know the person
who’s coding the module or functional-
ity that they’re responsible for. Usually
this results in greater degrees of ad-hoc
communication between them.
For example, if the QA person
finds that something isn’t working as
expected, he simply walks over to the
developer who owns this code and
describes the observed behavior. The
two work together and quickly deter-
mine whether this is the way it’s really
supposed to work, or whether they
have a bug.
In my experience, collocated teams
tend to have higher efficiencies of
problem discovery and resolution.
Strangely, though, collocation is not
often practiced. Even when developers
and QA personnel are located on the
same floor but separated by another
group, communication tends to be
minimal.
For collocation to work, teams have
to be located cheek-by-jowl before syn-
ergy occurs.
Size Matters
For optimum team size, the QA/QC
personnel should number slightly more
than a third of the developers working
on the project. The QA manager
should have a QA lead with a QA team
and a QC lead with a QC team for each
project.
Using the processes outlined
here, a well-trained team of this
size should be able to complete
manual testing of new features,
validate fixed bugs, write new
tests and scripts for the new
functionality and bugs, all with-
in five days of build availability.
To implement this concept
of quality assurance/control,
the QA manager must put in
place the processes and tech-
niques I’ve outlined with sup-
port from key stakeholders.
Once such a system is in
place, the process of qualifying a build
becomes simple—a question of the QA
team manually testing new features
and bug fixes as specified in the
release notes for that build, and the
QC team creating and running scripts
and reporting errors. In this way and
with suitable staffing, it should be pos-
sible to turn around a build in three to
five days.
SCRIPTING’S UNIQUE PROBLEM
The tools you choose for test scripting should offer more than record and playback capa-
bilities. After creating a recorded script, it’s helpful to break the script into modules.This
allows you to save common function tests into libraries and reuse them whenever you
encounter a similar need in the future. Nowadays, fairly comprehensive, prewritten frame-
works and libraries can be found on the Web. They’re usually free, and often of surpris-
ingly good quality.
I’ve found that scripts built using the record and playback technique tend to be brittle.
Navigation from field to field is often “tab-driven” and susceptible to changes in field
order. In the early stages of development, when GUIs are often not stable, fields may dis-
appear or be moved, causing scripts to break.
A less brittle approach is to use a library-based method that includes object identifiers for
navigation. This way, as long as the object name doesn’t change, the script will work
regardless of where the object may have moved in subsequent builds.
The key here is for developers to tag their objects with unique identifiers, so be sure to
advise the development team about this requirement.
•
For collocation to work, teams
must be located cheek-by-jowl
before synergy occurs.
•