and Quality Assurance:
The Royalty Distribution
April 25, 2007
Prepared by:Simon Hood
Student Number: 0240170
Work Term: 4
Last Academic Semester Completed:
University of Guelph-Humber,
Place of Employment: NVP Software Testing Inc.
Job Title: Software Tester/Web Developer
Supervisor: Neil Price-Jones
April 25, 2007
Dr. Michael Liu
University of Guelph-Humber
Toronto, ON, M9W 5L7
Re: Work Term Report, Software Testing and Quality Assurance Practices: The Royalty Distribution System
Dear Dr. Liu,
Attached please find my final work term report entitled Software Testing and Quality Assurance Practices:
The Royalty Distribution System Project. During the past two semesters I have been employed at NVP
Software Testing Inc. in Bolton under the supervision of the company’s Owner and President Neil
Price-Jones. My time at NVP was spent testing the latest version of a software project intended to
help our client distribute royalties based on television shows or movies aired by cable companies
throughout Canada since 1990. The report details some of the methodology and strategies used to
complete the testing of RDS, but as of this writing the eight year project is far past overdue and
there is no end in sight for the near future.
I began my first semester by learning the software as built, but quickly moved to testing the various
modules and returned bugs in the project. My second semester was largely spent managing, testing,
and retesting the more than 1700 known issues and 5000 existing testcases. The sheer scale of the
numbers should provide a testament to the size and complexity of the RDS software. Despite the
overwhelming state of affairs, I was able to advance NVP’s role on the project to a point where the
company was strictly waiting on the programmers for new releases; all possible testcases had already
been executed, and all new or existing issues had been dealt with appropriately. In general, my
responsibilities grew steadily throughout the two semesters, and by the end of my coop I was
responsible for the lion’s share of the work on RDS by NVP.
Testing RDS has taught me a great deal about quality assurance and software engineering. It has
given me the opportunity to see development from the end user’s point of view, rather than the
view of the designer and programmer I am used to at Guelph-Humber. As I plan to do
development work at some point after graduation, the knowledge I have gained will prove both
practical and useful to my career. Although RDS has not yet been completed, NVP hopes the
project will be finished sometime in 2008, and it is possible I will return to NVP with my degree in
hand in time to see the project come to a close. If you have any questions regarding my report,
please feel free to contact me.
The software lifecycle is not complete without some measure of quality
assurance and software testing. Many computer science graduates spend a lifetime
learning the intricacies of the field, and there is always more to learn. My coop
semesters at NVP Software Testing Inc. have offered me some insight on both the
best and worst practices, as applied to the long running and now overdue RDS
project. I have learned many things in the last eight months, and a few of my most
outstanding lessons are illustrated. For example, I have learned that it is critical
that testers choose the best strategy and methodology while tackling large projects.
Furthermore, a tester must be very specific with regard to both testcases and the
resulting issues. I am now aware that testers and programmers are opposing
forces, and balance must be achieved if the project is to succeed. Additionally,
complete testing is rarely possible; quality assurance itself is a balance of priorities
and a best effort situation. Finally, I believe I have achieved some insight on what
the future holds in store for software testing in general. That said, it is obvious that
the intricacies of the field can be maddening at times, but the rewards for a job well
done are well worth it - even if they go unnoticed by the end user.
Table of Contents
Abstract ................................................................................................................. 3
Table of Contents..................................................................................................... 4
Glossary ................................................................................................................. 4
Introduction ............................................................................................................ 5
Black-box Testing.................................................................................................. 9
Specificity .......................................................................................................... 11
Programmer Hostility ........................................................................................... 13
Complete Testing ................................................................................................ 14
The Future ......................................................................................................... 16
DOS: Disk Operating System
QA: Quality Assurance
MSSQL: Microsoft SQL
NVP: No Variances Please!
RDS: Royalty Distribution System
SQL: Structured Query Language
VB: Visual Basic
I spent the last eight months working for NVP Software Testing Inc. in
Bolton. NVP provides businesses with complete software testing solutions.
Services offered include: software testing consulting, management, development,
execution, automation, training, staffing, and outsourcing. However, it may appear
strange that a for company specializing in software, NVP itself does very little
programming. The software industry in general, and software testing in particular,
have grown to the point where many companies are able to survive just by
outsourcing niche services. At NVP, for example, the application development is
usually left up to the contracting company, or a separate company altogether, as
need’s dictate. In most cases, NVP is called in for very large software projects
where test management and accountability are outside the abilities of our clients.
Unfortunately, even with expert, professional quality assurance and test
management, for some software projects costs can still skyrocket, and schedules
can become years out of date – as seen in the RDS3 project.
The project I worked on is intended for divvying out a pool of money that has
grown to many millions of dollars. In much the same way that radio stations must
pay out funds for each song played on the radio, every Canadian cable company is
required to pay royalties on every television show or movie aired on any channel at
any time to compensate the rightful owners, provided those shows cannot also be
seen locally for free with a TV antenna. Two of NVP’s clients, known as The
Collectives to the testers, distribute the money using a program called RDS, short
for Royalty Distribution System.
The previous version of RDS has been in use for many years, but is now a
legacy application; RDS2 runs in a DOS shell and is based on an obsolete FoxPro
database. Therefore, The Collectives commissioned an upgrade for the program,
expecting to bring it up to current computer standards in only a few months of
programming. The upgrade to RDS3 was supposed to be a line for line conversion
of the source code to VB 6.0 and an MSSQL database server. Roughly eight years
later, the new version is finally nearing completion despite much mismanagement
and poor programming.
A deep-seated paradox exists with regards to software quality assurance. If
software development is well executed, it is widely accepted that the designers and
programmers did a good job, and costs are kept to a minimum. If, on the other
hand, software development results in a poor quality product, it is perceived as the
fault of the testers for not catching the bugs, and the costs go up trying to iron out
the bugs. In reality, high quality software products are, more often than not, the
result of meticulous testing, while the quality of substandard products can be
directly linked to their lack of testing in their lifecycle as a whole.
NVP is not responsible for the problems with RDS3. My company has done its
part endlessly testing and retesting frequent new releases with the latest bugs
fixed, and then shipping the new bugs back to the programmers so that they can
be fixed in turn. In reality, the problem seems to be the lack of a clear manager for
the project; programming, testing, and auditing are all outsourced to different
companies because the client is incapable of directing the technical aspects. That,
coupled with the fact that the total funds to be allocated still exceed the total
development costs to date by many millions of dollars, have lulled The Collectives
into a kind of complacency. In June of 2006, the client’s board of directors finally
realized that the eight year project had spun out of control and decided to put a
due date on it for November 1st, 2006. Unbelievably, the date came and went, and
as of April 2007 RDS3 is still at least a year away from completion.
The problem of how to finish up RDS3 in the shortest time possible, with all
eyes watching, fell to the three outsourced companies – NVP Software Testing Inc.
for testing, Synaptic Systems Inc. for programming, and KPMG for auditing. It was
more or less at this time that my first coop semester began, and I was asked to
help test RDS3 on my first day. Towards the end of my second semester, the vast
majority of the testing and a good part of NVP’s completion strategy had fallen on
RDS3 is by no means a small program. Certainly it does not compete in
scale with programs like the new Windows Vista, but it does perform some very
complex functions. If the functionality of the application is reduced to its simplest
form, it can be said that television shows aired by Canadian cable companies are
placed as properties, the owners and their banking information are recorded as
registrants, and the properties and registrants are linked to folders containing the
show. The money for the many different broadcasts of the show can then be paid
to the owner. The amount of payment is based on the weight of the show, or how
many people actually watched it, multiplied by a viewing factor to take the number
of people watching at any given time of the day into consideration. Functionality
for uploading new data into the system and importing old data from RDS2 is also
provided. Needless to say, the application’s learning curve was steep.
As I was faced with entire filing cabinets of documentation to familiarize
myself with, initially I was forced to limit myself to learning only the modules I was
assigned. Reams of documentation were available, and with nearly every new
release, more documents, albeit modified versions of the old documents, arrived.
The documentation itself came in the form of use cases, sequence diagrams, and
short write-ups. Every use case was directly translated into dozens of specific
testcases, with the number of testcases depending on the particular use case’s level
of detail. For example, the details of a use case might state that when folder-
registrant links are created from the Folder Management module, first “The actor
selects one or more folders from the Folder List.” The details become a
corresponding testcase with three major sections: setup, script, and expected
results. The setup lists the conditions that must be present or met before the
testcase can be executed. The script section contains the exact procedure to follow
while carrying out the test. Lastly, the expected results inform the user what
should happen after the procedure is followed. My first module, Payment
Management, was based on a 120 page document translated into some 1040
testcases, no small endeavour.
Gradually, I became familiar with both the testcases and documentation, and
I was able to turn my attention to optimizing the process. I eventually came to the
conclusion that testing RDS3 using a black-box strategy was the only method
possible. I also learned that specificity is of critical importance in both testcases
and issues. Furthermore, despite my best efforts to be innocuous and politically
correct, the programmers were still occasionally confused and sometimes became
hostile. I discovered that complete testing in large systems is impossible, and that
testing is a matter of providing the best effort. Finally, I now feel qualified to have
an opinion on the future of software testing in general.
Putting it bluntly, NVP is not responsible for carrying out the code conversion
to RDS3. NVP has been contracted to only test the program as built, while the
actual programming is up to the team at Synaptic Systems. Over two semesters, I
saw none of the actual code for RDS3, save a few lines of MSSQL database pseudo-
code used to speed up the program. Even though I am familiar with both SQL and
VB 6.0, it was simply not my job to fix the program. Therefore, any and all testing
I performed was strictly black-box.
Black-box testing refers to “the testing that you do when you put yourself in
the software user’s place to verify that [the software] behaves the way it’s
supposed to, no matter how it may be built.” (Bezier 1995) NVP’s RDS3 testcases
are entirely based on the design documentation and serve a dual purpose: to
eliminate any bugs encountered during the execution of the various modules, and
to ensure the design specifications are met. Without knowledge of the source code,
the RDS3 testing can only focus on creating the “subset of all possible testcases
that has the highest probability of detecting the most errors.” (Myers 1979)
Granted, few other testing methodologies were possible in the case of RDS3
because NVP does not have access to the source code, and it may appear at first
that black-box testing would not be the most effective method. White-box testing
strategies like stubs and drivers are often easier to employ when evaluating code
created from scratch. Stubs and drivers are a few lines of code created to make
output to other modules possible without the actual finished modules, or to make
required input arguments, often from other unfinished modules, available to the
module being tested, respectively. It is often the case that large sections of code
are missing while testing is performed, as the sections have not been written yet.
As well, some program modules are so interrelated that it might seem as though
testing cannot begin without a finished product, but stubs and drivers allow testing
to continue without the completed sections. On the other hand, Synaptic Systems
was already in possession of a finished, tested program in the form of RDS2, and
essentially NVP was only required to test the system as a whole; RDS3 is simply a
line by line conversion to a new SQL server and more modern programming
language. Experiments comparing the effectiveness of black-box versus white-box
forms of software testing found “white-box testing proves insufficient… for
situations in which testing isolated components may not reveal integration errors
with other components.” (Bryson 2003) Truth be told, RDS3 is a highly integrated
series of modules, and NVP was able to discover over 1700 individual issues using
only the black-box method.
Regrettably, issues and testcases are often misinterpreted. The systems
used by NVP, Tracker for testcase management and CHGLOG for issue
management, rely strictly on the written word. As such, the two applications do
not always permit complex procedures to be conveyed clearly in the few short
sentences allowed. Moreover, the lack of normal body language clues given
through face to face interaction, or even the conversational meter given through
two-way interaction in real time, make it only natural that some things will be
misconstrued. Therefore, it is imperative for a software tester to be as specific as
possible while writing both testcases and issues so as to avoid any
misunderstandings. Simply put, misunderstandings can cost time and money at
best, and create misapprehension and conflict at worst.
Normal testing procedure for RDS3 involves writing issues as they occur into
CHGLOG as a logical series of steps needed to reproduce the problem. The
programmers then read the issues, verify the problems, fix the code or database
where required, and return the issues to the testers so they can be tested again.
When working properly, the cycle generally takes a few weeks to a few months to
complete, depending on the priority of the problem. So if an issue is returned for
clarification, rather than being corrected, the resulting delay is expensive. There
are, in fact, still issues in CHGLOG that were entered almost nine years ago,
although they have usually been escalated to The Collectives for a final decision.
As mentioned previously, the RDS3 project has been running on overtime for quite
a while, so delays are to be avoided wherever possible.
Testcase execution can also be problematic. Requirements change on any
project as programming progresses; features are occasionally dropped to save
money, or even added where absolutely necessary, so from time to time the
functionality of the program changes. Executing the related testcases is impossible
if a feature is dropped, and because no notice is given in Tracker it is generally up
to the tester to figure out why it cannot be completed. Likewise, no testcases exist
to ensure a newly added feature is bug free, so new ones must be written.
Sometimes running ambiguous testcases is a best effort situation anyway, as the
wording of the script or expected results is unclear. In practice, all software
testcases should be as precise and clear-cut as possible, or the intended test may
not be carried out correctly. Inaccurate testing can result in logged failures where
the program in fact passed, and days or weeks may be wasted.
The programmers also become more than mildly unhappy when incorrect or
duplicate issues are entered. Synaptic Systems and NVP are under a great deal of
pressure to complete RDS3, and the programming team does not like to waste their
time any more than we do. Unfortunately, the written word does not always carry
the same intention as words spoken aloud, and a simple paragraph can appear
disrespectful or insulting if not phrased correctly. Specificity in both testcases and
issues is paramount.
Programmers and testers clash in exactly the same way artists clash with
their critics. Both artists and programmers produce creative works, while critics
and testers pick apart the finer details of the work making both positive and
negative comments. In the case of RDS3’s programmers though, praise for a job
well done is rarely heard; instead, only an endless barrage of criticism about their
mistakes is received. The praise due to the programmers for a job well done is
often not even verbalized by the testers, and issues are usually closed if fixed
without any notice to the Synaptic Systems team. Therefore, some level of hostility
is only natural, and it is up to both programmers and testers to manage their
personal feelings so that work can be completed and business can profit.
Synaptic Systems’ programming team has been told they made a mistake
over 1700 times during the last eight years. Nevertheless, it is not NVP’s fault that
we discover and report so many software bugs. Indeed, it is our contracted role on
the RDS3 project to report every bug found so that it can be fixed. Without
attempting to speak for the programmers, it seems like they forget that fact from
time to time. In eight months with the project I have personally been yelled at via
email, been given an endless run around for no real reason, and been made to look
a fool to my superiors, albeit unsuccessfully. My biggest realization has been that
programmers deserve empathy, but not necessarily sympathy or pity.
As they say, the show, and the work, must go on. Admittedly, I have been
dismayed when the RDS3 progression actually reverses, and things that were
already fixed break again. Other times, any number of errors will be fixed, only to
reveal the next error in the same process that still won’t complete. With that said,
RDS3 has been slowly making a logical progression towards completion. The total
number of outstanding issues has been shrinking at the rate of one or two a week
for the last few months. What's more, I feel I have garnered some measure of
respect within my organization for biting my tongue when necessary. Whether the
programmers know it or not, NVP’s role as tester is as critical as their own to the
project. “Approximately 50% of the elapsed time and over 50% of the total cost
are expended in testing the program or system being developed.” (Myers 1979)
Consequently, it is crucial that a good software tester be able to roll with the
punches throughout the lifetime of the project.
“Testing is potentially endless, both theoretically and pragmatically. Yet
testing must stop at some time, even if bugs remain, because otherwise the effort
itself is pointless.” (Bezier 1995) In a complex software package it is certainly not
practical, and in actual fact, virtually impossible, to test all the ways the features
can interact. The truth of it is that testing is a best effort. The tester must put
every effort forward to discover all the bugs, but complex software offers nearly
infinite possibilities, and testing them all is futile. Testing endless possibilities
ultimately comes down to a matter of probability; when dealing with infinity, testing
should focus on whether the sequence is ever likely to occur, rather than is the
program likely to fail if it does. There must come a point at which the designers,
the programmers, the testers, and the management all agree that the product is as
good as it is ever likely to be - even if that point is motivated by financial reasoning
rather than reality.
Most software is never finished, so realistic goals must be set as to the level
of testing necessary to complete the project and keep overall costs to a minimum.
Obviously, testing software for a nuclear reactor should have a large QA budget set
aside, and great care must be taken to eliminate even the smallest issues.
Alternatively, web pages may not need high levels of accuracy before all parties are
satisfied. If a project overruns for even one day because bugs still exist even
though a reasonable effort was made to eliminate them, money and time are
wasted. That is not to say that products should be released before a best effort is
given, but “if the software in question controls a large industrial factory or process
whose cost is $500,000,000, at an interest rate of 10% per year that works out to
$137,000 per day in interest costs, or in terms of prevailing programmer costs, a
burn rate of 15 man-months per day.” (Bezier 1984) It doesn’t take many days
for the cost of the idle plant to exceed the cost of some of the most expensive
software many times over, even in 1984 terms. Every project is different, and
requires different levels of quality assurance and control. Goals should be
reasonably achievable while still adhering to the intended use and environment for
RDS3 does not function as expected, and testing for the project is currently
far from complete. Towards the end of my coop semesters I saw real progression
in the testcases and outstanding issues. Despite the progression towards
completion, there are still modules that have not been programmed yet. Testing
began during the requirements phase of the project, and testcases have been
written for the unfinished modules, but they have never been executed. Therefore,
RDS3 has not reached a point, notwithstanding the fact that it has taken eight
years to date, where testing is reasonably complete and the project can be closed.
Using best estimates, it will be at least a year before NVP is able to make a decision
on whether enough effort has been expended finding bugs.
Preventing a bug is cheaper than discovering one, plain and simple. Oddly
enough, testers must strive to put themselves out of business; a good tester does
not just find current bugs, they must anticipate where future bugs are likely to
occur and take an active role in ensuring even potential problems are fixed.
Relatively new developments like computer generated code, automated testing, and
revised design methods all have their place in reducing the total number of issues,
but when all is said and done, testers will always be needed. In the short term,
there will never be a practical way to eliminate every bug, and as software projects
increase in size and complexity, the role of the tester can only become more
Conversely, automated testing, although time consuming at first, pays for
itself in the long run. “Attempted justification for manual testing usually is based
on a gross underestimate of the number of times tests will be run. …an order of
magnitude or more.” (Bezier 1995) If tests are only run once, the extra time and
effort spent automating them is wasted, but the fact of the matter is all tests are
run many times over. On RDS3, I have executed all 448 Folder module testcases
at least 4 times, and others have executed them many times more before me.
Testing the module required a solid week’s worth of painstaking effort each time.
Had the tests been automated, they could have been executed in a matter of
minutes or hours. It is important to keep in mind though that considerable time
must be spent automating the tests in the first place. Furthermore, even testers
are not immune to the occasional bug, and the tests themselves must be tested.
Finally, requirements change over the course of a long project, old tests become
invalid, and new ones must be created, all of which require that automated tests be
adjusted. Setting up automated testing is far more time consuming than just
manually testing, but sooner or later the cost and time savings are substantial.
The future of software testing is developing along two distinctly different
schools of thought: fully documented processes and procedures versus ingenuity
and invention of test practices to suit each specific need. Testers are required to
watch over projects, but few people if any watch the watchmen. Metrics have
already been developed that quantify the procedures, results, and effectiveness of
testing, and those metrics will only continue to improve. Nevertheless, many
testers believe “testing is a set of skills that allow the tester to select or invent
testing practices to suit each unique situation. This belief directly contradicts the
IEEE 829 Test Documentation Standard, and organizations such as the Food and
Drug Administration who promote them.” (Software Testing 2007) Testers must
be prepared to fully document standard procedures and results, and such practices
help to ensure effective, complete testing. They must also be resourceful because
every test is different, and there is no single rule, or even set of rules, that can test
for every condition. In the end, maturity models and agile testing are not
contradictions in terms; the tester is required to work creatively while still following
the best practices thus far. Therefore, future testers must closely follow both
schools of thought.
“The perfect development process does not exist, and the probability of it
existing in the foreseeable future is close to nil.” (Perry 1995) Quality assurance
and software testing are as essential to any large programming project as the
programming itself. At this point, my foray into the field of software testing is
complete, even though the RDS3 project itself is still far from completion. In fact, if
current rates continue, I may be able to return to NVP after graduation in
December to complete the work. As far as The Collectives, Synaptic Systems, and
KPMG are concerned, I am simply another tester who has come and gone during
RDS3’s eight year saga. However, I feel I will be missed at NVP, and all in all, the
experience has been overwhelmingly positive. With the knowledge I have gained,
my future as a tester or developer looks bright.
Inevitably, testing software is as just as complex a process as writing it.
Quality assurance is a vast field where travelers are more likely to lose their way
than to discover the shortest true path. Testers must choose the best strategies
and methodologies, like black-box testing in the case of RDS3, if they are to
succeed. They must always remember that their work should be as specific as
possible so that misunderstandings can be avoided, and so that the work is
reproducible, because generations of testers will follow - true for both RDS3 and in
general. Testers must keep in mind that ultimately it is the people who drive any
business, and simple differences must be set aside for the greater good. They
should know that complete testing on large systems is impossible, but with their
best efforts, high quality products, and a job well done, can still be achieved.
Finally, testers must be aware that quality assurance as a field is continually
evolving, and that they must evolve with it in order to stay effective and efficient.
Though the road to completion may be long and hard, in the end everyone benefits.
So why is it that a buggy software project is blamed on the testers and not
the programmers, while a product of exceptional quality is thought to be the result
of excellent programming and design? There is certainly some truth to the fact that
a good design can reduce the potential number of bugs. The design phase,
however, comes at the start of the project when all eyes are sharp and the future
looks bright. Ideally, a product is tested throughout the software lifecycle, but it is
often the case that professional testers are only brought in after a project has spun
out of control. Furthermore, “professional programmers average 1.2 bugs for every
200 lines of code.” (Paracoder Parametric 2006) Without software testing, large
projects quickly become so buggy as to be useless. The long and short of it is,
when software testing is done correctly, the end user is never conscious of a bug
having existed in the first place, whereas when it is done incorrectly, the user
cannot help but be painfully aware of the imperfections. When software works as
expected, the results can be impressive, or even thrilling, but the hard fought
hours, days, or years of testing that made it that way are invisible. Truly, the job
of software tester can be a thankless one, but my time at NVP has taught me the
rewards are great.
Bezier, Boris. 1995. Black-Box Testing: Techniques for Functional Testing of
Software and Systems. New York: John Wiley & Sons, Inc.
Bezier, Boris. 1984. Software Testing and Quality Assurance. New York: Van
Myers, Glenford J. 1979. The Art of Software Testing. New York: John Wiley &
IBM. 2003. Bridging the Gap Between Black-Box and White-Box Testing.
(accessed April 24, 2007).
Hendrickson, Elisabeth. 2002. Managing Technical People (When You’re No
Techie). STQE Magazine. http://testobsessed.com/wordpress/wp-
content/uploads/2007/01/mtpwynt.pdf (accessed January 10, 2007).
Marks, David M. 1992. Testing Very Big Systems. New York: McGraw-Hill, Inc.
Spectrum Research. 2006. Paracoder Parametric ASP and SQL Source Code
(accessed April 24, 2007).
Wikipedia. 2007. Black-box Testing.
http://en.wikipedia.org/wiki/Black_box_testing (accessed April 23, 2007).
Wikipedia. 2007. Software Testing. http://en.wikipedia.org/wiki/Software_testing
(accessed January 11, 2007).
Perry, William. 1995. Effective Methods for Software Testing. New York: John
Wiley & Sons, Inc.