Software Testing and Quality Assurance:


Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Software Testing and Quality Assurance:

  1. 1. Software Testing and Quality Assurance: The Royalty Distribution System Project April 25, 2007 Prepared by:Simon Hood Student Number: 0240170 th Work Term: 4 6th Last Academic Semester Completed: University of Guelph-Humber, Computing Coop Place of Employment: NVP Software Testing Inc. Job Title: Software Tester/Web Developer Supervisor: Neil Price-Jones
  2. 2. April 25, 2007 Dr. Michael Liu University of Guelph-Humber Toronto, ON, M9W 5L7 Re: Work Term Report, Software Testing and Quality Assurance Practices: The Royalty Distribution System Project Dear Dr. Liu, Attached please find my final work term report entitled Software Testing and Quality Assurance Practices: The Royalty Distribution System Project. During the past two semesters I have been employed at NVP Software Testing Inc. in Bolton under the supervision of the company’s Owner and President Neil Price-Jones. My time at NVP was spent testing the latest version of a software project intended to help our client distribute royalties based on television shows or movies aired by cable companies throughout Canada since 1990. The report details some of the methodology and strategies used to complete the testing of RDS, but as of this writing the eight year project is far past overdue and there is no end in sight for the near future. I began my first semester by learning the software as built, but quickly moved to testing the various modules and returned bugs in the project. My second semester was largely spent managing, testing, and retesting the more than 1700 known issues and 5000 existing testcases. The sheer scale of the numbers should provide a testament to the size and complexity of the RDS software. Despite the overwhelming state of affairs, I was able to advance NVP’s role on the project to a point where the company was strictly waiting on the programmers for new releases; all possible testcases had already been executed, and all new or existing issues had been dealt with appropriately. In general, my responsibilities grew steadily throughout the two semesters, and by the end of my coop I was responsible for the lion’s share of the work on RDS by NVP. Testing RDS has taught me a great deal about quality assurance and software engineering. It has given me the opportunity to see development from the end user’s point of view, rather than the view of the designer and programmer I am used to at Guelph-Humber. As I plan to do development work at some point after graduation, the knowledge I have gained will prove both practical and useful to my career. Although RDS has not yet been completed, NVP hopes the project will be finished sometime in 2008, and it is possible I will return to NVP with my degree in hand in time to see the project come to a close. If you have any questions regarding my report, please feel free to contact me. Warmest Regards, Simon Hood
  3. 3. Abstract The software lifecycle is not complete without some measure of quality assurance and software testing. Many computer science graduates spend a lifetime learning the intricacies of the field, and there is always more to learn. My coop semesters at NVP Software Testing Inc. have offered me some insight on both the best and worst practices, as applied to the long running and now overdue RDS project. I have learned many things in the last eight months, and a few of my most outstanding lessons are illustrated. For example, I have learned that it is critical that testers choose the best strategy and methodology while tackling large projects. Furthermore, a tester must be very specific with regard to both testcases and the resulting issues. I am now aware that testers and programmers are opposing forces, and balance must be achieved if the project is to succeed. Additionally, complete testing is rarely possible; quality assurance itself is a balance of priorities and a best effort situation. Finally, I believe I have achieved some insight on what the future holds in store for software testing in general. That said, it is obvious that the intricacies of the field can be maddening at times, but the rewards for a job well done are well worth it - even if they go unnoticed by the end user.
  4. 4. Table of Contents Abstract ................................................................................................................. 3 Table of Contents..................................................................................................... 4 Glossary ................................................................................................................. 4 Introduction ............................................................................................................ 5 Problem.................................................................................................................. 7 Black-box Testing.................................................................................................. 9 Specificity .......................................................................................................... 11 Programmer Hostility ........................................................................................... 13 Complete Testing ................................................................................................ 14 The Future ......................................................................................................... 16 Conclusions........................................................................................................... 18 References............................................................................................................ 20 Glossary DOS: Disk Operating System QA: Quality Assurance MSSQL: Microsoft SQL NVP: No Variances Please! RDS: Royalty Distribution System SQL: Structured Query Language VB: Visual Basic
  5. 5. Introduction I spent the last eight months working for NVP Software Testing Inc. in Bolton. NVP provides businesses with complete software testing solutions. Services offered include: software testing consulting, management, development, execution, automation, training, staffing, and outsourcing. However, it may appear strange that a for company specializing in software, NVP itself does very little programming. The software industry in general, and software testing in particular, have grown to the point where many companies are able to survive just by outsourcing niche services. At NVP, for example, the application development is usually left up to the contracting company, or a separate company altogether, as need’s dictate. In most cases, NVP is called in for very large software projects where test management and accountability are outside the abilities of our clients. Unfortunately, even with expert, professional quality assurance and test management, for some software projects costs can still skyrocket, and schedules can become years out of date – as seen in the RDS3 project. The project I worked on is intended for divvying out a pool of money that has grown to many millions of dollars. In much the same way that radio stations must pay out funds for each song played on the radio, every Canadian cable company is required to pay royalties on every television show or movie aired on any channel at any time to compensate the rightful owners, provided those shows cannot also be seen locally for free with a TV antenna. Two of NVP’s clients, known as The Collectives to the testers, distribute the money using a program called RDS, short for Royalty Distribution System.
  6. 6. The previous version of RDS has been in use for many years, but is now a legacy application; RDS2 runs in a DOS shell and is based on an obsolete FoxPro database. Therefore, The Collectives commissioned an upgrade for the program, expecting to bring it up to current computer standards in only a few months of programming. The upgrade to RDS3 was supposed to be a line for line conversion of the source code to VB 6.0 and an MSSQL database server. Roughly eight years later, the new version is finally nearing completion despite much mismanagement and poor programming. A deep-seated paradox exists with regards to software quality assurance. If software development is well executed, it is widely accepted that the designers and programmers did a good job, and costs are kept to a minimum. If, on the other hand, software development results in a poor quality product, it is perceived as the fault of the testers for not catching the bugs, and the costs go up trying to iron out the bugs. In reality, high quality software products are, more often than not, the result of meticulous testing, while the quality of substandard products can be directly linked to their lack of testing in their lifecycle as a whole. NVP is not responsible for the problems with RDS3. My company has done its part endlessly testing and retesting frequent new releases with the latest bugs fixed, and then shipping the new bugs back to the programmers so that they can be fixed in turn. In reality, the problem seems to be the lack of a clear manager for the project; programming, testing, and auditing are all outsourced to different companies because the client is incapable of directing the technical aspects. That, coupled with the fact that the total funds to be allocated still exceed the total development costs to date by many millions of dollars, have lulled The Collectives
  7. 7. into a kind of complacency. In June of 2006, the client’s board of directors finally realized that the eight year project had spun out of control and decided to put a due date on it for November 1st, 2006. Unbelievably, the date came and went, and as of April 2007 RDS3 is still at least a year away from completion. Problem The problem of how to finish up RDS3 in the shortest time possible, with all eyes watching, fell to the three outsourced companies – NVP Software Testing Inc. for testing, Synaptic Systems Inc. for programming, and KPMG for auditing. It was more or less at this time that my first coop semester began, and I was asked to help test RDS3 on my first day. Towards the end of my second semester, the vast majority of the testing and a good part of NVP’s completion strategy had fallen on my shoulders. RDS3 is by no means a small program. Certainly it does not compete in scale with programs like the new Windows Vista, but it does perform some very complex functions. If the functionality of the application is reduced to its simplest form, it can be said that television shows aired by Canadian cable companies are placed as properties, the owners and their banking information are recorded as registrants, and the properties and registrants are linked to folders containing the show. The money for the many different broadcasts of the show can then be paid to the owner. The amount of payment is based on the weight of the show, or how many people actually watched it, multiplied by a viewing factor to take the number
  8. 8. of people watching at any given time of the day into consideration. Functionality for uploading new data into the system and importing old data from RDS2 is also provided. Needless to say, the application’s learning curve was steep. As I was faced with entire filing cabinets of documentation to familiarize myself with, initially I was forced to limit myself to learning only the modules I was assigned. Reams of documentation were available, and with nearly every new release, more documents, albeit modified versions of the old documents, arrived. The documentation itself came in the form of use cases, sequence diagrams, and short write-ups. Every use case was directly translated into dozens of specific testcases, with the number of testcases depending on the particular use case’s level of detail. For example, the details of a use case might state that when folder- registrant links are created from the Folder Management module, first “The actor selects one or more folders from the Folder List.” The details become a corresponding testcase with three major sections: setup, script, and expected results. The setup lists the conditions that must be present or met before the testcase can be executed. The script section contains the exact procedure to follow while carrying out the test. Lastly, the expected results inform the user what should happen after the procedure is followed. My first module, Payment Management, was based on a 120 page document translated into some 1040 testcases, no small endeavour. Gradually, I became familiar with both the testcases and documentation, and I was able to turn my attention to optimizing the process. I eventually came to the conclusion that testing RDS3 using a black-box strategy was the only method possible. I also learned that specificity is of critical importance in both testcases
  9. 9. and issues. Furthermore, despite my best efforts to be innocuous and politically correct, the programmers were still occasionally confused and sometimes became hostile. I discovered that complete testing in large systems is impossible, and that testing is a matter of providing the best effort. Finally, I now feel qualified to have an opinion on the future of software testing in general. Black-box Testing Putting it bluntly, NVP is not responsible for carrying out the code conversion to RDS3. NVP has been contracted to only test the program as built, while the actual programming is up to the team at Synaptic Systems. Over two semesters, I saw none of the actual code for RDS3, save a few lines of MSSQL database pseudo- code used to speed up the program. Even though I am familiar with both SQL and VB 6.0, it was simply not my job to fix the program. Therefore, any and all testing I performed was strictly black-box. Black-box testing refers to “the testing that you do when you put yourself in the software user’s place to verify that [the software] behaves the way it’s supposed to, no matter how it may be built.” (Bezier 1995) NVP’s RDS3 testcases are entirely based on the design documentation and serve a dual purpose: to eliminate any bugs encountered during the execution of the various modules, and to ensure the design specifications are met. Without knowledge of the source code, the RDS3 testing can only focus on creating the “subset of all possible testcases that has the highest probability of detecting the most errors.” (Myers 1979)
  10. 10. Granted, few other testing methodologies were possible in the case of RDS3 because NVP does not have access to the source code, and it may appear at first that black-box testing would not be the most effective method. White-box testing strategies like stubs and drivers are often easier to employ when evaluating code created from scratch. Stubs and drivers are a few lines of code created to make output to other modules possible without the actual finished modules, or to make required input arguments, often from other unfinished modules, available to the module being tested, respectively. It is often the case that large sections of code are missing while testing is performed, as the sections have not been written yet. As well, some program modules are so interrelated that it might seem as though testing cannot begin without a finished product, but stubs and drivers allow testing to continue without the completed sections. On the other hand, Synaptic Systems was already in possession of a finished, tested program in the form of RDS2, and essentially NVP was only required to test the system as a whole; RDS3 is simply a line by line conversion to a new SQL server and more modern programming language. Experiments comparing the effectiveness of black-box versus white-box forms of software testing found “white-box testing proves insufficient… for situations in which testing isolated components may not reveal integration errors with other components.” (Bryson 2003) Truth be told, RDS3 is a highly integrated series of modules, and NVP was able to discover over 1700 individual issues using only the black-box method.
  11. 11. Specificity Regrettably, issues and testcases are often misinterpreted. The systems used by NVP, Tracker for testcase management and CHGLOG for issue management, rely strictly on the written word. As such, the two applications do not always permit complex procedures to be conveyed clearly in the few short sentences allowed. Moreover, the lack of normal body language clues given through face to face interaction, or even the conversational meter given through two-way interaction in real time, make it only natural that some things will be misconstrued. Therefore, it is imperative for a software tester to be as specific as possible while writing both testcases and issues so as to avoid any misunderstandings. Simply put, misunderstandings can cost time and money at best, and create misapprehension and conflict at worst. Normal testing procedure for RDS3 involves writing issues as they occur into CHGLOG as a logical series of steps needed to reproduce the problem. The programmers then read the issues, verify the problems, fix the code or database where required, and return the issues to the testers so they can be tested again. When working properly, the cycle generally takes a few weeks to a few months to complete, depending on the priority of the problem. So if an issue is returned for clarification, rather than being corrected, the resulting delay is expensive. There are, in fact, still issues in CHGLOG that were entered almost nine years ago, although they have usually been escalated to The Collectives for a final decision. As mentioned previously, the RDS3 project has been running on overtime for quite a while, so delays are to be avoided wherever possible.
  12. 12. Testcase execution can also be problematic. Requirements change on any project as programming progresses; features are occasionally dropped to save money, or even added where absolutely necessary, so from time to time the functionality of the program changes. Executing the related testcases is impossible if a feature is dropped, and because no notice is given in Tracker it is generally up to the tester to figure out why it cannot be completed. Likewise, no testcases exist to ensure a newly added feature is bug free, so new ones must be written. Sometimes running ambiguous testcases is a best effort situation anyway, as the wording of the script or expected results is unclear. In practice, all software testcases should be as precise and clear-cut as possible, or the intended test may not be carried out correctly. Inaccurate testing can result in logged failures where the program in fact passed, and days or weeks may be wasted. The programmers also become more than mildly unhappy when incorrect or duplicate issues are entered. Synaptic Systems and NVP are under a great deal of pressure to complete RDS3, and the programming team does not like to waste their time any more than we do. Unfortunately, the written word does not always carry the same intention as words spoken aloud, and a simple paragraph can appear disrespectful or insulting if not phrased correctly. Specificity in both testcases and issues is paramount.
  13. 13. Programmer Hostility Programmers and testers clash in exactly the same way artists clash with their critics. Both artists and programmers produce creative works, while critics and testers pick apart the finer details of the work making both positive and negative comments. In the case of RDS3’s programmers though, praise for a job well done is rarely heard; instead, only an endless barrage of criticism about their mistakes is received. The praise due to the programmers for a job well done is often not even verbalized by the testers, and issues are usually closed if fixed without any notice to the Synaptic Systems team. Therefore, some level of hostility is only natural, and it is up to both programmers and testers to manage their personal feelings so that work can be completed and business can profit. Synaptic Systems’ programming team has been told they made a mistake over 1700 times during the last eight years. Nevertheless, it is not NVP’s fault that we discover and report so many software bugs. Indeed, it is our contracted role on the RDS3 project to report every bug found so that it can be fixed. Without attempting to speak for the programmers, it seems like they forget that fact from time to time. In eight months with the project I have personally been yelled at via email, been given an endless run around for no real reason, and been made to look a fool to my superiors, albeit unsuccessfully. My biggest realization has been that programmers deserve empathy, but not necessarily sympathy or pity. As they say, the show, and the work, must go on. Admittedly, I have been dismayed when the RDS3 progression actually reverses, and things that were already fixed break again. Other times, any number of errors will be fixed, only to
  14. 14. reveal the next error in the same process that still won’t complete. With that said, RDS3 has been slowly making a logical progression towards completion. The total number of outstanding issues has been shrinking at the rate of one or two a week for the last few months. What's more, I feel I have garnered some measure of respect within my organization for biting my tongue when necessary. Whether the programmers know it or not, NVP’s role as tester is as critical as their own to the project. “Approximately 50% of the elapsed time and over 50% of the total cost are expended in testing the program or system being developed.” (Myers 1979) Consequently, it is crucial that a good software tester be able to roll with the punches throughout the lifetime of the project. Complete Testing “Testing is potentially endless, both theoretically and pragmatically. Yet testing must stop at some time, even if bugs remain, because otherwise the effort itself is pointless.” (Bezier 1995) In a complex software package it is certainly not practical, and in actual fact, virtually impossible, to test all the ways the features can interact. The truth of it is that testing is a best effort. The tester must put every effort forward to discover all the bugs, but complex software offers nearly infinite possibilities, and testing them all is futile. Testing endless possibilities ultimately comes down to a matter of probability; when dealing with infinity, testing should focus on whether the sequence is ever likely to occur, rather than is the program likely to fail if it does. There must come a point at which the designers, the programmers, the testers, and the management all agree that the product is as
  15. 15. good as it is ever likely to be - even if that point is motivated by financial reasoning rather than reality. Most software is never finished, so realistic goals must be set as to the level of testing necessary to complete the project and keep overall costs to a minimum. Obviously, testing software for a nuclear reactor should have a large QA budget set aside, and great care must be taken to eliminate even the smallest issues. Alternatively, web pages may not need high levels of accuracy before all parties are satisfied. If a project overruns for even one day because bugs still exist even though a reasonable effort was made to eliminate them, money and time are wasted. That is not to say that products should be released before a best effort is given, but “if the software in question controls a large industrial factory or process whose cost is $500,000,000, at an interest rate of 10% per year that works out to $137,000 per day in interest costs, or in terms of prevailing programmer costs, a burn rate of 15 man-months per day.” (Bezier 1984) It doesn’t take many days for the cost of the idle plant to exceed the cost of some of the most expensive software many times over, even in 1984 terms. Every project is different, and requires different levels of quality assurance and control. Goals should be reasonably achievable while still adhering to the intended use and environment for the software. RDS3 does not function as expected, and testing for the project is currently far from complete. Towards the end of my coop semesters I saw real progression in the testcases and outstanding issues. Despite the progression towards completion, there are still modules that have not been programmed yet. Testing began during the requirements phase of the project, and testcases have been
  16. 16. written for the unfinished modules, but they have never been executed. Therefore, RDS3 has not reached a point, notwithstanding the fact that it has taken eight years to date, where testing is reasonably complete and the project can be closed. Using best estimates, it will be at least a year before NVP is able to make a decision on whether enough effort has been expended finding bugs. The Future Preventing a bug is cheaper than discovering one, plain and simple. Oddly enough, testers must strive to put themselves out of business; a good tester does not just find current bugs, they must anticipate where future bugs are likely to occur and take an active role in ensuring even potential problems are fixed. Relatively new developments like computer generated code, automated testing, and revised design methods all have their place in reducing the total number of issues, but when all is said and done, testers will always be needed. In the short term, there will never be a practical way to eliminate every bug, and as software projects increase in size and complexity, the role of the tester can only become more important. Conversely, automated testing, although time consuming at first, pays for itself in the long run. “Attempted justification for manual testing usually is based on a gross underestimate of the number of times tests will be run. …an order of magnitude or more.” (Bezier 1995) If tests are only run once, the extra time and effort spent automating them is wasted, but the fact of the matter is all tests are run many times over. On RDS3, I have executed all 448 Folder module testcases
  17. 17. at least 4 times, and others have executed them many times more before me. Testing the module required a solid week’s worth of painstaking effort each time. Had the tests been automated, they could have been executed in a matter of minutes or hours. It is important to keep in mind though that considerable time must be spent automating the tests in the first place. Furthermore, even testers are not immune to the occasional bug, and the tests themselves must be tested. Finally, requirements change over the course of a long project, old tests become invalid, and new ones must be created, all of which require that automated tests be adjusted. Setting up automated testing is far more time consuming than just manually testing, but sooner or later the cost and time savings are substantial. The future of software testing is developing along two distinctly different schools of thought: fully documented processes and procedures versus ingenuity and invention of test practices to suit each specific need. Testers are required to watch over projects, but few people if any watch the watchmen. Metrics have already been developed that quantify the procedures, results, and effectiveness of testing, and those metrics will only continue to improve. Nevertheless, many testers believe “testing is a set of skills that allow the tester to select or invent testing practices to suit each unique situation. This belief directly contradicts the IEEE 829 Test Documentation Standard, and organizations such as the Food and Drug Administration who promote them.” (Software Testing 2007) Testers must be prepared to fully document standard procedures and results, and such practices help to ensure effective, complete testing. They must also be resourceful because every test is different, and there is no single rule, or even set of rules, that can test for every condition. In the end, maturity models and agile testing are not
  18. 18. contradictions in terms; the tester is required to work creatively while still following the best practices thus far. Therefore, future testers must closely follow both schools of thought. Conclusions “The perfect development process does not exist, and the probability of it existing in the foreseeable future is close to nil.” (Perry 1995) Quality assurance and software testing are as essential to any large programming project as the programming itself. At this point, my foray into the field of software testing is complete, even though the RDS3 project itself is still far from completion. In fact, if current rates continue, I may be able to return to NVP after graduation in December to complete the work. As far as The Collectives, Synaptic Systems, and KPMG are concerned, I am simply another tester who has come and gone during RDS3’s eight year saga. However, I feel I will be missed at NVP, and all in all, the experience has been overwhelmingly positive. With the knowledge I have gained, my future as a tester or developer looks bright. Inevitably, testing software is as just as complex a process as writing it. Quality assurance is a vast field where travelers are more likely to lose their way than to discover the shortest true path. Testers must choose the best strategies and methodologies, like black-box testing in the case of RDS3, if they are to succeed. They must always remember that their work should be as specific as possible so that misunderstandings can be avoided, and so that the work is reproducible, because generations of testers will follow - true for both RDS3 and in
  19. 19. general. Testers must keep in mind that ultimately it is the people who drive any business, and simple differences must be set aside for the greater good. They should know that complete testing on large systems is impossible, but with their best efforts, high quality products, and a job well done, can still be achieved. Finally, testers must be aware that quality assurance as a field is continually evolving, and that they must evolve with it in order to stay effective and efficient. Though the road to completion may be long and hard, in the end everyone benefits. So why is it that a buggy software project is blamed on the testers and not the programmers, while a product of exceptional quality is thought to be the result of excellent programming and design? There is certainly some truth to the fact that a good design can reduce the potential number of bugs. The design phase, however, comes at the start of the project when all eyes are sharp and the future looks bright. Ideally, a product is tested throughout the software lifecycle, but it is often the case that professional testers are only brought in after a project has spun out of control. Furthermore, “professional programmers average 1.2 bugs for every 200 lines of code.” (Paracoder Parametric 2006) Without software testing, large projects quickly become so buggy as to be useless. The long and short of it is, when software testing is done correctly, the end user is never conscious of a bug having existed in the first place, whereas when it is done incorrectly, the user cannot help but be painfully aware of the imperfections. When software works as expected, the results can be impressive, or even thrilling, but the hard fought hours, days, or years of testing that made it that way are invisible. Truly, the job of software tester can be a thankless one, but my time at NVP has taught me the rewards are great.
  20. 20. References Bezier, Boris. 1995. Black-Box Testing: Techniques for Functional Testing of Software and Systems. New York: John Wiley & Sons, Inc. Bezier, Boris. 1984. Software Testing and Quality Assurance. New York: Van Nostrand Reinhold. Myers, Glenford J. 1979. The Art of Software Testing. New York: John Wiley & Sons, Inc. IBM. 2003. Bridging the Gap Between Black-Box and White-Box Testing. (accessed April 24, 2007). Hendrickson, Elisabeth. 2002. Managing Technical People (When You’re No Techie). STQE Magazine. content/uploads/2007/01/mtpwynt.pdf (accessed January 10, 2007). Marks, David M. 1992. Testing Very Big Systems. New York: McGraw-Hill, Inc. Spectrum Research. 2006. Paracoder Parametric ASP and SQL Source Code Generator. (accessed April 24, 2007). Wikipedia. 2007. Black-box Testing. (accessed April 23, 2007). Wikipedia. 2007. Software Testing. (accessed January 11, 2007). Perry, William. 1995. Effective Methods for Software Testing. New York: John Wiley & Sons, Inc.