Stephen Allott - Testing Techniques - Are they of any Practical Use? - SoftTest Ireland
Testing Techniques – Are they of any Practical Use? Stephen Allott Bsc (Hons) MBCS CITP ElectroMind Ltd firstname.lastname@example.org Abstract solving or analytical skills are missing from the tester’s repertoire. Or is it the case that the techniques are out Testing techniques are taught in Universities and of date and no longer apply to testing of a modernon many industrial training courses yet there is software development project? These issues areevidence to suggest that, in the corporate world, explored and illustrated using case studies from thetesters often find it difficult to apply the techniques and financial services, software and travel industries.instead rely on domain knowledge and experiencewhen designing tests. This is a shame as testing 2. Backgroundtechniques are very effective at reducing the cost oftesting by providing better coverage with less effort. Many testers are swamped by the day to day pressuresWhen applying techniques it is important to be of having to get through so much testing in whatpragmatic and understand where they can add value to appears to be so little time. The problem is thatyour test process. A business focused approach, based ineffective approaches to testing tend to create tooon risk, is essential. Commercial strength tools must many of the wrong sorts of test which leads tobe considered to support the test process although duplicated effort and consumes extra time andhome grown solutions should not be ruled out in some resources. With a large number of tests it is moresituations. An appropriate investment in effective difficult to set up, control and manage the test project.testing will yield dividends in the longer term and so Introducing test automation is also more challengingfurther research is required to understand why testing and testers generally do not have any slack time totechniques are not much more widely adopted think about making improvements to the test process.throughout industry. At the unit testing level, Reid  has shown the1. Introduction effectiveness of test case design techniques such as equivalence class partitioning and boundary valueThis paper provides some practical insights into the analysis. These formal approaches, and related testapplication of testing techniques in industry and is case design techniques such as Pairwise, shouldbased on the author’s experience and observations as a provide better coverage, reduce the number of physicalsenior consultant working on software testing projects test cases and help facilitate test automation. Practicein the financial services and travel industry sectors. based models, for example TPI® – test processThis paper examines the relationship between testing improvement , can offer stepwise improvements andtheory and testing practice with particular reference to generate real benefits over time. The testing theorythe techniques used for test case design, automation of also suggests, for example, adopting a risk-basedtest execution and test process improvement. In approach will allow testers to prioritize their tests sogeneral, the theory is well understood yet it appears that they run the most important tests in the timethat some testing groups often shy away from using the available. In practice it is sometimes difficult to get allvery techniques that will help them to solve their most the stakeholders to agree on the real risks and provideimmediate and pressing problems. Is it possible that, sufficient information for testers to make an informedalthough the theory of test case design is well taught, choice about what to test.there is something missing from the skill set of thetester when he or she tries to apply the technique to areal life problem? It may be that some problem
3. Testing Techniques may feel are important. However, the beauty of using these techniques is that having created a reduced testAccording to Beizer  there are perhaps as many as set the team can then engage with other stakeholders to57 different varieties of test case design techniques and explore any additional testing that may be required,Copeland  has produced an excellent practical due to specific risks being identified with a particulardescription of some of the most popular and useful business function for example.techniques. Others prefer to use the term ‘techniques’in the higher level sense of an approach or 4.2 Management information systemmethodology, for example the ‘techniques’ of riskbased testing , or ‘this is our technique for This project aimed to consolidate financial data fromproducing a test strategy’. over 15 independent corporate repositories for a major software organisation. The technology architectureThe more common usage of the term test technique comprised an SQL data warehouse with MS Excelrelates to either black box or white box techniques. support for XML. The application characteristicsBoth types require the designer to specify inputs and included over 30 data feeds, 100 locations, multipleexpected outputs in advance of execution. Black box data warehouses three profit & loss forecasts, 30techniques  help to design tests based on the million rows, and several terabytes of storage.functionality of a software component without the testanalyst having to know about the details of how the The original testers were struggling to test this highcomponent works. White box techniques are based on profile application. There was no overall planning,the internal structure of a component and are mainly poor communication, a scattergun approach to theused by developers or technical testers. testing and no evidence of risk-based testing. They were attempting to test everything, logging defects thatFormal descriptions of test case design techniques, were extraneous, and did not keep any coverageexamples and their related coverage measures can be measures to monitor progress.found in BS7925 . The advantages of usingtechniques are many and include consistency, Average Change Request/Month=56, Average Churn of Change Requests/Month = 148, Average Churn/Change Request =3.5improved coverage, auditability and repeatability of 300 300tests. 250 250 Churn of Change Requests 200 200 Change Request4. Case Studies 150 150 100 1004.1 Front end trading application 50 50 0 0 June July August September October November December January A European bank reduced their test set for a GUI Monthsbased trading application from several thousand to just Change Requests Change Request Churn Bugsa few hundred key tests using test case designtechniques. This enabled automated regression testing Figure 1. Requirements churnfor an important migration from one software releaseto another. There are two popular approaches to Between June and October the application wasPairwise testing techniques which reduce the number unstable. The number of change requests continued toof physical test cases required: Orthogonal Arrays  rise and the churn, the number of times a change wasand the “all pairs” algorithm . Instead of testing all made after the testers received the change request, wasthe combinations for all the values for all the variables, also on the increase (figure 1).Pairwise techniques require that you be more selectiveand test just all pairs of variables. This saves time as An analysis of the defects (figure 2) by severityyou have significantly fewer tests to run. Also, 100% (impact of the defect) and priority (when will it beall pairs coverage gives the team more confidence in fixed) concludes that 62% of severity 1 and severity 2the testing achieved. If you later decide to automate defects, the key release criteria, had no priority setyour test execution, there are fewer scripts to develop, (356/574) leading to overload of both the developersmaintain and manage thus enabling the automation and the original test team.process. It’s not a magic solution and may miss someimportant combinations that you or other stakeholders
of $300,000 from the introduction of the new team. If Severity they had taken this approach from the start the savings would have been almost $825,000 over the year. There 1 2 3 4 Total were no defects found from January onwards, despitePriority 1 57 81 14 152 further changes, and there were no defects reported inPriority 2 19 52 12 1 84 production during release 1.Priority 3 6 14 10 30 The business team was content and the test group’sPriority 4 2 1 3 reputation was enhanced by their success. There wereNo priority 68 288 114 10 480 additional savings in the sales & marketing function due to the overall increased stability of the application. Total 146 428 154 21 749 The test lead in this project offered to help with unit Figure 2. Snapshot of defects by severity/priority testing and was able to get an unofficial early release to execute all test cases in advance of the officialAdditional investigation of defects by resolution code release into test. Curiously, after a while, the(figure 3) suggests that almost a third (32%) of the developers started to produce better code, which wastotal defects reported are questionable (*241/749) an interesting and useful side benefit that had not beenwhich is not what you want from a testing perspective. expected at the outset. Severity 4.3 Investment banking middleware Resolution 1 2 3 4 Total By Design* 10 37 29 4 80 A global leader in corporate financial services found that, despite having a good track record of Duplicate* 6 33 7 1 47 formal acceptance testing, there was still an External 2 16 4 0 22 unacceptable level of defects found during production. Fixed 86 254 95 12 447 This was not confined to one or two projects but an Not Repro* 29 56 4 0 89 across the board problem for the group which is Postponed 12 17 6 4 39 responsible for a variety of “middle office” systems Will Not Fix* 1 15 9 0 25 that process the bank’s business transactions after the front end capture has taken place. Total 146 428 154 21 749 An initiative to downsize a variety of key systems Figure 3. Snapshot of defects by resolution code throughout the company meant that various transaction flows had to be re-routed as systems through whichThe test lead was a very experienced manager who they originally traveled were taken out of service.selected from the original test team those that had the Having moved the flows, functionality which had beenmost appropriate technical skills. They had the SQL working correctly suddenly stopped working. This is aexperience and technical ability to build their own classic regression testing problem faced by manyautomation tool specifically tailored for this project. companies and not easy to solve without automation.The tool was well documented, reusable, andautomated 80% of the existing manual test suite. The Employing more people was not an option as thetest lead did not do any test execution. What he did overhead of manually maintaining the regression testwas to involve the new test team from the beginning pack was too high. Therefore a decision was quicklyand impose control on the project to reduce the churn taken to look into automation options. At the timein requirements and change requests. This had the there did not seem to be an automated regressioneffect of stabilizing the application and the defect testing tool on the market that would help. Primarilycount dropped dramatically in November (figure 1). this was because most tools assumed that there was some sort of GUI interface which could be used toThe original one-year project cost was projected at drive the tests (either through capture replay or$2.6m and after 6 months approx. 50% of this cost had scripting). The majority of data was received frombeen incurred with little to show in terms of a stable other systems within the company; none of them had aapplication. For the period November to April, the user interface where the same data could be entered.cost dropped to approx. $1m yielding an overall saving
For the people developing on these other data 4.4 Travel industry web applicationcapture systems they had already avoided usingcommercially available tools and developed their own. Table 1 shows a section of a test maturity matrix asTheir UI was effectively a ‘dumb’ interface and easy to a result of a quick scan assessment, using the TPI®test manually. All the serious processing was done model , of a testing process for a web application‘server side’ which led to the decision being taken to in the travel industry. The TPI® model proposes thatdevelop their own automated regression testing tool. progress should be made (from level A, to B, to C) in each of the 20 key areas at a steady rate and so the area After a short study the group decided that they under the scales 1 – 5, called controlled, should all becould not quite quantify the potential savings and so a highlighted before any attempt is made to make thedecision was taken to “just do it” based on the feeling process more efficient (scales 6 to 10) or optimizedshared by all stakeholders that this was “the right thing (scales 11 to 13 – not shown). The scales, levels, keyto do” for the organization. A small amount of seed areas and dependencies between key areas are basedmoney was allocated to the project to help develop the on practical experience and are fully described in theinitial concepts on a trial basis. If it worked out, which model. In this organization an attempt was made toof course it did, additional funding would be made develop a risk-based test strategy and introduceavailable to roll out the tool throughout the automation. However, the model requires that anyorganization. improvement is dependent upon testing techniques. Improvement of key area 1, test strategy, is dependent A development resource from a NY team who had on techniques (5B). For test automation (8) the modeltest tool experience was assigned to the development also shows a dependency on techniques (5A & 5B).activities to work alongside the VP in charge of the Progress on key area 2, testing life-cycle modelproject. The underlying test automation framework beyond level A is dependent on static test techniquesused was based on IBM’s STAF and STAX software (6A).which is freely available from the Internet. The VPput together a vision document (30 functions) which Table 1. Test Maturity Matrix (extract)was walked through with various senior managers.The original tool was developed in Java which is a Scale 1 2 3 4 5 6 7popular platform in the US (there is a possibility that it Test strategy (1) A Bwill be moved to C#). Life-cycle model (2) A B Moment of involvement (3) A B The design of the tests and the creation of test data Estimating and planning (4) Aevolved over time; some issues remain. Large Test techniques (5) A Bvolumes of data are not necessarily complex althoughfurther work needs to be done in this area; the Static test techniques (6) A Bapplications and message flows are very complex due Metrics (7) Ato the large number of switches and parameters within Test automation (8) A Bthe applications. Test environment (9) A B Office environment (10) A There were many benefits. Regression tests are Commitment & motivation A Bnow executed unattended (overnight) and so regression Test functions and training A Btesting is now done in days instead of weeks. It isrepeatable and tests can be executed in any Scope of methodology (13) Aenvironment. Errors were detected earlier thus saving Communication (14) A Bdollars. There is a dedicated test team that owns the Reporting (15) A B Ctest product, scripts and regression test data. There is Defect management (16) A B Can improved focus of key business and technical Testware management (17) A Bresources on more critical items (this allows the Test process management A Bbusiness to focus on Acceptance Testing rather than Evaluation (19) ARegression Testing). Finally, maintenance costsreduced due to improved quality of code. Low-level testing (20) A B
5. Summary & Conclusions 6. ReferencesMany testers try to use techniques such as equivalence  Reid S, Module Testing Techniques – which are the mostclasses, boundary value analysis and decision tables effective? 1997yet Pairwise techniques appear to be difficult to applywithout tool support. Several commercial and freeware  Koomen & Pol, Test Process Improvement, Addison Wesley, 1999, ISBN 0-201-59624-5.tools are available on the market to support Pairwisebut considerable skills are required to use them  Beizer, Software Testing Techniques (Second Edition),effectively. Many companies see test automation as a Van Nostrand Reinhold, 1990, ISBN 0-442-20672-0.“silver bullet” to solve all their testing problems.However, if they do not have the basics in place these  Copeland, A Practitioner’s Guide to Software Testautomation attempts often fail. The TPI® model shows Design, Artech House, 2004, ISBN 1-58053-791-X.the dependencies required for introducing testautomation tools and this can be a useful guide when  Gerrard & Thompson, Risk-based E-Business Testing,considering automation. You must at least have some Artech House, 2002, ISBN 1-58053-314-0.basic test process in place and an understanding of test  Beizer, Black-Box Testing: Techniques for Functionalcase design. Risk-based approaches are of course Testing of Software and Systems, John Wiley & Sons, 1995,useful yet they are often difficult to implement in ISBN 0-471-12094-4.practice and need strong leadership and consultancyskills within the test group if they are to be  Reid et al, BS7925, A Standard for Software Componentsuccessfully implemented. Process improvement Testing, British Standards Institute, 1998techniques require a strong business case and buy-infrom everyone, including senior management, to kick  N. J. A. Slone maintains a library of 200 Orthogonalstart these activities. In the corporate world, Arrays www.research.att.com/~njas/oadir/index.htmlineffective approaches to testing tend to create too  Pairwise techniques – The original allpairs algorithmmany of the wrong sorts of test which leads to devised by James Bach can be found at www.satisfice.com ;duplicated effort and consumes extra time and for a list of either commercial or freeware test case designresources. Effective testing using formal techniques and test data generation tools using several techniquescan save you time and money in the long run by including Equivalence Classes, Boundary Value Analysis,creating fewer tests with increased coverage. and Pairwise please visit www.pairwise.orgObviously there is some initial investment required intraining the test analysts to use the techniques and  TPI® is a registered trademark of Sogeti and refersshowing them how to apply them in practice in real life to a model for test process improvement which is nowsituations. The reluctance of companies to adopt test in the public domain www.sogeti.nl/tpitechniques probably stems partly from a lack ofknowledge, but perhaps also from a lack of confidenceas people seem to be too busy these days to take timeout to try new ideas.