Pairwise Testing


Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • A program with 10 inputs. If we select 10 test data values for each input, and want to test all combinations of inputs, boom! ... we would need 10 billion tests. So what do we do?
  • May end up guessing, but don’t want to start with guessing. Another technique is to use combinatorial design techniques, or experimental design techniques. These provide a systematic approach to running a subset of combinations, since you can’t run all combinations. Several approaches exist, at AT&T we had the OATS orthogonal array test system, and CATS constrained array test system. At Bellcore, AETG, or the automatic efficient test generator was developed to address this problem. All of these techniques DO reduce the number of combinatorial tests; however, it’s difficult to determine the impact on the fault-detection capability of the test set using these techniques. All of these tools require you to build a “test data model.” If the reduced test data model matches the software implementation, the fault-detection capability of the test set may be very good. If it doesn’t match you may have problems. So that’s the current state of things, lets look at how we defined the problem.
  • Pairwise Testing

    1. 1. Testing Software Engineering in Practice CS420
    2. 2. Review: What is Testing? <ul><li>IEEE: </li></ul><ul><ul><li>“The process of operating a system or component under specified condition, observing or recording the results, and making an evaluation of some aspect of the system or component.” </li></ul></ul><ul><li>Specified Condition = Test Cases </li></ul>
    3. 3. Review: What is testing? <ul><li>Rick Craig and Stefan Jaskiel: </li></ul><ul><ul><li>“ Testing is a concurrent lifecycle process of engineering, using and maintaining testware in order to measure and improve the quality of the software being tested” </li></ul></ul><ul><li>Includes planning, analysis, and design that lead to test cases </li></ul>
    4. 4. Review: What is testing? <ul><li>Testing is the process of comparing “what is” with “what ought to be” </li></ul>
    5. 5. Review: Test Cases <ul><li>Need to be designed: </li></ul><ul><ul><li>Systematic approach </li></ul></ul><ul><ul><li>Documentation & Planning </li></ul></ul><ul><ul><li>Goal and purpose </li></ul></ul>
    6. 6. Review: Test Case <ul><li>Inputs </li></ul><ul><li>Expected Outputs </li></ul><ul><li>Order of Execution </li></ul><ul><li>(and Testing Criteria) how do I know when I’m done testing??? </li></ul>
    7. 7. Review: Types of Testing <ul><li>Black Box </li></ul><ul><ul><li>Reqs and spec, no knowledge of internal structure </li></ul></ul><ul><li>White Box </li></ul><ul><ul><li>Based on internal paths, structure </li></ul></ul><ul><ul><ul><li>Ex1 - definition use pairs </li></ul></ul></ul>
    8. 8. Pairwise Testing <ul><li>A Web Site must operate with various </li></ul><ul><ul><li>Browsers: Netscape, IE, Opera, </li></ul></ul><ul><ul><li>Plug-ins: RealPlayer, Media Player, or none </li></ul></ul><ul><ul><li>Operating Systems: 95, 98, ME, NT, 2000, XP </li></ul></ul><ul><ul><li>Web Servers: Apache, Weblogic, IIS </li></ul></ul><ul><ul><li>OS Sever: Linux, NT, 2000 </li></ul></ul><ul><ul><li>-Large Number of combinations: ~1200 </li></ul></ul>
    9. 9. Worst Case <ul><li>Don’t test at all </li></ul><ul><li>Test all combinations as miss release date </li></ul><ul><li>Choose one or two and hope for best </li></ul><ul><li>Choose test you have already run </li></ul><ul><li>Choose test that are easy to create </li></ul><ul><li>Make a list of combos and choose first few </li></ul><ul><li>Make a list of all the combinations choose a random subset </li></ul>
    10. 10. Best Case <ul><li>Choose a specially selected, fairly small subset that find a great many defects, more that you would expect from a small subset. </li></ul><ul><li>But how? </li></ul>
    11. 11. Combinatorial Testing Problems <ul><li>Developers/Testers frequently face situations where testing combinations of inputs to a system is important. </li></ul><ul><li>Examples of systems that require combinatorial testing include: transaction-based systems, form-based systems, APIs, others. </li></ul>
    12. 12. Combinatorial Testing Problem Combinatorial testing problems generally follow a simple input-process-output model; the “state” of the system is not the focus of combinatorial testing. X 1 X 2 X 3 . . . X n System S
    13. 13. Combinatorial Testing <ul><li>Testers begin the process of combinatorial testing by selecting test data values for each system input. </li></ul><ul><li>The number of combinations of the selected test data values is very large; phenomenon referred to as the “combinatorial explosion”. </li></ul>
    14. 14. OPEN TICKET MAINTENANCE TICKET DATA Tkt Id: ____ Entity for Tkt: __________________________ Referred to Entity: __________ Work Type: ___________ Loc/Fic: _______ Loc/Unit: ________ Title Loc: _________________________ Related Entities: _______________________________ Ticket Create Time: _________ Date: ______ History Log Id: __________________ Date: __/__/__ Rptd Code: _______ Rpt Description: ________________ Report Type: ____ Direction: ______ CLFI: ____________ Do you want to enter comments?: ____ With 3 values per variable, there are 3 20  3.5 × 10 9 inputs to test I.S. Dunietz, W.K. Ehrlich, B.D. Szablak, C.L. Mallows, and A. Iannino, “Applying Design of Experiments to Software Testing,” Proceedings of the Nineteenth International Conference on Software Engineering , pp. 205-215, ACM Press, May 1997.
    15. 15. Combinatorics - Combinations <ul><li>The number of ways of picking k unordered outcomes from n possibilities. </li></ul>Example:
    16. 16. Combinatorics - Permutations <ul><li>The number of ways of obtaining an ordered subset of k elements from a set of n elements is given by </li></ul>The number of permutations on a set of n elements is given by n!    
    17. 17. Combinatorial Test Reduction <ul><li>Testing all combinations of selected test data values is usually impossible! </li></ul><ul><li>Not to test any combinations of input may increase the risk that a software fault escapes into a fielded system. </li></ul><ul><li>Testers look for ways to select an effective and economical subset of test cases. </li></ul>
    18. 18. Pairwise Testing Don’t test all combinations. Test all pairs. - Bernie Berger, STAREast 2003 International Conference on Software Testing Pairwise testing is a popular, frequently written about technique. Covered in Books by Kaner, Bach, & Pettichord; Copeland; McGregor & Sykes; Splaine & Jakiel. More than 50 conference and journal papers on the topic.
    19. 19. Pairwise Definition <ul><li>Pairwise testing – an approach to combinatorial testing that executes a pairwise test data set. </li></ul><ul><li>Pairwise test data set - A set of test cases that covers all combinations of the selected test data values for every pair of a system’s input variables. </li></ul><ul><li>Pairwise testing is also known as 2-way testing; other approaches include 3-way, 4-way, … , n-way. </li></ul>
    20. 20. All-pairs testing or pairwise testing <ul><li>is a combinatorial testing method that, for each pair of input parameters to a system tests all possible discrete combinations of those parameters. This can be done much faster than an exhaustive search of all combinations The number of tests is typically O( nm ), where n and m are the number of possibilities for each of the two parameters with the most choices. </li></ul><ul><li>The reasoning behind all-pairs testing: the simplest bugs in a program are generally triggered by a single input parameter. The next simplest category of bugs consists of those dependent on interactions between pairs of parameters, which can be caught with all-pairs testing. Bugs involving interactions between three or more parameters are progressively less common, whilst at the same time being progressively more expensive to find by exhaustive testing, which has as its limit the exhaustive testing of all possible inputs. </li></ul>
    21. 21. Pairwise Example A={1, 2} B={NW, SE} C={TDK, BDM} System T
    22. 22. Pairwise Example A={1, 2} B={NW, SE} C={TDK, BDM} A, B, C ------- 1,NW,- 1,SE,- 2,NW,- 2,SE,- A, B, C ------- 1,-,TDK 1,-,BDM 2,-,TDK 2,-,BDM A, B, C ------- -,NW,TDK -,NW,BDM -,SE,TDK -,SE,BDM Don’t care condition indicated with by a dash ( - )
    23. 23. Pairwise Reductions 15 1.6 x 10 6 3 13 21 1.2 x 10 19 3 40 8 128 2 7 Size of pairwise test set Number of combinations Number of selected test data values Number of inputs
    24. 24. Motivation for Using Pairwise Testing <ul><li>Generates small test sets, relative to the combinatorial test data set. Size of the pairwise test set is usually 1-20% of the combinatorial test data set </li></ul><ul><li>Many combinations of selected test data values tested. </li></ul>
    25. 25. Motivation for Using Pairwise Testing This method finds all double-mode faults that are two parameters conflicting with each other . G.T. Daich, “New Spreadsheet Tool Helps Determine Minimal Set of Test Parameter Combinations,” CrossTalk , August 2003, pp. 26-30. An analysis of field data at Bellcore indicated that most field faults are caused by interactions of 1 or 2 fields. G.C. Patton, “DAT (Defect Analsysis Team) 1986-1990 Overview,” Internal Bellcore Technical Memo, 1991.
    26. 26. Motivation for Using Pairwise Testing Wallace and Kuhn determined that 98% of the reported software defects in recalled medical devices could have been detected by testing all pairs of parameter settings. D.R. Wallace, D.R. Kuhn, “Failure Modes in Medical Device Software: An Analysis of 15 Years of Recall Data,” Intl. Journal of Reliability, Quality, and Safety Engineering, vol. 8, no. 4.
    27. 27. Creating Pairwise Test Sets <ul><li>Generation using Orthogonal Arrays </li></ul><ul><li>Algorithmic generation </li></ul>
    28. 28. Orthogonal Array <ul><li>Mathematical curiosity discovered by monks in the 1400s </li></ul><ul><li>Promoted by Dr. Genichi Taguchi in the 1950s for use in statistical test design in manufacturing, agriculture, medicine, etc. </li></ul><ul><li>Orthogonal Array: Two dimensional array of numbers in which any two columns in the array receive an even distribution of all pairwise combinations of values in the array. </li></ul>
    29. 29.     ORTHOGONAL ARRAY 2 1 1 2 1 2 2 8 1 2 2 1 1 2 2 7 1 2 1 2 2 1 2 6 2 1 2 1 2 1 2 5 1 1 2 2 2 2 1 4 2 2 1 1 2 2 1 3 2 2 2 2 1 1 1 2 1 1 1 1 1 1 1 1 Col.7 Col.6 Col.5 Col.4 Col.3 Col.2 Col.1 Expt. No.
    30. 30. Orthogonal Array <ul><li>Orthogonal arrays do not exist for all setting of input and values </li></ul><ul><li>Sloane’s catalog of orthogonal arrays. </li></ul><ul><li>rdExpert – commercial pairwise testing tool based on orthogonal arrays. </li></ul>
    31. 31. Algorithmic Generation <ul><li>Couldn’t I just write a program to do this? </li></ul><ul><li>AETG – Commercial n-way test generations tool. </li></ul><ul><li>Allpairs – Free perl script from James Bach </li></ul><ul><li>tvguwm – Free java source code </li></ul>
    32. 32. How Well Does Pairwise Testing Work? The all-pairs and all-values tests detected 88% of the correctness and convergence bugs, but only half of the interface and engine bugs. B. Smith, M. S. Feather, and N. Muscettola, &quot;Challenges and Methods in Testing the Remote Agent Planner,&quot; in Proc. 5th Conf. on Artificial Intelligence Planning and Scheduling (AIPS 2000), 2000, pp. 254-263.
    33. 33. Pairwise vs. Random Selection We would like to know that our testing techniques work at least as well as random testing. Make a list of all the combinations and choose a random subset.
    34. 36. How Many Pairs does a Random Set Cover? 99.99% 99.99% 99.6% DMAS 99.07% 97.66% 94.5% LAS 4Rand 3Rand 2Rand System
    35. 37. Pairwise Weakness Fault detection capability of a combinatorial test set is determined by: 1) The selected test data values. 2) The combinations of these values that are tested. 3) The combinatorial characteristics of the system under test. The pairwise technique addresses point 2 only!
    36. 38. Pairwise Conclusion <ul><li>Pairwise testing is the execution of a test data set that covers all combinations of the selected test data values for every pair of a system’s input variables. </li></ul><ul><li>It is used on combinatorial testing problems that appropriate for pairwise testing (i.e., program outputs influenced by at most two inputs). </li></ul>
    37. 39. Pairwise Conclusion <ul><li>The strength of pairwise testing is that it systematically generates small test sets that combine many test data values. </li></ul><ul><li>The weakness of the pairwise technique is that is does not take into account the combinatorial characteristics of the system under test. </li></ul>
    38. 40. Testing with Use-Cases <ul><li>Use-Cases do not specify input: </li></ul><ul><ul><li>Must select inputs: </li></ul></ul><ul><ul><ul><li>Try equivalence class </li></ul></ul></ul><ul><ul><ul><li>Boundary value techniques </li></ul></ul></ul><ul><li>Create at least one test case for every extension </li></ul><ul><li>Violate pre-conditions </li></ul><ul><li>Try different ordering of steps </li></ul><ul><li>Post-Conditions can be used as an Oracle </li></ul>