Combination Testing Strategies


Published on

  • Be the first to comment

Combination Testing Strategies

  1. 1. T-76.5613 Software Testing and Quality Assurance Lecture 5, 25.9.2006 Combination Testing Strategies Juha Itkonen SoberIT HELSINKI UNIVERSITY OF TECHNOLOGY
  2. 2. Typical testing task Juha Itkonen 2 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  3. 3. Testing combinations Typical testing task includes testing features with many interacting options In principle, all combinations of all input and output conditions and options should be tested In practice, the number of combinations grows too big Systematic techniques are needed to select a good subset of the combinations Testing combinations of two or more input variables that affect the outcome Intelligent and systematic way needed Juha Itkonen 3 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  4. 4. Combination Strategies Combination testing strategies identify test combinations by combining values of different test object parameters based on some combinatorial strategy Algorithms for selecting the combinations At least 16 different strategies have been presented M. Grindal, J. Offutt and S. F. Andler. “Combination testing strategies: a survey”, Journal of Software Testing, Verification and Reliability. 2005. Vol. 15:167-199. Juha Itkonen 4 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  5. 5. Example of combinatorial testing PowerPoint save options 5 checkboxes One numeric field One dropdown menu 5 options One freetext field Two radio buttons Basic testing of these options without knowledge of their interrelations leads to many combinations: 2x2x2x2x2x4x5x3x2 = 3840 15 min per test case = 960h Juha Itkonen 5 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  6. 6. Combination strategies on different levels Test cases Combinations of inputs and outputs Combinations of options Combinations of data etc. Test environments Operating systems Platforms (java, application server) Databases Versions of libraries or components etc. Juha Itkonen 6 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  7. 7. From EP to combinations Combination strategies complement other techniques Equivalence partitioning or boundary analysis produces a large set of independent values for the input parameters The next step towards more exhaustive testing is to cover the combinations of these individual values Juha Itkonen 7 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  8. 8. Coverage criteria for combination strategies Each-used (1-wise) coverage Every interesting value of each parameter is covered with some test case Pair-wise coverage Every possible pair of interesting values of any two parameters is covered t-wise coverage Every possible combination of interesting values of t parameters is covered N-wise coverage All possible combinations of interesting values of all N parameters are covered Juha Itkonen 8 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  9. 9. Variable strength coverage Requires higher coverage among a subset of parameters and lower across all parameters For example, for parameters A, B, C, and D 3-wise coverage among parameters B, C, and D 2-wise coverage among all parameters Can be used to reduce the total amount of tests Based on assumption of more tight dependencies and interaction between certain parameters Juha Itkonen 9 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  10. 10. Normal and error values in combinations Normal (valid) and error (invalid) values should be treated differently Error values typically lead to some error message or termination of execution Only one error value of any parameter should be included in each test case To avoid one error value masking another Single error coverage Normal coverage Each valid used coverage Pair-wise valid coverage t-wise valid coverage Juha Itkonen 10 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  11. 11. Testing based on combinatorial strategies All combinations strategy All values once strategy Base Choice (default) strategy Pair-wise Testing strategy Decision Tables Juha Itkonen 11 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  12. 12. Combinatorial testing example Testing a find function A simple find dialog Three inputs Text string Case sensitivity Direction of the search Assume following partitions (ignore illegal values for simplicity) Text: uppercase / lowercase / mixed Case: match / do not match Direction: normal / backwards Searched text is found / is not found Juha Itkonen 12 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  13. 13. All combinations Test no. Text Match Case Direction Is found All combinations for the three 1 U Y F Y 2 U Y F N 3 4 U U Y Y B B Y N input fields 5 U N F Y plus the outcome of the test (last 6 U N F N 7 U N B Y 8 9 U L N Y B F N Y column) 10 L Y F N 11 L Y B Y 12 L Y B N 13 L N F Y 14 L N F N 15 L N B Y 16 L N B N 17 M Y F Y 18 M Y F N 19 M Y B Y 20 M Y B N 21 M N F Y 22 M N F N 23 M N B Y 3x2x2x2 = 24 combinations 24 M N B N Juha Itkonen 13 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  14. 14. All values once strategy Text Match Case Direction Is found 1 L Y F N 2 U N F N 3 M N B Y Simplest method for selecting tests is to test each option at least once No combination coverage only single-mode defects covered systematically Some typical cases easily missed Can be used to test every feature once with minimal amount of test cases Then select which ones have interactions and create e.g. pair-wise combinations 3 combinations for those Juha Itkonen 14 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  15. 15. Base choice strategy Test no. Text Match Case Direction Is found 1 U Y F Y 2 L Y F Y 3 M Y F Y 4 U N F Y 5 U Y B Y 6 U Y F N Set each variable in the default value (first row) 6 combinations and vary one variable at time in each test case until all values of each variable are covered Good approach as long as the parameters are independent The most typical cases best covered Easy to analyse what caused the failure No systematic coverage for combinations Juha Itkonen 15 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  16. 16. Pair-wise strategy (a.k.a all-pairs testing) (Orthogonal array testing in some sources) The tested combinations are selected so that the test cases cover all possible pairs of the values of the parameters Based on assumption that most of the defects in software systems are related to interaction between two variables (double-mode defects) A certain values of some pair of variables causes the failure, not e.g. a certain combination of three or more variables Pair-wise strategy reduces the number of tested combinations radically 10 variables with ten values each would require 10,000,000,000 test cases. Pair-wise strategy only requires 177 cases. Juha Itkonen 16 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  17. 17. Pair-wise testing example Test Text Match Case Direction Is Found 1 Upper Yes Forward Yes 2 Upper No Backwards No 3 Lower Yes Backwards Yes 4 Lower No Forward No 5 Mixed Yes Forward No 6 Mixed No Backwards Yes All combinations of the values of each pair of parameters get tested Testing all pairs can catch defects caused by interaction of any two variables (i.e. double-mode defects) Note! Does not guarantee that 6 combinations typical cases are covered Juha Itkonen 17 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  18. 18. Decision tables Rule 1 Rule 2 … Modelling complicated business rules Conditions Focusing to interesting Condition 1 combinations Condition 2 Business logic viewpoint … Different combinations of conditions lead to different Actions expected outcome Action 1 (actions) … Does not have systematic coverage criteria Juha Itkonen 18 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  19. 19. Decision table example: Discount rules Young Pensioner Sale Sale with Sale with age coupon discount Conditions Age < 15 > 65 15>= - < 15 or age > 65 <=65 Sale 0% 0% -50% -50% -50% Discount - - - Yes - coupon Actions Discount 10% 15% 50% 70% 55% percent Bonus limit 10 € 25€ 100€ 100€ 50€ Juha Itkonen 19 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
  20. 20. More information More information on decision table testing and pairwise testing delivered as lecture notes Allpairs software for selecting pairwise test cases Can be downloaded from course material pages Simple command-line tool Documentation is included Zip-file includes sample files for the above covered “find feature” example allpairs find_feature_options.txt > find_feature_tests.txt Juha Itkonen 20 HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT