Download to read offline
More Related Content
Similar to Combinatorial testing
Combinatorial testing
- 1. SOFTWARE TESTING-ITE2004 Final Report UsingCombinatorial Testing to reduce Software Rework Submitted by Submitted to -Arnab Saikia 15BIT0072 Vijay Anand R -Kedar Kumar 15BIT0268
- 2. Introduction Combination testing refersto tests that involve more than one variable. A basic issue for multi-variable testing is called combinatorial blast. The more factors you join, the higher the quantity of tests. Combinatorial testing is about picking a couple tests that can show us what we have to think about, recommending that 6-way combinatorial testing would be adequate to trigger and recognize them. NIST built up a way to deal with consequently creating, executing, and investigating such tests. Why CT turns out to be cost effective? Error detection in the code at the late stage is one of the most unwanted things that should happen because detecting errors at the later stages needs more capital and much more software reworking. Basically CT provides the path to avoid such an issue What causes software failures? • Logic errors • Calculation errors • Interaction faults • Inadequate input checking
- 3. Combinatorial testingis a moderately new type of programming testing. It depends on straightforward standards, for example, association control, pairwise testing, t-way connection testing, and so on. Combinatorial testing can radically lessen testing times and expenses by diminishing the quantity of tests (i.e., decreasing number of experiments) to be performed. Combinatorial testing permits us to choose a few parameters at any given moment, be it setups or sources of info, and test the blends in a manner that greater part of the deficiencies will become visible with least number of tests. The difficulties can be outlined into these four classifications: • Test Modelling - recognition and straightforwardness • Generating comprehensible experiments • Facilitating simultaneous test execution • Integration with existing devices/forms
- 4. What are someof the most typical test issues? • Too many test scenarios to create and execute • Lack of time/effort to test all those scenarios • Detection of defects late in the cycle • Ad-hoc test plans (it is a commonly used term for software testing performed without planning and documentation. The tests are intended to be run only once, unless a defect is discovered. Ad hoc testing is the least formal test method.) • Duplicate/redundant test scenarios • Inadequate understanding of the test space Many teams today use an ad hoc design method for test planning which results in: • Spending a lot time and effort on testing while still finding simple problems late in the cycle • Tests that mix requirements with implementation • Too many test cases to create , execute and maintain
- 5. • Lackof ability to identify which tests are missing • Lack of ability to quantify the risk of testing less or the value of testing more • Insufficient hardware to execute all of your tests. • Lengthy and often painful test plan reviews • Volumes of test result data that makes it difficult, or impossible, to analyse results Combinatorial Test Design provides a structured, tool based design methodology which results in: • A repeatable test design process that doesn't depend solely on the experience of the test team • Improved the test planning process • Consistent coverage without test duplication • A test plan that is well positioned for effective test automation strategies. • The minimum number of test cases needed for known interaction level • Improved ability to predict code quality
- 6. Pairwise testing Testinformation demonstrates that around 60-95% of issues happen because of collaboration of two parameters. So if every one of the 2-way blends are tried, a high rate of mistakes could be identified. Testing 2-route blends of designs or potentially information sources is known as pairwise testing. Three-way collaborations and higher However pairwise testing won't be adequate relying upon the use of the product. There is a possibility of 3% errors. So by three way or higher degree its more accurate and precise. It is proven that 6-way test can attain 100% accuracy and precision but for the time being we just have softwares that can undertake pairwise testing. Pairwise Testing ‘Pairwise Testing’ is also known as ‘All-Pairs Testing’. Smart testing is the need of the hour. 90% of the times system testing team has to work with tight schedules. So test design techniques should be very effective for maximum test coverage and high defect yield rate. Pairwise Testing is a test design technique that delivers high per cent of test coverage. The output of a software application depends on many factors e.g. input parameters, state variables and environment configurations. Techniques like boundary value analysis and
- 7. equivalence partitioning canbe useful to identify the possible values for individual factors. But it is impractical to test all possible combinations of values for all those factors. So instead a subset of combinations is generated to satisfy all factors. It is a black-box test design technique in which test cases are designed to execute all possible discrete combinations of each pair of input parameters. All-Pairs technique is very helpful for designing tests for applications involving multiple parameters. Tests are designed such that for each pair of input parameters to a system, there are all possible discrete combinations of those parameters. The test suite covers all combinations; therefore it is not exhaustive yet very effective in finding bugs. Car ordering application The car ordering application allows for Buying and Selling cars. It should support trading in Delhi and Mumbai. It should have a registration number, may be valid or invalid. It should allow the trade of following cars: BMW, Audi, and Mercedes. Two types of booking can be done: E-booking and In Store. Orders can be placed only during trading hours. Step #1: Let’s list down the variables involved. 1) Order category a. Buy b. Sell 2) Location a. Delhi
- 8. b. Mumbai 3) Car brand a. BMW b. Audi c. Mercedes 4) Registration numbers a. Valid (5000) b. Invalid 5) Order type a. E-Booking b. In store 6) Order time a. Working hours b. Non-working hours If we want to test all possible valid combinations: = 2 X 2 X 3 X 5000 X 2 X 2 = 240000 There are also an infinite number of invalid combinations. Step #2: Let’s simplify – Use a smart representative sample. – Use groups and boundaries, even when data is non-discrete. – Reduce Registration Number to Two
- 9. 1. Validregistration number 2. Invalid registration number Now let’s calculate number of possible combinations = 2 X 2 X 3 X 2 X 2 X 2 = 96 Arranging variables and values involved. Arrange variables to create test suite After comparing, substituting and simplifying, these 96 test cases turns out to be just 8 test cases So CT reduces the number of test cases significantly. Tools are available that applies all-pairs testing technique that facilitates us to effectively automate the Test Case Design process by generating a compact set of parameter value choices as the desired Test Cases. Tools for pairwise testing Some well-known tools from the industry are: • PICT – ‘Pairwise Independent Combinatorial Testing’, provided by Microsoft Corp. • IBM FoCuS – ‘Functional Coverage Unified Solution’, provided by IBM. • ACTS – ‘Advanced Combinatorial Testing System’, provided by NIST, an agency of the US Government. • Hexawise
- 10. PICT – PairwiseIndependent Combinatorial Testing Given a set of N independent test factors—f1, f2, ..., fN—with each factor fi having Li possible levels—fi = {li,1, ..., li,Li}—a set of tests R is produced. Each test in R contains N test levels, one for each test-factor fi; and, collectively, all tests in R cover all possible pairs of test-factor levels (belonging to different parameters); in other words, for each pair of factor levels li,p and lj,q—where 1 ! p ! Li, 1 ! q ! Lj, and i " j—there exists at least one test in R that contains both li,p and lj,q. This concept can easily be extended from covering all pairs to covering any t- wise combinations in which 1 ! t ! N. Why is PICT the best till date? It provides the most flexible environment for testing. Generating all possible valid test cases Creating parameter hierarchy-User can give a priority sequence for testing Excluding unwanted combinations
- 11. All inall there is a need for improvement and such new softwares so that CT advances in the near future and this can then lessen the software rework upto a very great extent. Implementation by using the software PICT For PICT to run on your system, Strawberry Perl , Visual Studio and of course PICT 3.3 must be installed We took the following sample data to find the pairwise test cases by the use of PICT- OS: Windows, Linux, MacOS, SteamOS, Ubuntu Browser: Chrome, Internet Explorer, Firefox, Safari Database: Database A, Database B, Database Chrome
- 13. Result- Based onthe following variables- OS: Windows, Linux, MacOS, SteamOS, Ubuntu Browser: Chrome, Internet Explorer, Firefox, Safari Database: Database A, Database B, Database Chrome
- 14. We foundout that we need to run just 20 valid test cases to verify. As a result we do not need to run all those useless test cases and waste our precious time and efforts also Conclusion of pairwise testing (a type of CT) Pairwise testing technique can reduce the number of combinations to be covered but remains very effective in terms of fault detection. It is indeed a smart test design technique that guarantees a win-win situation for both test effort and test effectiveness. During the Test planning phase of software testing, Pairwise testing technique should always be taken into consideration. Either we are doing it manually or using any tool to generate test cases, it becomes a necessary component of the test plan because it in turn affects Test estimation. References Applying Combinatorial Testing in Industrial Settings Xuelin Li, Ruizhi Gao, W. Eric Wong 2016 IEEE International Conference on Software Quality, Reliability and Security
- 15. An IndustryProof-of-Concept Demonstration of Automated Combinatorial Test Redge Bartholomew Rockwell Collins Cedar Rapids, Ia., USA Internet sources-Google,ieee,wikipedia