Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Test Smarter (with Pair-wise Testing)

9,216 views

Published on

  • Be the first to comment

Test Smarter (with Pair-wise Testing)

  1. 1. Test Smarter (with Pair-wise Testing) Presented by: Bill Lewis CEO, Smartware Technologies, Inc. (469) 951-7548 www.smartwaretechnologies.com
  2. 2. Agenda <ul><li>Challenges of application development </li></ul><ul><li>Automated Testing </li></ul><ul><li>What is orthogonal arrays? </li></ul><ul><li>Power of pair-wise testing </li></ul><ul><li>Transforming theory to practice </li></ul><ul><li>Summary </li></ul><ul><li>Q&A </li></ul>
  3. 3. “V” Model Coding Unit Testing Design Integration Testing Architecture System Testing Functional & NF Requirements UA Testing User Requirements Project Success Business Requirements Static Testing Verify Dynamic Testing Validate 50 % 50 %
  4. 4. Tractability Matrix <ul><li>A Trace Matrix can be used to track test cases during development and execution </li></ul><ul><ul><li>Identification of uniquely “testable” requirements </li></ul></ul><ul><ul><ul><li>Eliminate wishes, hopes, dreams, and other un-testable statements </li></ul></ul></ul><ul><ul><ul><li>Eliminate duplicates </li></ul></ul></ul><ul><ul><ul><li>Assign a unique ID for each requirement and design objective </li></ul></ul></ul><ul><ul><ul><li>Determine its priority if possible </li></ul></ul></ul><ul><ul><li>Verification of implementation of all requirements </li></ul></ul>
  5. 5. Traceability Matrix Example
  6. 6. Issues Facing Software Testing Organizations <ul><li>Today’s software is getting more and more complex with thousands if not millions of different combinations that need to be tested. </li></ul><ul><li>Humans can’t think of all the test scenarios </li></ul><ul><ul><ul><li>Takes too much time </li></ul></ul></ul><ul><ul><ul><li>Too error prone </li></ul></ul></ul><ul><ul><ul><li>Which scenarios do you test ? </li></ul></ul></ul><ul><li>The length of time to test modern software is increasing. </li></ul><ul><li>Requirements documentation is often ambiguous or inadequate </li></ul><ul><li>Testing process is poorly defined or non-existent </li></ul>
  7. 7. Simple Example: Configuration Testing
  8. 8. Simple Example (Con’t) <ul><li>Suppose you have an application that needs to be tested on several hardware and software configurations.  </li></ul><ul><li>You are told that the application must be able to run on a Windows NT, Windows 2000 and Windows XP machine.  </li></ul><ul><li>It also has to work on a system with a minimum of 128Meg and a max of 512Meg of RAM.  </li></ul><ul><li>Finally, the application must also run on a Pentium II, III, and IV processor and work with Oracle, SQL and Access Databases. </li></ul>
  9. 9. Simple Example (Con’t) Determine test cases for testing the an application on the following: Access Pentium IV 512 Meg Windows XP SQL Pentium III 256 Meg Windows 2000 Oracle Pentium II 128 Meg Windows NT Database PC Processor RAM Operating System
  10. 10. Combinations of Test Cases <ul><li>3 x 3 x 3 x 3 = 81 Test Cases </li></ul>
  11. 11. Why Automate? <ul><li>Over time, number of functional features changes/ increase </li></ul><ul><li>and </li></ul><ul><li>Number of people and amount of testing time for each new release </li></ul><ul><ul><li>Remains flat </li></ul></ul><ul><ul><li>May even decline </li></ul></ul><ul><li>Test coverage steadily decreases </li></ul><ul><ul><li>Increases the risk of failure </li></ul></ul><ul><ul><li>Potential business losses </li></ul></ul><ul><li>Source: Linda Hayes, “Why Certify” </li></ul>
  12. 12. <ul><li>Latest generation of automated testing tools support variable test data </li></ul><ul><li>Difference between static and variable capture/ replay </li></ul><ul><ul><li>With Static the inputs and outputs are fixed </li></ul></ul><ul><ul><li>With variable inputs and outputs are variable </li></ul></ul><ul><ul><li>Replacing the captured inputs and expected outputs with variables </li></ul></ul><ul><ul><li>Variable capture/ replay reduces the risk of not performing regression testing on existing/ new features </li></ul></ul>Today’s Test Automation Variable Capture/ Replay Tools
  13. 13. Test Automation Framework Source: Ross Collard Source: Ross Collard
  14. 14. What’s the Missing Link? Answer: scientifically generated test data
  15. 15. Simple Example Revisited Determine test cases for testing the an application on the following: * Constraint: Windows XP cannot run on a Pentium II PC Processor. Access Pentium IV 512 Meg Windows XP SQL Pentium III 256 Meg Windows 2000 Oracle Pentium II 128 Meg Windows NT Database PC Processor RAM Operating System
  16. 16. Optimal Test Cases (via Pair-Wise Analysis) Oracle Pentium III 512 Meg Windows XP 8 SQL Pentium IV 128 Meg Windows XP 7 SQL Pentium II 512 Meg Windows 2000 6 Oracle Pentium IV 256 Meg Windows 2000 5 Access Pentium III 128 Meg Windows 2000 4 Access Pentium IV 512 Meg Windows NT 3 SQL Pentium III 256 Meg Windows NT 2 Oracle Pentium II 128 Meg Windows NT 1 Database PC Processor RAM Operating System Test Case #
  17. 17. Pairwise Definition <ul><li>Pairwise testing – an approach to combinatorial testing that executes a pairwise test data set. </li></ul><ul><li>Pairwise test data set - A set of test cases that covers all combinations of the selected test data values for every pair of a system’s input variables. </li></ul><ul><li>Pairwise testing is also known as 2-way testing; other approaches include 3-way, 4-way, … , n-way. </li></ul>
  18. 18. Secret Sauce (Orthogonal Arrays) <ul><li>Mathematical curiosity discovered by monks in the 1400s </li></ul><ul><li>Promoted by Dr. Genichi Taguchi in the 1950s for use in statistical test design in manufacturing, agriculture, medicine, etc. </li></ul><ul><li>Orthogonal Array: Two dimensional array of numbers in which any two columns in the array receive an even distribution of all pairwise combinations of values in the array. </li></ul><ul><li>Orthogonal arrays are a standard construction used for statistical experiments. </li></ul>
  19. 19. Example Orthogonal Arrays and Notation L 4 (2 3 ) L 18 (3 6 6 1 ) L 9 (3 4 ) 5 1 0 0 1 2 2 18 4 0 1 0 2 1 2 17 3 1 1 2 0 0 2 16 2 2 0 1 0 1 2 15 1 0 2 1 1 0 2 14 0 2 2 2 2 2 2 13 5 0 2 2 0 1 1 12 4 2 0 2 1 0 1 11 3 0 0 1 2 2 1 10 2 1 2 0 2 0 1 9 1 2 1 0 0 2 1 8 0 1 1 1 1 1 1 7 5 2 1 1 2 0 0 6 4 1 2 1 0 2 0 5 3 2 2 0 1 1 0 4 2 0 1 2 1 2 0 3 1 1 0 2 2 1 0 2 0 0 0 0 0 0 0 1 7 6 5 4 3 2 1 0 1 1 4 1 0 1 3 1 1 0 2 0 0 0 1 3 2 1 0 1 2 2 9 2 0 1 2 8 1 2 0 2 7 1 0 2 1 6 0 2 1 1 5 2 1 0 1 4 2 2 2 0 3 1 1 1 0 2 0 0 0 0 1 4 3 2 1
  20. 20. Why is Pair-wise Testing So Efficient? <ul><li>Example: Insurance Application (GUI) </li></ul><ul><li>Suppose a bug exists wherein the application produces a garbled record when the spouse's name is empty, but only when the &quot;married&quot; radio button is selected </li></ul><ul><li>With pair-wise testing we’ll find the bug! </li></ul>Empty Divorced Adult Children Greater than 20 characters Married Dependent Children Mary Single No Children Spouse First Name (text field less or equal 20 characters) Marital Status (radio buttons) Dependents (checkbox)
  21. 21. Exhaustive Testing … 3 x 3 x 3 = 27 tests Mary Divorced No Children Greater than 20 characters Divorced No Children Empty Divorced No Children Mary Married No Children Greater than 20 characters Married No Children Empty Married No Children Greater than 20 characters Married Dependent Children Mary Married Dependent Children Empty Single Dependent Children Greater than 20 characters Single Dependent Children Mary Single Dependent Children Empty Single No Children Greater than 20 characters Single No Children Mary Single No Children
  22. 22. Pair-wise Results When “business rules” (functional coverage) are also applied to the results, the probability of detecting a defect is dramatically increased . (Only 9 test cases are required)
  23. 23. Real Results of Pair-wise Testing <ul><li>Identify a minimum set of tests that will find all multi-mode defects (particularly serious defects) </li></ul><ul><ul><li>Clarification </li></ul></ul><ul><ul><ul><li>All single-mode defect will be found if every option is tested at least once (unit elements) </li></ul></ul></ul><ul><ul><ul><li>2-way or “Pair-wise” is combinations of 2 items (parameters) that cause a defect </li></ul></ul></ul><ul><ul><ul><li>3-way or “Tri-wise” is combinations of 3 items (parameters) that cause a defect </li></ul></ul></ul><ul><ul><li>“ Pair-wise” defect detection (pair-wise testing) finds most defects, ex. testing all pairs typically finds 75% of defects Source: Kuhn </li></ul></ul><ul><ul><li>NASA Deep Space Mission - study showed that 88% of bugs discovered using “pair-wise” defect detection testing </li></ul></ul><ul><ul><li>Source: 27 th NASA/ IEEE Software Engineering Workshop, NASA Goddard Space Flight Center, 4-6 Dec, 2002. </li></ul></ul><ul><ul><li>U.S. Food and Drug Administration – study showed 98% bugs discovered using “ pair-wise ” defect detection Source: 27 th NASA/ IEEE Software Engineering Workshop, NASA Goddard Space Flight Center, 4-6 Dec, 2002. </li></ul></ul><ul><ul><li>“ The Combinatorial Design Approach to Automatic Test Generation” Source: IEEE, Software 1996, Vol. 13, No. 5 </li></ul></ul><ul><ul><ul><li>Baseline application under study resulted in a reduction in test plan time from 1 month to less than 1 week </li></ul></ul></ul><ul><ul><ul><li>In several experiments, “pair-wise” demonstrated good code coverage and defect detection ability </li></ul></ul></ul>
  24. 24. Managing the Test Combination Explosion
  25. 25. Transforming Theory to Practice
  26. 26. How does Smart Test ™ work?
  27. 27. Demonstration
  28. 28. Invoke Smart Test ™ Generator
  29. 29. Smart Test ™ Generated Test Data
  30. 30. QTP Script Ready to Executed
  31. 31. Agenda <ul><li>Challenges of application development </li></ul><ul><li>Evolution of software testing </li></ul><ul><li>What is orthogonal arrays? </li></ul><ul><li>Power of pair-wise testing </li></ul><ul><li>Transforming theory to practice </li></ul><ul><li>Summary </li></ul><ul><li>Q&A </li></ul>
  32. 32. Questions ?
  33. 33. Contact Information <ul><li>If you want to contact me </li></ul><ul><ul><li>Bill Lewis, CEO & President Smartware Technologies, Inc. </li></ul></ul><ul><ul><li>www.smartwaretechnologies.com </li></ul></ul><ul><ul><li>(469) 951-7548 </li></ul></ul>

×