Presented by Stephen Allott CaseMaker International A Diaz & Hilterscheid group company Effective Testing Using industry s...
Agenda <ul><li>Why do we test software? </li></ul><ul><li>How do we test effectively? </li></ul><ul><li>Risk based testing...
Why test software?
Read all about it! US site  – down for 3 days, forced to return $30 fee to nearly 260,000 online “fantasy baseball” player...
Ineffective Testing <ul><li>Creates too many of the wrong tests </li></ul><ul><li>Duplicates effort unnecessarily </li></u...
Effective Testing <ul><li>Is aligned with the business need </li></ul><ul><li>Uses a  risk-based  approach and  formal tec...
Risk Based Testing <ul><li>Identify business risks </li></ul><ul><li>Assess the impact </li></ul><ul><ul><li>Severity, if ...
Test Case Design Techniques – 1  <ul><li>Black box and white box techniques have been proven to be effective </li></ul><ul...
Test Case Design Techniques – 2  <ul><li>Analysis of the test conditions </li></ul><ul><li>A formal design process gives y...
Test Case Design Techniques – 3 <ul><li>The design activity alone can find faults </li></ul><ul><li>Helps drive the need f...
Black Box Techniques  <ul><li>BS7925 techniques </li></ul><ul><ul><li>Equivalence class partitioning </li></ul></ul><ul><u...
White Box Techniques
Test Data <ul><li>Copy of live data </li></ul><ul><li>Realistic data volumes </li></ul><ul><li>Range of values and inputs ...
Equivalence Class Partitioning <ul><li>Can you ever see a need to:  </li></ul><ul><ul><li>try the view from every seat of ...
Equivalence Class Partitioning  <ul><li>Equivalence class or partition </li></ul><ul><li>Ranges of data </li></ul><ul><li>...
Equivalence Class Partitioning <ul><li>An online car insurance quotation system allows a 30% discount for good drivers age...
Boundary Value Analysis <ul><li>Builds on Equivalence Class technique </li></ul><ul><li>Assumes errors are made on the bou...
Boundary Value Analysis – Example <ul><li>An online car insurance quotation system  </li></ul><ul><ul><li>Under 17 – unabl...
Boundary Value Analysis – Example  <ul><li>Online Banking (Account balance) </li></ul><ul><li>Lower boundaries </li></ul><...
Boundary Value Analysis – Example <ul><li>Online Banking (Set up standing order) </li></ul><ul><li>Boundaries for  from da...
Decision Tables <ul><li>Another intuitive black box technique </li></ul><ul><li>Express complex business rules in a table ...
Example – Train Tickets Pay £30 Pay £20 Pay £15 Pay £10 Actions N Y N Y Travels after 9 am N N Y Y Holds a railcard Rule 4...
Example – Train Tickets Test Cases Pay £30 Pay £20 Pay £15 Pay £10 Expected results N Y N Y Travels after 9 am N N Y Y Hol...
Example – ATM  Reject Reject Reject Auth Actions N Y N Y Enter valid PIN N N Y Y Enter valid card Rule 4 Rule 3 Rule 2 Rul...
Example – ATM “don’t care”  Reject Reject Auth Actions ---  N Y Enter valid PIN N Y Y Enter valid card Rule 3 Rule 2 Rule ...
Example – File Backup Y Y Y N Stick inserted Works Fails Fails Fails Actions Y N --- --- Sufficient space Y Y N --- Stick ...
State Transition Testing Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.u...
State Transition Testing Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.u...
0-Switch coverage Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.uk/~cise...
1-Switch coverage Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.uk/~cise...
Pairwise Testing Acknowledgement This session on pairwise testing techniques is based on the book by Lee Copeland: A pract...
Pairwise Testing <ul><li>Pairwise is based on testing pairs of combinations of variables rather than all combinations </li...
Pairwise Example <ul><li>Consider a simple database enquiry screen: </li></ul><ul><ul><li>It has 4 fields </li></ul></ul><...
Pairwise Example <ul><li>Holiday destination </li></ul><ul><ul><li>UK, USA, Europe </li></ul></ul><ul><li>Price range (for...
Orthogonal Array L 9  (3 4 ) This orthogonal array has 9 rows, 4 columns and three ‘levels’ per column   1 2 3 3 Row 9 3 1...
Orthogonal Array L 9  (3 4 ) Each variable becomes a column in the orthogonal array   1 2 3 3 Row 9 3 1 2 3 Row 8 2 3 1 3 ...
Orthogonal Array L 9  (3 4 ) For each column we fill out the possible choices 1 2 3 Europe Row 9 3 1 2 Europe Row 8 2 3 1 ...
Orthogonal Array L 9  (3 4 ) Now the array is complete   Now 4 star High Europe Row 9 Later Basic Medium Europe Row 8 Soon...
Orthogonal Array L 9  (3 4 ) Each row becomes a test case – you can add more if you wish   Now 4 star High Europe Test 9 L...
How to use Orthogonal Arrays <ul><li>Identify the variables </li></ul><ul><li>Determine the number of choices for each </l...
What size array do we need? How many possible combinations are there?   Choices Variable NT, XP, Linux Server O/S IIS, Apa...
Using Orthogonal Arrays <ul><li>We require an array with 5 columns </li></ul><ul><ul><li>Column 1 needs 8 levels (browsers...
Allpairs Algorithm <ul><li>Devised by James Bach </li></ul><ul><li>Generates fewer tests </li></ul><ul><li>In our example,...
Pairwise Testing – Summary  <ul><li>Test all pairs rather than all combinations </li></ul><ul><ul><li>With 4 variables and...
Effective Testing – Summary <ul><li>Effective Testing uses </li></ul><ul><ul><li>A risk-based approach </li></ul></ul><ul>...
After all, what is quality?
Presented by Stephen Allott CaseMaker International A Diaz & Hilterscheid group company Effective Testing Using industry s...
Upcoming SlideShare
Loading in …5
×

Steven K Allott - Effective Testing - SoftTest Ireland

943 views

Published on

Visit SoftTest Ireland www.softtest.ie and sign up for access to free Irish Software Testing events.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
943
On SlideShare
0
From Embeds
0
Number of Embeds
85
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Steven K Allott - Effective Testing - SoftTest Ireland

  1. 1. Presented by Stephen Allott CaseMaker International A Diaz & Hilterscheid group company Effective Testing Using industry standard techniques SoftTest Ireland 16 th February 2006
  2. 2. Agenda <ul><li>Why do we test software? </li></ul><ul><li>How do we test effectively? </li></ul><ul><li>Risk based testing </li></ul><ul><li>Black Box techniques </li></ul><ul><li>Pairwise testing </li></ul>Photo: Courtesy of NASA
  3. 3. Why test software?
  4. 4. Read all about it! US site – down for 3 days, forced to return $30 fee to nearly 260,000 online “fantasy baseball” players. US site down for 2 days, $280,000 in lost sales. Shopping Site: 6-hour downtime: $400,000 in direct costs plus additional in discounts to inconvenienced customers. UK - Concerns over bugs and delays in new air traffic control software UK – Payment clearing system failure at month end delayed wages to many customers UK - Televisions, Cameras sold at very low prices
  5. 5. Ineffective Testing <ul><li>Creates too many of the wrong tests </li></ul><ul><li>Duplicates effort unnecessarily </li></ul><ul><li>Costs too much and takes too long </li></ul><ul><li>Has an adverse impact on the business as a whole </li></ul>
  6. 6. Effective Testing <ul><li>Is aligned with the business need </li></ul><ul><li>Uses a risk-based approach and formal techniques to create fewer tests with increased coverage </li></ul><ul><li>Facilitates communication with the business </li></ul><ul><li>Provides pre-production confidence </li></ul><ul><li>Saves time and money </li></ul>
  7. 7. Risk Based Testing <ul><li>Identify business risks </li></ul><ul><li>Assess the impact </li></ul><ul><ul><li>Severity, if it were to fail </li></ul></ul><ul><ul><li>Probability of failure </li></ul></ul><ul><li>Focus more testing on high risk areas </li></ul><ul><li>Continually measure and monitor the risks </li></ul>
  8. 8. Test Case Design Techniques – 1 <ul><li>Black box and white box techniques have been proven to be effective </li></ul><ul><li>Techniques are described in BS7925 </li></ul><ul><li>Analysis/design separated from execution </li></ul><ul><li>Improved and demonstrable test coverage </li></ul>
  9. 9. Test Case Design Techniques – 2 <ul><li>Analysis of the test conditions </li></ul><ul><li>A formal design process gives you </li></ul><ul><ul><li>Objectivity </li></ul></ul><ul><ul><li>Repeatability </li></ul></ul><ul><ul><li>Consistency </li></ul></ul><ul><ul><li>Documentation </li></ul></ul><ul><ul><li>Auditability </li></ul></ul><ul><ul><li>Traceability </li></ul></ul><ul><ul><li>Confidence </li></ul></ul>
  10. 10. Test Case Design Techniques – 3 <ul><li>The design activity alone can find faults </li></ul><ul><li>Helps drive the need for good requirements </li></ul><ul><li>Use of formal techniques helps with estimates of overall test effort required </li></ul><ul><li>Supports test automation </li></ul>
  11. 11. Black Box Techniques <ul><li>BS7925 techniques </li></ul><ul><ul><li>Equivalence class partitioning </li></ul></ul><ul><ul><li>Boundary value analysis </li></ul></ul><ul><ul><li>Decision tables </li></ul></ul><ul><ul><li>State transition testing </li></ul></ul><ul><ul><li>Syntax testing </li></ul></ul><ul><ul><li>Random testing </li></ul></ul><ul><li>Other techniques </li></ul><ul><ul><li>Pairwise </li></ul></ul><ul><ul><li>Domain analysis </li></ul></ul><ul><ul><li>Use case testing </li></ul></ul>
  12. 12. White Box Techniques
  13. 13. Test Data <ul><li>Copy of live data </li></ul><ul><li>Realistic data volumes </li></ul><ul><li>Range of values and inputs </li></ul>
  14. 14. Equivalence Class Partitioning <ul><li>Can you ever see a need to: </li></ul><ul><ul><li>try the view from every seat of a cinema or theatre? </li></ul></ul><ul><ul><li>test drive every car in the showroom? </li></ul></ul><ul><ul><li>withdraw cash from all the ATMs in your country? </li></ul></ul><ul><li>If you answered “No, of course not!” </li></ul><ul><ul><li>what would you do instead, and why? </li></ul></ul>
  15. 15. Equivalence Class Partitioning <ul><li>Equivalence class or partition </li></ul><ul><li>Ranges of data </li></ul><ul><li>Based on an assumption about the software’s behaviour </li></ul><ul><li>Reduces the number of test cases whilst maintaining reasonable test coverage </li></ul>
  16. 16. Equivalence Class Partitioning <ul><li>An online car insurance quotation system allows a 30% discount for good drivers aged between 25 and 50 but charges full price if you are under 25. For older careful drivers a 20% discount is allowed and if you are retired we will be happy to quote you a special price </li></ul>Under 17 17 to 24 25 to 49 50 to 64 65 and over £1,000 £700 £800
  17. 17. Boundary Value Analysis <ul><li>Builds on Equivalence Class technique </li></ul><ul><li>Assumes errors are made on the boundaries </li></ul><ul><li>Applicable to value and structure of data </li></ul><ul><ul><li>where there is a continuous range of values </li></ul></ul><ul><ul><li>length and character type </li></ul></ul><ul><li>Generates three test cases per boundary </li></ul><ul><ul><li>one on the boundary </li></ul></ul><ul><ul><li>one either side of the boundary </li></ul></ul><ul><ul><li>the smallest distance away </li></ul></ul>
  18. 18. Boundary Value Analysis – Example <ul><li>An online car insurance quotation system </li></ul><ul><ul><li>Under 17 – unable to quote </li></ul></ul><ul><ul><li>17 to 24 (premium price) </li></ul></ul><ul><ul><li>25 to 49 (30% discount) </li></ul></ul><ul><ul><li>50 to 64 (20% discount) </li></ul></ul><ul><ul><li>65 and over (price on application) </li></ul></ul>Under 17 17 to 24 25 to 49 50 to 64 65 and over 16,17,18 49,50,51 24,25,26 64,65,66
  19. 19. Boundary Value Analysis – Example <ul><li>Online Banking (Account balance) </li></ul><ul><li>Lower boundaries </li></ul><ul><ul><li>EUR 0.01 (DR) </li></ul></ul><ul><ul><li>EUR 0.00 (CR) </li></ul></ul><ul><ul><li>EUR 0.01 (CR) </li></ul></ul><ul><li>Upper boundaries </li></ul><ul><ul><li>EUR 999,999.99 (CR) </li></ul></ul><ul><ul><li>EUR 1,000,000.00 (CR) </li></ul></ul><ul><ul><li>EUR 1,000,000.01 (exceeds max for this account type) </li></ul></ul>
  20. 20. Boundary Value Analysis – Example <ul><li>Online Banking (Set up standing order) </li></ul><ul><li>Boundaries for from date </li></ul><ul><ul><li>Yesterday (invalid) </li></ul></ul><ul><ul><li>Today (should be allowed) </li></ul></ul><ul><ul><li>Tomorrow (should be allowed) </li></ul></ul><ul><li>Boundaries for until date </li></ul><ul><ul><li>12 months from today (ok) </li></ul></ul><ul><ul><li>12 months less 1 day (ok) </li></ul></ul><ul><ul><li>12 months and one day from today (invalid) </li></ul></ul>
  21. 21. Decision Tables <ul><li>Another intuitive black box technique </li></ul><ul><li>Express complex business rules in a table </li></ul><ul><li>Cause-Effect graphing helps with the analysis </li></ul><ul><li>Define the combination of conditions (causes) </li></ul><ul><li>Define the resulting actions (effects) </li></ul><ul><li>Easy to create test cases </li></ul>
  22. 22. Example – Train Tickets Pay £30 Pay £20 Pay £15 Pay £10 Actions N Y N Y Travels after 9 am N N Y Y Holds a railcard Rule 4 Rule 3 Rule 2 Rule 1 Conditions
  23. 23. Example – Train Tickets Test Cases Pay £30 Pay £20 Pay £15 Pay £10 Expected results N Y N Y Travels after 9 am N N Y Y Holds a railcard Test 4 Test 3 Test 2 Test 1 Inputs
  24. 24. Example – ATM Reject Reject Reject Auth Actions N Y N Y Enter valid PIN N N Y Y Enter valid card Rule 4 Rule 3 Rule 2 Rule 1 Conditions
  25. 25. Example – ATM “don’t care” Reject Reject Auth Actions --- N Y Enter valid PIN N Y Y Enter valid card Rule 3 Rule 2 Rule 1 Conditions
  26. 26. Example – File Backup Y Y Y N Stick inserted Works Fails Fails Fails Actions Y N --- --- Sufficient space Y Y N --- Stick formatted Rule 4 Rule 3 Rule 2 Rule 1 Conditions
  27. 27. State Transition Testing Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.uk/~cised/sreid/BCS_SIG/index.htm.
  28. 28. State Transition Testing Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.uk/~cised/sreid/BCS_SIG/index.htm.
  29. 29. 0-Switch coverage Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.uk/~cised/sreid/BCS_SIG/index.htm. S2 S4 S1 S1 S3 S2 Finish State D AD T T AT D Expected Output DS R CM TS R CM Input S4 S2 S2 S3 S1 S1 Start State 6 5 4 3 2 1 Test Case
  30. 30. 1-Switch coverage Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.uk/~cised/sreid/BCS_SIG/index.htm. S4 S1 S2 S3 S2 S3 S2 S1 S4 S1 Finish State AD T D AT D AT D T AD T Exp. Output R CM DS R CM R CM TS R CM Input S2 S2 S4 S1 S1 S1 S1 S3 S2 S2 Next State D D AD T T T T AT D D Exp. Output DS DS R CM CM TS TS R CM CM Input S4 S4 S2 S2 S2 S3 S3 S1 S1 S1 Start State 10 9 8 7 6 5 4 3 2 1 Test Case
  31. 31. Pairwise Testing Acknowledgement This session on pairwise testing techniques is based on the book by Lee Copeland: A practitioner’s guide to software test design, published by Artech House, 2004, ISBN 1-58053-791-X
  32. 32. Pairwise Testing <ul><li>Pairwise is based on testing pairs of combinations of variables rather than all combinations </li></ul><ul><li>It reduces the number of test cases </li></ul><ul><li>Many faults are either </li></ul><ul><ul><li>Single mode </li></ul></ul><ul><ul><li>Double mode </li></ul></ul><ul><li>There are two recognised methods: </li></ul><ul><ul><li>Orthogonal arrays </li></ul></ul><ul><ul><li>Allpairs algorithm </li></ul></ul>
  33. 33. Pairwise Example <ul><li>Consider a simple database enquiry screen: </li></ul><ul><ul><li>It has 4 fields </li></ul></ul><ul><ul><li>Each field has three possible choices </li></ul></ul><ul><ul><li>Giving 3 X 3 X 3 X 3 = 81 possible combinations to test </li></ul></ul><ul><li>With Pairwise Testing </li></ul><ul><ul><li>You can reduce this to 9 tests! </li></ul></ul>
  34. 34. Pairwise Example <ul><li>Holiday destination </li></ul><ul><ul><li>UK, USA, Europe </li></ul></ul><ul><li>Price range (for family of 5) </li></ul><ul><ul><li>Budget: Under £2000 </li></ul></ul><ul><ul><li>Medium: £2000 to £5000 </li></ul></ul><ul><ul><li>Expensive: Over £5000 </li></ul></ul><ul><li>Standard of hotel </li></ul><ul><ul><li>Basic, 4 star, luxury </li></ul></ul><ul><li>Availability </li></ul><ul><ul><li>Now: Within 7 days from today </li></ul></ul><ul><ul><li>Soon: After a week but up to one month from today </li></ul></ul><ul><ul><li>Later: Later than one month from today </li></ul></ul>
  35. 35. Orthogonal Array L 9 (3 4 ) This orthogonal array has 9 rows, 4 columns and three ‘levels’ per column 1 2 3 3 Row 9 3 1 2 3 Row 8 2 3 1 3 Row 7 2 1 3 2 Row 6 1 3 2 2 Row 5 3 2 1 2 Row 4 3 3 3 1 Row 3 2 2 2 1 Row 2 1 1 1 1 Row 1 D C B A
  36. 36. Orthogonal Array L 9 (3 4 ) Each variable becomes a column in the orthogonal array 1 2 3 3 Row 9 3 1 2 3 Row 8 2 3 1 3 Row 7 2 1 3 2 Row 6 1 3 2 2 Row 5 3 2 1 2 Row 4 3 3 3 1 Row 3 2 2 2 1 Row 2 1 1 1 1 Row 1 When Hotel Price Where
  37. 37. Orthogonal Array L 9 (3 4 ) For each column we fill out the possible choices 1 2 3 Europe Row 9 3 1 2 Europe Row 8 2 3 1 Europe Row 7 2 1 3 US Row 6 1 3 2 US Row 5 3 2 1 US Row 4 3 3 3 UK Row 3 2 2 2 UK Row 2 1 1 1 UK Row 1 When Hotel Price Where
  38. 38. Orthogonal Array L 9 (3 4 ) Now the array is complete Now 4 star High Europe Row 9 Later Basic Medium Europe Row 8 Soon Luxury Budget Europe Row 7 Soon Basic High US Row 6 Now Luxury Medium US Row 5 Later 4 star Budget US Row 4 Later Luxury High UK Row 3 Soon 4 star Medium UK Row 2 Now Basic Budget UK Row 1 When Hotel Price Where
  39. 39. Orthogonal Array L 9 (3 4 ) Each row becomes a test case – you can add more if you wish Now 4 star High Europe Test 9 Later Basic Medium Europe Test 8 Soon Luxury Budget Europe Test 7 Soon Basic High US Test 6 Now Luxury Medium US Test 5 Later 4 star Budget US Test 4 Later Luxury High UK Test 3 Soon 4 star Medium UK Test 2 Now Basic Budget UK Test 1 When Hotel Price Where
  40. 40. How to use Orthogonal Arrays <ul><li>Identify the variables </li></ul><ul><li>Determine the number of choices for each </li></ul><ul><li>Locate an orthogonal array of the right size </li></ul><ul><li>Map the testing problem onto the array </li></ul><ul><li>Construct the test cases </li></ul>
  41. 41. What size array do we need? How many possible combinations are there? Choices Variable NT, XP, Linux Server O/S IIS, Apache, WebLogic Server W95, W98, W2000, NT, XP, XP-PRO Operating systems None, Real player, Media Player Plug-ins IE5, IE6, …. 8 in total Browsers
  42. 42. Using Orthogonal Arrays <ul><li>We require an array with 5 columns </li></ul><ul><ul><li>Column 1 needs 8 levels (browsers) </li></ul></ul><ul><ul><li>Column 2 would need 3 levels (plug ins) </li></ul></ul><ul><ul><li>Column 3 would need 6 levels (O/S) </li></ul></ul><ul><ul><li>Columns 4 & 5 would need 3 levels (Server, Server O/S) </li></ul></ul><ul><li>Ideally we need 8 1 6 1 3 3 but it doesn’t exist! </li></ul><ul><li>The next largest is L 64 (8 2 4 3 ) </li></ul><ul><ul><li>2 columns of 1 – 8 </li></ul></ul><ul><ul><li>3 columns of 1 – 4 </li></ul></ul><ul><li>So rather than 1,296 tests we need only 64 tests! </li></ul><ul><li>Remember - one test case per row </li></ul>
  43. 43. Allpairs Algorithm <ul><li>Devised by James Bach </li></ul><ul><li>Generates fewer tests </li></ul><ul><li>In our example, it generates just 48 tests compared to 64 from using orthogonal arrays </li></ul><ul><li>Consider 100 variables, 2 choices </li></ul><ul><ul><li>Using orthogonal arrays you need 101 tests </li></ul></ul><ul><ul><li>Using all pairs you need only 10 tests </li></ul></ul>
  44. 44. Pairwise Testing – Summary <ul><li>Test all pairs rather than all combinations </li></ul><ul><ul><li>With 4 variables and 3 choices for each you would require 81 test cases whereas you can cover all pairwise input combinations with only 9 tests </li></ul></ul><ul><ul><li>For 13 variables and 3 choices for each you would require1,594,323 test cases whereas you can cover all pairwise input combinations with only 15 tests </li></ul></ul><ul><li>Pairwise guides us to help find </li></ul><ul><ul><li>Single mode defects </li></ul></ul><ul><ul><li>Double mode defects </li></ul></ul><ul><li>You should always add any known risky combinations </li></ul>
  45. 45. Effective Testing – Summary <ul><li>Effective Testing uses </li></ul><ul><ul><li>A risk-based approach </li></ul></ul><ul><ul><li>Formal test case design techniques </li></ul></ul><ul><li>The benefits are </li></ul><ul><ul><li>Reduced number of test cases </li></ul></ul><ul><ul><li>Repeatable well documented tests </li></ul></ul><ul><ul><li>Improved planning, estimation and communication </li></ul></ul><ul><ul><li>Facilitates earlier involvement with the business </li></ul></ul><ul><ul><li>Find faults earlier </li></ul></ul><ul><ul><li>Give you a consistent approach to testing </li></ul></ul><ul><ul><li>Save time and money </li></ul></ul><ul><ul><li>Enables test automation </li></ul></ul>
  46. 46. After all, what is quality?
  47. 47. Presented by Stephen Allott CaseMaker International A Diaz & Hilterscheid group company Effective Testing Using industry standard techniques SoftTest Ireland 16 th February 2006

×