Your SlideShare is downloading. ×
Steven K Allott - Effective Testing - SoftTest Ireland
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Steven K Allott - Effective Testing - SoftTest Ireland

660
views

Published on

Visit SoftTest Ireland www.softtest.ie and sign up for access to free Irish Software Testing events.

Visit SoftTest Ireland www.softtest.ie and sign up for access to free Irish Software Testing events.


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
660
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Presented by Stephen Allott CaseMaker International A Diaz & Hilterscheid group company Effective Testing Using industry standard techniques SoftTest Ireland 16 th February 2006
  • 2. Agenda
    • Why do we test software?
    • How do we test effectively?
    • Risk based testing
    • Black Box techniques
    • Pairwise testing
    Photo: Courtesy of NASA
  • 3. Why test software?
  • 4. Read all about it! US site – down for 3 days, forced to return $30 fee to nearly 260,000 online “fantasy baseball” players. US site down for 2 days, $280,000 in lost sales. Shopping Site: 6-hour downtime: $400,000 in direct costs plus additional in discounts to inconvenienced customers. UK - Concerns over bugs and delays in new air traffic control software UK – Payment clearing system failure at month end delayed wages to many customers UK - Televisions, Cameras sold at very low prices
  • 5. Ineffective Testing
    • Creates too many of the wrong tests
    • Duplicates effort unnecessarily
    • Costs too much and takes too long
    • Has an adverse impact on the business as a whole
  • 6. Effective Testing
    • Is aligned with the business need
    • Uses a risk-based approach and formal techniques to create fewer tests with increased coverage
    • Facilitates communication with the business
    • Provides pre-production confidence
    • Saves time and money
  • 7. Risk Based Testing
    • Identify business risks
    • Assess the impact
      • Severity, if it were to fail
      • Probability of failure
    • Focus more testing on high risk areas
    • Continually measure and monitor the risks
  • 8. Test Case Design Techniques – 1
    • Black box and white box techniques have been proven to be effective
    • Techniques are described in BS7925
    • Analysis/design separated from execution
    • Improved and demonstrable test coverage
  • 9. Test Case Design Techniques – 2
    • Analysis of the test conditions
    • A formal design process gives you
      • Objectivity
      • Repeatability
      • Consistency
      • Documentation
      • Auditability
      • Traceability
      • Confidence
  • 10. Test Case Design Techniques – 3
    • The design activity alone can find faults
    • Helps drive the need for good requirements
    • Use of formal techniques helps with estimates of overall test effort required
    • Supports test automation
  • 11. Black Box Techniques
    • BS7925 techniques
      • Equivalence class partitioning
      • Boundary value analysis
      • Decision tables
      • State transition testing
      • Syntax testing
      • Random testing
    • Other techniques
      • Pairwise
      • Domain analysis
      • Use case testing
  • 12. White Box Techniques
  • 13. Test Data
    • Copy of live data
    • Realistic data volumes
    • Range of values and inputs
  • 14. Equivalence Class Partitioning
    • Can you ever see a need to:
      • try the view from every seat of a cinema or theatre?
      • test drive every car in the showroom?
      • withdraw cash from all the ATMs in your country?
    • If you answered “No, of course not!”
      • what would you do instead, and why?
  • 15. Equivalence Class Partitioning
    • Equivalence class or partition
    • Ranges of data
    • Based on an assumption about the software’s behaviour
    • Reduces the number of test cases whilst maintaining reasonable test coverage
  • 16. Equivalence Class Partitioning
    • An online car insurance quotation system allows a 30% discount for good drivers aged between 25 and 50 but charges full price if you are under 25. For older careful drivers a 20% discount is allowed and if you are retired we will be happy to quote you a special price
    Under 17 17 to 24 25 to 49 50 to 64 65 and over £1,000 £700 £800
  • 17. Boundary Value Analysis
    • Builds on Equivalence Class technique
    • Assumes errors are made on the boundaries
    • Applicable to value and structure of data
      • where there is a continuous range of values
      • length and character type
    • Generates three test cases per boundary
      • one on the boundary
      • one either side of the boundary
      • the smallest distance away
  • 18. Boundary Value Analysis – Example
    • An online car insurance quotation system
      • Under 17 – unable to quote
      • 17 to 24 (premium price)
      • 25 to 49 (30% discount)
      • 50 to 64 (20% discount)
      • 65 and over (price on application)
    Under 17 17 to 24 25 to 49 50 to 64 65 and over 16,17,18 49,50,51 24,25,26 64,65,66
  • 19. Boundary Value Analysis – Example
    • Online Banking (Account balance)
    • Lower boundaries
      • EUR 0.01 (DR)
      • EUR 0.00 (CR)
      • EUR 0.01 (CR)
    • Upper boundaries
      • EUR 999,999.99 (CR)
      • EUR 1,000,000.00 (CR)
      • EUR 1,000,000.01 (exceeds max for this account type)
  • 20. Boundary Value Analysis – Example
    • Online Banking (Set up standing order)
    • Boundaries for from date
      • Yesterday (invalid)
      • Today (should be allowed)
      • Tomorrow (should be allowed)
    • Boundaries for until date
      • 12 months from today (ok)
      • 12 months less 1 day (ok)
      • 12 months and one day from today (invalid)
  • 21. Decision Tables
    • Another intuitive black box technique
    • Express complex business rules in a table
    • Cause-Effect graphing helps with the analysis
    • Define the combination of conditions (causes)
    • Define the resulting actions (effects)
    • Easy to create test cases
  • 22. Example – Train Tickets Pay £30 Pay £20 Pay £15 Pay £10 Actions N Y N Y Travels after 9 am N N Y Y Holds a railcard Rule 4 Rule 3 Rule 2 Rule 1 Conditions
  • 23. Example – Train Tickets Test Cases Pay £30 Pay £20 Pay £15 Pay £10 Expected results N Y N Y Travels after 9 am N N Y Y Holds a railcard Test 4 Test 3 Test 2 Test 1 Inputs
  • 24. Example – ATM Reject Reject Reject Auth Actions N Y N Y Enter valid PIN N N Y Y Enter valid card Rule 4 Rule 3 Rule 2 Rule 1 Conditions
  • 25. Example – ATM “don’t care” Reject Reject Auth Actions --- N Y Enter valid PIN N Y Y Enter valid card Rule 3 Rule 2 Rule 1 Conditions
  • 26. Example – File Backup Y Y Y N Stick inserted Works Fails Fails Fails Actions Y N --- --- Sufficient space Y Y N --- Stick formatted Rule 4 Rule 3 Rule 2 Rule 1 Conditions
  • 27. State Transition Testing Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.uk/~cised/sreid/BCS_SIG/index.htm.
  • 28. State Transition Testing Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.uk/~cised/sreid/BCS_SIG/index.htm.
  • 29. 0-Switch coverage Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.uk/~cised/sreid/BCS_SIG/index.htm. S2 S4 S1 S1 S3 S2 Finish State D AD T T AT D Expected Output DS R CM TS R CM Input S4 S2 S2 S3 S1 S1 Start State 6 5 4 3 2 1 Test Case
  • 30. 1-Switch coverage Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.uk/~cised/sreid/BCS_SIG/index.htm. S4 S1 S2 S3 S2 S3 S2 S1 S4 S1 Finish State AD T D AT D AT D T AD T Exp. Output R CM DS R CM R CM TS R CM Input S2 S2 S4 S1 S1 S1 S1 S3 S2 S2 Next State D D AD T T T T AT D D Exp. Output DS DS R CM CM TS TS R CM CM Input S4 S4 S2 S2 S2 S3 S3 S1 S1 S1 Start State 10 9 8 7 6 5 4 3 2 1 Test Case
  • 31. Pairwise Testing Acknowledgement This session on pairwise testing techniques is based on the book by Lee Copeland: A practitioner’s guide to software test design, published by Artech House, 2004, ISBN 1-58053-791-X
  • 32. Pairwise Testing
    • Pairwise is based on testing pairs of combinations of variables rather than all combinations
    • It reduces the number of test cases
    • Many faults are either
      • Single mode
      • Double mode
    • There are two recognised methods:
      • Orthogonal arrays
      • Allpairs algorithm
  • 33. Pairwise Example
    • Consider a simple database enquiry screen:
      • It has 4 fields
      • Each field has three possible choices
      • Giving 3 X 3 X 3 X 3 = 81 possible combinations to test
    • With Pairwise Testing
      • You can reduce this to 9 tests!
  • 34. Pairwise Example
    • Holiday destination
      • UK, USA, Europe
    • Price range (for family of 5)
      • Budget: Under £2000
      • Medium: £2000 to £5000
      • Expensive: Over £5000
    • Standard of hotel
      • Basic, 4 star, luxury
    • Availability
      • Now: Within 7 days from today
      • Soon: After a week but up to one month from today
      • Later: Later than one month from today
  • 35. Orthogonal Array L 9 (3 4 ) This orthogonal array has 9 rows, 4 columns and three ‘levels’ per column 1 2 3 3 Row 9 3 1 2 3 Row 8 2 3 1 3 Row 7 2 1 3 2 Row 6 1 3 2 2 Row 5 3 2 1 2 Row 4 3 3 3 1 Row 3 2 2 2 1 Row 2 1 1 1 1 Row 1 D C B A
  • 36. Orthogonal Array L 9 (3 4 ) Each variable becomes a column in the orthogonal array 1 2 3 3 Row 9 3 1 2 3 Row 8 2 3 1 3 Row 7 2 1 3 2 Row 6 1 3 2 2 Row 5 3 2 1 2 Row 4 3 3 3 1 Row 3 2 2 2 1 Row 2 1 1 1 1 Row 1 When Hotel Price Where
  • 37. Orthogonal Array L 9 (3 4 ) For each column we fill out the possible choices 1 2 3 Europe Row 9 3 1 2 Europe Row 8 2 3 1 Europe Row 7 2 1 3 US Row 6 1 3 2 US Row 5 3 2 1 US Row 4 3 3 3 UK Row 3 2 2 2 UK Row 2 1 1 1 UK Row 1 When Hotel Price Where
  • 38. Orthogonal Array L 9 (3 4 ) Now the array is complete Now 4 star High Europe Row 9 Later Basic Medium Europe Row 8 Soon Luxury Budget Europe Row 7 Soon Basic High US Row 6 Now Luxury Medium US Row 5 Later 4 star Budget US Row 4 Later Luxury High UK Row 3 Soon 4 star Medium UK Row 2 Now Basic Budget UK Row 1 When Hotel Price Where
  • 39. Orthogonal Array L 9 (3 4 ) Each row becomes a test case – you can add more if you wish Now 4 star High Europe Test 9 Later Basic Medium Europe Test 8 Soon Luxury Budget Europe Test 7 Soon Basic High US Test 6 Now Luxury Medium US Test 5 Later 4 star Budget US Test 4 Later Luxury High UK Test 3 Soon 4 star Medium UK Test 2 Now Basic Budget UK Test 1 When Hotel Price Where
  • 40. How to use Orthogonal Arrays
    • Identify the variables
    • Determine the number of choices for each
    • Locate an orthogonal array of the right size
    • Map the testing problem onto the array
    • Construct the test cases
  • 41. What size array do we need? How many possible combinations are there? Choices Variable NT, XP, Linux Server O/S IIS, Apache, WebLogic Server W95, W98, W2000, NT, XP, XP-PRO Operating systems None, Real player, Media Player Plug-ins IE5, IE6, …. 8 in total Browsers
  • 42. Using Orthogonal Arrays
    • We require an array with 5 columns
      • Column 1 needs 8 levels (browsers)
      • Column 2 would need 3 levels (plug ins)
      • Column 3 would need 6 levels (O/S)
      • Columns 4 & 5 would need 3 levels (Server, Server O/S)
    • Ideally we need 8 1 6 1 3 3 but it doesn’t exist!
    • The next largest is L 64 (8 2 4 3 )
      • 2 columns of 1 – 8
      • 3 columns of 1 – 4
    • So rather than 1,296 tests we need only 64 tests!
    • Remember - one test case per row
  • 43. Allpairs Algorithm
    • Devised by James Bach
    • Generates fewer tests
    • In our example, it generates just 48 tests compared to 64 from using orthogonal arrays
    • Consider 100 variables, 2 choices
      • Using orthogonal arrays you need 101 tests
      • Using all pairs you need only 10 tests
  • 44. Pairwise Testing – Summary
    • Test all pairs rather than all combinations
      • With 4 variables and 3 choices for each you would require 81 test cases whereas you can cover all pairwise input combinations with only 9 tests
      • For 13 variables and 3 choices for each you would require1,594,323 test cases whereas you can cover all pairwise input combinations with only 15 tests
    • Pairwise guides us to help find
      • Single mode defects
      • Double mode defects
    • You should always add any known risky combinations
  • 45. Effective Testing – Summary
    • Effective Testing uses
      • A risk-based approach
      • Formal test case design techniques
    • The benefits are
      • Reduced number of test cases
      • Repeatable well documented tests
      • Improved planning, estimation and communication
      • Facilitates earlier involvement with the business
      • Find faults earlier
      • Give you a consistent approach to testing
      • Save time and money
      • Enables test automation
  • 46. After all, what is quality?
  • 47. Presented by Stephen Allott CaseMaker International A Diaz & Hilterscheid group company Effective Testing Using industry standard techniques SoftTest Ireland 16 th February 2006