Testing TestsIdeas for better quality test software
Brian Colfer     brian.colfer@gmail.com                     @briancolfer
Introduction• Testing software is subject to error• False negatives are the main problem• Use Various strategies to help d...
Defects in tests• Same source as product defects:• Misunderstood requirements• Logic errors
Two types• False positives: type I errors we detect  something when it doesnt exist• False negatives: type II errors we do...
Testing the test          framework• Not controversial• Normal test tools and techniques• Frameworks are normal products
Track defects in tests• Have QA framework and tests as a part of  defect tracking system• Have a release process for softw...
Type I errors• Not as much of a worry but costs• Further examination will catch the error• Costs in delays and analysis ef...
Type II errors• Insidious, you dont know that you dont know• Costs of the defect: normal user defect costs,  repair costs ...
Strategies• Unit tests• Reviews• Quality Control metrics
Unit tests• Unit tests validate that test code is correct not  the application under test• Use TDD or BDD to write tests• ...
Reviews• Step through the code looking for defects• Make it a frequent practice• Keep the scope digestible
Quality control metrics• Traditional QC doesn’t apply to software • QC will look for changes in power and   sensitivity of...
Multi method testing• Use fault injection to find test errors and  ungraceful application failure• Exploratory testing• Do...
Upcoming SlideShare
Loading in …5
×

Testing tests

377 views
349 views

Published on

Strategies for improving the quality of automated tests.

Wednesday Last Track Selenium Conference 2011

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
377
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Testing tests

    1. 1. Testing TestsIdeas for better quality test software
    2. 2. Brian Colfer brian.colfer@gmail.com @briancolfer
    3. 3. Introduction• Testing software is subject to error• False negatives are the main problem• Use Various strategies to help detect when there is a problem
    4. 4. Defects in tests• Same source as product defects:• Misunderstood requirements• Logic errors
    5. 5. Two types• False positives: type I errors we detect something when it doesnt exist• False negatives: type II errors we dont detect something when it does exit
    6. 6. Testing the test framework• Not controversial• Normal test tools and techniques• Frameworks are normal products
    7. 7. Track defects in tests• Have QA framework and tests as a part of defect tracking system• Have a release process for software framework and tests
    8. 8. Type I errors• Not as much of a worry but costs• Further examination will catch the error• Costs in delays and analysis effort
    9. 9. Type II errors• Insidious, you dont know that you dont know• Costs of the defect: normal user defect costs, repair costs and test repair costs.• Deterioration of confidence in software
    10. 10. Strategies• Unit tests• Reviews• Quality Control metrics
    11. 11. Unit tests• Unit tests validate that test code is correct not the application under test• Use TDD or BDD to write tests• Run unit tests as part of a CI practice for test development
    12. 12. Reviews• Step through the code looking for defects• Make it a frequent practice• Keep the scope digestible
    13. 13. Quality control metrics• Traditional QC doesn’t apply to software • QC will look for changes in power and sensitivity of the testing • Regression tests are supposed to not find errors• Track defects found by automated tests• Establish a baseline rate of defect detection
    14. 14. Multi method testing• Use fault injection to find test errors and ungraceful application failure• Exploratory testing• Don’t make your tests too DRY

    ×