Configuration identification Configuration Identification Configuration Control Status Accounting Configuration Auditing Configuration Structures CI Planning Version/issue Numbering Baseline/release Planning Naming Conventions CI: Configuration item: stand alone, test alone, use alone element Selection criteria
Configuration control Configuration Identification Configuration Control Status Accounting Configuration Auditing CI Submission Withdrawal/ Distribution control Status/version Control Clearance Investigation Impact Analysis Authorised Amendment Review/ Test Controlled Area/library Problem/fault Reporting Change Control Configuration Control Board
Status accounting & Configuration Auditing Configuration Identification Configuration Control Status Accounting Configuration Auditing Status Accounting Database Input to SA Database Queries and Reports Data Analysis Traceability, impact analysis Procedural Conformance CI Verification Agree with customer what has been built, tested & delivered
If it takes 10 mins to write a fault report, how many can be written in one day?
The more fault reports you write, the less testing you will be able to do.
Test Fault analysis & reporting Mike Royce: suspension criteria: when testers spend > 25% time on faults
Measuring test execution progress 1 tests run tests passed tests planned now release date what does this mean? what would you do?
Diverging S-curve poor test entry criteria ran easy tests first insufficient debug effort common faults affect all tests software quality very poor tighten entry criteria cancel project do more debugging stop testing until faults fixed continue testing to scope software quality Note: solutions / actions will impact other things as well, e.g. schedules Possible causes Potential control actions
Measuring test execution progress 2 tests planned run passed action taken old release date new release date
Measuring test execution progress 3 tests planned run passed action taken old release date new release date
Case history Source: Tim Trew, Philips, June 1999
analysis (s/w fault, test fault, enhancement, etc.)
assignment to fix (if fault)
fixed not tested
fixed and tested OK
Use of incident metrics Is this testing approach “wearing out”? What happened in that week? We’re better than last year How many faults can we expect?
Report as quickly as possible? report 5 test can’t reproduce - “not a fault” - still there can’t reproduce , back to test to report again insufficient information - fix is incorrect dev 5 reproduce 20 fix 5 re-test fault fixed 10 dev can’t reproduce incident report test 10
Summary: Key Points Independence can be achieved by different organisational structures Configuration Management is critical for testing Tests must be estimated, monitored and controlled Incidents need to be managed Standards for testing: quality, industry, testing Test Management 1 2 4 5 3 6 ISTQB / ISEB Foundation Exam Practice