Boost Your Testing Processes October 29, 2008 www.uTest.com

Choosing a Testing Process X Development Major Release! Traditional (Waterfall) Agile Requirements Analysis System Design Coding QA & Bug Fixes Rqmts Dsgn Code Release Test Test Rqmts Dsgn Code Release Test Test Rqmts Dsgn Code Release Test Test

It’s not One or The Other! Traditional Agile Adopt practices from each that make sense for you

Avoiding Us vs. Them QA is delaying the release. QA is not cost-effective. QA gets in the way. Mgmt doesn’t get it. Mgmt keeps changing priority between Quality and Time-To-Market. Management QA

Avoiding Us vs. Them QA is delaying the release. QA is not cost-effective. QA gets in the way. Mgmt doesn’t get it. Mgmt keeps changing priority between Quality and Time-To-Market. Management QA X X Mgmt doesn’t use my data. QA doesn’t give me valuable data.

Measurements: Not just: # Test Cases (written / performed / successful / failed / by severity etc.) Find statistics that mean something: Business Impact Missed Sales Revenues Customer Renewal Rate Punitive Costs Development Process Defect Density In-Step Containment Cost per Defect

Not just: # Test Cases (written / performed / successful / failed / by severity etc.)

Find statistics that mean something:

Business Impact

Missed Sales Revenues

Customer Renewal Rate

Punitive Costs

Development Process

Defect Density

In-Step Containment

Cost per Defect

Defect Density Step Size Defects Defect Density Requirements 120 pgs 52 0.43 / pg Design 175 objects 142 0.81 / obj Coding 50,000 LoC 724 14.4 / KLoC

Use Historical Data! New Project: Step Size Historical Defect Density Expected Defects Requirements 150 pgs 0.43 / pg 64 Design 220 objects 0.81 / obj 178 Coding 80,000 LoC 14.4 / KLoC 1152 This is Actionable Data! Resource Planning / Budgeting

Relative Cost of Defect by Time of Discovery

In-Step Containment: Errors vs. Defects Error = Mistake captured within the step Defect = Mistake captured downstream Containment Rate = E / (E+D) Step Errors Defects Containment Rate Requirements 3 52 5% Design 281 142 66% Coding 1183 724 62% This is Actionable Data! Requirements Review Process

Error = Mistake captured within the step

Defect = Mistake captured downstream

Containment Rate = E / (E+D)

Cost per Defect In Person-Hours In $ Step Defects Effort (person-hours) Direct Cost (non-HR) Total Cost per Defect Requirements 52 67 $8K $282 Design 142 83 $22K $213 Coding 724 242 $35K $81 This is Actionable Data! Resource Optimization

In Person-Hours

In $

Self-measurement Capture relevant data Description, How it was caught, When it should have been caught, Cost Side benefit: Better scheduling (Evidence-Based Scheduling) http://www.joelonsoftware.com/items/2007/10/26.html

Capture relevant data

Description, How it was caught, When it should have been caught, Cost

Side benefit: Better scheduling (Evidence-Based Scheduling) http://www.joelonsoftware.com/items/2007/10/26.html

What is Your Target? Find every bug? Make the software ‘good’? Testing is the infinite process of comparing the invisible to the ambiguous in order to avoid the unthinkable happening to the anonymous - James Bach

Find every bug?

Make the software ‘good’?

Define Testing Early Software Requirements = Test Plan! Creates the team early Eliminate “SRS-to-TestPlan” Alchemy

Software Requirements = Test Plan!

Creates the team early

Eliminate “SRS-to-TestPlan” Alchemy

Test Case = Spec Quality Experts know how to find errors. Product Managers need this during Spec Developers need this during Design/Coding Test Driven Development Write the test before you code

Quality Experts know how to find errors.

Product Managers need this during Spec

Developers need this during Design/Coding

Test Driven Development

Write the test before you code

Test Case = Spec Do them! Test Plan Review is not sole domain of QA Review Review

Do them!

Test Plan Review is not sole domain of QA

Define the Testing Matrix Matrix coverage = Spec Give guidance early to PM, Dev Consider matrix variables Browser Type and Version # – IE, Firefox, Opera, etc. Browser settings – Allow pop-ups, Enable active content, etc. Existence of plug-ins and their version # – Flash, Acrobat, various toolbars, etc. Firewall Anti-Virus Operating System Language and Regional Settings Make sure your QA process can handle the Matrix

Matrix coverage = Spec

Give guidance early to PM, Dev

Consider matrix variables

Browser Type and Version # – IE, Firefox, Opera, etc.

Browser settings – Allow pop-ups, Enable active content, etc.

Existence of plug-ins and their version # – Flash, Acrobat, various toolbars, etc.

Firewall

Anti-Virus

Operating System

Language and Regional Settings

Make sure your QA process can handle the Matrix

Use Case X X Storytelling

Storytelling… Is more fun than Use Casing! Encourages exploratory testing (without getting too informal) Empowers the team (without using the ‘empower’ buzzword) Ties well into Regression Testing

Is more fun than Use Casing!

Encourages exploratory testing (without getting too informal)

Empowers the team (without using the ‘empower’ buzzword)

Ties well into Regression Testing

= Don’t worry about that Failure… The Feature Is Not For This Release Don’t worry about that Failure… It’s a Known Bug

Testing and the End-User

Thank you! Q&A For more info: http://www.utest.com/webinar_test_processes.htm