Welcome IPMA and SolutionsIQ Professional Event Testing, Testing, 1…2…3… Improving software quality -- one bug at a time

Agenda Building the Test Framework Jan McCollum, SolutionsIQ Break Practical Panel Discussion Cheryl Hainje – AFRS Product Manager, OFM Dotti Lane – QA Project Manager, OFM Tim Vessey – POS Project Manager, LCB Stein Wang – Quality Assurance Lead, SolutionsIQ Break Testing Templates & Checklists

Building the Test Framework

Jan McCollum, SolutionsIQ

Break

Practical Panel Discussion

Cheryl Hainje – AFRS Product Manager, OFM

Dotti Lane – QA Project Manager, OFM

Tim Vessey – POS Project Manager, LCB

Stein Wang – Quality Assurance Lead, SolutionsIQ

Break

Testing Templates & Checklists

SolutionsIQ Overview SolutionsIQ is a full-spectrum IT services company 25 years of technology services and solutions 400+ consultants Corporate headquarters (Bellevue, WA) Professional Services (Bellevue, WA) Oregon Branch Office (Lake Oswego, OR) 8+ years of serving the State of WA DOC, AOC, LCB, DNR, DOL, LNI, & DSHS

SolutionsIQ is a full-spectrum IT services company

25 years of technology services and solutions

400+ consultants

Corporate headquarters (Bellevue, WA)

Professional Services (Bellevue, WA)

Oregon Branch Office (Lake Oswego, OR)

8+ years of serving the State of WA

DOC, AOC, LCB, DNR, DOL, LNI, & DSHS

SolutionsIQ Expertise Professional Services Division Consulting and Analytical Solutions Project management Assessments and feasibility studies Design and architecture roadmaps Development and Test Solutions Full life cycle development projects Custom application development EAI, portals, and business intelligence Quality assurance and testing solutions

Professional Services Division

Consulting and Analytical Solutions

Project management

Assessments and feasibility studies

Design and architecture roadmaps

Development and Test Solutions

Full life cycle development projects

Custom application development

EAI, portals, and business intelligence

Quality assurance and testing solutions

Building the Testing Framework Jan McCollum Manager, Quality Assurance and Testing Solutions

Setting Goals Knowing WHAT you want is as important as knowing how to get it Defining the vision Defining the timeline Gaining acceptance and buy in

Knowing WHAT you want is as important as knowing how to get it

Defining the vision

Defining the timeline

Gaining acceptance and buy in

Defining the Vision To define the vision look at what came before What went well What went badly What now Where do you want to go

To define the vision look at what came before

What went well

What went badly

What now

Where do you want to go

Testing vs. Quality Assurance Testing is about finding bugs Quality Assurance is about preventing them!

Testing is about finding bugs

Quality Assurance is about preventing them!

Quality Assurance Takes time Is about the overall effort – including development Methodologies can be very formal

Takes time

Is about the overall effort – including development

Methodologies can be very formal

QA Applied to Testing Quality assurance principals applied to the testing effort will produce higher quality work

Quality assurance principals applied to the testing effort will produce higher quality work

Establishing a Timeline The 6 month / 1 year / 3 year plan Implement processes and strategies that give the best return on investment

The 6 month / 1 year / 3 year plan

Implement processes and strategies that give the best return on investment

Quality Testing Roadmap After the goals and objectives are complete, make them real by publishing the quality testing roadmap

After the goals and objectives are complete, make them real by publishing the quality testing roadmap

Quality Testing Roadmap Roadmap should include… Test team structure Communications plans Test processes Test procedures

Roadmap should include…

Test team structure

Communications plans

Test processes

Test procedures

Quality Testing Roadmap Test scope Test dependencies and impacts Automation transition plan Test deliverables

Test scope

Test dependencies and impacts

Automation transition plan

Test deliverables

Gaining Acceptance and Buy In Development Business management Project management IT management Customer/product support

Development

Business management

Project management

IT management

Customer/product support

Making it Happen! Organizational structure Qualified candidates Roles and responsibilities

Organizational structure

Qualified candidates

Roles and responsibilities

Test Planning The master test plan: a one-stop shopping guide for your project Contents Contributing documents Sign-off procedures

The master test plan: a one-stop shopping guide for your project

Contents

Contributing documents

Sign-off procedures

Test Planning Test matrix and test suites Detailed test steps Pass/Fail results Tester who performed tests

Test matrix and test suites

Detailed test steps

Pass/Fail results

Tester who performed tests

Test Planning Test case design – what is a good test case? Accurate – tests what it’s designed to test Repeatable, reusable – has a life after this release Economical – no unnecessary steps

Test case design – what is a good test case?

Accurate – tests what it’s designed to test

Repeatable, reusable – has a life after this release

Economical – no unnecessary steps

Test Planning Test case design Traceable to a requirement Appropriate for test environment, testers Self-standing has enough information for anyone to run

Test case design

Traceable to a requirement

Appropriate for test environment, testers

Self-standing has enough information for anyone to run

Test Planning Test case design: How to make good test cases better Setup, environment, data Steps, actions and expected results Use active voice in expected results System displays this, does that Simple, conversational language

Test case design: How to make good test cases better

Setup, environment, data

Steps, actions and expected results

Use active voice in expected results

System displays this, does that

Simple, conversational language

Test Planning Test case design: Why work to improve test cases? Productivity – less time to write and maintain cases Testability – less time to execute them Scheduling – better reliability in estimates

Test case design: Why work to improve test cases?

Productivity – less time to write and maintain cases

Testability – less time to execute them

Scheduling – better reliability in estimates

Defect (Bug) Management Deciding upon a tool Easy of configuration Ability to add/change fields Reporting capabilities Integrated solution

Deciding upon a tool

Easy of configuration

Ability to add/change fields

Reporting capabilities

Integrated solution

Defect (Bug) Management The bug lifecycle Who can create bugs Who can assign bugs Who can close bugs

The bug lifecycle

Who can create bugs

Who can assign bugs

Who can close bugs

Defect (Bug) Management The bug triage meeting Purpose and who should go Reporting Determining a trend Bug metrics Number of bugs found Bugs found in production vs. test cycle

The bug triage meeting

Purpose and who should go

Reporting

Determining a trend

Bug metrics

Number of bugs found

Bugs found in production vs. test cycle

Moving On Improving the process: Requirements traceability Test cases for each requirement Requirements matrix Tracing requirements to defects

Improving the process: Requirements traceability

Test cases for each requirement

Requirements matrix

Tracing requirements to defects

Moving On Improving the process: Risk-based testing You can’t test everything so test what is important The risk list and how to use it to drive test strategy

Improving the process: Risk-based testing

You can’t test everything so test what is important

The risk list and how to use it to drive test strategy

Broadening Your Scope Build verification testing Also called smoke or acceptance tests Is a subset of the major functional areas Integration testing Testing the entire system

Build verification testing

Also called smoke or acceptance tests

Is a subset of the major functional areas

Integration testing

Testing the entire system

Broadening Your Scope Compatibility testing How application works with other apps Configuration testing Testing on different configurations Setup testing Testing the installation Regression testing Verify if bug fixes are successful

Compatibility testing

How application works with other apps

Configuration testing

Testing on different configurations

Setup testing

Testing the installation

Regression testing

Verify if bug fixes are successful

Broadening Your Scope Black box testing White box testing Grey box testing

Black box testing

White box testing

Grey box testing

Improving Quality Testing metrics – measure your success Bug tracking metrics Number found Number found per component Daily bug find rate

Testing metrics – measure your success

Bug tracking metrics

Number found

Number found per component

Daily bug find rate

Improving Quality Test case effectiveness Metric: Test case effectiveness; test case effectiveness = bugs found in test/total found * 100 Test coverage Metric: Test coverage (absolute) = tests conducted/total tests * 100

Test case effectiveness

Metric: Test case effectiveness; test case effectiveness = bugs found in test/total found * 100

Test coverage

Metric: Test coverage (absolute) = tests conducted/total tests * 100

Improving Quality Test team performance Metric: Test process effectiveness: test process effectiveness = bugs fixed/bugs found * 100 Metric: Planned days vs. actual days in test

Test team performance

Metric: Test process effectiveness: test process effectiveness = bugs fixed/bugs found * 100

Metric: Planned days vs. actual days in test

Improving Quality QA and test involvement early! Design reviews Why testers should attend Develop and use checklists Project closeout meetings You should have them

QA and test involvement early!

Design reviews

Why testers should attend

Develop and use checklists

Project closeout meetings

You should have them

Questions? For additonal information, email [email_address]

For additonal information, email

[email_address]