Test Concepts and Practices From Concept to Deliverable Presented by T. James LeDoux, Test Management Consultant 1 An Executive-level overview of quality assurance and test management practices and considerations

Topics Differences between QA and Test Goals of Testing Phases / Levels of Testing Types of Testing Basic Concepts of Test Planning and Creation Pass/Fail Criteria (Discussion) Traps and Pitfalls (Discussion) Successful Completion (Discussion) 2

Differences between QA and Test

Goals of Testing

Phases / Levels of Testing

Types of Testing

Basic Concepts of Test Planning and Creation

Pass/Fail Criteria (Discussion)

Traps and Pitfalls (Discussion)

Successful Completion (Discussion)

Differences Between QA and Test QA Focused on managing organization-wide process for acceptance of upgrade Targets affected system Performs by developing management processes Involves Quality Manager, Quality Director or QA Manager Test Focused on validating level of confidence in upgrade Targets a Release Performs from planned script Involves Test Engineer or Test Manager 3

QA

Focused on managing organization-wide process for acceptance of upgrade

Targets affected system

Performs by developing management processes

Involves Quality Manager, Quality Director or QA Manager

Test

Focused on validating level of confidence in upgrade

Targets a Release

Performs from planned script

Involves Test Engineer or Test Manager

Goals of Testing Increase Quality of the Upgrade Quality = Confidence Reduce risks to the system or business Stabilize the system Meet the needs of the Users Determine capability of the system Define monitoring requirement in production 4

Increase Quality of the Upgrade

Quality = Confidence

Reduce risks to the system or business

Stabilize the system

Meet the needs of the Users

Determine capability of the system

Define monitoring requirement in production

Phases / Levels of Testing Requirements Unit Tests Test Plan Test Scenarios Performance Tests String Tests System Tests UAT Tests Integration Tests Staging Tests 5

Types of Testing Functional Security User Exception Etc. 6

Functional

Security

User

Exception

Etc.

System Tests Unit Tests String Tests System Tests Integration Tests Functional Tests Security Tests Exception Tests Developer System Testing Test Group System Testing 7

Performance Tests Performance Tests System Tests UAT Tests Staging Tests Throughput Tests Stress Tests Benchmark Tests Load Tests 8

User Tests Performance Tests System Tests UAT Tests Staging Tests User Acceptance Tests Business Exception Tests 9

Deployment Tests Performance Tests System Tests UAT Tests Staging Tests Deployment Tests Release Notes Test 10

The Test Harness Acts as the initiator of the object’s task Typically structured with three parts: Input data and form Setup process Output data handler Must also be capable of creating and determining exception responses 11

Acts as the initiator of the object’s task

Typically structured with three parts:

Input data and form

Setup process

Output data handler

Must also be capable of creating and determining exception responses

Test Harness Example Arithmetic Adder A + B = C Method ARADD (A,B,C) Input data Iteration A B 1 2 3 4 5 Except 6 Except 7 1 2 4 5 7 21 S 1 3 7 8 12 K 8 Expected data Iteration C P/F 1 2 3 4 5 Except 6 Except 7 2 5 11 13 19 Error Error P P P P P F F 12

Test Plan Page 1 Describes purpose. Example: One of the primary goals of the Web To Compass project at Compassion International is to streamline the process of getting a web transaction from the web site (Compassion.com) to the Compass database. Currently, the web site and Compass are disconnected, and a series of manual and semi-automated processes are in place to facilitate this work. 1. Introduction 1.1 Purpose 1.2 Goals Describes the goals of the testing. Example: This Test Plan for the Compassion W2C Release supports the following Goals: Defines the activities required to prepare for and conduct Unit, System, Load, and Performance Testing. Communicates to all responsible parties the testing strategy. 1.3 Test Scope Describes scope of testing. Example: Testing encompasses System, Load and Regression testing under a manual test process. 1.4 System Overview Describes the system layout along with a diagram 13

Describes the goals of the testing.

Example:

This Test Plan for the Compassion W2C Release supports the following Goals:

Defines the activities required to prepare for and conduct Unit, System, Load, and Performance Testing.

Communicates to all responsible parties the testing strategy.

Example of a System Overview Diagram 14

Test Plan Page 2 2.0 Test Strategy Overview (From Test Strategy document) 15 2.1 Purpose, 2.2 Approach 2.3 Validation

Test Plan Page 3 3.1 Unit/String Testing Unit/String Testing Environment Test Data Installation, Verification and Control Unit/String Test to be Performed Unit/String Test Logging and Reporting 3.1 Integration Testing Integration Environment Test Data Installation, Verification and Control Test Team Logistics Logging and Reporting 3.0 Test Levels

Unit/String Testing Environment

Test Data

Installation, Verification and Control

Unit/String Test to be Performed

Unit/String Test Logging and Reporting

Integration Environment

Test Data

Installation, Verification and Control

Test Team Logistics

Logging and Reporting

Test Plan Page 4 System Test Environment Test Data Installation, Verification and Control Test Team Logistics System Test to be Performed System Test Logging and Reporting Defect Management Quality Requirements and Metrics (see next slide) Roles and Responsibilities 16 3.2 System Test 3.3 Performance Test

System Test Environment

Test Data

Installation, Verification and Control

Test Team Logistics

System Test to be Performed

System Test Logging and Reporting

Defect Management

Quality Requirements and Metrics (see next slide)

Roles and Responsibilities

Quality Requirements and Metrics Section Entrance Criteria Test Plan complete Test Cases complete Test Procedures complete Test Scenarios complete Execution Schedule complete Test data acquired User names, passwords and access rights issued Test environment ready Exit Criteria Testing completed Test results complete Screen captures complete Test logs up to date Defects identified and reported or fixed 17

Test Plan complete

Test Cases complete

Test Procedures complete

Test Scenarios complete

Execution Schedule complete

Test data acquired

User names, passwords and

access rights issued

Test environment ready

Testing completed

Test results complete

Screen captures complete

Test logs up to date

Defects identified and reported or

fixed

Test Plan Page 5 4.0 Test Identification and Definition 18 Requirements Testability Matrix Functional Scenarios Exception Scenarios End-To-End Functional Test Cases Test Cases not associated with a Use Case Exception Tests End-To-End 4.1 Test Scenarios

Requirements Testability Matrix

Functional Scenarios

Exception Scenarios

End-To-End

Functional Test Cases

Test Cases not associated with a

Use Case

Exception Tests

End-To-End

Test Plan Page 6 4.2 Performance Test Scenarios 4.3 Regression Test Scenarios 4.4 Test Procedures 19 Unit/String Test Schedule Integration Test Schedule System Test Schedule Performance Test Schedule Manual Test Procedures Automated Test Procedures 5.0 Test Schedule

Unit/String Test Schedule

Integration Test Schedule

System Test Schedule

Performance Test Schedule

Manual Test Procedures

Automated Test Procedures

Test Case Example 20 Section Description Procedure Name ABC1-TC100.0 Use Case 1 Happy Path Procedure Overview This test covers the 2 steps of the UC1 happy path; 1. Tester confirms no web comments 2. Tester logs into system and navigates to the constituent screen and enters the constituent number Procedure Dependencies 1. Records have been properly created for the constituents being pulled up. Functional Requirements 1. No web comments are placed in the comments fields in the database 2. The IIS is properly set up. Procedure Test Data Test Database Image Suite TDB100.0 Test Procedure Conducted by __________________________________ Date Conducted ________________________ Build Label/Version Pre-Amble Instructions All databases need to be initialized to a known condition. This is typically done by copying the golden database images to their respective databases. Note that for Release 1, the databases will be small and the restoration of the databases will be more manual than is typical with a golden database environment. Procedure Instructions Steps Passed/Fail Data Input Results Comments Setup steps Step 1 – Tester confirms no web comments Step 2 – Tester logs into the system Post-Amble Instructions Do not restore the golden database to this environment after this test. Selected constituents in the database are set up for each test scenario to reduce the need to restore the database.

Summary QA is organization-wide; Test is Project-wide QA requires that testing be performed at the development, validation and deployment phases The Test Strategy defines the validation approach (test, acceptance process) for the business need The Test Plan defines how testing is managed The Test Cases define how testing is executed 21

QA is organization-wide; Test is Project-wide

QA requires that testing be performed at the development, validation and deployment phases

The Test Strategy defines the validation approach (test, acceptance process) for the business need

The Test Plan defines how testing is managed

The Test Cases define how testing is executed