SOFTWARE TESTING PROCESS Byungwook Cho C. (bwcho75@gmail.com) Sr Consultant 2008.08.08

ICE BREAK

AGENDA Test model Test process Static testing

Test model

Test process

Static testing

Test model

Test oriented development process model V-Model Enhanced traditional waterfall model Verification & Validation < V-Model > < Verification & Validation > Dynamic testing Static testing Testing with system Review & Inspection with artifacts from each development step Valid & estimate the system To verify the artifacts that has been produced in each development cycle. Validation Verification

V-Model

Enhanced traditional waterfall model

Verification & Validation

Test Level ※ Integration strategies Lead by QA Team or Customer Lead by QA Team (Specialized for system test) Lead by Dev Team Lead by Dev Team Responsibility Alpha Test Beta Test Use Mock Object (Test Stub ,Driver) Use Code coverage Black box test Black box test White box test White box test Testing Type Verify customer’s requirement (End User) Verify customer’s Legal issue (Legal) Verify customer’s maintanance issue (SM) Acceptance Test Test system over production environment to verify system itself and production environment Include Functional & Non Functional Test (Availibility, Stablity, Extendability, Performance etc) System Test Verify integration between component. Verify software flow ,interface & interaction Integration Test Verify software component Unit Test Description Level Small project SOA svc BPM Ex. Short time Can test implementation logic easily find architectural defect Adv Most important & high risk component first Integrate all in time Integrate from bottom (use test driver) Integrate from top (use test stub) Desc Backbone Big-bang Bottom up Top down

Test Cycle Each test is consists of test cycle Test cycle Pre test - check up testability Main test Conformation test - check up the defect found in previous “Main test” Regression test - check up the impact from change TEST CYCLE Pre Test Main Test Conformation Test Regression Test

Each test is consists of test cycle

Test cycle

Pre test - check up testability

Main test

Conformation test - check up the defect found in previous “Main test”

Regression test - check up the impact from change

Test Process

Test Process Master Test Plan Organization Policy Strategy Mgmt sub tests Acceptance Test Plan System Test Plan Integration Test Plan Unit Test Plan Test plan Define objective of Test and scope Define Test strategy Test estimation Set up test organization Test analysis and design Evaluation test object & basis Designing and prioritizing test case Identify test Identify & prepare test data Designing test env & identify required infra & tools Make test procedure Test Implementation & Execution Implement TC Implement test script Set up test environment Pre Test Run Test Logging Test Result Test Closure activity Store test-ware into repository Analyzing lesson learned for future test maturity Evaluating exit criteria and reporting Write a test summary report for stakeholders. Assessing if more test are needed Test control Test mgmt & control Test monitoring & reporting Test plan Define objective of Test and scope Define Test strategy Test estimation Set up test organization Test analysis and design Evaluation test object & basis Designing and prioritizing test case Identify test Identify & prepare test data Designing test env & identify required infra & tools Make test procedure Test Implementation & Execution Implement TC Implement test script Set up test environment Pre Test Run Test Logging Test Result Test Closure activity Store test-ware into repository Analyzing lesson learned for future test maturity Evaluating exit criteria and reporting Write a test summary report for stakeholders. Assessing if more test are needed Test control Test mgmt & control Test monitoring & reporting Test plan Define objective of Test and scope Define Test strategy Test estimation Set up test organization Test analysis and design Evaluation test object & basis Designing and prioritizing test case Identify test Identify & prepare test data Designing test env & identify required infra & tools Make test procedure Test Implementation & Execution Implement TC Implement test script Set up test environment Pre Test Run Test Logging Test Result Test Closure activity Store test-ware into repository Analyzing lesson learned for future test maturity Evaluating exit criteria and reporting Write a test summary report for stakeholders. Assessing if more test are needed Test control Test mgmt & control Test monitoring & reporting Test plan Define objective of Test and scope Define Test strategy Test estimation Set up test organization Test analysis and design Evaluation test object & basis Designing and prioritizing test case Identify test Identify & prepare test data Designing test env & identify required infra & tools Make test procedure Test Implementation & Execution Implement TC Implement test script Set up test environment Pre Test Run Test Logging Test Result Test Closure activity Store test-ware into repository Analyzing lesson learned for future test maturity Evaluating exit criteria and reporting Write a test summary report for stakeholders. Assessing if more test are needed Test control Test mgmt & control Test monitoring & reporting

Test plan

Define objective of Test and scope

Define Test strategy

Test estimation

Set up test organization

Test analysis and design

Evaluation test object & basis

Designing and prioritizing test case

Identify test

Identify & prepare test data

Designing test env & identify required infra & tools

Make test procedure

Test Implementation & Execution

Implement TC

Implement test script

Set up test environment

Pre Test

Run Test

Logging Test Result

Test Closure activity

Store test-ware into repository

Analyzing lesson learned for future test maturity

Evaluating exit criteria and reporting

Write a test summary report for stakeholders.

Assessing if more test are needed

Test plan

Define objective of Test and scope

Define Test strategy

Test estimation

Set up test organization

Test analysis and design

Evaluation test object & basis

Designing and prioritizing test case

Identify test

Identify & prepare test data

Designing test env & identify required infra & tools

Make test procedure

Test Implementation & Execution

Implement TC

Implement test script

Set up test environment

Pre Test

Run Test

Logging Test Result

Test Closure activity

Store test-ware into repository

Analyzing lesson learned for future test maturity

Evaluating exit criteria and reporting

Write a test summary report for stakeholders.

Assessing if more test are needed

Test plan

Define objective of Test and scope

Define Test strategy

Test estimation

Set up test organization

Test analysis and design

Evaluation test object & basis

Designing and prioritizing test case

Identify test

Identify & prepare test data

Designing test env & identify required infra & tools

Make test procedure

Test Implementation & Execution

Implement TC

Implement test script

Set up test environment

Pre Test

Run Test

Logging Test Result

Test Closure activity

Store test-ware into repository

Analyzing lesson learned for future test maturity

Evaluating exit criteria and reporting

Write a test summary report for stakeholders.

Assessing if more test are needed

Test plan

Define objective of Test and scope

Define Test strategy

Test estimation

Set up test organization

Test analysis and design

Evaluation test object & basis

Designing and prioritizing test case

Identify test

Identify & prepare test data

Designing test env & identify required infra & tools

Make test procedure

Test Implementation & Execution

Implement TC

Implement test script

Set up test environment

Pre Test

Run Test

Logging Test Result

Test Closure activity

Store test-ware into repository

Analyzing lesson learned for future test maturity

Evaluating exit criteria and reporting

Write a test summary report for stakeholders.

Assessing if more test are needed

Test Plan Test planning & control Define activity of each testing level Define milestone ,resource ,schedule. Make a plan based on analyzed risk factor Define test strategy based on risk Define approach & techniques for testing (testing techniques, coverage, test item, test ware) Must include time & resource for preparing testware Define completion condition Define test scope objective Risk anal Set up strategy Estimate resource & time Planning PRE TEST TEST Monitoring Report

Test planning & control

Define activity of each testing level

Define milestone ,resource ,schedule.

Make a plan based on analyzed risk factor

Define test strategy based on risk

Define approach & techniques for testing (testing techniques, coverage, test item, test ware)

Must include time & resource for preparing testware

Define completion condition

Test Plan Risk analysis Risk = Likelihood * impact Risk identification Recommend < 36 identification . Likelihood : Complexity, Implementation technical Level, Size,Developer skill Impact : Biz impact Erik van neneendaal, Risk Based Testing, STAREAST,2006 factor Risk item Risk item factor factor factor Impact Likelihood Likelihood Impact STA (Severe Test Area) SSTA (Strong Test Area) FTA (Fundamental Test Area) ITA (Intensive Test Area)

Risk analysis

Risk = Likelihood * impact

Risk identification

Recommend < 36 identification .

Likelihood : Complexity, Implementation technical Level, Size,Developer skill

Impact : Biz impact

Test Plan Risk based test strategy Other strategies Analytical approaches Model based approaches Methodical approaches Dynamic and heuristic approaches ( Low Level Test) ( High Level Test) - Unit Test - Integration Test - System Test - Acceptance Test Likelihood Impact STA (Severe Test Area) SSTA (Strong Test Area) FTA (Fundamental Test Area) ITA (Intensive Test Area) Likelihood Impact STA (Severe Test Area) SSTA (Strong Test Area) FTA (Fundamental Test Area) ITA (Intensive Test Area)

Risk based test strategy

Other strategies

Analytical approaches

Model based approaches

Methodical approaches

Dynamic and heuristic approaches

Test Analysis and Design Test analysis Review - test basis, testability Identify test requirement & test data Identify test infra & tools Test design Identify test condition Test case specification Define test procedure

Test analysis

Review - test basis, testability

Identify test requirement & test data

Identify test infra & tools

Test design

Identify test condition

Test case specification

Define test procedure

Test Analysis and Design Design technique Specification based Equivalence partitioning, Boundary value analysis Pairwise, Decision table testing State transition testing, Usecase testing Structure based Control Flow Test Basic Path Test – Based on Cyclomatic complexity (Edge – Node + 2P) P = number of connected component Elementary comparision test – Based on MC/DC Coverage * Coverage : Statement, Condition, Decision, Condition Decision Coverage, Modified Condition/Descision Coverage(MC/DC) , Multiple Condition Coverage Experience based Exploratory Testing – Adhoc testing, based on test charter, define timezone Classification Tree method CheckList

Design technique

Specification based

Equivalence partitioning, Boundary value analysis

Pairwise, Decision table testing

State transition testing, Usecase testing

Structure based

Control Flow Test

Basic Path Test – Based on Cyclomatic complexity (Edge – Node + 2P) P = number of connected component

Elementary comparision test – Based on MC/DC Coverage

* Coverage : Statement, Condition, Decision, Condition Decision Coverage, Modified Condition/Descision Coverage(MC/DC) , Multiple Condition Coverage

Experience based

Exploratory Testing – Adhoc testing, based on test charter, define timezone

Classification Tree method

CheckList

Test Execution Execution Define serverity & priority Develop TC,Test data,Test stub & driver,Test script,Test suite Reporting Logging expected result, real result Current status Time & resource usage Defect tracking Use Bug(Issue) tracking system ( Mantis,Bugzilla,JIRA,Trac) Define tracking workflow Test Execution Test Report Defects Issue Tracking System Progress Report Test logs TEST DRIVER TEST STUB TARGET SYSTEM TEST CASE TESTING TOOL TEST DATA MODULE MODULE MODULE MODULE TEST SCRIPT TEST SUITE1 TEST SUITE2 TEST SIMULATOR

Execution

Define serverity & priority

Develop TC,Test data,Test stub & driver,Test script,Test suite

Reporting

Logging expected result, real result

Current status

Time & resource usage

Defect tracking

Use Bug(Issue) tracking system ( Mantis,Bugzilla,JIRA,Trac)

Define tracking workflow

Evaluating exit criteria and reporting Test exit criteria Coverage, Duration, number of unresolved defects Test result report Progress report Release advice Final report per Level Matrix Defects per KLOC, Days test effort per requirement, Unresolved defect impact analysis, Defects / Hour ,Cumulative defects / Day ,Test coverage / Day , Risk vs number of TC

Test exit criteria

Coverage, Duration, number of unresolved defects

Test result report

Progress report

Release advice

Final report per Level

Matrix

Defects per KLOC, Days test effort per requirement, Unresolved defect impact analysis, Defects / Hour ,Cumulative defects / Day ,Test coverage / Day , Risk vs number of TC

Test closure activities Test closure activities Evaluating Evaluate test target Evaluate test process for maturing next text Final reporting Store test-ware

Test closure activities

Evaluating

Evaluate test target

Evaluate test process for maturing next text

Final reporting

Store test-ware

Static Testing

Static testing techniques Performed in Verification phase Test with artifacts not a runtime software Test by human

Performed in Verification phase

Test with artifacts not a runtime software

Test by human

Manual Review Inspection Objective : To find defects Inspect artifacts Lead by moderator Walkthrough Objective : Knowledge sharing Knowledge transfer by author Code review Objective : Find defects and share knowledge Lead by Programming Leader Pair programming XP Lead by mentor or coworker Share just “ONE” keyboard Formal action Informal action

Inspection

Objective : To find defects

Inspect artifacts

Lead by moderator

Walkthrough

Objective : Knowledge sharing

Knowledge transfer by author

Code review

Objective : Find defects and share knowledge

Lead by Programming Leader

Pair programming

XP

Lead by mentor or coworker

Share just “ONE” keyboard

Static Analysis Static Analysis Test code or diagram with “static analysis tool” Analysis Can find syntax error or well known bugs Validate naming rules Can find dead code or missing parameter Can analysis code complexity & dependency Quality? Not so smart It need a rule and need a maturing Easy to appeal your manager Tools Open source – PMD, Find Bugs Commercial – Klockworks, Jtest etc.

Static Analysis

Test code or diagram with “static analysis tool”

Analysis

Can find syntax error or well known bugs

Validate naming rules

Can find dead code or missing parameter

Can analysis code complexity & dependency

Quality?

Not so smart

It need a rule and need a maturing

Easy to appeal your manager

Tools

Open source – PMD, Find Bugs

Commercial – Klockworks, Jtest etc.