Testing Framework: Basic testing approach By Javier Fernández-Pello Alvargonzález

Agenda 05/06/09 Introduction Testing Methodologies: TMMi (Test Maturity Model) Basic Concept The problem Implementation Objectives Roles and Responsibilities Testing Activities Testing Life Cycle The processes Metrics: The Dashboard The tool The Final Picture

Introduction

Testing Methodologies: TMMi (Test Maturity Model)

Basic Concept

The problem

Implementation

Objectives

Roles and Responsibilities

Testing Activities

Testing Life Cycle

The processes

Metrics: The Dashboard

The tool

The Final Picture

Introduction What is a Testing Framework? “A framework is a set methods, tools and objects used to address complex issues” There could be different kinds of frameworks or frameworks could be used for different purposes. A testing framework provides a schema that organizes testing activities into logical units to help control and integrate testing throughout your application life cycle. Why my organization needs a Testing Framework ? A Testing Framework provides a end to end approach for implementing testing activities. It is necessary to provide a flexible, scalable and cost effective ‘on-demand’ testing approach for all non-functional and functional testing needs and setup the overall testing process. Increase the long-term success of a project. Testing activities are easier to manage. Repeatability 05/06/09

What is a Testing Framework? “A framework is a set methods, tools and objects used to address complex issues”

There could be different kinds of frameworks or frameworks could be used for different purposes.

A testing framework provides a schema that organizes testing activities into logical units to help control and integrate testing throughout your application life cycle.

Why my organization needs a Testing Framework ?

A Testing Framework provides a end to end approach for implementing testing activities.

It is necessary to provide a flexible, scalable and cost effective ‘on-demand’ testing approach for all non-functional and functional testing needs and setup the overall testing process.

Increase the long-term success of a project.

Testing activities are easier to manage.

Repeatability

Testing Framework Testing Methodologies - TMMi 05/06/09 The model consist on 5 levels of maturity. The model is inspired by the CMM (Capability Maturity Model) Identify areas that can be improved. Identify testing processes that can be adopted organization-wide. Provide a method for measuring the improvement results. TMMi (Test Maturity Model): The TMMi framework is a guideline for test process improvement.

The model consist on 5 levels of maturity.

The model is inspired by the CMM (Capability Maturity Model)

Identify areas that can be improved.

Identify testing processes that can be adopted organization-wide.

Provide a method for measuring the improvement results.

TMMi (Test Maturity Model):

The TMMi framework is a guideline for test process improvement.

Testing Framework Testing Methodologies - TMMi Why do we need to use TMMi ? Allows you to know where you are standing. It can be used as a assessment tool and easily incorporated into your software quality procedures. Provides guides for incremental improvements. As the testing process moves up the maturity levels, your organization saves testing costs. 05/06/09 TMMi Structure: The structure of each level consists of a set of objectives of maturity supported by sub-objectives of maturity that are reached through activities, tasks and responsibilities. The activities and tasks represent the activities that must be carried out in each level in order to improve the capacity of the test. The activities and tasks are assigned to the managers, developers, testers , users ….

Why do we need to use TMMi ?

Allows you to know where you are standing.

It can be used as a assessment tool and easily incorporated into your software quality procedures.

Provides guides for incremental improvements.

As the testing process moves up the maturity levels, your organization saves testing costs.

TMMi Structure:

The structure of each level consists of a set of objectives of maturity supported by sub-objectives of maturity that are reached through activities, tasks and responsibilities.

The activities and tasks represent the activities that must be carried out in each level in order to improve the capacity of the test.

The activities and tasks are assigned to the managers, developers, testers , users ….

Testing Framework Basic Concepts 05/06/09 According to the V-model, the correct testing approach states that each project phase has to be tested with suitable methods

Testing Framework The Problem 05/06/09 Four different environments. Testing activities must be done in all the defined environments. Necessity to define and standardize all the testing activities and concepts. Different vendors all around the world. Vendors working Onsite.

Four different environments.

Testing activities must be done in all the defined environments.

Necessity to define and standardize all the testing activities and concepts.

Different vendors all around the world.

Vendors working Onsite.

Testing Framework Implementation Identify Business Goals. ( Objectives ) Analyze how many people you need to accomplice Business goals. ( Roles ) Define the testing activities that you want to do. Select your team based on those activities. ( Testing Activities ) Begin defining the Life Cycle that you want to implement in your organization. Define the testing processes to be implemented. Examine what to measure and how to measure it. ( Metrics ) Select the appropriate tool to ensure your testing activities. 05/06/09

Identify Business Goals. ( Objectives )

Analyze how many people you need to accomplice Business goals. ( Roles )

Define the testing activities that you want to do. Select your team based on those activities. ( Testing Activities )

Begin defining the Life Cycle that you want to implement in your organization.

Define the testing processes to be implemented.

Examine what to measure and how to measure it. ( Metrics )

Select the appropriate tool to ensure your testing activities.

Testing Framework Objectives 05/06/09 Define your testing objectives. What is the company expecting from the testing activities? Testing, like many other aspects of software engineering, lends itself to a principle-based approach, where core principles are applied, regardless of the implementation technology or application scenario. Some examples: Ensure every function produces the expected outcome ( Functional Testing ). Ensure the system integrates with and does not adversely affect other enterprise systems ( System Integration Testing ) . Ensure the customer is satisfied with the system ( User Acceptance Test (UAT)) Validate the software meets all specifications and requirements. Identify application's higher-risk aspects. Reduce software development costs.

Define your testing objectives.

What is the company expecting from the testing activities?

Testing, like many other aspects of software engineering, lends itself to a principle-based approach, where core principles are applied, regardless of the implementation technology or application scenario.

Some examples:

Ensure every function produces the expected outcome ( Functional Testing ).

Ensure the system integrates with and does not adversely affect other enterprise systems ( System Integration Testing ) .

Ensure the customer is satisfied with the system ( User Acceptance Test (UAT))

Validate the software meets all specifications and requirements.

Identify application's higher-risk aspects.

Reduce software development costs.

Testing Framework Roles and Responsibilities Define an appropriate testing team. Hire team members with testing skills. Roles Description: Test Director: Responsible of the team’s management. Test Manager: Test Management, Test Plan, Customer interaction Team Management, Work allocation to the team, Reviews and Status reporting . Test Lead: Test planning, Customer interaction, Team management, Reviews criticality/Severity of defects and Status reporting. Tester Engineer: Development of Test cases and Scripts, Test Execution, Result capturing and analyzing, Defect Reporting and Status reporting. 05/06/09

Define an appropriate testing team.

Hire team members with testing skills.

Roles Description:

Test Director: Responsible of the team’s management.

Test Manager: Test Management, Test Plan, Customer interaction Team Management, Work allocation to the team, Reviews and Status reporting .

Test Lead: Test planning, Customer interaction, Team management, Reviews criticality/Severity of defects and Status reporting.

Tester Engineer: Development of Test cases and Scripts, Test Execution, Result capturing and analyzing, Defect Reporting and Status reporting.

Testing Framework Testing Team : Onsite – Offshore Model 05/06/09

Testing Framework Testing Activities Describe the most important test activities that you need to accomplice. Include white box, black box, automatic and performance testing. Determinate testing techniques to be used. Determinate test environment. Describe all the processes that involve these testing activities. Define the responsible of each activity. 05/06/09

Describe the most important test activities that you need to accomplice.

Include white box, black box, automatic and performance testing.

Determinate testing techniques to be used.

Determinate test environment.

Describe all the processes that involve these testing activities.

Define the responsible of each activity.

Testing Framework Testing Activities: Smoke Test 05/06/09 INPUT: Code is promoted to SIT ACTIVITIES: Smoke Test will be executed. Test result is reviewed. The code won't be promoted unless all the test cases of the SMOKE TEST are PASSED . If defects are detected while Smoke Test execution the test must be executed again until the exit criteria is met. After the Smoke Test execution result is OK, the code will be promoted and a Smoke Test executed again in UAT environment. OUTPUT : Smoke Test Result: PASSED . Application ready to be tested in UAT.

INPUT:

Code is promoted to SIT

ACTIVITIES:

Smoke Test will be executed.

Test result is reviewed. The code won't be promoted unless all the test cases of the SMOKE TEST are PASSED .

If defects are detected while Smoke Test execution the test must be executed again until the exit criteria is met.

After the Smoke Test execution result is OK, the code will be promoted and a Smoke Test executed again in UAT environment.

OUTPUT :

Smoke Test Result: PASSED .

Application ready to be tested in UAT.

Testing Framework Testing Life Cycle 05/06/09 Define the Testing Life Cycle in conjunction of your company's Software Development Life Cycle. Phase 1 – Approach: Outlines the detailed objectives, scope of tests to be executed, entry/exit criteria, test tools and test environment for the respective test phase. Create Work Plan: Outlines the detailed tasks, resources and schedule for the test phase. Phase 2 - Plan: The objective of this phase is to develop an overall test plan. Estimation & Schedule. Defining Test Pass/Fail Criteria. Test Environment definition. Resource Requirements Identification. Test Coverage. Test deliverable.

Define the Testing Life Cycle in conjunction of your company's Software Development Life Cycle.

Phase 1 – Approach:

Outlines the detailed objectives, scope of tests to be executed, entry/exit criteria, test tools and test environment for the respective test phase.

Create Work Plan: Outlines the detailed tasks, resources and schedule for the test phase.

Phase 2 - Plan: The objective of this phase is to develop an overall test plan.

Estimation & Schedule.

Defining Test Pass/Fail Criteria.

Test Environment definition.

Resource Requirements Identification.

Test Coverage.

Test deliverable.

Testing Framework Testing Life Cycle Phase 3 - Prepare: Complete Test Model: Built on the Test Plan created in the planning stage by defining input data and expected results, scripting the Test Cycles and preparing the test cycle schedule. The deliverable is a Test Model that includes all Test Scripts. Prepare Test Environment: Ensures that the environment is established and proven before test execution. Manage Test Phase: Ensures that the progress of the test phase is periodically measured. Phase 4 - Execute: Based on the Test plan and test cases, both manual and automated test cases would be executed. This phase includes the following activities: Test Execution Environment setup. Automated and Performance Test script execution. Test Script/Cases Execution. Capturing the Test Results. Report Defects. Record Defects. Phase 5 – Close: Sign-Off: Ensures that the test phase has been completed in line with requirements. All exit criteria have been met. Analysis and Reporting. Retrospective meeting. 05/06/09

Phase 3 - Prepare:

Complete Test Model: Built on the Test Plan created in the planning stage by defining input data and expected results, scripting the Test Cycles and preparing the test cycle schedule. The deliverable is a Test Model that includes all Test Scripts.

Prepare Test Environment: Ensures that the environment is established and proven before test execution.

Manage Test Phase: Ensures that the progress of the test phase is periodically measured.

Phase 4 - Execute: Based on the Test plan and test cases, both manual and automated test cases would be executed. This phase includes the following activities:

Test Execution Environment setup.

Automated and Performance Test script execution.

Test Script/Cases Execution.

Capturing the Test Results.

Report Defects.

Record Defects.

Phase 5 – Close:

Sign-Off: Ensures that the test phase has been completed in line with requirements.

All exit criteria have been met.

Analysis and Reporting.

Retrospective meeting.

Testing Framework Testing Life Cycle 05/06/09

Testing Framework Processes 05/06/09 Provide steps for test modelling. For simplification and tangible results, the life cycle is decomposed into activities with inputs, main activities (steps), and outputs. Define for each process the input details, activities and output details. Define the process responsible. Define the “Test Entry Criteria” and the “Test Exit Criteria” and “Sing-off responsibilities”.

Provide steps for test modelling. For simplification and tangible results, the life cycle is decomposed into activities with inputs, main activities (steps), and outputs.

Define for each process the input details, activities and output details.

Define the process responsible.

Define the “Test Entry Criteria” and the “Test Exit Criteria” and “Sing-off responsibilities”.

Testing Framework Processes : Test Execution Process 05/06/09 Test Execution Process is defined in the Testing Framework and it is controlled by the Testing Management Tool. The process describes the entry criteria and the exit criteria of the process. The process is executed by different team members. The process describes their roles and responsibilities. Test Exit Criteria Responsible for delivery 100% of Test scripts executed with 90% success rate . Test Director All priority 1 and 2 defects resolved or agreed to be taken over the next environment. Test Director Error and defect resolution documentation from previous test phases completed. Test Director

Test Execution Process is defined in the Testing Framework and it is controlled by the Testing Management Tool.

The process describes the entry criteria and the exit criteria of the process.

The process is executed by different team members. The process describes their roles and responsibilities.

Testing Framework Metrics: The Dashboard 05/06/09 Why is Dashboard necessary?: Dashboards provide alignment, visibility of the collaborations across the organization by allowing business users to define, monitor and analyze business performance. Business Issues: • Inaccurate and incomplete information. • Too much data to sift through – or not enough. • Difficulties locating the necessary information. • Inability to analyze data for details and root causes.

Why is Dashboard necessary?:

Dashboards provide alignment, visibility of the collaborations across the organization by allowing business users to define, monitor and analyze business performance.

Testing Framework Metrics : The Dashboard 05/06/09 What can we include?: Today’s Execution Summary: Nº of total TC executed % of TC Executed Nº of TC Passed % of TC Passed Test execution Progress. Defects Summary: Number of Defects per Functional Area and Priority Defect per status and Priority. Defect Metrics History. Key issues.

What can we include?:

Today’s Execution Summary:

Nº of total TC executed

% of TC Executed

Nº of TC Passed

% of TC Passed

Test execution Progress.

Defects Summary:

Number of Defects per Functional Area and Priority

Defect per status and Priority.

Defect Metrics History.

Key issues.

Testing Framework The Tool: Commercial tool Allows you to centralize all your testing activities. Track your Test coverage. Helps management in the measure of progress and effectiveness of quality activities. Access testing assets anytime, anywhere via a browser interface. Manage manual and automated tests. Accelerate testing cycles by scheduling and running tests automatically, unattended, 24x7. (Functional automatic testing and Performance testing) Analyze the testing process with integrated graphs and reports. (Dashboard) Easy to use for all the team members (managers, testers, developers…) Integrates his own bug tracking system. 05/06/09

Allows you to centralize all your testing activities.

Track your Test coverage.

Helps management in the measure of progress and effectiveness of quality activities.

Access testing assets anytime, anywhere via a browser interface.

Manage manual and automated tests.

Accelerate testing cycles by scheduling and running tests automatically, unattended, 24x7. (Functional automatic testing and Performance testing)

Analyze the testing process with integrated graphs and reports. (Dashboard)

Easy to use for all the team members (managers, testers, developers…)

Integrates his own bug tracking system.

Testing Framework The Tool: Open Source Track requirements with your test cases. Create, manage and organize your Test cases. Organize your test cases in test suites. Identify test coverage. Prioritize your test cases. Create metrics of your test execution. Customize “TestLink” to fit your requirements and processes. Easy to integrate with Mantis or Bugzilla (a Bug tracking system ) Web based tool. (PHP + mySQL ) 05/06/09

Track requirements with your test cases.

Create, manage and organize your Test cases.

Organize your test cases in test suites.

Identify test coverage.

Prioritize your test cases.

Create metrics of your test execution.

Customize “TestLink” to fit your requirements and processes.

Easy to integrate with Mantis or Bugzilla (a Bug tracking system )

Web based tool. (PHP + mySQL )

Testing Framework The final picture 05/06/09

Questions? 05/06/09 References TMMI - http:// www.tmmifoundation.org / TestLink - http:// testlink.org/wordpress / Contact Details Personal Website: http://www.softqanetwork.com

References

TMMI - http:// www.tmmifoundation.org /

TestLink - http:// testlink.org/wordpress /

Contact Details

Personal Website: http://www.softqanetwork.com