NYC Technology Forum Introduction to Test Automation 11/2/07 All rights reserved Not to be reproduced without permission Bill Rinko-Gay Solutions Director, Software Quality Management John Stephens Practice Director, Software Quality Management

Get familiarized with basic concepts of Test Automation Understand what it takes to implement effective Test Automation Introduce techniques to emphasize maintainability Objectives

Get familiarized with basic concepts of Test Automation

Understand what it takes to implement effective Test Automation

Introduce techniques to emphasize maintainability

What is Test Automation?

Benefits of Automated Testing Examples Run automated tests unattended (overnight) Reliable, consistent testing Increased speed in test execution Increased test coverage

Run automated tests unattended (overnight)

Reliable, consistent testing

Increased speed in test execution

Increased test coverage

General Misconceptions About Automated Testing Automation is simply “record and playback” Automation is the “silver bullet” to solve all testing problems Return on Investment will be achieved immediately Automation will replace all manual test cases

Automation is simply “record and playback”

Automation is the “silver bullet” to solve all testing problems

Return on Investment will be achieved immediately

Automation will replace all manual test cases

What is Test Automation? An enhancement on manual testing Utilizes technology to provide repeatable, consistent testing of the Application Under Test (AUT) Reduces costs and time for regression testing

An enhancement on manual testing

Utilizes technology to provide repeatable, consistent testing of the Application Under Test (AUT)

Reduces costs and time for regression testing

Primary Goal of Automated Testing To reduce number of tests to be run manually, not to eliminate manual testing altogether Automated Testing is an enhancement upon existing methodology and manual process

To reduce number of tests to be run manually, not to eliminate manual testing altogether

So, You Want to Implement Test Automation…

Challenges in Automated Testing Test Automation is a development effort! Automation is time consuming to create (and maintain, if poorly implemented) Limitations of the testing tools and developer expertise Automation of the wrong tests for the given constraints Unrealistic expectations

Test Automation is a development effort!

Automation is time consuming to create (and maintain, if poorly implemented)

Limitations of the testing tools and developer expertise

Automation of the wrong tests for the given constraints

Unrealistic expectations

Test Automation Checklist Criteria for Automation Tool – AUT compatibility Test Automation Requirements Implementation Design Stable, isolated test environment Test data Implementation and coding standards Maintenance

Criteria for Automation

Tool – AUT compatibility

Test Automation Requirements

Implementation Design

Stable, isolated test environment

Test data

Implementation and coding standards

Maintenance

What to Automate? Choosing Ideal Manual Test Cases

Criteria for Automating a Test Case Why Automate? Selecting the right tests to automate is very critical to effective implementation!! Repetitiveness Test Cases that are to be performed repeatedly Test Cases that are business-critical and/or high risk Criticality Tedious Test Cases that are tedious to perform manually Time Consuming Test Cases that require considerable amount of time to perform manually

Test Automation Checklist Criteria for Automation Tool – AUT compatibility Test Automation Requirements Implementation Design Stable, isolated test environment Test data Implementation and coding standards Maintenance

Criteria for Automation

Tool – AUT compatibility

Test Automation Requirements

Implementation Design

Stable, isolated test environment

Test data

Implementation and coding standards

Maintenance

Tool - AUT Compatibility Conduct Proof of Concept of Tool on AUT Navigate through functional areas to test Verify Object Recognition of AUT Determine whether tool selection is valid

Conduct Proof of Concept of Tool on AUT

Navigate through functional areas to test

Verify Object Recognition of AUT

Determine whether tool selection is valid

Test Automation Checklist Criteria for Automation Tool – AUT compatibility Test Automation Requirements Implementation Design Stable, isolated test environment Test data Implementation and coding standards Maintenance

Criteria for Automation

Tool – AUT compatibility

Test Automation Requirements

Implementation Design

Stable, isolated test environment

Test data

Implementation and coding standards

Maintenance

Test Automation Requirements Definition The set of criteria that the automated test suite must meet to be successful - the requirements for an automation project.

Test Automation Requirements What are the Success Criteria for the project? Meeting the goals and objectives of the automation effort Tool capabilities Capabilities and skills of the test automation team Example Requirement: The test tool shall test the login functionality of a web application, on a Windows 2000 platform, and run unattended simultaneously on three PCs.

Meeting the goals and objectives of the automation effort

Tool capabilities

Capabilities and skills of the test automation team

Planning for Test Automation Test Automation Requirements help drive and determine the approach and design of the implementation. Test Automation Requirements Test Automation Design/Approach

Test Automation Checklist Criteria for Automation Tool – AUT compatibility Test Automation Requirements Implementation Design Stable, isolated test environment Test data Implementation and coding standards Maintenance

Criteria for Automation

Tool – AUT compatibility

Test Automation Requirements

Implementation Design

Stable, isolated test environment

Test data

Implementation and coding standards

Maintenance

Test Automation Methodology Analysis & Strategy Execution & Results Management Construction Design

Automation Architecture encompasses the environment for execution and development of an automated test suite. It is the overall scheme in which the test scripts will be developed. Implement a Test Automation Architecture

Automation Architecture encompasses the environment for execution and development of an automated test suite. It is the overall scheme in which the test scripts will be developed.

Designing the Test Automation Architecture Maximizing Maintainability

Automation Architecture Design Options Capture/replay Framework-based Data-driven Keyword-driven Hybrid

Capture/replay

Framework-based

Data-driven

Keyword-driven

Hybrid

Design Goals Evaluate feasible design options for implementation Anticipate likely changes to AUT Consider skill level of team members developing and/or inheriting the automated scripts Design of automated tests should address test automation requirements

Evaluate feasible design options for implementation

Anticipate likely changes to AUT

Consider skill level of team members developing and/or inheriting the automated scripts

Design of automated tests should address test automation requirements

Typical Design Components Subscripts Startup Script Data Table Driver Script Function Library Compiled module containing application-specific routines and functions A script that initiates automated test execution A script that performs a discrete portion of a business process to be tested A script that sets up and standardizes test automation environment Data sources (such as spreadsheets) used to store test data for use during test execution

Compiled module containing application-specific routines and functions

A script that initiates automated test execution

A script that sets up and standardizes test automation environment

Test Automation Checklist Criteria for Automation Tool – AUT compatibility Test Automation Requirements Implementation Design Stable, isolated test environment Test data Implementation and coding standards Maintenance

Criteria for Automation

Tool – AUT compatibility

Test Automation Requirements

Implementation Design

Stable, isolated test environment

Test data

Implementation and coding standards

Maintenance

Automation Test Environment Stable and isolated environment is important for reliable execution of automated tests GUI must be identical to Production

Stable and isolated environment is important for reliable execution of automated tests

GUI must be identical to Production

Test Automation Checklist Criteria for Automation Tool – AUT compatibility Test Automation Requirements Implementation Design Stable, isolated test environment Test data Implementation and coding standards Maintenance

Criteria for Automation

Tool – AUT compatibility

Test Automation Requirements

Implementation Design

Stable, isolated test environment

Test data

Implementation and coding standards

Maintenance

Test Data Strategy Create a test environment that is restorable to a known constant, such that the test cases can be repeated each time Identify the procedure required for backup and recovery of data Test Data is critical to reliable automated testing!!

Create a test environment that is restorable to a known constant, such that the test cases can be repeated each time

Identify the procedure required for backup and recovery of data

Creating Test Data Various Methods Create test data through application by doing data entry Import test data from production environment, if available Develop automated test scripts to generate test data as prerequisite

Create test data through application by doing data entry

Import test data from production environment, if available

Develop automated test scripts to generate test data as prerequisite

Test Automation Checklist Criteria for Automation Tool – AUT compatibility Test Automation Requirements Implementation Design Stable, isolated test environment Test data Implementation and coding standards Maintenance

Criteria for Automation

Tool – AUT compatibility

Test Automation Requirements

Implementation Design

Stable, isolated test environment

Test data

Implementation and coding standards

Maintenance

Coding Standards Specify coding practices and naming conventions May be based on standards in place for project’s AUT developers

Specify coding practices and naming conventions

May be based on standards in place for project’s AUT developers

Test Automation Checklist Criteria for Automation Tool – AUT compatibility Test Automation Requirements Implementation Design Stable, isolated test environment Test data Implementation and coding standards Maintenance

Criteria for Automation

Tool – AUT compatibility

Test Automation Requirements

Implementation Design

Stable, isolated test environment

Test data

Implementation and coding standards

Maintenance

Maintenance Start small and build on Build modular components Stay within scope of automation requirements Adhere to defined standards and conventions Emphasize robustness and flexibility

Start small and build on

Build modular components

Stay within scope of automation requirements

Adhere to defined standards and conventions

Maintenance Must be a good tester Has programming background Must be aware of maintenance and reliability Must be motivated to develop and create automated test scripts Building an Automation Test Team Ideal characteristics of an Automation Engineer:

Must be a good tester

Has programming background

Must be aware of maintenance and reliability

Must be motivated to develop and create automated test scripts

Get familiarized with basic concepts of Test Automation Understand what it takes to implement effective Test Automation Introduce techniques to emphasize maintainability Learning Objectives Revisited

Get familiarized with basic concepts of Test Automation

Understand what it takes to implement effective Test Automation

Introduce techniques to emphasize maintainability

Q&A