Software Testing

Slide 1: NYC Technology Forum Introduction to Test Automation 11/2/07 Bill Rinko-Gay Solutions Director, Software Quality Management John Stephens Practice Director, Software Quality Management All rights reserved Not to be reproduced without permission

Slide 2: Objectives  Get familiarized with basic concepts of Test Automation  Understand what it takes to implement effective Test Automation  Introduce techniques to emphasize maintainability 2

Slide 3: What is Test Automation?

Slide 4: Benefits of Automated Testing Examples  Run automated tests unattended (overnight)  Reliable, consistent testing  Increased speed in test execution  Increased test coverage 4

Slide 5: General Misconceptions About Automated Testing  Automation is simply “record and playback”  Automation is the “silver bullet” to solve all testing problems  Return on Investment will be achieved immediately  Automation will replace all manual test cases 5

Slide 6: What is Test Automation?  An enhancement on manual testing  Utilizes technology to provide repeatable, consistent testing of the Application Under Test (AUT)  Reduces costs and time for regression testing 6

Slide 7: Primary Goal of Automated Testing  To reduce number of tests to be run manually, not to eliminate manual testing altogether Automated Testing is an enhancement upon existing methodology and manual process 7

Slide 8: So, You Want to Implement Test Automation…

Slide 9: Challenges in Automated Testing  Test Automation is a development effort!  Automation is time consuming to create (and maintain, if poorly implemented)  Limitations of the testing tools and developer expertise  Automation of the wrong tests for the given constraints  Unrealistic expectations 9

Slide 10: Test Automation Checklist  Criteria for Automation  Tool – AUT compatibility  Test Automation Requirements  Implementation Design  Stable, isolated test environment  Test data  Implementation and coding standards  Maintenance 10

Slide 11: What to Automate? Choosing Ideal Manual Test Cases

Slide 12: Criteria for Automating a Test Case Why Automate? Criticality Test Cases that are business-critical and/or high risk Repetitiveness Test Cases that are to be performed repeatedly Test Cases that are tedious to perform manually Tedious Test Cases that require considerable amount of time to perform manually Time Consuming Selecting the right tests to automate is very critical to 12 effective implementation!!

Slide 13: Test Automation Checklist  Criteria for Automation  Tool – AUT compatibility  Test Automation Requirements  Implementation Design  Stable, isolated test environment  Test data  Implementation and coding standards  Maintenance 13

Slide 14: Tool - AUT Compatibility  Conduct Proof of Concept of Tool on AUT  Navigate through functional areas to test  Verify Object Recognition of AUT  Determine whether tool selection is valid 14

Slide 15: Test Automation Checklist  Criteria for Automation  Tool – AUT compatibility  Test Automation Requirements  Implementation Design  Stable, isolated test environment  Test data  Implementation and coding standards  Maintenance 15

Slide 16: Test Automation Requirements Definition The set of criteria that the automated test suite must meet to be successful - the requirements for an automation project. 16

Slide 17: Test Automation Requirements What are the Success Criteria for the project?  Meeting the goals and objectives of the automation effort  Tool capabilities  Capabilities and skills of the test automation team Example Requirement: The test tool shall test the login functionality of a web application, on a Windows 2000 platform, and run unattended simultaneously on three PCs. 17

Slide 18: Planning for Test Automation Test Automation Test Automation Requirements Design/Approach Test Automation Requirements help drive and determine the approach and design of the implementation. 18

Slide 19: Test Automation Checklist  Criteria for Automation  Tool – AUT compatibility  Test Automation Requirements  Implementation Design  Stable, isolated test environment  Test data  Implementation and coding standards  Maintenance 19

Slide 20: Test Automation Methodology Analysis & Strategy Design Construction Execution & Results Management 20

Slide 21: Implement a Test Automation Architecture  Automation Architecture encompasses the environment for execution and development of an automated test suite. It is the overall scheme in which the test scripts will be developed. 21

Slide 22: Designing the Test Automation Architecture Maximizing Maintainability

Slide 23: Automation Architecture Design Options  Capture/replay  Framework-based  Data-driven  Keyword-driven  Hybrid 23

Slide 24: Design Goals  Evaluate feasible design options for implementation  Anticipate likely changes to AUT  Consider skill level of team members developing and/or inheriting the automated scripts  Design of automated tests should address test automation requirements 24

Slide 25: Typical Design Components Compiled module containing application-specific routines Function Library and functions A script that initiates automated test execution Driver Script A script that performs a discrete portion of a business process to Subscripts be tested A script that sets up and standardizes test automation Startup Script environment Data sources (such as spreadsheets) used to store test data for use Data Table during test execution 25

Slide 26: Test Automation Checklist  Criteria for Automation  Tool – AUT compatibility  Test Automation Requirements  Implementation Design  Stable, isolated test environment  Test data  Implementation and coding standards  Maintenance 26

Slide 27: Automation Test Environment  Stable and isolated environment is important for reliable execution of automated tests  GUI must be identical to Production 27

Slide 28: Test Automation Checklist  Criteria for Automation  Tool – AUT compatibility  Test Automation Requirements  Implementation Design  Stable, isolated test environment  Test data  Implementation and coding standards  Maintenance 28

Slide 29: Test Data Strategy  Create a test environment that is restorable to a known constant, such that the test cases can be repeated each time  Identify the procedure required for backup and recovery of data Test Data is critical to reliable automated testing!! 29

Slide 30: Creating Test Data Various Methods  Create test data through application by doing data entry  Import test data from production environment, if available  Develop automated test scripts to generate test data as prerequisite 30

Slide 31: Test Automation Checklist  Criteria for Automation  Tool – AUT compatibility  Test Automation Requirements  Implementation Design  Stable, isolated test environment  Test data  Implementation and coding standards  Maintenance 31

Slide 32: Coding Standards  Specify coding practices and naming conventions  May be based on standards in place for project’s AUT developers 32

Slide 33: Test Automation Checklist  Criteria for Automation  Tool – AUT compatibility  Test Automation Requirements  Implementation Design  Stable, isolated test environment  Test data  Implementation and coding standards  Maintenance 33

Slide 34: Maintenance Emphasize robustness and flexibility  Start small and build on  Build modular components  Stay within scope of automation requirements  Adhere to defined standards and conventions 34

Slide 35: Maintenance Building an Automation Test Team Ideal characteristics of an Automation Engineer:  Must be a good tester  Has programming background  Must be aware of maintenance and reliability  Must be motivated to develop and create automated test scripts 35

Slide 36: Learning Objectives Revisited  Get familiarized with basic concepts of Test Automation  Understand what it takes to implement effective Test Automation  Introduce techniques to emphasize maintainability 36

Slide 37: 37