The document discusses key aspects of successful test automation including: 1. Applying a software development process to automation to improve reliability and maintainability. 2. Improving testing processes with robust manual testing and defect management before automating. 3. Clearly defining requirements for what to automate and goals of the automation effort.

1. Key to successful Automation

2. Murphy’s Rules for Testers Never assume anything. There is no such thing as stupid question. Record everything thoroughly. If you don’t understand, it’s not your fault.

3. What is Testing ? Testing is a process of executing the program with intent to finding the bugs Testing is the set of processes which ensure that the functionality and performance planned through design has been delivered through the system. Testing is required to improve the quality of the product before handling over the product to the customer.

4. Type of testing Manual Testing Automated testing

5. Manual Testing Traditional form of testing Slow , tedious , boring Costly , more time & effort Difficult to Manage Reliable ? Flexible (A lot have been said about the Slow and tediousness of the manual testing but it’s the flexibility that it provides that overshadows all other points. With the frequent change in the software its not viable to change the test script regularly. So when changes are brought into the software regularly, the manual testing is the best option).

6. Automation and its Need Automation is done to reduce repeated and redundant testing. Automation maintains a single standard of quality for your application over time — across releases, platforms, and networks. It is used to replace or supplement manual testing with a suite of test programs. Benefits include increased software quality, improved time and resource efforts , repeatable test procedures, and reduced testing costs.

7. Myths about Automated testing Find more bugs:- In most places the test cases are written by test engineers who are familiar with the application they are testing. From test cases to test scripts, automation does not add anything in the process to find more bugs. The test scripts will work only as good as the test cases when comes to finding bugs. Eliminate or reduce manual testers:-For the stable application we can say this, but for an application that required any change this is totally a myth. As said earlier that the test scripts are as good as the test cases itself so for the new test cases manual tester will definitely be required. Automated Testing doesn't mean Automatic Testing …. It means Computer aided testing

8. Advantages of Automated testing Ease the testing process Quicken the testing, by reducing the time and effort involved in manual testing . A typical automated test suite will run in less than 24 hours, without any human intervention required. For a sophisticated product, manual testing may require dozens of staff months to perform the same testing Maintain consistency – After a new drop is taken, the same script would be run against the same data, thereby eliminating the chances of any inconsistency. With a complex testing process manual testing often yields inconsistent coverage and results depending on the staff and schedule employed. An automated test suite ensures the same scope and process is used repeatable each time testing is performed. Accelerate release schedule by testing at any point in the development cycle IMPROVED TESTING PRODUCTIVITY. With its much shorter execution time an automated test suite can be run multiple times over the course of a product development cycle. By testing earlier and more often bugs are detected and corrected earlier and at much reduced expense. IMPROVED PRODUCT QUALITY. The sum of improved test procedures and testing productivity is a substantial improvement in product quality. Automated testing detects functional and performance issues more efficiently, allowing test staff to focus on quality in areas such as documentation, installation, hardware compatibility, etc. Reuses test across platforms Encourage more thorough testing without straining manual resources

9. Advantages of Automated testing Cost effective - Automated testing has an upfront cost to develop, but over the lifetime of a product it will offer substantial net savings. An average automated test suite development is 3-5 times the cost of a complete manual test cycle. Over multiple product releases with multiple cycles per release, this cost is quickly recouped. Decrease the product’s time to market and prevent QA from being the bottleneck in product delivery Improve coverage of regression testing Improve the quality of testing by reducing the human error Make test cycle more maintainable, consistent and thorough Improve the reusability of tests Provide the organized, detailed test log and audit trail of executed tests Simplifies the debugging by allowing the exact test case to be replicated Allows cross-referencing of automated test cases against bug logged in the bug tracking system.

10. Starting with Automation… The tester works begin once the developer is ready with the software. On an average effort require to automate 1 test case is equivalent to effort required for executing the test case 8 times manually. There are several factors to consider before anyone go for the automated testing. Cost effectiveness of the automation. Resources availability. Reusability of the automated test script. Tool selection.

11. Typical Automated testing process Create a test plan Record a test script ( Use agreed norms ) Customise Code , Use Reusable components and functions Add Check points and parameterize Use standard programming conventions Save and run Test Script Create a Test Suite

12. Automation Tools WinRunner (From Mercury Interactive www.merc-int.com ) . Quick Test Pro (From Mercury Interactive www.merc-int.com ) . Silk Test (From Segue www.segue.com ) . Visual Test (From Rational www.rational.com ) . Rational Robot (From Rational www.rational.com ) . QA Run and Test Partner ( www.compuware .com)

13. Maintainability Reliability Most important criteria for Automation

14. Current Trend …. Programmers quickly create application using GUI tools which has lead to increase in their productivity Pressure on Testers Test more and more code in less time Improve their productivity Move towards automation for GUI , CLI and API based application Test Tools – a misnomer Lack of Programming skills Record and play back scripts , very sensitive to change How to effectively Design , Develop and maintain automated suites?

15. Who should Automate Tests Good Testing and development skills Understand Testing Requirements and situations Testers face. Testers who want to be programmers ? Automate 100 % testing ? A test automator needs to know how to develop software. He needs to be particularly aware of issues such as maintenance and reliability. Making the system easy to update with changes to the product under test should be the priority. Independent contractors . Who will maintain tests after they have left ? Rejects from Development or Testing as Automators ?

16. What and When to Automate Is 100 % automation possible ? Testing is the art of pragmatic – Good software testing requires good judgment The area where one spends a lot of time testing manually Regression Testing Smoke Testing Don’t focus on areas that might otherwise go untested ( Difficult to maintain ) Rigorous Manual testing of the functionality before Very Strong Manual Testing Process and Procedures Automating everything to make job exciting ? Problems associated Test automation takes 8 times the time to test manually continued ….

17. What and When to Automate Desire to be a programmer Maintainability should be the top priority – Aim to run tests unattended Usage of Reusable components and distribution of work First run on test and then build test suite – Focus not on how much code has been written but how many test set have been automated and how reliable and maintainable they are . All areas that are run frequently should be automated . Don’t automate everything .. Look for parts that are big paybacks. Introduction of Automation in Development cycle – Timing is important.

18. Building Maintainable and Reliable Test Suites Biggest challenge – Product Interface / components changes Accurate automated Test suites No False Positives Abortions and False negatives are still o.k. Usability and comprehension Standard naming conventions and programming practices Using Error Recovery Systems - Exception handling Test Case Independence – base state restoration

19. Key to successful Automation Apply Software Development Process Improve the Testing Process Define Requirements Prove the concept Champion Product Testability Design for Sustainability Plan for Deployment Face the Challenges of Success

20. Key to successful Automation Apply Software Development Process Avoid Pattern Zero status (no distinction between user and developer ) Dedicated resources to Test Automation and treating like development activity Plan , Design Effective coding practices Configuration management , version control Bug Tracking and testing Usage of comments , functions etc Usage of Re-usable components Comprehensive documentation including Best practices guide

21. Key to successful Automation Improve the Testing Process Robust Manual Testing and Defect Management process and procedures. Objective is to streamline Test process , allowing things to move more quickly without delays. Effective Regression and documentation of the same Document Testing Approach Names , data for tests , preconditions etc Expected results Test Designs Creation of Logs Identify product improvements that would help Manual Testing ( eg simplify installation process ) More computers for Testing

22. Key to successful Automation Define Requirements What needs to be tested – Test Design Automation Requirements – Goals for automation Speed up testing to accelerate releases Allow testing to happen more frequently Reduce costs of testing by reducing manual labor Improve test coverage Ensure consistency Improve the reliability of testing Allow testing to be done by staff with less skill Define the testing process and reduce dependence on the few who know it Make testing more interesting Develop programming skills Agree on all party expectations

23. Key to successful Automation Prove the concept Proof of concept – A meaningful test suites that demonstrates suitability for Regression Testing Configuration Testing GUI / NON GUI Testing Demonstrate advantages of automation to higher management Secure support and necessary approvals Tool Evaluation and Feasibility study Need to investigate if the tool and approach will work for your product and staff . Is it even possible to automate tests for your product ? Vendor issues, support , license etc Budget and other considerations

24. Key to successful Automation Champion Product Testability CLI / API / GUI Testing GUI is toughest to automate Heavy Customization of automated scripts ( programming) Technical challenge of getting the tool to work with the product Add inns Non standard components Keeping up with frequent design change to GUI . API Automation – Unit Testing CLI Testing simpler – For e.g. .Install shield silent installation

25. Key to successful Automation Design for Sustainability Long term focus on the need for maintenance and reliability so that the product remains functional and relevant with new releases. Integrity of the tests are paramount Trust automation results , All checkpoints covered No False positive or False Alarms Tests have PASS, FAILED and NOTRUN Status Performance Attention on testing automation , automated code etc. Proper Code Reviews , remove redundant code etc Ease of Analysis Analyze failure due to product defects or improper automation False Alarm Improve Error Reporting Remove unreliable tests that are redundant or obsolete

26. Key to successful Automation Reviewability Ease of understanding the objectives and working of each test Good documentation and code coverage Maintainability Keep in mind product changes by using Centralized object repository , libraries etc Independence Must for successful execution of a test suite as one failure could cascade others. Ideally two tests should be independent unlike Manual Testing Each test should create its own test environment Though redundancy increases there is more reliability Object is to run tests unattended .

27. Key to successful Automation Repeatability Tests should execute in the same way each and every time they are run Action Libraries Reusable components Effective documentation Differentiate between errors in Code or called function Data Driven Tests Also called Table Driven or third generation automation Tests are written in simplified table format Parser is written to interpret and execute test statements. Heuristic Verification Correct Result verification

28. Key to successful Automation Plan for Deployment Document setup/installation, how to run tests and analyze failures Package tests for other people to use Helpful error messages Treat Test Suites as a product – Least dependencies on external libraries /services. Test suites readily available to concerned parties Face the challenges of success Staff should know how to diagnose failures . Minimize calling automators for every issue Defects should be reproduced manually Repair Old tests / Add new / Maintenance etc Formal Review of Test suites after each release. – Avoid Old Oak syndrome

29. Key to successful Automation Reviewability Ease of understanding the objectives and working of each test Good documentation and code coverage Maintainability Keep in mind product changes by using Centralized object repository , libraries etc Independence Must for successful execution of a test suite as one failure could cascade others. Ideally two tests should be independent unlike Manual Testing Each test should create its own test environment Though redundancy increases there is more reliability Object is to run tests unattended .