Test automation is software that automates any aspect of testing of an application system
Test automation includes capabilities to generate test inputs and expected results, it has no manual intervention and it can evaluate pass/no pass
Automated testing depends on testing goals, budget, software process, kind of application under development, and particulars of the development and target environment
As project nears completion, two undesirable options are faced:
Rerun fewer baseline test cases and focus on the tests necessary to validate the final increments
Add more people to enter and judge the increasing number of baseline test cases
Both options involve risk: The first option has risk concerning error detection, the tester is hoping that software fixes have not introduced errors in other test cases. The second option has risk concerning adding more staff on a budget that cannot change and adding test cases that increases the testing schedule.
Automating a test is typically more expensive than just running the application, if no need to repeat the tests is evident, then automation is probably not justified
When testing applications with significant user interaction, the tester can use this knowledge to try features and combinations that might be hard to identify when designing a GUI script
The cost of maintenance may be quite high if the requirements and implementation of the IUT change frequently. If the test automation is not maintained, it will become unusable and the expected return will not be realized
Test Cases – How should test message be implemented?
Test Control – How should the servers of the IUT and its environment be controlled and observed?
Test drivers – How should test code be organized so that it is modular, has a meaningful correspondence to the IUT, and achieves reuse where possible? How can drivers support controllability and observability?
Test Framework – How should test cases, test suites, drivers, and a tester interface be organized for an entire application system?
Deciding when automated testing is appropriate involves reviewing past metrics on various programs, thus, a CMMI level of at least 3 is ideal
Automated testing can be used at any level of the CMM, however, recording and following process standards is evident at level 3 and higher, which makes automated testing easier to implement
At CMM levels 4 and 5, automated testing can be firmly established in the company’s processes and standards and the organization can predict trends in product quality and how automated testing will achieve this or if it is cost effective
Automated testing may involve training staff members and CMM levels of 3 and higher have training processes in place and have been used many times on other projects
Vendors tools are impressive during demonstrations of basic things, however, when the tester needs to test software in depth, more functionality needs to be added to generated test scripts which can be complex and time consuming
Automated test tools can be expensive and will only be a good investment if the automated test scripts are reused frequently
Considerations (Theresa Lanowitz)
When considering to adopt automated tools, ensure there are skilled professionals in place and a good process
Automated tools are only useful when proper training is given to bring staffing up to speed on the tools
Documented, repeatable development and test process should be already established (i.e. CMMI level 3 and higher is valuable)
Test automation can significantly reduce risk only if processes and standards are followed version after version