Designing a Successful Test Automation Strategy- Impetus White Paper


Published on

For Impetus’ White Papers archive, visit-

The paper discusses the important factors that need to be kept in mind while designing a robust and successful test automation strategy.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Designing a Successful Test Automation Strategy- Impetus White Paper

  1. 1. Designing a successful TestAutomation strategy:Connecting the dotsW H I T E P A P E RAbstractThis white paper discusses how designing an effective testautomation strategy can help address major testing challengesfaced by organizations.It questions the illusionary benefits of Test Automation andexamines ways in which they can actually be converted into realones.The white paper also discusses the important factors that needto be kept in mind while designing a robust and successful testautomation strategy. It additionally provides a brief overview ofan automation platform that can help address the testautomation problems that companies encounter.Impetus Technologies,
  2. 2. Designing a successful Test Automation strategy: Connecting the dots2Table of ContentsIntroduction...........................................................................................................3Deriving benefits from Testing Automation ..........................................................3Challenges associated with Test Automation........................................................4Designing the right automation strategy...............................................................5The Impetus solution.............................................................................................6Summary..............................................................................................................10
  3. 3. Designing a successful Test Automation strategy: Connecting the dots3IntroductionSoftware Testing is emerging as a critical component of the product life cycletoday. With the fast pace that software development has entered with theupcoming methodologies and trends, there are increased expectations fromsoftware testing. Considering the pressure on product companies to deliverhigh-quality offerings within shorter-and-shorter timelines, the importance of,and expectations from the function is only expected to increase in the future.Recent research also supports this assumption. According to another study, inthe year 2004 the number of automated tests that were conducted globally asagainst manual tests were a mere five percent. By 2006, however, this numberjumped to 20 percent and today, would be in the region of 50 percent!Clearly, companies are keen to derive benefits out of test automation. The last5-6 years have seen an increase in the availability of commercial testing tools.Customers are also requesting for Open Source tools, thanks to the dynamic andchangeable economic scenario.The need of the hour is that software testing becomes more mature, strong andshould fit into quicker delivery cycle.Deriving benefits from Testing AutomationDue to this increase in the expectation from software testing, Test Automationis gaining popularity day by day. Test automation, as we know, is a process ofwriting a computer program to perform testing that would otherwise be donemanually.Over the last few years organizations have slowly gravitated towardsautomation testing. The reasons for this are many. In the initial days of anyproduct, testing involves manual execution of test suites, that involves varioustest cases in a given test cycle. Over time, as the product matures and addsmore features, the test team also keeps on adding more test cases to validatethe functionality that is getting developed. With this addition of test cases,testing teams start demanding more hours for testing. Eventually allstakeholders look towards test automation and acknowledge the benefits thattest automation can bring to improvement the testing process. It is felt thatautomation will enable quick testing cycles and will keep a check on the testingcosts.According to astudy conducted bythe US NIST,software producerslose 21.2 billiondollars annuallybecause ofinadequate testing.
  4. 4. Designing a successful Test Automation strategy: Connecting the dots4Some of the other stated benefits of test automation include:Improved test coverage: As the repetitive tests are scripted and can be runin an unattended mode, the manual test engineers can dedicatedly focus onthe new functionality and unexplored areas of the product.Reduced test cycle time: With different combinations of browsers, thesingle automation suite can be executed, in parallel, in an unattendedmode. This helps in saving manual testers efforts that needs to be putseparately for each browser.Repetitive execution: The automation suite can be executed any number oftimes after creation.Reliable tests: Tests that perform precisely the same operations each timethey are run, eliminating human error.Reusable tests: Tests that can run on different versions of the application.Cost reduction: Regression/Sanity suite can be executed throughautomation and at the same time existing resources can be better utilized.Typically, the first area in software testing that companies usually consider fortest automation in a project is the functional testing via User Interface. It isbelieved that the functional UI test cases help in reducing the manual effortsthat are being put in by the testing team and allow the team to focus on newand critical areas of the product. It is true that automation is the best solutionin this scenario as it promises to save effort and time.Challenges associated with Test AutomationSeveral large and mid-sized organizations continue to struggle with existing testautomation activities. The first and foremost reason for this situation is theinsufficient test coverage that is achieved by the existing automated test suites.As the complexities of applications keep increasing, the percentage of testcoverage achieved using the functional UI automated test cases is not enough.Technologies like the Cloud or Big Data are also adding to the existing challengesthat prevent companies from achieving better test coverage through testautomation. Therefore, it is a false interpretation that says test coverage willimprove by automating the functional UI tests only.In the initial stages of test automation, when a cost-benefit analysis isundertaken, it is assumed that as the team will be involved in automation, RoIwill increase exponentially for the testing activities. The diminishing impacts ofSome studiesindicate thatglobally, close to50 percent of alltest automationprojects do not givethe desired RoI. Itmeans that the testautomationoutcomes eitherfail to deliver onfinancialexpectations or donot satisfystakeholderexpectations.
  5. 5. Designing a successful Test Automation strategy: Connecting the dots5automation are often not considered while doing such calculations. Typically,what is seen is that once the scripts stabilize along with the product,stakeholders even stop feeling the use of executing the automation suite on aregular basis. In fact, the criteria that were decided earlier for automating thetest cases, themselves come under scrutiny.It is also seen that the cost of maintaining existing test scripts exceeds the initialinvestments. This also poses a bigger challenge in sustaining consistent RoI. Asfunctional UI automation mainly focuses on the regression test suite, the overallefficiency of the test teams become somewhat constant as the time progresses.Looking at the above challenges, it is obvious that automating only thefunctional UI tests will not be going to yield maximum benefits fromautomation. Organizations need to look beyond functional UI test automationto maximize test coverage and achieve real advantages from automationactivities. The best way to do this is to identify the gaps in the current testingprocess, revisit the process, and work to plug the identified loopholes.It has been found that the only way to correct all automation testing wrongs isdesigning the right Automation Strategy for the product.Designing the right automation strategyWhile there are definite advantages of automation in software testing, it isnecessary to use it intelligently and diligently.As we saw in the last section, one of the biggest challenges facing TestAutomation is Automation scoping, which plays a key role in deciding the fate ofautomation activities. Currently, the vision of automation in many of the cases islimited to the User Interface only.Let us take a look at the bigger picture of automation, where other importantaspects such as API/web services, non-functional areas like security orperformance and integration based on product and business requirements areconsidered. These not only enable the testing team to bring in the dedicatedtest focus on all the major layers of the product, but also enable theidentification of critical bugs early in the test cycle.Companies need to design test solutions that can help their testing teams toautomate test cases in the identified areas. This solution should also optimizetesting activities and integrate them to achieve Business Process automation. Itshould be capable enough to run in extended environments and should provide
  6. 6. Designing a successful Test Automation strategy: Connecting the dots6integration points with multiple automation tools so that each layer gets testedthoroughly. Finally, the testing solution should also be able to measure coverageand provide various metrics that help analyze its benefits.A successful and effective Test Automation Strategy should also address someof the major challenges of software test automation including:High maintenance of test scriptsHuge learning curve required to maintain test scriptsConsistent RoIThere are inherent benefits associated with this Test Automation Strategy.This strategy widens the overall scope of test automation. It not only helpsone validate the functionality from the User Interface, but also enablesintroduction of test automation into areas like API, integration, non-functional etc that are mostly overlooked during automation, in any givenproduct.The strategy enables test automation for each and every layer which alsohelps in increasing the overall test effectiveness. Hence it helps in achievinga higher degree of test coverage and enables testing of the product morethoroughly.Another benefit is in the area of RoI. Getting consistent RoI has always beena challenge for any automation initiative. But a robust test automatonstrategy can change this. Having a right test automation strategy enhancesthe benefits of the automation activity by maximizing the test coverage andoptimizing the resources involved in the process of automation.The Impetus solutionThe Impetus R&D team is working on a test automation platform that can helpin implementing such a test strategy and address the major test automationchallenges.Impetus is using iLeap 2.0, a SMART and hybrid test automation platform, that isbased on proven best practices and approaches that includes keyword driven,data-driven and functional decomposition.
  7. 7. Designing a successful Test Automation strategy: Connecting the dots7Figure 1 iLeap 2.0 WorkflowTo automate test cases, the following steps need to be performed in iLeap 2.0:The first step is to identify the automation candidates and mark them in thetest case management tool itself to properly categorize the test cases withinthe test suites. This information is utilized by iLeap 2.0 to produce theautomation suite.The testing team must create and store the object repository in accordancewith the format supported by the test tool.The automated test scripts are written as per the steps/actions that need tobe performed on the application in tabular format. iLeap 2.0 actuallyenables engineers to do this using the user interface exposed by theexisting tool.The next step is to create Test Data. This can be stored in either Excel orXML or a database, whichever is found to be best from an applicationperspective.Next is defining the execution strategy. In existing test automationsolutions, the automation team needs to re-define the test suites in thefunctional testing tool to group and execute the test cases. This is a hugetask in itself as the test suites needs to be modified every time there are anychanges made by the manual testing team. iLeap 2.0 enables the definitionof test suites based on the settings done in the test case management tool.
  8. 8. Designing a successful Test Automation strategy: Connecting the dots8Therefore, if the test team is making any changes in the test suitedefinitions, the test automation suite will always remain up-to-date withoutany manual intervention.Now comes the process of executing the test scripts in the desiredenvironment. iLeap 2.0 enables the definition of this environment and otherpre-requisites required to execute the test cases in a separate configurationfile.Once all the settings are done, the test scripts are executed, and iLeap 2.0enables updation of results in test case management tool directly.It is essential to follow such a workflow for effective and maximum testing.iLeap 2.0 is being designed for the widely used functional test automation toolslike Selenium and QTP and can be extended to any of the other popular toolsavailable in the industry. It is designed in such a way that the testing team cantake out the maximum benefits of the functional testing tool and lowers thelearning curve towards automation.Furthermore, iLeap 2.0 supports Open Source Testing tools and frameworkssuch as TestNG, w3af, autoit, etc., over and beyond the functional testing tools.In addition to supporting automation of the functional UI test suite, iLeap 2.0 ismature enough to enable API, web services, integration, UAT and basic securitytesting as needed to perform and achieve a complete product testing.
  9. 9. Designing a successful Test Automation strategy: Connecting the dots9Figure 2 iLeap 2.0 ArchitectureThe iLeap 2.0 architecture has several components that also provide significantbenefits:The Object Repository component stores the object/control’s informationsuch as properties, locater values, etc.The Test Data component stores the set of data that is needed to performthe operations on the application under test for single/multiple iterations.There is the Test case Sequences component that defines the actual teststeps for the test cases that are planned to be executed on any applicationunder test in tabular format.iLeap 2.0 has its own set of robust custom libraries that consist of thecustom functions being developed at Impetus.There are Configuration and Environment Settings that help in initializingthe parameter values that are necessary for any test run like various pathsof framework, application, database or browsers etc.iLeap 2.0 has an Execution Strategy component that helps in creating theautomation suites based on testing needs. iLeap 2.0 creates and stores theautomation suites as defined in the TCM tool and executes the desiredsuite.
  10. 10. Designing a successful Test Automation strategy: Connecting the dots10There is the Result Reporting component which ensures that the testexecution results are automatically updated in the TCM tool after each run.It also maintains an in-memory database for the results so that they can beexported in the form of XML.SummaryAs this white paper indicates, Test Automation definitely needs to go beyondthe boundaries of the User Interface or functional testing. Testing teams needto focus on designing effective Test Automation Strategies that can help addressthe common testing challenges such as inappropriate test coverage and low RoI.Also, there are major bottlenecks in testing automation, which can be overcomethrough automation platforms such as iLeap 2.0. At the same time, iLeap 2.0defines the best practices in automation and utilizes the existing features offunctional testing tools to develop a robust automation platform. It is importantfor automation teams therefore, to familiarize themselves with the automationcapabilities, in order to derive its benefits.About ImpetusImpetus is a Software Solutions and Services Company with deeptechnical maturity that brings you thought leadership, proactiveinnovation, and a track record of success. Our Services and Solutionsportfolio includes Carrier grade large systems, Big Data, Cloud,Enterprise Mobility, and Test and Performance Engineering.Website: | Email:© 2013 Impetus Technologies, Inc.All rights reserved. Product andcompany names mentioned hereinmay be trademarks of theirrespective companies. May 2013