How to Maximize ROI from your Test Automation Strategy and Budget


Published on

This webinar will cover five proven strategies for a successful testing automation strategy and how to best allocate your test automation budget.

Key Takeaways:

A practical guide to developing a successful automation strategy including where to spend your budget dollars and allocate resources.
Case studies outlining strategies to choosing the following: End-to-End scenarios, Automation Frameworks, Implementation Approaches and Test Data
Definitions of metrics and how to capture them to validate that your strategies are working

Presenter: Sreekanth Singaraju, VP, QA and Testing

As Vice President, QA and Testing, Sreekanth has more than 12 years of senior technology leadership experience and leads Alliance's QA & Testing organization in developing cutting edge solutions. He is a respected technology thought leader with focus in testing Agile projects, test process automation and metrics-based development approaches and utilizing lean techniques in software development.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

How to Maximize ROI from your Test Automation Strategy and Budget

  1. 1. Maximize ROI from Automation Budget12-12-12Sreekanth SingarajuVice President of QA & Testing
  2. 2. 2© Alliance Global Services 2012Automation
  3. 3. 3© Alliance Global Services 2012Promise of Test AutomationAcceleratedTesting CycleDecrease time-to-market for new applications and products – especially forregression testingIncreasedCoverageEnable more extensive and detailed testingRepeatable andAuditableImplement formal testing processes that provide evidence of execution and areeasy to executeAccuracy inExecution andReportingNot prone to human judgment or errorsEnhancedProductivityExecute repeatedly and often; Future test cycles will be more productiveUse of correctly implemented automated frameworks aligned with organizationobjectives, deliver substantial benefits to the organization.
  4. 4. 4© Alliance Global Services 2012Reality of Most Test AutomationImplementationMore expensive than plannedTakes longerFewer manual testing resources replacedROI is not metSteady StateCosts more to maintain than plannedNever seems to run successfullyDevelopment loses buy-inOut of sync with the applicationMany automation projects unfortunately fail before they deliver projected ROI orsubstantially under-perform on the objectives.
  5. 5. 5© Alliance Global Services 2012Impediments to Achieving Best in Class AutomationObjectives Non-existent or Unrealistic automation objectivesFocused around technical goals rather than business goalsAutomationStrategyHighest automation ROI areas not identifiedSubscale initiatives or insufficient commitmentStart-UpCostsTool licenses and framework creation costsUp front Labor cost for implementing test casesExpertise &AssetsManual test teams lack development expertiseKnowledge of test case lifecycle mgmt. missingMeasuringEffectivenessLack of a metrics program to measure alignment with overall objectivesEvaluate effectiveness of testing programs and fine-tune strategy
  7. 7. 7© Alliance Global Services 2012Objectives
  8. 8. 8© Alliance Global Services 2012Automation ObjectivesTesting objectives need to be clearly defined and derived from business objectives.These then serve as a guidepost for managing Automation programs.• Execute a rapid Test Assessment for anyone planning to build orrefactor an automated regression test suite to identify businessobjectives and current state of capabilities of the test organization• Automation objectives are inherently easy to quantify, since abaseline for costs, timelines and resources exists from comparablemanual testing• Current state of capabilities and assets provides an understanding ofresources needed to implement automation• Sets accurate expectations to key stakeholders• Provides benchmarks for evaluating the progress of the automationinitiative
  9. 9. 9© Alliance Global Services 2012Automation Business Objectives – ExampleA high level business objective was provided to decrease cost of testing whileproviding same or better level of quality control.• Utilized Alliance’s Assessmentkit to perform a rapidassessment in matter ofweeks• Mapped business objectivesto specific Testing objectives• Defined automation initiativesthat need to be implementedto achieve Testing objectives• Baselined existing metrics onthe key objectives to be usedfor measuring ROI and coursecorrections
  10. 10. 10© Alliance Global Services 2012Automation Business Objectives – ExampleA high level business objective was provided to decrease cost of testing whileproviding same or better level of quality control.Business Impact Area Current StateQualityDocumented Regression Test scenarios Less than 50Regression testing for Mini Releases No Exhaustive Regression testing for MiniReleasesUAT Test Validations process Business and IT teams validate tests and isvery expensiveTest data planning Test data setup after UAT start dateextends the testing cycle timeSpeed /EfficiencyReduced # of Testing Cycles of MajorReleases4 CyclesDuration of Test execution of MajorReleases25 calendar daysInternal business team effort reduction 221.75 days of effort(60% of IT (10) and 60% Biz (4) over 25 days)External team effort reduction 13.25 days of effort(25% of 4 resources over 25 days)
  11. 11. 11© Alliance Global Services 2012Automation Strategy - What To Automate?
  12. 12. 12© Alliance Global Services 2012Automation Strategy - What To Automate?AutomationCandidatesHigh BusinessValueWorkflows basedon large numberof test datacombinationsFrequently usedfunctionalityLong usefullifespanEnd-to-Endscenarios acrosscomplex sub-systems andintegration pointsTedious and/orresourceintensive toconduct testmanuallyWhat to Automate is the single most important planning step as it defines directlythe achieved business value of the automation program
  13. 13. 13© Alliance Global Services 2012Prioritization•Obtain the candidates andassign them scores onBusiness Value, ExpectedUse of Scenario and Easeof Automation•Interface with businessand development team toobtain inputsCode•Cyclomatic complexity ofmodules anddependencies•Interactions with internalexternal systemsData & Realtime Tracing•Volume of Data generated•Importance of data•End-user usage needsWhat To Automate (cont’d.)Analyse the applications source code, database and real-time tracing to highlightcandidates that should be then prioritized by business value, longevity and ease ofautomation
  14. 14. 14© Alliance Global Services 2012Upfront Costs
  15. 15. 15© Alliance Global Services 2012High Upfront CostsToolsExplore multiple tools strategy if feasible todecrease the cost of testingResourcesExplore jumpstart programs to get valuableexpertise into your teams rapidlyFramework &Build-out Attempt the smallest but highest businessvalue modules first – outsource if necessaryInfrastructureUtilize cloud or virtual models for TestEnvironment and Automation Infrastructure
  16. 16. 16© Alliance Global Services 2012Expertise - FrameworksIn order for automated tests to be reusable, repeatable and maintainable, testdevelopment standards need to be defined and followed. These can be achieved bydefining the framework and guidelines document has to be taken as the base and followthe development architecture finalized.There are some basic features that a framework is expected to have which include:Application IndependentEasy to UseEasy to MaintainIs ScalableExecute on Demand
  17. 17. 17© Alliance Global Services 2012Automation Framework• Automation dependent on anefficient framework andgovernance• Utilize quality measurementand reporting to evaluateeffectiveness• Execute and automate allaspects of testing fromFramework to TestLab• Reusable test data and testenvironment managementprocessesRESOURCESExperience ExpertiseFlexible ScalableASSETSAcceleratorsFrameworks TemplatesTDM &STRATEGYE2E TESTENVIRONMENTMETRICSDASHBOARDAUTOMATED TESTINGFRAMEWORKSuccessful automation frameworks focus on all aspects of automation and plan forresources and reusable assets
  18. 18. 18© Alliance Global Services 2012Titanium xAFT – Enterprise Test Automation FrameworkTitanium xAFT (eXcel Automated Functional Tester) isa robust enterprise test automation framework – SingleScript can be executed against Multiple executionengines and browsersSupports HPQTP, Selenium,TWIN executionenginesFlexible &MaintainableframeworkBased on Openand proventechnologiesTitanium xAFT is a Open-Source Test Automation Framework to develop Automationrapidly across the enterprise while creating highly maintainable suites.30% productivityimprovementsover SeleniumautomationCollaborativescriptdevelopmentby BAs andTesters inExcel & OpenOfficeCross-browser Mobile - Web WebServices
  19. 19. 19© Alliance Global Services 2012Measuring Effectiveness
  20. 20. 20© Alliance Global Services 2012Measuring EffectivenessAutomation initiatives need to have metrics, reporting and analysis as an integralpart of the initiative to ensure success.• Metrics should capture both business level and execution level valuesto ensure efficiency is accurately captured and reported• Metrics should not be left for the end of the project to implement• Metrics measurements make implementation level course correctionsand expectations management easier• Business Objectives identified should be directly tied to the metricsprogram and measured throughout the initiative• Assumptions and approximations can be made at the beginning of theinitiative
  21. 21. 21© Alliance Global Services 2012Business MetricsValue AddBefore Titanium TestingServicesAfter Titanium TestingServicesQualityDocumented Regression TestcasesLess than 50 1900+ implemented in testmanagement toolEnabled Mini Support releasesand Regression testing for HotfixesNo Exhaustive Regressiontesting for Mini ReleasesEnabled 2 Mini releases and 4Hot fixes into release calendarUAT Test Validations Business and IT teams validatetestsTesting team validates testswhile Biz, IT approve resultsTest data planning Test data setup after UAT startdateTest data setup and approvalprior to UAT start dateSpeed /EfficiencyReduced # of Testing Cycles ofMajor Releases4 Cycles 2.5 CyclesDuration of Test execution ofMajor Releases25 calendar days 13 calendar daysInternal business team effortreduction221.75 days of effort(60% of IT (10) and 60% Biz (4) over 25 days)74.5 days of effort(40% of IT (10) and 40% of Biz (4) over 13 days)External team effort reduction 13.25 days of effort(25% of 4 resources over 25 days)0 days of effort(Regression Test bed also supports Releases)
  22. 22. 22© Alliance Global Services 2012Execution Metrics Dashboard• Should provide risk-based actionable informationwith drill down capabilities• Aggregates information from all aspects of testing• Metrics should include– Testing Confidence– Testing Design and Execution Velocity– Design Effectiveness– Automation Coverage– Automation Backlog– Defect Identification Efficiency– Cost of Quality
  23. 23. 23© Alliance Global Services 2012Thank YouSreekanth SingarajuTitle – VP of QA & Testing ServicesEmail - ssingaraju@allianceglobalservices.comWebsite -
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.