Telco grade testautomationsolution

688 views
607 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
688
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
20
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Telco grade testautomationsolution

  1. 1. Telco Grade – Test Automation Solution Proven & Cost Effective Approach Author: Rajiv Ranjan Feb 2011Abstract:Test Automation has been around but has not yet become part of the testing DNA for both Telco’sand Equipment Manufacturers. Tech Mahindra has been providing automation solution to both usingtools. It has also developed an in-house automation framework to provide a cost effective automationsolution. The learning’s of 200+ person years of experience has enabled Tech Mahindra to providevalue enhancing, vendor agnostic automation solutions.This paper describes the approach needed to make automation a successful and cost effectivesolution. © Tech Mahindra Limited 2010 © Tech Mahindra Limited 2011
  2. 2. Table of ContentsExecutive Summary ......................................................................................................................... 2Business Need .................................................................................................................................. 3Network Automation Approach ................................................................................................. 4 Automation Assessment............................................................................................................ 4 Arriving at the solution .............................................................................................................. 5 Design ............................................................................................................................................ 6 Planning.......................................................................................................................................... 6Benefit Measurement ...................................................................................................................... 7Conclusion ........................................................................................................................................ 8 1 © Tech Mahindra Limited 2011
  3. 3. Executive SummaryAutomation is a much clichéd expression in today’s business environments. In thenetworks area, automation has been notorious in soaking up expensive investments andproducing little in terms of the ballyhoo of the benefits promised. Telco’s andcommunication equipment providers (CEP) therefore tread with extreme caution when the‘A’ word is flaunted as the solution to all its problems. In the end automation in mostorganizations ends up being nothing more than an ‘also-ran’ and in many cases a pariahwhich is unlikely to be invoked in the testing scheme of things. Less than 40% of Telco’s and CEP’s use some kindTech Mahindra (TechM) understands this hesitation faced by the Telecom industry in of automation currentlyadopting automation as a viable approach in solving many of the latter’s Achilles heel. Less than 15% of themThis white paper intends to carefully describe the approach needed to make automation have actually claimed that automation hassucceed at Telco’s and CEP’s. Tech Mahindra’s assertion to this approach is drawn from provided tangibleits expertise in providing bespoke and turnkey automation solutions to its Telecom benefitscustomers. Tech Mahindra has more than 200 person years of Test AutomationExperience which spans across various Testing tools, Network devices/equipments &languages used for automating network test scenarios.The stages of TechM’s automation approach covered as part of this whitepaper includeassessment, solution formulation, and automation implementation strategy. The first stagewill probe into the need for automation assessment and will delve into TechM’s automationmethodology. The automation solution development stage will cover both the toolevaluation & selection strategy and the automation design aspects. The planning stage willexamine the key aspects necessary to operationalize the automation plan. The narrative,at each stage, is strengthened with examples from a successful engagement and will providelegitimacy to the approach recommended through this white paper.Tech Mahindra’s vendor and technology agnostic position allows it to provide valueenhancing automation solutions to its customers. Tech Mahindra, as part of theassessment strategy, also evaluates, identifies & provides recommendations of the tools andtechnologies that will bring in the desired benefits. The approach presented in thiswhitepaper together with the right experience will enable unlocking some of the utopianbenefits promised by automation. 2 © Tech Mahindra Limited 2011
  4. 4. Business NeedTelecom industry is highly dynamic both in terms of technology & competition. Changingbusiness needs & market compulsions make Telco’s and CEP’s to perform more and moretesting in order to ensure that the desired levels of quality and performance are met.Time-to-market pressures often could mean deploying new features or devices into livewithout adequate testing which invariable leads to increased support costs. These costsassociated with fixing problems from field are exponentially high when compared withfixing them during pre-deployment stages. Yet, many a times Telco’s and CEP’s are forcedto cut corners during the test cycles in order to avoid missing out on bringing a product orservice to market.In the absence of Test Automation the pre deployment testing is done manually. Thismethodology has its own pros & cons. The human element involved in it brings its ownintelligence but also brings along its human limitation in terms of getting tired & bored 1 in every 4 dollars of R&D/Design spend goeswith repetitive work. This leads to a time consuming human incentive process which towards Testingcomes out with a not-so-fully tested system which again is not-so-dependable. Over 45% of defects inNetwork test automation can help Telco’s and CEP’s to fasten the test cycle times live is attributed tothereby enabling them to meet market opportunities while maintaining quality. seepage from testingRegression tests are a ‘low-hanging fruit’ when it comes to selecting automationcandidates owing primarily to their repetitive nature and wider test coverage. Network Automation could potentially help plug thistest automation can be used primarily for defect seepage upto 90%  Regression / repetitive tests for a specific test setup  Sanity Tests prior to upgrades or deployments  Stress tests to measure performance of the desired service, feature or product  Interoperability tests to validate desired features or product behavior 3 © Tech Mahindra Limited 2011
  5. 5. Network Automation ApproachWhile a lot of emphasis is seen around the execution aspects of automation, not much iswritten or mentioned regarding the stages prior to execution, namely the assessment,design & planning stages. Ironically, it is these pre-execution stages that govern theeffectiveness of the automation outcome. While there is no cookie-cutter approach toassess and build automation solutions, a robust guiding framework will go a long way inhelping automation planners to achieve the desired benefits. The automation frameworkdescribed in the sections below follows a 3-step approach which can be applied by the Case AnalysisQA/Test teams to formulate their automation strategy. Customer Profile: Tier 1 US - Existing Test - Identify - Define roles and Telco Solution Planning Assessment Plan & Test Bed Automation Tool responsibilities Analysis - Identify clearly - Setup test Business Need: E2E - Arrive at Points the inclusions and environment automation of its Next of Automation Exclusions - Generate Generation Mobile Access - Assessment - Design the Automation Test Report solution using the Plan Gateway platform tool Test Setup: Multiple network elements and platforms a. Network layer – RAN, SGSN, GGSN, Cisco load Fig 1. Automation Assessment Stages balancer, load sharing NGG & MAG b. Content – Web,Automation Assessment application servers, IMAbsence of an objective assessment before automation implementation leads to several gateways, MMSC, demo setup of third partypitfalls. Limited understanding of the automation tools and the requirements leads to a half services like MediaNet c. Provisioning - MIPS+ R2,baked approach resulting in a costly solution which fixes only a part of the problem. Fact Radius Server, MINDremains that automation is not the answer to all the functionalities which a Tester LDAP subscriber database d. User devices & trafficperforms. A detailed assessment is a must to evaluate the present state objectively and in generators – 2.5G/3Gtotality to arrive at the automation recommendations. Handsets, Laptop with 3G data cardThe assessment begins with the detailed analysis of the existing test plan & test casedocuments to determine whether or not they can be automated (fully/partially automated).Assessment involves understanding and defining the various elements of an existing systemlike Base line Configuration, Interoperability, Failover, Capacity Validation, Security, ApplicationTest, System Upgrades, Regression Tests, New and Enhanced features, protocols etc.In this stage, specific points of assessment (POA) are created, which are specific devices Case Analysisand interfaces that currently require manual intervention and therefore need to be TechM identified 5 POA’s for theassessed for automation. These points of assessment cover the three areas of test client.operations: pre-execution test configuration, test case execution and test validation. If 4 © Tech Mahindra Limited 2011
  6. 6. these points of assessment can be automated then these points are termed as points ofautomation.In addition the entire Test Bed setup is taken into consideration and evaluated. Test Case AnalysisConfigurations are identified & analyzed for automation. Finally a summary report with a Tools Evaluatedbreakup of fully automatable, partial and manual test case counts is prepared which then is Handset Automation Toolsgrouped logically into test categories that can be implemented as discrete test suites. 1. AToM 2. TestQuestFinally, automation test case candidates are prioritized based on customer business 3. Custom Application based Handset Simulatorrequirements. 1. Silk Test 2. Selenium (open source)Arriving at the solution 3. Watir (open source) 4. Custom simulators (three)The next task, once the assessment of the current environment is complete, is to identify Automation Platforms 1. Fanfare iTestthe right automation tool that would fit the requirement. Three options are typically 2. TestShellavailable: 3. Quick Test Pro (QTP) 1. COTS (Commercial Off The Shelf) Tools 2. Open Source Tools Case Analysis 3. Bespoke or Custom Built Tools Proposed Solution: 2-tiered automation solutionAll the options have their inherent advantages and disadvantages as shown in the table was identified where individual automation tools,below: driving the POA and POS will be centrally controlled by an Advantages Disadvantages automation platform. - Offers a l l i n one go: Sa ves cos t & ti me - Li cens e & s upport cos ts a re expens i ve COTS Tool s - Better tra cea bi l i ty a nd ma na gea bi l i ty - Di ffi cul t to s ca l e - Ava i l a bi l i ty of dri vers & pl ugi ns for l a rger COTS - Integra ti on cha l l enges wi th other products Open - Genera l l y upto da te wi th dependa bl e qua l i ty - Unrel i a bl e s upport Source - Cos t effecti ve - Tool roa dma p not wel l defi ned Tool s - Communi ty contri buti on - Poor tra cea bi l i ty a nd ma na gement functi ons - Offers s ca l a bi l i ty & fl exi bi l i ty - Ti me cons umi ng tool devel opment cycl e Cus tom - Al l ows better cos t control - Robus tnes s a nd s ta bi l i ty i s s ues Bui l t Tool s - Ta i l or ma de to fi t exa ct requi rements - Support i s s ue once core devel opers l ea ve Fig 2. Automation Tool AnalysisRecommendation: Perform a PoC using the identified tool on a subset of automatablefeatures in order to validate the automation feasibility and the tool selection.Once the right automation tool is identified, the solutioning aspect comes into play.Besides the network or device under test, the automation environment variables need tobe chosen. These include elements such as simulators, protocol analyzers, platforms forhosting automation suites & management applications etc. Inclusions & exclusion of theautomation are clearly stated and agreed upon. The Library procedures to be automated 5 © Tech Mahindra Limited 2011
  7. 7. are identified & categorized as fully/partially automated. Response Maps (templates forresponses from the Network Elements & Test Equipment), Test Suites & Test cases areidentified & finalized during this stage.DesignDesign considerations vary based upon the test tool selected for automation. It entails thefollowing:  Identification of all common procedures by examining existing test cases for generic execution and validation steps  Building a library of common procedures  Maximizing design of analysis rules in the tool, instead of having this logic reside on other automation elements  Ensuring that the test cases are test bed and user independent  Building modular test cases using library procedures, test beds, session profiles, global and local variables  Design hierarchy bases test suites to allow flexibilityThe Key design features of the automation solution should be  Reusability – generic & reusable library procedures reduce development time  Data Driven – input data stored in external data files eliminates need for hard coding data in script  Variation Testing – variation testing for any number of pre-configured handset user agents  Test Bed Portability – test bed data saved in separate configuration files for easy script portability  Flexible Validation – user configured criterion for protocol & content validationIn addition, the design phase may involve exploring ways to integrate the automationmanagement system with the existing tools available in the organization for TestManagement, Defect Tracking & Reporting Tools etc.PlanningBusinesses that dont build a careful plan for automated testing may end up wastingexpensive resources: time, effort, and money. The automation planning phase defines 6 © Tech Mahindra Limited 2011
  8. 8. roles and responsibilities, project test schedule, test planning and design activities. The key Case Analysisactivities that should be part of this phase are: Planning  Test environment setup 24 week plan covering - Dev & test  Preliminary test schedule - Requirement analysis  Define roles and responsibilities - Design of automation test suits  Define performance measures - Coding and unit tests  Identify risks and develop contingencies - Perform integration and acceptance tests  Develop the automation acceptance criteriaAs part of the test environment setup the test team schedules and track environmentactivities. The planning phase should also involve installation and integration of testenvironment hardware, software, and network resources. All these inputs are captured aspart of the automation test plan that will be used by the Test teams during the executionphase. The test manager can use the plan to ensure that tests are executed according toschedule, and test personnel are allocated and redirected when necessary to handleproblems that arise during the test execution process.Recommendation: TechM suggests a phased approach to Test Automationimplementation to help progressively derive the benefits of Automation and also allowingyou to measure the benefits in a phased manner.Benefit MeasurementThe basic economic premise of network test automation is to automate if cost of test Case Analysisautomation and cost of automation execution is below cost of manual test execution. The Benefitsbusiness case for automation will require that the benefits are measurable. A few - Execution time per test case reduced from 96 minquantitative methods to measure automation benefits include : to 1.5 min  Savings derived from cost of automating versus cost of manually executing test - Test Cycle time Reduction from 4 weeks to 8 hrs cases in a given period of time - Enabled multiple device testing as compared to  Savings derived from reduction in cycle time max of 5 device testing earlier  Cost savings from avoidance of defects found in field 7 © Tech Mahindra Limited 2011
  9. 9. ConclusionNetwork test automation has the potential to significantly reduce OPEX of CEP’s andTelco’s. Currently the network automation uptake in the telecom industry is patchy whichis attributed to inadequate assessment and planning. Detailed automation assessment incombination with proper design and planning are a must to enable an efficient automationstrategy and to unlock the true potential of network test automation. The white paperprovides test planners with the framework required to make the right automation decision.The white paper identified critical success factors needed for making network testautomation successful -  Shoring-up the right skills required to perform assessment of current Test Methodology  Understanding the customers existing test management framework How to reach us:  Analysis & validation of test scripts & procedures- which all procedures can be Americas automated (if yes to what extent). Tech Mahindra (Americas), Inc. sales.am@techmahindra.com  Identification of right Test Automation tool, which can interact with all Tel: +1 972 991-2900 network Elements. Europe Tech Mahindra Ltd.  Designing flexibility in Test scripts, to test in the ever changing & varied sales.eu@techmahindra.com Tel : +44 01908 553400 business requirements. Fax : +44 01908 553499 Middle-east & Africa  Understanding various interface aspects (of network elements, network test Tech Mahindra Ltd. tools, traffic simulators, existing defect and test management tools etc.). sales.mea@techmahindra.com Tel : +002 02 5806608 Fax : +002 02 5806601This framework was validated by depicting instances where it was successfully applied andthe resulting benefits highlighted. Asia-Pacific Tech Mahindra Ltd. sales.apac@techmahindra.comIn the near future as testing in networks gets more complex, there will be an increasing Tel: +91 80 40243000 Fax: +91 80 25527027demand to improve test efficiencies and effectiveness in test cycles through automation.The key to success here will be to adopt winning strategies realized through betterautomation planning that will enable users to keep testing costs and quality under control. 8 © Tech Mahindra Limited 2011

×