Why On-Demand Provisioning Enables Tighter Alignment of Test and Production Environments


Published on

To improve their test environments and application quality, organizations are turning to the cloud for on-demand provisioning, as well as build and deployment automation.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Why On-Demand Provisioning Enables Tighter Alignment of Test and Production Environments

  1. 1. • Cognizant ReportsWhy On-Demand Provisioning EnablesTighter Alignment of Test and ProductionEnvironments Executive Summary environment downtime and inaccurate con- Steve Smith looks like a worried man, moving rest- figuration of test environments impact testing lessly in his chair and making frantic phone calls effectiveness. These result in application roll-out late on a Friday evening, inquiring about what delays and test managers signing off on projects went wrong with the software his team had devel- with caveats, leading to applications misbehaving oped. All seemed well during testing, but errors once put into production. surfaced as soon as the application was deployed, leaving Steve and his team in a quandary. Organizations are currently exploring various ways to mitigate these risks. Attempts to use Tales such as Steve’s abound in the software test- virtualization to overcome test environment ing arena. From banking applications to mobile challenges have met with limited success. phones and airport systems, software defects Also on the rise is “on-demand provisioning” affect organizations’ business-critical and cus- via the cloud, as well as build and deployment tomer-facing applications, impacting their daily automation. Our experience shows that business operations and resulting in huge losses. cloud-based test environments offer bet- According to the National Institute of Standards ter results by providing testing teams with and Technology (NIST), software flaws cost the on-demand capacity. Cloud-based offerings U.S. economy a whopping $60 billion annually. also enable organizations to generate standard builds that are more closely and cost-effectively One of the primary reasons for such unpleasant aligned with real production environments. situations — in which errors are discovered in production or only after end-users are affected On-demand provisioning has much to offer — is low-quality testing, often impacted by organizations seeking to instill greater discipline poorly built test environments. Test environ- in managing builds and elevating code quality. ments rarely match production environments Application quality depends on test environ- due to the huge cost and effort involved in mim- ments that support functional testing require- icking them. Further, issues such as unplanned ments; moreover, it is also important to have cognizant reports | december 2011
  2. 2. a production-like environment to complete the applications, it may not be possible to predict alltesting process without an exception. This syn- the likely scenarios in which the application willchronization of test and production environ- be deployed.ments is achieved by building a service frame-work around the test environment, supported by The conventional approach of manually creatingcloud-based computing grids. This approach, in in-house testing environments that fully mirrorour view, has already delivered great benefits to these complexities and multiplicities consumesorganizations that are piloting and enacting full- huge capital and resources. This has causedfledged on-demand test environment strategies. application developers to take the risky route of conducting testing in their production envi- ronments. However, the Sarbanes–Oxley Act, asTest vs. Production Environments well as other regulations, restricts access toAccording to a 2008 Aberdeen study, only 42% production systems, which has forced organi-of over 200 organizations surveyed were happy zations across industries to conduct testingwith the performance of their business-critical in separate environments. Further, it is notapplications.1 Primary reasons for this include uncommon for organizations that cannot affordthe inability to identify issues before end-users large-scale test environments to use sizing/are impacted and the challenges of testing appli- extrapolation techniques during performancecation performance in pre-production stages testing to understand how an application behaves(see Figure 1). Sadly, our experience with com- in the real world. This approach, however, doespanies across industries indicates that not much not guarantee success.has changed. At a time when companies arelaser-focused on operational cost-cutting and Poorly built test environments often limit thetime-to-market acceleration, software defects ability of testing teams to test applicationsthat emerge in production environments can for various scenarios. Test environments differseverely undercut financial performance and from production environments in terms of OS,cause reputational damage. patch levels, software version, configuration, etc. The wider the gap between test and produc-With applications becoming increasingly complex, tion environments, the greater the chance of anthe scope of testing has expanded tremendously, application misbehaving once deployed or for acreating unforeseen challenges for testing teams. defect to leak into live systems. The following areFor instance, to assess their likely performance among the most common issues that afflict testin real-time, Web and mobile applications must environments:be tested for multiple operating systems andupdates, multiple browser platforms and versions, Creating test environments can bedifferent types of hardware and a large number time-consuming and delay-prone. It canof concurrent users. Further, in the case of new sometimes take weeks or month to set upTop Challenges for Application Performance Management Inability to identify issues before 57% end-users are impacted Increase in complexity of applications 51% Inability to measure SLAs around 37% application performance Inability to test application 34%performance in pre-production stages Increase in complexity of 32% network trafficSource: Aberdeen GroupFigure 1 cognizant reports 2
  3. 3. a simple test environment due to delays in opportunity to maintain an up-to-date testfinding servers, procuring licenses for pro- environment.grams and testing tools and installing them. Loosely managed asset control necessi-Roughly 30% of defects arise due to tates undue investments in the infrastruc-inaccurate test environment configurations, ture. Inadequate access control reducesstudies show. Further, configuration changes confidence in the state of the environmentmade in response to errors often go untracked, and its configuration control. Together, thiscausing errors when applications are moved adds to the cost of the project and to the risk-into production. based sign-off of the test plan.Scheduling of the test environment iscrucial, especially when environmentsare shared by two projects. If not properly The Impactmanaged, this can lead to unnecessary delays. Test environment issues can impact deliveryTesting teams often clone production schedules and result in the risk of subsequentdatabases and write extract scripts to software failures, as well as additional businesscreate test data. This approach is resource expenditures. It is estimated that 40% of theconsumptive and error prone due to a lack of effort involved in a software development lifecy-audit trails. cle goes toward managing defects that arise dueSimulating the production issue in the to environment-related issues.2 The ever-presenttest environment to identify the root cause pressure to meet service level agreements (SLAs)is challenging due to a lack of test environ- can force IT managers to resort to risk-basedment alignment with production versions at all sign-off on a product/application release. Thissystem component layers. means that the product/application is tested forTest environment downtime and changes only those risks that have a high probability ofare often not controlled and communicated occurrence.properly. This poses challenges in managingtesting projects and achieving KPIs. This risked-based sign-off can have theLack of a single view makes the environ- unintended consequence of putting thement less flexible to support changes organization’s reputation and financial condi-due to code modifications during the tion at risk (i.e., errors that cause customers totest cycle. Further, it is difficult to tune a question product quality — or, worse, causecomplex test environment to suit the advance- monetary harm — can be grounds for lawsuits).ments in software development methods such Further, with the cost of fixing defects in pro-as Agile. duction being significantly greater than in theThe test environment refresh is not early stages (see sidebar below), testing teamsconsidered while planning for the test must ensure that defects do not pass through toproject execution phase. This narrows the later stages.Cost of Fixing DefectsAccording to a study by dynaTrace on application performance management, about 50% ofsoftware problems are discovered in production only after they have affected end users andthat resolving these issues can involve huge costs.3Business losses from system downtime can exceed $1 million per hour. The majority of defectscreep into code during the early stages of the development lifecycle. The longer a defect goesundetected, the greater the costs of fixing it. According to studies, the ratio of the cost to fix adefect at different stages of a project is 1:10:100. This means the cost of fixing a coding defect inproduction is 10 times the cost of fixing it during testing and 100 times the corresponding cost offixing it during the design or requirements phase (see Figure 2, next page). According to Gartner,“The cost of fixing defects ranges from a low of approximately $70 (cost to fix a defect at therequirements phase) to a high of $14,000 (cost to fix a defect in production).”4 cognizant reports 3
  4. 4. Index of the Relative Cost of Fixing Defects 110 90 70 50 30 10 0 Requirements Design Code Test ProductionSource: “The Economics of Testing,” www.riceconsulting.comNote: Analysis based on base cost of fixing a defect in the requirements and design stages, measured as one unit (hour, dollar, etc.).Figure 2On-Demand Provisioning Cloud-based testing leverages on-demandThe criticality of business applications for today’s computing grids that can be more easily andorganizations goes without saying. As such, flexibly deployed, eliminating the need for thedelivering quality applications requires a proper upfront capital expenditures (Cap-Ex) requiredtest strategy complemented by an efficient test to procure hardware and software. The pay-per-environment. Testing helps minimize errors but use model provides several benefits, includingdoes not guarantee their removal. Organizations providing testing organizations with freedomstrive to deliver quality applications by reducing from owning assets; enhanced collaborationthe majority of defects through rigorous testing among testers, developers and managers;while preparing themselves to fix minor bugs greater levels of efficiency; and, importantly,during production. In the case of a new version reduced time-to-market for key businessof an existing application, organizations must applications. These benefits are delivered byensure that the new version works without dis- leveraging access to cloud-based resources overturbing the current systems environment. This the Internet without compromising securitycreates the need for a production-like test envi- requirements.ronment, which might not offer the same process-ing capacity but delivers the same functionality The pursuit to reduce time-to-market and theand enables all required components to interact expense of application releases is often ham-as they would in a production environment. pered by huge costs and long delays in setting up an appropriate test environment. Production-likeOrganizations have tried numerous options to environments at reduced costs can be achievedimprove the test process. Recently, system virtu- through on-demand provisioning, combined withalization has provided some respite, but it is not efficient processes and governance that createsa panacea. It has met base-level operational and the discipline to develop quality build images andfinancial objectives by eliminating the need for version control.large capital investments in hardware, softwareand related infrastructure; however, given the To a large extent, the testing model can alsorequisite set-up costs, many pioneering compa- eliminate configuration errors. This will helpnies have yet to achieve the operational flexibility organizations achieve their test plan objectivesand scalability required to deliver the forecasted and minimize defect leakages when applicationsproductivity and ROI on assets. enter production due to incorrect test environ- ment setup/configurations. Organizations wouldWe believe that on-demand provisioning of test do well to combine on-demand provisioningenvironments via the cloud offers an effective with industry best practices like shift-left (seesolution for achieving better ROI on assets. Cloud sidebar, next page) to improve the overall qualitycomputing minimizes the volume of issues that of software and reduce the cost of applicationplague current approaches to test environments. development, maintenance and testing. cognizant reports 4
  5. 5. Shift-Left Practice Finding coding flaws early in the development lifecycle is one way to reduce the number of defects that surface in production. Shift-left is a practice where efforts are made to identify defects early and fix them before submitting the code to a testing team. It also includes sharing the project’s progress with clients at regular intervals, which ensures that the project is heading in the right direction. This relieves testing teams from incurring additional costs and effort required to fix defects at later stages, when teams are hard-pressed for time. The process can be time-consuming, and care must be taken to ensure it does not cause unnecessary delays in product development. Automating tests and running automated tests with every build can help in identifying bugs quickly.Footnotes1 “Application Performance Management: The Lifecycle Approach Brings IT and Business Together,” Aberdeen Group, June 30, 2008.2 Julian Brook, “Testing Environment Management Challenges With Virtualisation,” ComputerWorldUK, Sept. 6, 2008.3 Alois Reitbauer, “Is There a Business Case for Application Performance?” Dynatrace, May 26, 2010.4 “Hype Cycle for Application Development, 2011,” Gartner, July 27, 2011.BibliographyMartin Perlin, "Notes from the Trenches: Obstacles and Challenges to IT Environment Stability,"www.evolven.com, October 2011.Wayne Ariola, "The Next Generation of Test Environment Management," Virtualization Journal,July 12, 2011."Maximizing the Value of Good Testing Practice in an Agile Environment: Delivering On Time, In Scope,On Budget and At the Tight Level of Quality," Sogeti, September 2010.“Is Shift-Left Agile? And Death by Build,” AgileFall, March 22, 2010.Vinaya Kumar Mylavarapu, “Taking Testing to the Cloud,” Cognizant, March 2011."Official Fix for the Zune 30 Fail," www.gizmodo.com, December 31, 2008.Bojan Simic, "Application Performance Management: The Lifecycle Approach Brings IT and BusinessTogether," Aberdeen Group, June 2008."Improving Software Quality to Drive Business Agility," IDC, June 2008.Matt Adamson, “Root Out Elusive Production Bugs with These Effective Techniques,” July 2005."The Economics of Testing," www.riceconsulting.com, 2001. cognizant reports 5
  6. 6. Author and AnalystVinaya Kumar Mylavarapu, Cognizant Research CenterSubject Matter ExpertInamdar Mahendra, Associate Director, IT IS, Cognizant Technology SolutionsThe author and subject matter expert would like to acknowledge the inputs and support of Navin Rao,a Senior Director in Cognizant’s Testing Practice.About CognizantCognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business process out-sourcing services. Cognizant’s single-minded passion is to dedicate our global technology and innovation know-how,our industry expertise and worldwide resources to working together with clients to make their businesses stronger.With over 50 global delivery centers and more than 130,000 employees as of September 30, 2011, we combine a uniqueglobal delivery model infused with a distinct culture of customer satisfaction. A member of the NASDAQ-100 Index andS&P 500 Index, Cognizant is a Forbes Global 2000 company and a member of the Fortune 1000 and is ranked amongthe top information technology companies in BusinessWeek’s Hot Growth and Top 50 Performers listings.Visit us online at www.cognizant.com for more information. World Headquarters European Headquarters India Operations Headquarters 500 Frank W. Burr Blvd. Haymarket House #5/535, Old Mahabalipuram Road Teaneck, NJ 07666 USA 28-29 Haymarket Okkiyam Pettai, Thoraipakkam Phone: +1 201 801 0233 London SW1Y 4SP UK Chennai, 600 096 India Fax: +1 201 801 0243 Phone: +44 (0) 20 7321 4888 Phone: +91 (0) 44 4209 6000 Toll Free: +1 888 937 3277 Fax: +44 (0) 20 7321 4890 Fax: +91 (0) 44 4209 6060 Email: inquiry@cognizant.com Email: infouk@cognizant.com Email: inquiryindia@cognizant.com© Copyright 2011, Cognizant. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by anymeans, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Cognizant. The information contained herein issubject to change without notice. All other trademarks mentioned herein are the property of their respective owners.