In the software development lifecycle, application testing is crucial - but often given short shrift by companies allocating temporary resources. Test environment management services (TEMS) is a perfect solution, reducing testing costs and errors while conducting test/QA environment management, monitoring and maintenance and cloud infrastructure provisioning.

1. • Cognizant Reports
The Business Case for Test Environment
Management Services
Application test environments managed by temporary resources can cause
spikes in release costs, high rates of defects that leak into production
systems and consequent losses. Test environment services delivered via
a dedicated managed services approach represents a superior alternative
that delivers greater monetary benefits and fosters enhanced reputation
as well as brand loyalty.
Executive Summary Managing environment-related issues and
This much about testing is incontrovertible: defects consumes nearly 40% of the effort
Poor test environment configuration, downtime, involved in the software development lifecycle.1
and the unavailability of test environments can Apart from delaying testing cycles and creating
impact the quality of the testing process as well additional costs, the test management problems
as the code itself. This is caused primarily by noted above force managers to unnecessarily
the lack of ownership and management of test rush through the testing process, conduct tests
environments. in environments that are far from optimal (i.e.,
environments that do not approximate produc-
The reasons are manifold. For starters, most tion scenarios) and result in risk-based sign-off
IT organizations lack a dedicated testing team (i.e., acknowledging the risks involved in releasing
that ensures code quality from top to bottom. a product that is not completely tested).
In fact, many organizations temporarily divert
employees from development and other respon- Against this backdrop, test environment manage-
sibilities to manage test environments. Such ment services (TEMS) provides a consultative
a setup leads to a lack of accountability and approach, thus making it a superior alternative.
creates issues that often impact the quality of The TEMS framework helps organizations to
testing, which in turn compromises the quality of address common test environment issues and
software applications. Our experience suggests achieve efficiencies in demand fulfillment,
that the cost of QA project/activity is increased capacity utilization and environment availability.
by at least 20% to 25% due to diversion of efforts Delivered as an end-to-end managed service,
by testers and developers. TEMS includes creating a centralized, single
cognizant reports | june 2012

2. ownership for environment management, moni- cycle times, organizations must ensure that test
toring and maintenance along with cloud-based environments closely mimic real-time situations
infrastructure provisioning. and are highly available. Reduced release cycle
time does not provide an allowance for defect
TEMS is designed to: leakage into production, and regardless of the
project timeframe all testing steps involved until
• Support in-flight projects, planned projects release sign-off must be rigorously followed.
and new application releases.
• Manage existing test environments, build new Test environments have a vital role to play in deliv-
test environments and provide additional ering fully tested code and in ensuring confidence
capacity on demand via the cloud. in the successful release of the application. Our
• Help organizations establish the processes experience with organizations across industries
and controls that are required to place an suggest that poorly built test environments often
application environment in the cloud, provided limit the testing team’s ability to test applications
the application is properly cloud-enabled. for various scenarios and increases required
testing cycles, which in turn results in unneces-
The monetary and reputational benefits of switch- sary costs and allows defects to pass through to
ing to TEMS are significant as its per-project costs the production environments.
are lower than test projects without managed
services. However, organizations need to be aware Poor software quality costs organizations
that the TEMS model has its own limitations. They worldwide $500 billion annually, according to a
must also exercise due diligence in choosing the recent survey by Caper Jones & Associates LLC.2
right provider in order to achieve desired results. Defects seep in at various stages of an applica-
tion’s development lifecycle; importantly, finding
Why Managed Test Environment and fixing these bugs in the early stages costs
Services? less than remediation during the later stages.
Jeff and his team are excited about their first big According to Gartner, “The cost of fixing defects
assignment — testing a new business application ranges from a low of approximately $70 (cost to
to be released shortly. They have left no stone fix a defect at the requirements phase) to a high
unturned in their preparation. of $14,000 (cost to fix a defect in production).”3
However, on the day of testing, Jeff faces a Testing is considered an important but not very
few setbacks. First, he discovers that the test business-critical activity; hence, test environ-
environment was not built to the project ments within many organizations receive a low
specifications due to lack of capacity and priority. Test environments are typically managed
configurations. Second, by the time he gets the not by a dedicated unit but by a team of employees
configuration right, he is staring at an unsched- temporarily diverted from development and
uled infrastructure maintenance activity. He is production support activities. With no unit
forced to hurry through the testing process. As exclusively responsible for testing, there is a
a result, defects surface immediately after the lack of ownership and accountability. Further,
application is rolled out. such ad hoc testing teams often have limited
capabilities in building and managing test
Such incidents are common in testing centers environments, leading to unnecessary delays
where lack of ownership and poor management in creating test environments and other issues
of testing impact quality, allowing defects to pass (see sidebar, page 4) that significantly impact
through to production. the effectiveness of testing and ultimately the
business. It is therefore not surprising that in an
Issues in Test Environment Management Aberdeen Group study, only 42% of over 200
The growing criticality of business applica- organizations surveyed were happy with the per-
tions needs no substantiation. The increasing formance of their business-critical applications.4
complexity of today’s applications has expanded
the scope of testing, creating unforeseen The need for efficient test environments is
challenges for testing teams. To deal with the endorsed by an old but still relevant National
complexity and the pressure to reduce release Institute of Standards and Technology (NIST)
cognizant reports 2

3. study. According to the study, improving test • Setting up new environment: service on-board-
infrastructure can save U.S. organizations ing and build-out.
one-third of their annual QA costs.5 • Environment monitoring and management:
hardware, networking, systems software and
TEMS to the Rescue applications.
By applying a combination of consulting and man- • Infrastructure monitoring: servers, databases,
aged services, the TEMS framework addresses mainframes, etc.
common issues that plague test environment • Service request management: issue resolution
management. In most deployments, a dedicated and coordination.
TEMS team works closely with an organization’s • Release management: systematic updates,
IT department to create a centralized resource terminations and reports.
that owns, monitors and controls test environ-
ments. A real-time project dashboard typically Further, TEMS offered via the cloud can generate
enables day-to-day tracking of project milestones additional benefits by eliminating the need for
and metrics, while an integrated delivery model costly upfront Cap-Ex investments and providing
synthesizes the efforts of multiple project teams. on-demand access to test environments.
One of TEMS’ key attributes is that it can be TEMS: A Better Alternative
offered modularly as well as an end-to-end In the traditional model, the cost of test
service (see Figure 1). It can also be delivered environment management is borne by various
in multivendor scenarios and with minimum teams. When organizations switch to the TEMS
disruption to daily business activities in a rapid model, those costs become visible since under
and robust fashion by following best practices for TEMS a separate group is accountable for
change management. Organizations can evaluate providing testing services through a direct cost
their current test environment management allocation model. While TEMS merely makes
setup and choose the most suitable service. the relevant costs visible, organizations may
mistakenly believe it necessitates additional
TEMS provides a single contact point for costs. On the contrary, the monetary losses due
resolution of issues, queries, changes and all to environment downtime — paying testers for the
other requirements, thus creating greater idle time and for the additional time they work
accountability for losses due to test environ- to finish the task — are far greater than the cost
ment issues. Broadly, TEMS’ end-to-end service of creating a separate team that minimizes the
includes the following: incidence of environment issues.
TEMS Suite
Module Services
Environment Management • Manage/monitor SLAs.
• Plan/analyze environment requests.
• Facilitate design workshops.
• Liaise with testing/infrastructure teams.
• Create/update service reports.
• Identify improvement areas.
Environment Monitoring • Additional responsibility for monitoring servers and notification.
Environment Maintenance • Handle infrastructure support or any combination of technology support levels.
• Incident management, problem management, change management.
• Build and smoke-test environment.
Cloud Infrastructure • Provide hosting services (using partner vendor).
Provisioning/Management • Provision virtual servers and test environments.
Figure 1
cognizant reports 3

4. In this regard, TEMS offers a superior alternative than 80% of customer-invested tools and
to the traditional approach, creating new efficien- technologies, thereby lowering the total cost
cies in managing test environments and deliv- of ownership (TCO).
ering greater monetary benefits (as described • Increased environment availability: Through
below). Such advantages arise out of the fact proactive monitoring, TEMS ensures the
that ownership and accountability within TEMS health of the environment as well as its
rests with an exclusive team as opposed to the availability. In some cases, average uptime of
traditional model where a heterogeneous team is more than 90% has been achieved from just
accountable for test environment management 68%.6 Higher availability allows testers to
activities. Such a centralized, dedicated TEMS perform extensive and exhaustive testing
team therefore frees up ad hoc resources drawn which vastly improves result efficacy.
together from multiple groups and instead allows • Proper scheduling replaces chaos: Improper
them to focus on their core activities. scheduling of tests when the environment is
shared by multiple testing teams often delays
TEMS’ major benefits include: testing projects. This forces test managers to
conduct nonfunctional testing in scaled-down
• Improved build efficiencies: With a dedicated environments and/or non-functional testing
team of specialists running the show, new that relies on stubbed environments, result-
test environments can be built and deployed ing in projects being signed-off with numerous
rapidly. This minimizes the defects that caveats and application problems in produc-
arise due to inaccurate test environment tion. In contrast, by continuously providing
configuration (the rate of which is 30% environment availability metrics through daily
according to studies) and achieve build standardized reports, TEMS allows teams to
efficiencies of more than 80%. book the environment in advance. Further,
• Reuse prevents waste: As a “tool-agnostic” through proper scheduling with set start
framework, TEMS enables reuse rates of more and end dates and institutionalization of the
Quick Take
Drawbacks of Traditional Test Environment Management
• Test environments differ from production environments in terms of operating systems, patch
levels, software versions, configuration, etc. The wider the gap between the test and production
environments, the greater the chance of an application failing after being deployed or a defect
leaking into live systems. Further, configuration changes made in response to errors often go
untracked, causing errors when applications are moved into production.
• Loosely managed asset control necessitates undue investment in infrastructure. Inadequate
access control reduces confidence in the state of the environment and its configuration control.
Together, this introduces additional risks and adds unnecessary costs to the project.
• Simulating the production issue in the test environment to identify root cause issues is
challenging due to a lack of complete alignment between the test environment and production
versions across system component layers.
• Testing teams often clone production databases or extract data by writing scripts to create test
data. This approach takes a lot of effort, is prone to errors and may not meet data protection
policies. Often, this activity is not change-controlled and is non-auditable.
• The conventional approach of manually creating in-house testing environments that fully
mirror the complexities and multiplicities of real-time environment consumes a lot of resources.
This has prompted application developers to adopt the risky approach of conducting testing in
production environments. However, the Sarbanes–Oxley Act and other regulations mandates
restricted access to production systems. This has forced organizations across industries to
conduct testing in separate environments. Further, it is not uncommon for organizations that
cannot afford large-scale test environments to use sizing or extrapolation techniques during
performance testing to understand how an application behaves in the real world. This approach,
however, leaves too much room for potential problems after implementation.
cognizant reports 4

5. release management process, TEMS helps controlled access management; maintaining
deliver fully tested projects on time. up-to-date inventory of environments and
• Better utilization of infrastructure: In the infrastructure components; compliance with
traditional model, approximately 50% to 70% security and standards; etc., TEMS offers organi-
of the technology infrastructure earmarked zations a one-stop solution to their test environ-
for testing is underutilized, according to both ment issues.
anecdotal and published reports.7 Through
effective capacity planning, TEMS ensures TEMS Efficiencies
that testing projects receive the right amount Figure 2 illustrates the QA project efficiencies
of resources and test environments are that TEMS can generate, with the dark red
optimally utilized, prompting improved returns and dark green bands representing the lowest
on existing environments. For instance, (pre-TEMS) and best-in-class (post-TEMS)
incidents such as over-provisioning, which efficiency levels respectively. In our experi-
result in additional costs, can be prevented. ence, most of the current test environment
• Communication replaces commotion: In efficiencies fall in the D and E bands, with an
the absence of ownership, test environment average efficiency of 57%. Deploying TEMS has
downtime and changes are often not increased the average efficiency levels to 75%,
controlled and communicated properly. This and has the potential to increase it to 95%.
poses challenges in managing testing projects
and achieving KPIs. TEMS creates an effective However, it must be noted that the TEMS
communication mechanism among the key framework does not establish rigorous processes
stakeholders in the event of environment or controls and has lower SLA commitments
issues by deploying same-day text alerts, etc. when compared with production environments.
Additionally, its scope is limited to the staging
With a host of other benefits such as targeted environment and does not include the change
problem management and root cause analysis; controls at the production environment and
TEMS-Efficiency Meter
Commonly seen TEMS Partly SLA and OLA
Pre-TEMS (%) Implemented (%) in place (%)
QA Environment Availability
Post-TEMS: Highly available managed test environments reduces
project costs.
(91-100) A 95
(86-90) B
(81-85) C 85
(60-80) D
(41-60) E 57
(21-40) F
(1-20) G
Pre-TEMS: Poorly managed test environments raises project costs.
Note:
1. Bands G to D indicate improvements to QA environment can be rapid as low availability indicator means the environment
is unstable or poorly supported.
2. Bands C and B represent environments that get timely support and are stable, but require more processes and controls
to improve availability.
3. Band A represents environments that are stable and their availability is high as committed SLA/OLAs are in place.
Source: Cognizant Research Center
Figure 2
cognizant reports 5

6. release management process. These limita- Organizations planning to deploy TEMS must
tions are by design of the service and separate find a partner that understands business
TEMS from production environment service challenges and has prior experience in offering
management. They add value by lowering the cost managed testing services. Reputable consulting
of operation and enabling the test environment to firms that have rich domain experience and the
align itself with release management objectives. ability to work in multivendor scenarios should
be considered to ensure a smooth transfor-
Moving Forward: Embracing TEMS mation, create new process efficiencies and
A stable, reliable and flexible test environ- accelerate payback, culminating in instilling
ment is important for companies to cope with greater discipline in managing test environments.
today’s testing demands. Poor test environment This developent will help ensure that managers
management affects delivery schedules enjoy their testing assignments and assure
and results in increased risk of subsequent application code quality and business continuity
software failures as well as additional business across the enterprise.
expenditure. TEMS has much to offer organiza-
tions seeking to overcome environment-related
challenges in an easy and convenient fashion.
Footnotes
1
“Testing Environment Management Challenges With Virtualization,” ComputerWorldUK, Sept. 6, 2008.
2
“Software Quality in 2011: A Survey of the State of the Art,” Caper Jones & Associates LLC, August 31,
2011.
3
“Hype Cycle for Application Development, 2011,” Gartner, July 27, 2011.
4
“Application Performance Management: The Lifecycle Approach Brings IT and Business Together,”
Aberdeen Group, June 30, 2008.
5
“The Economic Impacts of Inadequate Infrastructure for Software Testing,” NIST, May 2002.
6
“UK Bank Hits Peak Performance with Cognizant TEMS,” Cognizant, 2011.
7
“Taking Testing to the Cloud,” Cognizant, March 2011.
Bibliography
1. Martin Perlin, "Notes from the Trenches: Obstacles and Challenges to IT Environment Stability,"
www.evolven.com, October 2011.
2. Wayne Ariola, "The Next Generation of Test Environment Management," Virtualization Journal,
July 12, 2011.
3. “Determining the 'Right' Number of Nonproduction Environments,” Gartner, July 4, 2011.
4. “Why On-Demand Provisioning Enables Tighter Alignment of Test and Production Environments,”
Cognizant, December 2011.
5. “Transforming a Telecom Service Provider’s Test Environment for Maximum Responsiveness and
Efficiency,” Cognizant, 2011.
cognizant reports 6

7. Author and Analyst
Vinaya Kumar Mylavarapu, Senior Research Analyst, Cognizant Research Center
Subject Matter Expert
Inamdar Mahendra, Associate Director, Cognizant IT Infrastructure Management Services
Design
Harleen Bhatia, Creative Director
Suresh Sambandhan, Designer
About Cognizant
Cognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business process
outsourcing services, dedicated to helping the world’s leading companies build stronger businesses. Headquartered
in Teaneck, New Jersey (U.S.), Cognizant combines a passion for client satisfaction, technology innovation, deep
industry and business process expertise, and a global, collaborative workforce that embodies the future of work.
With over 50 delivery centers worldwide and approximately 140,500 employees as of March 31, 2012, Cognizant is a
member of the NASDAQ-100, the S&P 500, the Forbes Global 2000, and the Fortune 500 and is ranked among the
top performing and fastest growing companies in the world.
Visit us online at www.cognizant.com for more information.
World Headquarters European Headquarters India Operations Headquarters
500 Frank W. Burr Blvd. 1 Kingdom Street #5/535, Old Mahabalipuram Road
Teaneck, NJ 07666 USA Paddington Central Okkiyam Pettai, Thoraipakkam
Phone: +1 201 801 0233 London W2 6BD Chennai, 600 096 India
Fax: +1 201 801 0243 Phone: +44 (0) 207 297 7600 Phone: +91 (0) 44 4209 6000
Toll Free: +1 888 937 3277 Fax: +44 (0) 207 121 0102 Fax: +91 (0) 44 4209 6060
Email: inquiry@cognizant.com Email: infouk@cognizant.com Email: inquiryindia@cognizant.com
©
­­ Copyright 2012, Cognizant. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any
means, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Cognizant. The information contained herein is
subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.