Test Environment Management: A Critical Requirement for Effective CI/CD

IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Jeff Keyes
Director of Product Marketing
Plutora
Steve Hendrick
Research Director
Enterprise Management Associates (EMA)

Test Environment Management (TEM) is a fully managed and automated approach to
defining, configuring, booking, assigning, and recycling of test environments (TEs).
Page 3
What?
Scale. Large enterprises must manage hundreds or thousands of test environments.
Speed. TEM tools provision TEs faster and reduce development time.
Efficiency. TEs can be recycled faster and fewer headcount are needed to manage TEs.
Quality. TEM drives application quality up.
Oversight. TEM provides complete visibility and manageability into the lifecycle of all TEs.
Why?
EMA 2018 TEM Survey, June 2018

Why TEM Matters
EMA 2018 TEM Survey, June 2018
25 50 76 100 200 500
Number Annual Production Releases
Predicted TEM Cost Saving
Faster Provisioning Faster Dev/Test Reduced TEM Staff Improved App Quality Fewer Test Envs
TEM delivers cost savings even at a
modest production release volumes
If TEM tool benefits are viewed as
cost avoidance, ROI for most
enterprises will be almost immediate
$613,332
$7,981,790
$1,001,145
$1,404,471
$1,776,772
$3,328,027
Page 4

2018 DevOps TEM Survey
Survey Parameters
• Geographic Scope: North America
• Sample Size: 159
• Survey Fielding: June 2018
• Respondent: IT professionals with experience in TEM, DevOps environment
management, dev/test environment management or scheduling
• Enterprise size: 5,000 employees or more
• Custom app dev: Active enterprise involvement
• Vertical industries: All, except Education & 25% cap on IT vendors
Page 5EMA 2018 TEM Survey, June 2018

16%
17%
26%
41%
0% 10% 20% 30% 40% 50%
30,000 or more
20,000 to 29,999
10,000 to 19,999
5,000 to 9,999
Percent Responding (single response)
Mean=17,720
Enterprise Size
How many employees does your enterprise have worldwide?
EMA 2018 TEM Survey, June 2018, Q2, N=159 Page 6

Utilities
Government (State/local)
Professional Services
Government (Federal/National)
Telecommunications / ISP / Web Hosting
Life Sciences (biotech, pharmaceuticals, etc.)
Media
Oil and Gas
Transportation & Logistics
Business Services (accounting, consulting, legal, etc.)
Consumer Packaged Goods
Construction/Engineering
Information Technology
Process Manufacturing
Wholesale
Discrete Manufacturing
Retail
Financial Services (Banking/Insurance/Securities)
Health Care
0% 1% 2% 3% 4% 5% 6% 7% 8%
Primary Vertical Industry
What is your organization’s primary industry?
EMA 2018 TEM Survey, June 2018, Q5, N=159 7

CURRENT IT ENVIRONMENT
Page 8

13%
62%
25%
0% 10% 20% 30% 40% 50% 60% 70%
100-249
50-99
0-49
Production Release Volume
How many production releases does your enterprise perform annually?
Mean=76

8%
18%
22% 22%
4%
4%
21%
0%
5%
10%
15%
20%
25%
0- 50 51- 100 101- 150 151- 200 201- 250 251- 300 300+ Don’t know or not
sure
Percentresponding
Number of Test Environments
Number of Test Environments
How many test environments does your enterprise have?
Min=50
Mean=188
Max=685

4%
14%
19%
25%
21%
9%
4%
5%
34%
16%
21% 21%
0%
5%
10%
15%
20%
25%
30%
35%
40%
0 -10% 11 - 20% 21 - 30% 31 - 40% 41 - 50% 51 - 60% 61 - 70%
Percentresponding
Percent of TEs created in public cloud compared to TEs auto created
Public Cloud
Auto Created
Distribution of Test Environments
What percent of your test environments are created in an automated way compared to test
environments created in a public cloud?
EMA 2018 TEM Survey, June 2018, Q10, Q11, N=159 Page 11

12%
31%
22%
20%
15%
0%
5%
10%
15%
20%
25%
30%
35%
$100K to $250K $250K to $500K $750K to $1M $500K to $750K $1M to $2M
PercentRespondingHardware/Software Spend on Test Environments
What is your level of annual hardware and software spend on test environments?
Mean=$682,000

23%
2%
3%
4%
6%
12%
17%
33%
0% 5% 10% 15% 20% 25% 30% 35% 40%
We do not use a commercial TEM tool
Xebia Labs
HP
AWS
IBM
Apwide
Omnium Lite
Enov8
Plutora Environments/Plutora
TEM Products in Use
Do you use a commercial TEM tool and if so which one?
EMA 2018 TEM Survey, June2018, Q14, N=159 Page 13

8%
23%
66%
72%
72%
75%
0% 10% 20% 30% 40% 50% 60% 70% 80%
We do not use any other tools to manage our test
environments
Inhouse developed tool(s)
Configuration management tools (like Ansible, Chef,
Puppet…)
Workload automation tools (like CA Automic, Control-M,
Cron, IBM WLA…)
Spreadsheets
CI/CD tools (like AWS Code Pipeline, Bamboo, Jenkins,
Travis…)
Percent Responding (multiple response)
Other Tools Used to manage Test Environments
Do you use any other tools to manage your test environments?

4%
21%
16%
46%
35%
9%
0% 10% 20% 30% 40% 50%
TEM is part of our application development DNA
We are experienced in TEM and it is a part of many
development projects
IT recognizes the value of TEM and we are developing formal
TEM processes
Many teams have developed their own ad hoc TEM
processes
Some teams have developed their own ad hoc TEM
processes
We have no ad hoc or formal process for TEM
State of TEM Maturity
What statements best describe the state of TEM in your enterprise?
EMA 2018 TEM Survey, June2018, Q16, N=159
Single responses
Single responses
Page 15

Resolving booking or change request conflicts
Integration with CI/CD tools (like AWS Code Pipeline, Bamboo, Jenkins, Travis…)
Test environment charge back
Verifying & maintaining the health of test environments
Expediting the handoff from development to test
Reporting and analysis of test environment usage
Managing test environment change requests
Test environment scheduling
A single view into all test environments
Test environment capacity planning
Integration with ITSM tools (like BMC Remedy, ServiceNow…)
Integration with configuration management tools (like Ansible, Chef, Puppet…)
The provisioning and deployment of test environments
0% 20% 40% 60% 80% 100%
Should Provide
In Use Today
Capabilities TEM Should Provide vs. In Use Today
What capabilities should a TEM tool provide vs. capabilities in use today?
EMA 2018 TEM Survey, June 2018, Q8, Q17, N=159 Page 16

5%
4%
6%
10%
23%
19%
31%
11%
10%
14%
10%
17%
11%
15%
12%
10%
10%
10%
18%
14%
10%
15%
14%
0% 10% 20% 30% 40% 50% 60%
Missing audit trail on test environment configurations
Inability to manage test environment costs
Inefficient use of test environments
We have yet to see enough value in TEM specific tools
Inability to schedule and deliver test environments in a timely way
Lack of visibility into test environment inventory and availability
Configuration drift
Inability to instantiate test environments in an automated way
Percent of mentions
First Second Third
Top 3 Challenges Addressed by TEM Tools
What are your top 3 TEM challenges?

5%
4%
7%
18%
14%
19%
33%
12%
14%
12%
16%
13%
18%
15%
10%
16%
15%
10%
21%
16%
11%
0% 10% 20% 30% 40% 50% 60% 70%
Understanding the scope of what our enterprise should be doing in TEM
Transitioning from ad hoc to formal TEM processes
Providing a way to charge back for test environments used
Adopting TEM specific tools to manage our TEM process
Implementing a way to more effectively manage TEM activities
Integrating our TEM processes with other tools for dev/test/deploy
Developing a more automated and scalable approach to TEM
Percent of mentions
First Second Third
Top 3 Objectives Addressed by TEM Tools
What are your enterprise’s top 3 TEM objectives?

TEM COST SAVINGS ANALYSIS
Q20-Q29
Page 19

54%
58%
62%
74%
89%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
We are not far enough along in TEM to identify any
benefits
TEM enables us to improve the quality of the applications
we develop
We have been able to reduce the headcount necessary for
managing test environments
TEM enables us to reduce the time it takes to deliver
applications into production
TEM enables us to use and recycle our test resources
more efficiently
TEM allows us to provision test environments faster
TEM Benefits Realized
Which of the following benefits have you attained from yout TEM process?
EMA 2018 TEM Survey, June2018, Q20, N=159
Related to:
TEs
TEs
TEs
NAPRs
NAPRs
Page 20

TEM Enables Faster Provisioning of Test Environments
Expected cost savings range from $612 to $864 per test environment
• TEM tools reduced test environment
provisioning from a mean of 3.3 days
(before TEM) to 1.8 days (after TEM).
• Fully loaded TE management staff cost
is $450/day
Y = 586.34x + 18075
$0
$100,000
$200,000
$300,000
$400,000
$500,000
$600,000
$700,000
0 100 200 300 400 500 600 700 800
TESavings
Number Test Environments (NTE)
TE Provisioning Savings (TEPS) Data
$76,715
$128,318 $135,355
$311,275
$604,475
$0
$100,000
$200,000
$300,000
$400,000
$500,000
$600,000
$700,000
100 188 200 500 1,000
TESavings
Number of Test Environments (NTE)
Predicted TE Provisioning Savings (PTEPS)
• TEPS = (Provision days before-Provision
days after)*NTE*450
• PTEPS = (586.3*NTE)+18,075
• At mean number of test environments of
188, TE provisioning savings is $128,318
EMA 2018 TEM Survey, June2018, Q9, Q24, Q25, N=112 Page 21

TEM Reduces the Number of Test Environments Needed
Expected cost savings range between $3,500 and $8,685 per test environment eliminated
• TEM tools reduce the number of test
environments needed
• Test environment cost varies widely
from $565 to $18,293 per environment
• TEE = TE hardware & software spend
/NTE*number test environments eliminated
• PTEE = 173.7*NTE
• Implicit in NTE is between 2 and 34 test
environments eliminated
EMA 2018 TEM Survey, June2018, Q9, Q28, Q29, N=99
y = 173.7x
$0
$50,000
$100,000
$150,000
$200,000
$250,000
0 100 200 300 400 500 600 700 800
TestEnvironmentElimination(TEE)Savings
Test Environment Elimination (TEE) Data
$17,370
$32,656 $34,740
$86,850
$173,700
$0
$25,000
$50,000
$75,000
$100,000
$125,000
$150,000
$175,000
$200,000
100 188 200 500 1,000
TestEnvEliminationSavings
Predicted Test Environment Elimination (PTEE) Savings
Page 22

TEM Reduces Dev/Test Time for Each Release
Expected cost savings are $9,578 per release
• Reductions in dev/test time ranged
from 1 to 5 weeks with a mean of 1.6
• Fully loaded dev/test staff cost is
$850/day
• DTS = NAPR*dev/test days saved*850
• PDTS = 9,578*NAPR
• At mean number of annual production releases
of 76, TEM dev/test savings is $727,928
EMA 2018 TEM Survey, June2018, Q6, Q21, N=99
y = 9578.3x
$0
$500,000
$1,000,000
$1,500,000
$2,000,000
$2,500,000
$3,000,000
$3,500,000
$4,000,000
0 50 100 150 200 250
Dev/TestCostSavings
Number of Annual Production Releases (NAPR)
Dev/Test Cost Saving (DTS) Data
$239,450
$478,900
$727,928
$957,800
$1,915,600
$4,789,000
$0
$1,000,000
$2,000,000
$3,000,000
$4,000,000
$5,000,000
$6,000,000
25 50 76 100 200 500
Dev/TestCostSavings
Number of Annual Production Releases (NAPR)
Predicted Dev/Test Cost Savings (PDTS)
Page 23

TEM Reduces Test Environment Management Costs
Expected cost savings range from $1,010 to $3,726 per test environment
• TEM tools reduce the staffing level of
test environment managers needed
• TEM Staff reductions ranged between
1 and 5 headcount with a mean of 2.7
• TEMS = test env managers reduced*105,000
• PTEMS = 433.57*NTE+207,443
• At mean number of TEs of 188, test
environment manager savings are $288,954
y = 433.57x + 207443
$0
$100,000
$200,000
$300,000
$400,000
$500,000
$600,000
0 50 100 150 200 250 300 350 400
TestEnvManagerCostSavings
Test Env Manager Cost Saving (TEMS) Data
$250,800
$288,954 $294,157
$424,228
$641,013
$0
$100,000
$200,000
$300,000
$400,000
$500,000
$600,000
$700,000
100 188 200 500 1,000
TestEnvManagerCostSavings
Predicted Test Env Manager Cost Savings (PTEMS)
Page 24

TEM Improves Application Quality
Expected cost savings are $2,492 per release
• TEM tools reduce dev, QA, & help desk
time per release
• Between 1 and 7 days with a mean of
3.6 days was saved per release
• DTH = NAPR*dev/test days saved*650
• PDTH = 2492*NAPR
• At mean number of annual releases of 76,
TEM app quality savings is $189,392
y = 2492.3x
$0
$100,000
$200,000
$300,000
$400,000
$500,000
$600,000
$700,000
$800,000
0 50 100 150 200
StaffCostSavings
Number Annual Production Releases (NAPR)
TEM-Driven Dev/Test/Help Desk (DTH) Savings
$62,300
$124,600
$189,392
$249,200
$498,400
$1,246,000
$0
$200,000
$400,000
$600,000
$800,000
$1,000,000
$1,200,000
$1,400,000
25 50 76 100 200 500
StaffCostSavings
Number Annual Production Releases (NAPR)
Predictive TEM-Driven Dev/Test/Help Desk (PDTH) Savings
Page 25

TEM Tool Cost Savings Summary
What cost savings can the average enterprise expect from using a TEM tool?
EMA 2018 TEM Survey, June2018
$0
$1
$2
$3
$4
$5
$6
$7
$8
$9
25 50 76 100 200 500
TEM-drivenCostSavingsin$Millions
Number Annual Production Releases
Predicted TEM Cost Saving Based of Annual Number of Production Releases
Faster Provisioning Faster Dev/Test Reduced TEM Staff Improved App Quality Fewer Test Envs
• Cost savings were recognized
across five TEM categories
• Cost saving from using TEM
increase with the number of
annual production releases
• At the mean number of annual
production releases in the
sample, initial TEM tool cost
savings would be $1,404,471
• If TEM tool benefits are viewed
as cost avoidance, ROI for most
enterprises will be no more
than a few months
$613,332
$7,981,790
$1,001,145
$1,404,471
$1,776,772
$3,328,027
Page 26

FUTURE PRIORITIES
Q30-Q32
Page 27

27%
32%
32%
35%
36%
40%
42%
67%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Better integrate TEM tools with configuration management tools
Finding a way to define, provision, and redeploy test environments
Obtain insight into test environment demand and capacity for better…
Implementing a way to schedule and schedule test environments
Obtaining a single complete view into all pre-production environments
Defining a better process for managing test environments
Use TEM tools to remove cost and accelerate application development
Better integrate TEM tools with CI/CD tools for better dev/test automation
TEM Priorities for 2018 and 2019
What are your most important TEM priorities for the remainder of 2018 and 2019?

43%
48%
53%
57%
68%
79%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Improve our agility in addressing changing market needs
Improve the performance of applications
Improve the customer experience
Improve IT productivity and reduce cost
Improve the quality and reliability of applications
Improve the speed of software delivery
DevOps Priorities for 2018 and 2019
What are your most important DevOps priorities for the remainder of 2018 and 2019?

38%
43%
47%
53%
63%
74%
75%
79%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Reducing IT expenditures
Improving our DevOps strategy
Use of public cloud services
Data center modernization (i.e., virtualization, containerization, SDx…)
Becoming more agile
Use of private cloud services
Improving cybersecurity
Leveraging analytics for better business intelligence and real time insights
Overall IT Priorities for 2018 and 2019
What are your overall IT priorities for the remainder of 2018 and 2019?

Plutora Test Environment Management

Continuous Delivery Pipelines
Interactive view of projects, environments.

Questions
Steve Hendrick
Research Director
Boston, MA
617.899.5359
shendrick@emausa.com
www.enterprisemanagement.com
Page 33
Jeff Keyes
Product Marketing
Seattle, WA
617.899.5359
Jeff.keyes@Plutora.com
www.Plutora.com

Test Environment Management: A Critical Requirement for Effective CI/CD

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Test Environment Management: A Critical Requirement for Effective CI/CD

Similar to Test Environment Management: A Critical Requirement for Effective CI/CD (20)

More from DevOps.com

More from DevOps.com (20)

Recently uploaded

Recently uploaded (20)

Test Environment Management: A Critical Requirement for Effective CI/CD