est Environment (TEM) enables the efficient configuration, allocation, reporting, and management
of test environments.
Attend this webinar to get results and insights from the latest test environment research by Enterprise Management Associates (EMA) and the role that test environment management is playing in implementing effective CI/CD. The survey gathered insights and usage details from 160 of the largest North American enterprises. You'll hear Steve Hendrick, Research Director at EMA and Jeff Keyes, Director of Marketing at Plutora discuss the results and dive into the strategies, objectives, and experiences of large enterprises in using TEM tools. This research will show how proactive management of preproduction environments accelerates application development while generating significant cost savings across DevOps activities, resources, and staff. The result will be a roadmap for enterprises showing how best to leverage TEM technology.
This webinar summarizes the research findings into these key areas:
Test environment management strategies, priorities and maturity
Key functionality and top vendors providing capabilities
Real-world benefits with quantifiable results
Scaling API-first – The story of a global engineering organization
Test Environment Management: A Critical Requirement for Effective CI/CD
1. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Jeff Keyes
Director of Product Marketing
Plutora
Steve Hendrick
Research Director
Enterprise Management Associates (EMA)
2. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Test Environment Management (TEM) is a fully managed and automated approach to
defining, configuring, booking, assigning, and recycling of test environments (TEs).
Page 3
What?
Scale. Large enterprises must manage hundreds or thousands of test environments.
Speed. TEM tools provision TEs faster and reduce development time.
Efficiency. TEs can be recycled faster and fewer headcount are needed to manage TEs.
Quality. TEM drives application quality up.
Oversight. TEM provides complete visibility and manageability into the lifecycle of all TEs.
Why?
EMA 2018 TEM Survey, June 2018
3. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Why TEM Matters
EMA 2018 TEM Survey, June 2018
25 50 76 100 200 500
Number Annual Production Releases
Predicted TEM Cost Saving
Faster Provisioning Faster Dev/Test Reduced TEM Staff Improved App Quality Fewer Test Envs
TEM delivers cost savings even at a
modest production release volumes
If TEM tool benefits are viewed as
cost avoidance, ROI for most
enterprises will be almost immediate
$613,332
$7,981,790
$1,001,145
$1,404,471
$1,776,772
$3,328,027
Page 4
4. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
2018 DevOps TEM Survey
Survey Parameters
• Geographic Scope: North America
• Sample Size: 159
• Survey Fielding: June 2018
• Respondent: IT professionals with experience in TEM, DevOps environment
management, dev/test environment management or scheduling
• Enterprise size: 5,000 employees or more
• Custom app dev: Active enterprise involvement
• Vertical industries: All, except Education & 25% cap on IT vendors
Page 5EMA 2018 TEM Survey, June 2018
5. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
16%
17%
26%
41%
0% 10% 20% 30% 40% 50%
30,000 or more
20,000 to 29,999
10,000 to 19,999
5,000 to 9,999
Percent Responding (single response)
Mean=17,720
Enterprise Size
How many employees does your enterprise have worldwide?
EMA 2018 TEM Survey, June 2018, Q2, N=159 Page 6
6. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Utilities
Government (State/local)
Professional Services
Government (Federal/National)
Telecommunications / ISP / Web Hosting
Life Sciences (biotech, pharmaceuticals, etc.)
Media
Oil and Gas
Transportation & Logistics
Business Services (accounting, consulting, legal, etc.)
Consumer Packaged Goods
Construction/Engineering
Information Technology
Process Manufacturing
Wholesale
Discrete Manufacturing
Retail
Financial Services (Banking/Insurance/Securities)
Health Care
0% 1% 2% 3% 4% 5% 6% 7% 8%
Percent Responding (single response)
Primary Vertical Industry
What is your organization’s primary industry?
EMA 2018 TEM Survey, June 2018, Q5, N=159 7
7. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
CURRENT IT ENVIRONMENT
Page 8
8. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
13%
62%
25%
0% 10% 20% 30% 40% 50% 60% 70%
100-249
50-99
0-49
Percent Responding (single response)
Production Release Volume
How many production releases does your enterprise perform annually?
Mean=76
EMA 2018 TEM Survey, June 2018, Q6, N=159 Page 9
9. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
8%
18%
22% 22%
4%
4%
21%
0%
5%
10%
15%
20%
25%
0- 50 51- 100 101- 150 151- 200 201- 250 251- 300 300+ Don’t know or not
sure
Percentresponding
Number of Test Environments
Number of Test Environments
How many test environments does your enterprise have?
Min=50
Mean=188
Max=685
EMA 2018 TEM Survey, June 2018, Q9, N=159 Page 10
10. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
4%
14%
19%
25%
21%
9%
4%
5%
34%
16%
21% 21%
0%
5%
10%
15%
20%
25%
30%
35%
40%
0 -10% 11 - 20% 21 - 30% 31 - 40% 41 - 50% 51 - 60% 61 - 70%
Percentresponding
Percent of TEs created in public cloud compared to TEs auto created
Public Cloud
Auto Created
Distribution of Test Environments
What percent of your test environments are created in an automated way compared to test
environments created in a public cloud?
EMA 2018 TEM Survey, June 2018, Q10, Q11, N=159 Page 11
11. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
12%
31%
22%
20%
15%
0%
5%
10%
15%
20%
25%
30%
35%
$100K to $250K $250K to $500K $750K to $1M $500K to $750K $1M to $2M
PercentRespondingHardware/Software Spend on Test Environments
What is your level of annual hardware and software spend on test environments?
Mean=$682,000
EMA 2018 TEM Survey, June 2018, Q12, N=159 Page 12
12. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
23%
2%
3%
4%
6%
12%
17%
33%
0% 5% 10% 15% 20% 25% 30% 35% 40%
We do not use a commercial TEM tool
Xebia Labs
HP
AWS
IBM
Apwide
Omnium Lite
Enov8
Plutora Environments/Plutora
Percent Responding (single response)
TEM Products in Use
Do you use a commercial TEM tool and if so which one?
EMA 2018 TEM Survey, June2018, Q14, N=159 Page 13
13. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
8%
23%
66%
72%
72%
75%
0% 10% 20% 30% 40% 50% 60% 70% 80%
We do not use any other tools to manage our test
environments
Inhouse developed tool(s)
Configuration management tools (like Ansible, Chef,
Puppet…)
Workload automation tools (like CA Automic, Control-M,
Cron, IBM WLA…)
Spreadsheets
CI/CD tools (like AWS Code Pipeline, Bamboo, Jenkins,
Travis…)
Percent Responding (multiple response)
Other Tools Used to manage Test Environments
Do you use any other tools to manage your test environments?
EMA 2018 TEM Survey, June2018, Q15, N=159 Page 14
14. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
4%
21%
16%
46%
35%
9%
0% 10% 20% 30% 40% 50%
TEM is part of our application development DNA
We are experienced in TEM and it is a part of many
development projects
IT recognizes the value of TEM and we are developing formal
TEM processes
Many teams have developed their own ad hoc TEM
processes
Some teams have developed their own ad hoc TEM
processes
We have no ad hoc or formal process for TEM
Percent Responding (multiple response)
State of TEM Maturity
What statements best describe the state of TEM in your enterprise?
EMA 2018 TEM Survey, June2018, Q16, N=159
Single responses
Single responses
Page 15
15. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Resolving booking or change request conflicts
Integration with CI/CD tools (like AWS Code Pipeline, Bamboo, Jenkins, Travis…)
Test environment charge back
Verifying & maintaining the health of test environments
Expediting the handoff from development to test
Reporting and analysis of test environment usage
Managing test environment change requests
Test environment scheduling
A single view into all test environments
Test environment capacity planning
Integration with ITSM tools (like BMC Remedy, ServiceNow…)
Integration with configuration management tools (like Ansible, Chef, Puppet…)
The provisioning and deployment of test environments
0% 20% 40% 60% 80% 100%
Percent Responding (multiple response)
Should Provide
In Use Today
Capabilities TEM Should Provide vs. In Use Today
What capabilities should a TEM tool provide vs. capabilities in use today?
EMA 2018 TEM Survey, June 2018, Q8, Q17, N=159 Page 16
16. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
5%
4%
6%
10%
23%
19%
31%
11%
10%
14%
10%
17%
11%
15%
12%
10%
10%
10%
18%
14%
10%
15%
14%
0% 10% 20% 30% 40% 50% 60%
Missing audit trail on test environment configurations
Inability to manage test environment costs
Inefficient use of test environments
We have yet to see enough value in TEM specific tools
Inability to schedule and deliver test environments in a timely way
Lack of visibility into test environment inventory and availability
Configuration drift
Inability to instantiate test environments in an automated way
Percent of mentions
First Second Third
Top 3 Challenges Addressed by TEM Tools
What are your top 3 TEM challenges?
EMA 2018 TEM Survey, June 2018, Q18, N=154 Page 17
17. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
5%
4%
7%
18%
14%
19%
33%
12%
14%
12%
16%
13%
18%
15%
10%
16%
15%
10%
21%
16%
11%
0% 10% 20% 30% 40% 50% 60% 70%
Understanding the scope of what our enterprise should be doing in TEM
Transitioning from ad hoc to formal TEM processes
Providing a way to charge back for test environments used
Adopting TEM specific tools to manage our TEM process
Implementing a way to more effectively manage TEM activities
Integrating our TEM processes with other tools for dev/test/deploy
Developing a more automated and scalable approach to TEM
Percent of mentions
First Second Third
Top 3 Objectives Addressed by TEM Tools
What are your enterprise’s top 3 TEM objectives?
EMA 2018 TEM Survey, June 2018, Q19, N=154 Page 18
18. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
TEM COST SAVINGS ANALYSIS
Q20-Q29
Page 19
19. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
54%
58%
62%
74%
89%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
We are not far enough along in TEM to identify any
benefits
TEM enables us to improve the quality of the applications
we develop
We have been able to reduce the headcount necessary for
managing test environments
TEM enables us to reduce the time it takes to deliver
applications into production
TEM enables us to use and recycle our test resources
more efficiently
TEM allows us to provision test environments faster
Percent Responding (multiple response)
TEM Benefits Realized
Which of the following benefits have you attained from yout TEM process?
EMA 2018 TEM Survey, June2018, Q20, N=159
Related to:
TEs
TEs
TEs
NAPRs
NAPRs
Page 20
20. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
TEM Enables Faster Provisioning of Test Environments
Expected cost savings range from $612 to $864 per test environment
• TEM tools reduced test environment
provisioning from a mean of 3.3 days
(before TEM) to 1.8 days (after TEM).
• Fully loaded TE management staff cost
is $450/day
Y = 586.34x + 18075
$0
$100,000
$200,000
$300,000
$400,000
$500,000
$600,000
$700,000
0 100 200 300 400 500 600 700 800
TESavings
Number Test Environments (NTE)
TE Provisioning Savings (TEPS) Data
$76,715
$128,318 $135,355
$311,275
$604,475
$0
$100,000
$200,000
$300,000
$400,000
$500,000
$600,000
$700,000
100 188 200 500 1,000
TESavings
Number of Test Environments (NTE)
Predicted TE Provisioning Savings (PTEPS)
• TEPS = (Provision days before-Provision
days after)*NTE*450
• PTEPS = (586.3*NTE)+18,075
• At mean number of test environments of
188, TE provisioning savings is $128,318
EMA 2018 TEM Survey, June2018, Q9, Q24, Q25, N=112 Page 21
21. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
TEM Reduces the Number of Test Environments Needed
Expected cost savings range between $3,500 and $8,685 per test environment eliminated
• TEM tools reduce the number of test
environments needed
• Test environment cost varies widely
from $565 to $18,293 per environment
• TEE = TE hardware & software spend
/NTE*number test environments eliminated
• PTEE = 173.7*NTE
• Implicit in NTE is between 2 and 34 test
environments eliminated
EMA 2018 TEM Survey, June2018, Q9, Q28, Q29, N=99
y = 173.7x
$0
$50,000
$100,000
$150,000
$200,000
$250,000
0 100 200 300 400 500 600 700 800
TestEnvironmentElimination(TEE)Savings
Number of Test Environments (NTE)
Test Environment Elimination (TEE) Data
$17,370
$32,656 $34,740
$86,850
$173,700
$0
$25,000
$50,000
$75,000
$100,000
$125,000
$150,000
$175,000
$200,000
100 188 200 500 1,000
TestEnvEliminationSavings
Number of Test Environments (NTE)
Predicted Test Environment Elimination (PTEE) Savings
Page 22
22. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
TEM Reduces Dev/Test Time for Each Release
Expected cost savings are $9,578 per release
• Reductions in dev/test time ranged
from 1 to 5 weeks with a mean of 1.6
• Fully loaded dev/test staff cost is
$850/day
• DTS = NAPR*dev/test days saved*850
• PDTS = 9,578*NAPR
• At mean number of annual production releases
of 76, TEM dev/test savings is $727,928
EMA 2018 TEM Survey, June2018, Q6, Q21, N=99
y = 9578.3x
$0
$500,000
$1,000,000
$1,500,000
$2,000,000
$2,500,000
$3,000,000
$3,500,000
$4,000,000
0 50 100 150 200 250
Dev/TestCostSavings
Number of Annual Production Releases (NAPR)
Dev/Test Cost Saving (DTS) Data
$239,450
$478,900
$727,928
$957,800
$1,915,600
$4,789,000
$0
$1,000,000
$2,000,000
$3,000,000
$4,000,000
$5,000,000
$6,000,000
25 50 76 100 200 500
Dev/TestCostSavings
Number of Annual Production Releases (NAPR)
Predicted Dev/Test Cost Savings (PDTS)
Page 23
23. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
TEM Reduces Test Environment Management Costs
Expected cost savings range from $1,010 to $3,726 per test environment
• TEM tools reduce the staffing level of
test environment managers needed
• TEM Staff reductions ranged between
1 and 5 headcount with a mean of 2.7
• TEMS = test env managers reduced*105,000
• PTEMS = 433.57*NTE+207,443
• At mean number of TEs of 188, test
environment manager savings are $288,954
EMA 2018 TEM Survey, June2018, Q9, Q22, Q23, N=75
y = 433.57x + 207443
$0
$100,000
$200,000
$300,000
$400,000
$500,000
$600,000
0 50 100 150 200 250 300 350 400
TestEnvManagerCostSavings
Number of Test Environments (NTE)
Test Env Manager Cost Saving (TEMS) Data
$250,800
$288,954 $294,157
$424,228
$641,013
$0
$100,000
$200,000
$300,000
$400,000
$500,000
$600,000
$700,000
100 188 200 500 1,000
TestEnvManagerCostSavings
Number of Test Environments (NTE)
Predicted Test Env Manager Cost Savings (PTEMS)
Page 24
24. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
TEM Improves Application Quality
Expected cost savings are $2,492 per release
• TEM tools reduce dev, QA, & help desk
time per release
• Between 1 and 7 days with a mean of
3.6 days was saved per release
• DTH = NAPR*dev/test days saved*650
• PDTH = 2492*NAPR
• At mean number of annual releases of 76,
TEM app quality savings is $189,392
EMA 2018 TEM Survey, June2018, Q6, Q26, Q27, N=86
y = 2492.3x
$0
$100,000
$200,000
$300,000
$400,000
$500,000
$600,000
$700,000
$800,000
0 50 100 150 200
StaffCostSavings
Number Annual Production Releases (NAPR)
TEM-Driven Dev/Test/Help Desk (DTH) Savings
$62,300
$124,600
$189,392
$249,200
$498,400
$1,246,000
$0
$200,000
$400,000
$600,000
$800,000
$1,000,000
$1,200,000
$1,400,000
25 50 76 100 200 500
StaffCostSavings
Number Annual Production Releases (NAPR)
Predictive TEM-Driven Dev/Test/Help Desk (PDTH) Savings
Page 25
25. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
TEM Tool Cost Savings Summary
What cost savings can the average enterprise expect from using a TEM tool?
EMA 2018 TEM Survey, June2018
$0
$1
$2
$3
$4
$5
$6
$7
$8
$9
25 50 76 100 200 500
TEM-drivenCostSavingsin$Millions
Number Annual Production Releases
Predicted TEM Cost Saving Based of Annual Number of Production Releases
Faster Provisioning Faster Dev/Test Reduced TEM Staff Improved App Quality Fewer Test Envs
• Cost savings were recognized
across five TEM categories
• Cost saving from using TEM
increase with the number of
annual production releases
• At the mean number of annual
production releases in the
sample, initial TEM tool cost
savings would be $1,404,471
• If TEM tool benefits are viewed
as cost avoidance, ROI for most
enterprises will be no more
than a few months
$613,332
$7,981,790
$1,001,145
$1,404,471
$1,776,772
$3,328,027
Page 26
26. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
FUTURE PRIORITIES
Q30-Q32
Page 27
27. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
27%
32%
32%
35%
36%
40%
42%
67%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Better integrate TEM tools with configuration management tools
Finding a way to define, provision, and redeploy test environments
Obtain insight into test environment demand and capacity for better…
Implementing a way to schedule and schedule test environments
Obtaining a single complete view into all pre-production environments
Defining a better process for managing test environments
Use TEM tools to remove cost and accelerate application development
Better integrate TEM tools with CI/CD tools for better dev/test automation
Percent Responding (multiple response)
TEM Priorities for 2018 and 2019
What are your most important TEM priorities for the remainder of 2018 and 2019?
EMA 2018 TEM Survey, June2018, Q30, N=159 Page 28
28. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
43%
48%
53%
57%
68%
79%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Improve our agility in addressing changing market needs
Improve the performance of applications
Improve the customer experience
Improve IT productivity and reduce cost
Improve the quality and reliability of applications
Improve the speed of software delivery
Percent Responding (multiple response)
DevOps Priorities for 2018 and 2019
What are your most important DevOps priorities for the remainder of 2018 and 2019?
EMA 2018 TEM Survey, June2018, Q31, N=159 Page 29
29. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
38%
43%
47%
53%
63%
74%
75%
79%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Reducing IT expenditures
Improving our DevOps strategy
Use of public cloud services
Data center modernization (i.e., virtualization, containerization, SDx…)
Becoming more agile
Use of private cloud services
Improving cybersecurity
Leveraging analytics for better business intelligence and real time insights
Percent Responding (multiple response)
Overall IT Priorities for 2018 and 2019
What are your overall IT priorities for the remainder of 2018 and 2019?
EMA 2018 TEM Survey, June2018, Q32, N=159 Page 30
30. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Plutora Test Environment Management
31. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Continuous Delivery Pipelines
Interactive view of projects, environments.
32. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Questions
Steve Hendrick
Research Director
Boston, MA
617.899.5359
shendrick@emausa.com
www.enterprisemanagement.com
Page 33
Jeff Keyes
Product Marketing
Seattle, WA
617.899.5359
Jeff.keyes@Plutora.com
www.Plutora.com