In the era of DevOps, IT leaders are under constant pressure to be more responsive to changing business needs. However, the cost of failed IT changes can be high. While you may know that some deployed changes will fail, what you often don’t know is which ones — until it’s too late. In fact, a recent Gartner study estimates that 80% of major incidents are change-related.   Finally, new approaches are breaking through, based on a rigorous application of Machine Learning (ML) and predictive analytics.

1. AI-powered IT Business AnalyticsAI-powered IT Business Analytics
Mastering DevOps with AI-
powered Change Risk Prediction
Amit Shah, Director of Product Marketing
April 7th, 2020

2. 22
© Numerify, Inc. All Rights Reserved.
Our mission: Help organizations win through the
power of IT business analytics
$4.1M
3-year benefits
214%
ROI
< 3 Month
Payback period
Source: TEI of Numerify IT Business Analytics, April 2019
$ $

3. © Numerify, Inc. All Rights Reserved.
33
Typical customer
experience
Reducing change failures prevents incident resolution costs
Our customers typically experience between $0.2M-1M1 in annual recurring cost savings
Monthly change volume
Change failure
rate reduction
0.25%
0.5%
1%
2%
1K 2K 5K
$108K $216K $540K
$432K
$864K
$1,728K$864K
$216K
$432K
$4,320K
$1,080K
$2,160K
1. Assumes one P0/P1 incident per failed change, average change related incident resolution cost of $3,600
Source: Numerify value framework

4. 4
© Numerify, Inc. All Rights Reserved.
001010100101010>>101010101>01010101010101>0101010
11010101>01010101001010100101010>>101010101>01010
101101010101>0101010101010101>010101010010101001
01010>>101010101>01010101101010101>01010110101010
1>0101010101010100101010>>101010101>0101010101010
101>01010101101010101>01010101001010100101010>>10
010101>01010101101010101>01010101010101>010101000
1010100101010>>101010101>01010101010101>010101011
010101>01010101001010100101010>>101010101>0101010
1101010101>0101010101010101>01010101001010100101
010>>101010101>01010101101010101>010101101010101>
0101010101010100101010>>101010101>010101010101010
1>01010101101010101>01010101001010100101010>>1001
0101>01010101101010101>01010101010101>0101010
001010100101010>>101010101>01010101010101>0101010
11010101>01010101001010100101010>>101010101>01010
101101010101>0101010101010101>010101010010101001
01010>>101010101>01010101101010101>01010110101010
1>0101010101010100101010>>101010101>0101010101010
101>01010101101010101>01010101001010100101010>>10
010101>01010101101010101>01010101010101>010101000
1010100101010>>101010101>01010101010101>010101011
010101>01010101001010100101010>>101010101>0101010
1101010101>0101010101010101>01010101001010100101
010>>101010101>01010101101010101>010101101010101>
0101010101010100101010>>101010101>010101010101010
1>01010101101010101>01010101001010100101010>>1001
0101>01010101101010101>01010101010101>0101010
101010101>0101010101010101>01010101101010101>0101
001010100101010>>101010101>01010101010101>0101010
11010101>01010101001010100101010>>101010101>01010
101101010101>0101010101010101>010101010010101001
01010>>101010101>01010101101010101>01010110101010
1>0101010101010100101010>>101010101>0101010101010
101>01010101101010101>01010101001010100101010>>10
010101>01010101101010101>01010101010101>010101000
1010100101010>>101010101>01010101010101>010101011
010101>01010101001010100101010>>101010101>0101010
1101010101>0101010101010101>01010101001010100101
010>>101010101>01010101101010101>010101101010101>
0101010101010100101010>>101010101>010101010101010
1>01010101101010101>01010101001010100101010>>1001
0101>01010101101010101>01010101010101>0101010
101010101>0101010101010101>010101011010101010010
10100101010>>101010101>01010101010101>01010101101
0101>01010101001010100101010>>101010101>010101011
01010101>0101010101010101>0101010100101010010101
0>>101010101>01010101101010101>010101101010101>01
01010101010100101010>>101010101>0101010101010101>
01010101101010101>01010101001010100101010>>100101
01>01010101101010101>01010101010101>0101010001010
100101010>>101010101>01010101010101>0101010110101
01>01010101001010100101010>>101010101>01010101101
010101>0101010101010101>01010101001010100101010>>
DEVELOPMENT
OPERATIONS
PLANNING &
GOVERNANCE
AI-powered IT Business Analytics solutions
derived from multiple systems of record
A System of Intelligence for technology-driven organizations
Go beyond manual analysis and basic operational reports

5. 55
© Numerify, Inc. All Rights Reserved.
106
TIMES FASTER
lead time from
commit to deploy
208
TIMES MORE
frequent code
deployments
7
TIMES LOWER
Change failure rate
2,604
TIMES FASTER
Time to recover form
incidents
Source: “Accelerate: State of DevOps 2019” report by DORA
Benefits of a DevOps transformation

6. 66
© Numerify, Inc. All Rights Reserved.
Daily changes at Facebook
Source: Facebook Engineering blog
1,000+ code changes daily

7. 77
© Numerify, Inc. All Rights Reserved.
Levels of automation
Example from the auto industry
LEVEL 1 LEVEL 2 LEVEL 3 LEVEL 4 LEVEL 5 LEVEL 6
No Automation Driver Assistance Partial Automation Conditional Automation High Automation Full Automation
Manual
Steering
Blindspot
Warning
Automated
Parking
Lane-Keep
Assist
Autonomous
Parking
Highway
Assist
Highly
Automated
Driving
Autonomous
Driving
Self Driving Car
Source: US Department of Transportation, NHTSA

8. 88
© Numerify, Inc. All Rights Reserved.
Application to Change Management?
AI for decision support and automation
LEVEL 1 LEVEL 2 LEVEL 3 LEVEL 4 LEVEL 5 LEVEL 6
No Automation Driver Assistance Partial Automation Conditional Automation High Automation Full Automation
Manual
reports &
analysis
Manual
remediation
using
AI-powered
analytics
Automatically
identify root
cause & next
best action
Automate
simple tasks
(e.g., low risk
changes)
Automate
most tasks
(e.g., low &
medium risk
changes)
No Ops
Self-driving IT business services
AI for Decision Support AI for Decision Automation

9. 99
© Numerify, Inc. All Rights Reserved.
DEVELOPMENT DEPLOYMENT CUSTOMER EXPERIENCE
Software Development
• Test coverage
• Test results
• Production defects
+ ….
Build Automation
• Commits & rollbacks
• Build automation results
+ ….
Change Management
• Assignment group
• Change category
+ ….
CMDB
• Configuration item
• Dependencies
+ ….
Release Automation
APM
• Performance degradation
• Application error
+ ….
Incident Management
• Customer reported incident
• Autogenerated incident
+ ….
What makes change success hard?
Siloed Data captured in siloed systems

10. 1010
© Numerify, Inc. All Rights Reserved.
INCIDENT
CI
CHANGE
…
0100101010011010101
010010101001101011010100
0100101010011010101101
01001010100110101010101
01001010100110101
0100101010011010101101
0100101010011010101
01001010100110101011010
0100101010011010101101
010010101001101010101001
01001010100110101011
0100101010011010101101
Continuously uncover key risk
factors from historical data
Mitigate risks
of individual
changes
PREDICT
changes that
are most likely
to fail and why
ACT
to prevent
change failure
CORRELATE
root cause
analysis of
change failure
IMPLEMENT
improvements to
people, process,
technology
Remove systemic
causes of change
failure
ENRICH CHANGE PREDICTION DATA | MONITOR FOR NEW THREATS
A machine learning-driven approach to change success

11. © Numerify, Inc. All Rights Reserved.
1111
IDENTIFY
Understand what is
going on and identify
problems
What is the change
failure rate by CI type,
change type,
assignment group?
INVESTIGATE
Investigate which
problems are
occurring and why
Are the most
experienced change
implementers
concentrated in a few
teams?
CORRELATE
Find root cause,
discover hidden
relationships
Are there factors in
dev & test that may be
indicative of failed
production changes?
PREDICT
Prescribe actions
and prevent
problems before
they occur
What is the
probability that a
change will cause a
business-impacting
incident?
Change success requires
a comprehensive set of analytical capabilities

12. Change Credit Score
Drive desired change management behaviors through leadership accountability

13. Fortune 20 Healthcare company reduces Change related problems by 64%
67% CAB Meetings
Inefficient and ineffective
change governance
Change related
Problems
FROM
2.5 Hrs.
Use analytics to :
Reduce CAB meeting to 60 min
Calculate daily change credit score by team
Drive change accountability across IT
Help teams improve Change processes
TO

14. Step 1: Find your true Change Failures
Auto-Link Incidents to Change Root Cause

15. © Numerify, Inc. All Rights Reserved.
1515
15
Step 2: Predict which changes are most likely to fail…and why
COMMUNICATE SUCCESS
Track progress and gain
macro visibility into overall
change success metrics
IDENTIFY RISKY CHANGES
Predict risk of individual changes
and prioritize risk elimination or
mitigation efforts
MITIGATE OR ELIMINATE RISK
Sift through millions of
permutations and combinations of
risk factors and identify proactive
action

16. Top Financial Services Institution saves $1.4M annually
10%+ Changes monthly
Made it was hard to discern
problem patterns
Change failure rate
for legacy applications
FROM
10K
66% Annual SavingsChange Failure
reduction
by addressing change risk factors
TO
$1.4M
16

18. Demo

19. 19
Parting Thoughts
Parting Thoughts

20. Questions

21. Demo screenshots
(backup)

22. 22
© Numerify, Inc. All Rights Reserved.

23. 23
© Numerify, Inc. All Rights Reserved.

24. 24
© Numerify, Inc. All Rights Reserved.

25. 25
© Numerify, Inc. All Rights Reserved.