Real-world incident response, management, and prevention

| @ema_research
Real-world incident response
(management, and prevention)
Valerie O’Connell
Research Director - Digital Service Execution
Enterprise Management Associates (EMA)
Sponsored by

| @ema_research 2
Watch the On-Demand Webinar
• Real-world incident response, management, and prevention on-
demand webinar: https://info.enterprisemanagement.com/incident-
response-managemen-prevention-webinar-ss
• Check out upcoming webinars from EMA here:
https://www.enterprisemanagement.com/freeResearch
© 2023 Enterprise Management Associates, Inc.

| @ema_research
| @ema_research
Featured Speaker
Valerie O’Connell
Research Director
Digital Service Execution
Valerie O’Connell leads the Digital Service Execution
practice at Enterprise Management Associates (EMA).
Her practice encompasses intersections and innovations
across AIOps, asset management, end-user experience,
ITSM/ESM, and business context as they interact to
deliver excellence in digital service.
Valerie came to EMA with decades of senior-level
experience in the effective marketing of technology. Her
experience ranges from VP of product marketing at
what was then CA to a successful run as an independent
practitioner, serving industry giants such as Microsoft and
EMC, as well as cutting-edge startups.
© 2024 Enterprise Management Associates, Inc. 3

| @ema_research
| @ema_research
Sponsors

| @ema_research
| @ema_research
Incidents and outages:
some organizations are fighting a losing battle …
… others are taming the beast
An unsurprising 82% of an early research panel characterized incidents
and outages as increasing every year (19% stated that “increased reliance
on IT guarantees continued increases”)
A fascinating 18% of the panel ran countertrend, stating that incidents and
outages have decreased due to proactive systems they’ve put in place
New research: current/desired state of incident management processes, incidents, automation, AI …. )
• 50% North America; 25% EMEA; 25% APAC
• Even mix of IT service, ITOps, and IT security
• 27% IT professionals/team lead, 47% manager/director, and 26% IT executive VP/CIO
• Even distribution of company sizes: 1,000 – 10,000+ employees
• Mix of industries: high tech, manufacturing, financial services, retail
• Automation as a priority was a qualification

| @ema_research
| @ema_research
If you could choose one thing to do really well,
what would have the biggest positive impact?
“Preventative monitoring is worth more than incident
response after the fact, no matter how good it might be”
“Be proactive rather than reactive. We seem to wait until
things happen before we do anything to stop it.”
“I would like to see more automation and integration of
the tools used. That would help reduce the manual effort
and human error that is typical of the incident process.”
“If I could improve one thing about my organization’s
approach to incident response it would be to try to be a
bit bolder – a bit more unconventional and try out more
innovative methods using AI and automation.”
If you could improve one thing about your
organization’s approach to incident response,
effectiveness, or tools used, what would it be?
Also mentioned: better training, fewer levels of bureaucracy, knowledge sharing,
workflow automation, and increased headcount

| @ema_research
| @ema_research
The reality vs the wish list
What percentage of incidents are caught before causing an outage/user impact?
Only 18% successfully intercept more than half of incidents before they impact users.
All of that group report AIOps as strategic, mature, and implemented on an enterprise-wide platform

| @ema_research
| @ema_research
A fundamental disconnect ….. The business view and IT’s view
How does your organization define an IT service?

| @ema_research
| @ema_research
The disconnect is driving organizational change
Although oversimplified, how would you describe your IT’s organizational principle when it comes
to service availability/performance and incident management?

| @ema_research
| @ema_research
Process definition/documentation – process effectiveness
How would you rate the effectiveness of your
organization’s incident management processes?
Which statement best describes your organization’s
incident management processes?
73% of respondents say that these processes are “widely used, well understood, and frequently updated”.

| @ema_research
| @ema_research
Does the incident management process change during an outage?
When the incident is due to cybersecurity issues, which statement best represents your organization?

| @ema_research
| @ema_research
Detection/identification and logging
Categorization, triage, prioritization, and routing
Team engagement, incident communication and
collaboration
Response for analysis, diagnosis, and resolution
Closure and post-incident reporting
A common incident response process flow
Understanding that every organization is different, does the following incident response flow reflect the incident
management flow in your organization
76% said it is a typical flow
24% responded, “Our flow is more ad hoc – it depends on the incident”
Ad hoc had
• Lower level of: platform use, effectiveness, use of automation and AI, IT quality
• Higher level of: incidents coming in through user complaints …
Not one respondent said “no.”

| @ema_research
| @ema_research
Which steps or phases are the most challenging or most in need of improvement? (select two)

| @ema_research
| @ema_research
Which phase is most time consuming – biggest contributor to MTTR? (select one)

| @ema_research
| @ema_research
Which phase is the least important (gets skipped, overlooked, or only done slightly)?
Note: team engagement still gets honorable mention

| @ema_research
| @ema_research
Automation and AI process observations
Most automated phase is detection;
Least automated is closure followed by team engagement
AI/ML most used or applicable in detection and response phases
AI and automation use projected to grow …
…. but in the areas already used
Underserved processes are not on the investment radar

| @ema_research
| @ema_research
Incident prioritization
Slide
19
Does your organization prioritize IT incidents
by severity?
51% Business operations are disrupted
39% Compliance or regulations are at risk
38% Number of users impacted
36% External customers are involved
25% Revenue is at stake
11% SLA penalties
What determines the priority of an
incident? (select two)
An incident is any degradation in IT
service or availability
35% specifically include predicted or potential issues
28% specified disruption to business operations

| @ema_research
| @ema_research
Incident basics
How are incidents most often surfaced? Which statement best describes incident tracking in your
organization? “Incidents can originate from users or monitoring
systems, but they are created and tracked primarily through…”
36% A centralized ITOps or AIOps system or platform
31% Tickets in the service desk or ITSM system
31% A mix of ITSM tickets and ITOps logs
2% Decentralized DevOps/SRE “you build it, you run it”
When there is an incident, what groups are most responsible for identifying and fixing the cause? (select two)
54% ITOps 49% ITSM/service
28% DevOps 27% Engineering/dev 24% Network 15% SRE/agile teams

| @ema_research
| @ema_research
The “T” in MTTR (whatever your “R” is …. Reducing it is universally a top IT objective)
Once an incident is logged, how long does it usually take to resolve it?
70% 1-4 hours
19% more than 4 hours
11% minutes

| @ema_research
| @ema_research
MTTR’s “secret” ingredient
The least wasted time = mature
AIOPs and most AI/automation
What percentage of the MTTR is inactive time
spent waiting for information or response?
27% MTTR is 50% or more
wasted time
42% MTTR is 25% waste
19% MTTR is 10% waste
12% almost no wasted time

| @ema_research
| @ema_research
How long does it take to identify and engage the right response
teams from the time an incident is created?
Which phase is most time consuming – biggest contributor to MTTR? (select one)
Reminder: MTTR after logging
70% 1-4 hours
19% more than 4 hours
11% minutes
30% Team engagement/
communication
28% Response/resolution 25% Categorization/
prioritization/routing
Reducing time to identify and engage team looks like low-hanging fruit

| @ema_research
| @ema_research
Lack of information/insight increases MTTR
What are the biggest challenges to effective incident response and management? (select two)

| @ema_research
| @ema_research
Actionable incidents/alerts cut MTTR
On average, what percentage of incidents/events/alerts are actionable (turn out to be a problem
that requires a resolution and includes at least one piece of insight on how to respond)?
Most teams are reactive: 65% rapid and reactive 35% shift toward proactive and agile teams

| @ema_research
Automation and AI
26

| @ema_research
| @ema_research
How is predictive AI insight used for proactive action in your organization?
Note: automated proactive action increasing year over year

| @ema_research
| @ema_research
Automation and humans
What is the long-term goal for automation in incident management?
ITOps has primary responsibility for automation used in incident response
followed by cybersecurity, ITSM/service and DevOps

| @ema_research
| @ema_research
The state of automation and incident management
Without exception, the mature automation
group greatly out-performed the other
cohorts in all incident response metrics
including MTTR, reduction in incidents
and outages, cost, effectiveness, IT
productivity, and use of AI.
56% automation is a mature,
C-level strategic initiative
28% early
16% departmental
Mature, C-level:
• AIOps tends to be a strategic enterprise initiative
• Platform use predominates as opposed to siloed systems and tools
• There is a high level of well-defined/documented processes that are widely used
• Far fewer incidents come in through user complaints (24% vs 53% for lower automation priority)
• 73% of organizations that have automation as a C-level priority have reorganized to take
advantage of AI and automation.

| @ema_research
| @ema_research
If IT automation is “software that can replace repeatable processes, address complex processes,
and take critical actions,” what percentage of the incident response and management tasks or
processes do you estimate use automation in your organization?

| @ema_research
| @ema_research
What’s preventing your organization from adopting automation more broadly
for incident response and management? (select two)

| @ema_research
| @ema_research
Incident automation: drivers and results delivered = close map
Top drivers of automation in incident response
and management What are the top two benefits achieved so far through
incident management automation?
Top metrics to measure investments
in incident management automation (in order) :
Reduction in:
1 MTTR, 2 events and incidents, and 3 downtime ;
4 Security and compliance metrics,
Others: SLA performance, reduction in escalations
(L2, L3), reduction in the number of trouble tickets, and
mean time to assemble response team.
93% plan increased investments in incident automation
(43% very high increase/strategic)
Reduced MTTR and
downtime
Improved user experience
IT personnel productivity
Business impact/revenue
Cost savings
Cross-functional workflows

| @ema_research
| @ema_research
How would you characterize the current use
of AI/ML and analytics in your organization’s
incident management processes?
Mature AI group:
• Higher degree of automation in incident management
• Incident management effectiveness 55% very effective vs
21% early
• More autonomous and proactive actions
• A high percentage of incidents caught before an outage
or user impact.
• 34% report actionable alerts at 75% or more vs 11% early
• Use AI in change management for incident response as
well as for pre-change impact.
• More likely to be in pilot with genAI and more aggressive
in a timeframe for use in production.
AI in incident response and management lags behind automation

| @ema_research
| @ema_research
GenAI in incident response and management
What is your personal perception of GenAI?
“I think GenAI is …”
Interest is high in 95%
38% in research phase 57% in PoC or some form of
production
Timeframe
27% months
31% a year
25% 18 months+/-
12% 2 years or more
The main reason to delay implementation
29% regulation, data
privacy, and
compliance
24% security concerns
13% newness and unproven
value/accuracy
12% cost

| @ema_research
| @ema_research
Some hypothetical use cases for GenAI
If a real-time solution could
accurately determine, in
seconds, the impact of incidents
across distributed systems and
communicate it in clear
language, the value would be…
If a real-time solution could
generate an accurate summary
of alerts and incidents that
includes incident title,
description, and possible/likely
root causes in seconds, the value
would be…
If Knowledge Base articles could
be automatically and accurately
generated/ updated and easily
searched with normal, everyday
language, what percentage of
incidents, tickets, and cases
might be deflected?
38% transformative – priority actions
could be instantly identified at
speed and scale
45% high value – all impacted
response teams could be
immediately identified
17% valuable – it would increase the
effectiveness of incident
response
27% transformative – It would cut
MTTR by at least 20-30 minutes
per incident
55% High value – It would save
10-20 minutes per incident
11% Valuable – It would save 5-10
minutes per incident
7% Useful – it would save time
25% more than half
41% as much as half
26% between 10% - 25%
8% up to 10%

| @ema_research
| @ema_research
You can’t just throw technology at incidents …
How has your organization changed to improve incident management effectiveness?

| @ema_research
Concluding thoughts
37

| @ema_research
| @ema_research
Where we began -- outage frequency, cost, and duration
• An unsurprising 82% characterized incidents and outages as increasing every year
(19% stated that “increased reliance on IT guarantees continued increases”)
• A fascinating 18% stated that incidents and outages have decreased due to
proactive systems they’ve put in place – 100% had enterprise-level, mature AIOps
Automation, AI, and platform use are essentials for incident
response and management
• Explore new uses of automation and AI in underserved processes to
slash MTTR. (categorization and routing, team engagement, and
collaboration; closure and post-incident reporting}
• Become more bold in use of predictive/proactive incident interception
• Think of incident automation as a competitive advantage
(or disadvantage)
• Technology is essential, but it isn’t enough – organize for automation
and collaboration

| @ema_research
| @ema_research
Current/desired state of incident management:
processes, challenges, organization, automation, AI and GenAI
50% North America; 25% EMEA; 25% APAC
Even mix of IT service, ITOps, and IT security
Mix of industries: high tech, manufacturing, financial services, retail
Automation as a priority was a qualification

| @ema_research
| @ema_research
Check out the sponsor sites to learn more!

Real-world incident response, management, and prevention

Recommended

Recommended

More Related Content

Similar to Real-world incident response, management, and prevention

Similar to Real-world incident response, management, and prevention (20)

More from Enterprise Management Associates

More from Enterprise Management Associates (20)

Recently uploaded

Recently uploaded (20)

Real-world incident response, management, and prevention