The Top Outages of 2023: Analysis and Takeaways

Featured speakers
Mike Hicks
Principal Solutions Analyst
© 2024 Cisco Systems, Inc. and/or its affiliates. All rights reserved.

3
Before We Begin…
• If you have any questions, please type them in the Questions window.
• If you have any audio problems, please chat us for help.
• A recording of this presentation will be sent to you in a few days.
• Interested in more outage analysis and Internet insights? Check out the ThousandEyes
blog and The Internet Report podcast.

Anatomy of an Outage
• Understanding different types of
Internet outages is important to
mitigate their impact.
• Outages can vary in blast radius, be
planned or unplanned, and have
varying MTTR.
• Network outages depend on where
the problem occurs, with transit
network incidents impacting multiple
providers.
• Tracking outages can help teams
identify patterns and prevent
customer service disruptions.

5
Outage and Degradation Impacts
BGP
ISP
CDN
DNS SaaS Apps
Services
APIs
Data Center
Cloud
DDoS
Protection
SSE
RISK AND
COMPLIANCE
Is our traffic
getting routed
out of region?
SERVICE
AVAILABILITY
Which cloud
regions are
impacted?
SITUATIONAL
AWARENESS
Are regional
ISPs spoofing
our DNS
records?
SERVICE
RECOVERY
Did we
successfully
cut over to
our DDoS
mitigation
service?
NETWORK
SECURITY
Are SASE routing
policies working
as we expect?
CUSTOMER
SUPPORT
Is an Internet
outage
preventing users
from reaching our
service?
WORKFORCE
PRODUCTIVITY
Will our Salesforce
dev updates
degrade
performance
for some global
users?
$32,000
$120,000
$3,500
3474
REVENUE
PROTECTION
Is the payment
gateway down
or just
unreachable?

2023 Outages by the Numbers: ISP Compared to CSP
• ThousandEyes reported an increase in cloud service provider (CSP) outages in 2023.
• CSP outages are the second most common type of disruption after ISP outages.
• The ratio of CSP outages to ISP outages increased in 2023.

2023 Outages by the Numbers:
U.S.-centric Outages in Relation to Global Outages
• U.S.-centric outages increased to 37% in 2023 from 34% in 2022.
• Smaller, contained outages are becoming more common.
• Localized outages have different impacts and require different responses compared to global outages.

2023 Outages by the Numbers: Application Outages
• The number and frequency of application outages have been on the rise over the past year.
• Application-related disruptions can have a bigger impact than network outages, though they are not as common.

9
Connections Are Complex
Branch
Office
Employee
BYOD
Corp devices
IOT
Cameras
and sensors
IoT
VDI
People, places,
and things
Edge
BYOD
Data Center
IOT
Core
network
Mobile
networks
Core
network
Peering
Access
networks
Wireless
network
Wireless
gateway
DNS
Cloud
and SaaS
Cloud
providers
Datacenter
infrastructure
Cloud
connectivity
Direct
connect
ISP transit
providers
SaaS
onramp

10
Correlate Performance Across Every Layer
8
3
9
3
5
4
6
8
6
Time Correlated

Microsoft
(1/25)
Outlook
(2/7)
Virgin Media
(4/4)
AWS
(6/13)
Slack
(8/2)
Square
(9/8)
Workday +
Cloudflare (11/2)
2023 Outage Timeline
Purple = Application Outage
Red = Network Outage
Blue = Infrastructure Outage
Bookmark the Internet
Outages Timeline for outage
updates throughout the year.

Slack (Aug 2, 2023)
~2 hours
Unable to send/receive messages
AWS (Jun 13, 2023)
~2 hours
Latency, server timeouts, and HTTP errors
Virgin Media (Apr 4, 2023)
~7 hours
Network traffic loss/BGP route withdrawal
Microsoft Outlook (Feb 7, 2023)
~2 hours
Service unavailable/application errors
Microsoft 365 (Jan 25, 2023)
~90 minutes
Network issues due to BGP changes
#2
#3
#1
#4
#5
Square (Sept 8, 2023)
~12 hours
App errors and backend transactions failing
#6
Workday + Cloudflare (Nov 2, 2023)
~36 hours
Application and service outages
#7

Microsoft 365 (1/25/23)
• Microsoft started experiencing service related issues around
07:05 AM (UTC).
• The disruption was triggered by an external BGP change
by Microsoft that impacted connected service providers
• Microsoft BGP prefixes were withdrawn completely
but then almost immediately re-advertised.
• Affected smaller (/24) prefixes and summary prefixes (/12).
• Cascading impact on global routing tables, causing
significant churn.
• Prefixes were either withdrawn or re-advertised to
transit providers.
• Large amount of packet loss were seen as well as
HTTP and DNS timeouts.
• Timeouts seen in the application “Response,” further
indicating the effect of the network on service availability.

Microsoft Outlook (2/7/23)
• Starting around 03:55 UTC,
Outlook became unavailable.
• Network path was working
properly, but ThousandEyes
observed elevated server
response timeouts and slow
page loading.
• Majority of the errors were
HTTP server timeouts,
indicating an application issue.
• Incident was mostly
concentrated in the U.S. and
lasted ~2 hours.

Virgin Media (4/4/23)
• From approximately 00:30 to 17:30 UTC,
two outages impacted the reachability of
Virgin Media UK’s network and services.
• The first incident began at approximately
00:30 UTC and appeared to coincide with
a series of BGP route withdrawals.
• Second incident was shorter, but the
networks experienced similar BGP and
reachability issues.
• Outages were overnight and due to the
repeat nature, could indicate
maintenance issues.

AWS (6/13/23)
• Outage impacted services within US-
EAST-1.
• Lasted two hours and increased latency,
server timeouts, and HTTP server errors
were observed.
• AWS console access was also affected,
making troubleshooting difficult.
• AWS confirmed the issue was due to a
capacity management subsystem failure.
• Organizations leveraging cloud services,
such as those offered by AWS, should be
aware of the relationships in their digital
ecosystem, regardless of whether those
relationships are services or networks.

Slack (8/2/23)
• Application outage that lasted from 4:01
PM to 6 PM (UTC).
• Network paths and accessibility were
unaffected.
• Initially could be seen as HTTP 500
errors and higher-than-normal page load
times.
• During the outage, users were unable to
upload files or share screenshots.
• Root cause—work on a “routine
database cluster migration”—that
accidentally reduced database capacity
to the point that it could not support a
regularly scheduled job running.

Square (9/8/23)
• Outage lasted over 18 hours.
• Backend issue that prevented
the platform from processing
payment transactions.
• Users reported various
problems, from terminal
connections dropping out, to
payments appearing to
complete but then not showing
up in business accounts.
• ThousandEyes observed
intermittent dropouts and 503
‘service unavailable’ errors.

Workday + Cloudflare (11/2/23)
• Cloudflare and Workday
experienced a major outage
due to multiple infrastructure
provider failures.
• DR resources took 6 hours to
come online and full resolution
took 36 hours.
• Initial cause was a partial mains
power outage at a Flexential
data center in Portland.
• Further generator and grid
failures resulted in a complete
power loss and ungraceful
shutdown.

Takeaways
• Understanding how your application works is important for quickly identifying failures and making
improvements.
• Just because your application is working doesn't mean it's functioning optimally.
• Knowing how all parts of the service work together is crucial for ongoing design and future optimizations.
• Improved visibility and operational optimizations can prevent outages and minimize their impact.
• Tracking different categories of outages and degradations over time can be helpful.

• Subscribe to our blog to keep up-to-date!
thousandeyes.com/blog/
• Tune in to The Internet Report Podcast.
https://www.thousandeyes.com/the-internet-report/
21
Next Steps
• New tutorial videos on our features
thousandeyes.com/resources/?cat=tutorial
• New Getting Started Guides
docs.thousandeyes.com/product-documentation/getting-started
Blog and
Podcast
Learning
Resources
Support
Community
• Still have questions? Ask us on the ThousandEyes
Support Community AMA: http://bit.ly/2023Outages

Q&A
22

23

The Top Outages of 2023: Analysis and Takeaways

Recommended

Recommended

More Related Content

Similar to The Top Outages of 2023: Analysis and Takeaways

Similar to The Top Outages of 2023: Analysis and Takeaways (20)

More from ThousandEyes

More from ThousandEyes (20)

Recently uploaded

Recently uploaded (20)

The Top Outages of 2023: Analysis and Takeaways