The worst time to discover your processes are totally broken? During a live incident. Avoid incident management flops by running regular simulations that actually reflect a realistic incident environment. I'll teach you how. Palms are sweaty, knees weak, arms are heavy...sound like your first on call shift? One of the biggest challenges in incident response work, especially for newer SREs, is the lack of safe spaces to fail. Incident simulations can be an effective way to take the terror out of that first on-call shift, but they take careful planning. In this talk, I’ll explore different types of simulations (from tabletops to full-on realistic mock incidents), how and when to utilize them, and how to make sure you get the most out of them. Ashley Sawatsky is an expert in incident management and communication, with a focus on the SaaS world. In her 6+ years of experience building and scaling Shopify's incident response program, she developed the ability to fluently translate the technical aspects of SRE incident response to Legal, PR, Customer Support, and Executive stakeholders. Now, she works at Rootly as Senior Developer Relations Advocate, where she engages with the SRE and incident response communities, and consults with customers from the world's largest tech companies—like Canva, Figma, NVIDIA, and more—on their incident response strategies.

1. Fake It ‘Til
You Make It
How to get the most out of your incident simulations
Ashley Sawatsky
Reliability Advocate

2. Hey, I’m Ashley 👋
➔ Managed guest complaints /
escalations at Disney
➔ Joined Shopify in 2016
➔ Founding member of Shopify’s
Incident Response team, went on to
build and lead Incident
Communications
➔ Joined Rootly this year where I get
to help other organizations level up
their incident management
practices!

3. Obligatory photo of my pug, Whiskey.

4. What is an incident simulation?
An exercise in which you enact a fake (or simulated) incident
scenario as a means of preparing for the real thing.

5. Plan Build Run Reflect Improve

6. We already have real incidents,
why do we need fake ones?

7. Simulations
create a
safe place
to learn,
experiment,
and fail.

8. In an incident
simulation, your goal
is to fail.

9. ➔ Single points of failure
➔ Gaps in process
➔ Bad process
➔ Communication breakdowns
➔ Misalignment
➔ Broken escalation paths
➔ Slowdowns
👀 You’re looking for…

10. So we’re setting
people up for failure?
Isn’t that kind
of...terrible? 🤔

11. Setting the
right tone
starts
before the
simulation
begins.

12. Ground rules for
planning your
incident
simulation

13. This is
not a
performance
exercise.
1 Incident simulations shouldn’t feel
like a test for employees. Make it
clear to everyone participating that
the only consequence of making
mistakes during the simulation is
learning.

14. Be
transparent.
2 People should know they’re
participating in a simulation that is
intended to challenge them and
expose weaknesses in the incident
response process.

15. Make it
feel real.
3 The environment should be as
realistic and close to the actual
experience of running an incident
as possible.

16. Don’t stop
the clock.
4 We’ll get into the mechanics and
how to handle it when someone
needs to step “outside” of the
simulation soon. This will be done
in a way that doesn’t derail the
overall scenario.

17. Save
feedback for
the retro.
5
Encourage participants to avoid
breaking the “fourth wall”. No “next
time we run one of these we
should…” while the simulation is
active.

18. Make it fun.
Let the participants expense their
lunch that day, or send a fun gift
like a relaxing scented candle, a
specially designed sticker, etc.
6

19. Building the Scenario

20. Pick
something
unfamiliar.
1
Pull inspiration from past incidents,
but don’t replicate them exactly.

21. Don’t go
nuclear.
2
Scenarios that are too severe are
likely to be escalated to VP level or
higher. For crisis exercises, bring in
a neutral third-party.

22. Work in some
curveballs.
3
Think of your scenario as dynamic,
like a choose-your-own-adventure.
Have some curveballs ready to
throw to keep things interesting.

23. Keep timing
in mind.
4
You want to gauge how close your
expected vs. actual time to
resolution is.

24. Define
“done”.
5
Determine at what point will you
consider the incident resolved
before the timer runs out.

25. Choosing your
Participants

26. Facilitator
➔ Can be the organizer or someone else
➔ Ensures the simulation runs smoothly
by thinking on their feet
➔ Welcomes participants
➔ Available for questions from
participants
➔ Notifies participants when simulation
is over
➔ Runs post-simulation retrospective

27. Responders
➔ Identify responders for every incident
role
➔ Don’t default to your strongest
responders
➔ May be new to on-call or in need of
practice to build confidence
➔ Work with leads to identify who would
most benefit

28. Observers ➔ Your strongest incident responders
from each response team (SRE,
Communications, Legal, etc)
➔ Experts in your organization’s incident
response practices
➔ Silent observers throughout the
simulation
➔ Noting opportunities for improvement

29. Scribe
➔ Documents key moments, decisions,
observer discussion points, and
responder actions
➔ Prepares timelines for retrospective
➔ Supports the facilitator as needed

30. Constructing your
Environment

32. Clearly mark any
sandbox
environments to
avoid panic or
confusion

33. The
Viewing
Room
➔ Video call where the facilitator,
observers, and scribe will hang out
during the simulation
➔ Discuss how things are going, adjust
as needed
➔ Note any observations to discuss in
the retrospective
➔ Scribe records key points

34. The Big Day

35. 1. Hold the welcome meeting
2. Initiate the simulation
3. Observe and note
4. End the simulation
5. Take a break
6. Reflect
Run of Show

36. The Retro

37. The Good
The Bad
The Ugly
You can find existing
templates for this in Miro
or FunRetrospectives!

38. Action items need an
action plan.
✓ Owner
✓ Timeline
✓ Tracking

39. The Gift of Feedback

40. Manage incidents the
easy way with Rootly
Rootly brings your entire incident response tooling suite together so you can manage incidents
from start to finish right from Slack.
Create automated workflows that are completely customized to your incident response
process to handle tasks like paging, escalation, surfacing playbooks, syncing action items to
Jira and other tooling, updating your status page, and more.
Want to see it in action? Come find me between talks!
Trusted by