SlideShare a Scribd company logo
1 of 78
Download to read offline
Cognitive
Apprenticeship in
Action
Paige Cruz
Agenda
01 I’m on-call for WHAT?!
02 Alert Triage Hour of Power
03 Findings
04 Cognitive Apprenticeship
05 Outcomes
I am on-call
for WHAT!?
01
Read the
docs
*
On-call Onboarding Before
Read the
docs
Pair with
buddy
*
*
On-call Onboarding Before
Read the
docs
Pair with
buddy
Shadow
Primary
* *
*
On-call Onboarding Before
Read the
docs
Pair with
buddy
Shadow
Primary
* *
* *
On-call Onboarding Before
Reverse
Shadow
Read the
docs
Pair with
buddy
Shadow
Primary
* *
* *
On-call Onboarding Before
Reverse
Shadow
🎉
�
�
On-Call Knowledge Areas
System Telemetry Business
Context
The System
● Application
● Infrastructure
● Release Process
● Architecture
Available Telemetry
● Traces? Events?
Logs?
● Instrumentation
● Querying
● Tags/Attrs
Business Context
● Tool sprawl
● Alert hygiene
● Who owns what
Sarah H.
Staff SWE
Alert Triage
Hour of Power
02
“[Alert Triage] is the most
valuable meeting on my
calendar” - paigerduty
Alert Triage Hour of Power
Meeting Intention &
Roles
10m
Investigate alert 40m
Wrap up 10m
Facilitator
Scribe
Roles
Driver
Support
Facilitator
Select alert
Stay on track
Protect Recap
Set Tone
Driver
Advocate for your
learning needs
Externalize thought
process
Ask for guidance
Scribe
Capture learnings
Ask for clarification
Sharing is caring
Support
Differing perspectives
Cheerleaders
Agenda
Meeting Intention &
Roles
10m
Set Up
Alert Triage Hour of Power
Meeting Intention &
Roles
10m
Investigate alert 40m
ACK the “page”
Verify then trust the alert
Investigate
Alert Triage Hour of Power
Meeting Intention &
Roles
10m
Investigate alert 40m
Wrap up 10m
TUNE
KEEP DELETE
Alert Recommendations
Alert
Triage:
Support
x2
Alert
Triage:
Driver
*
*
On-call Onboarding After
Alert
Triage:
Support
x2
Alert
Triage:
Driver
Pair with
buddy
* *
* *
On-call Onboarding After
Shadow
🎉
*
Reverse
Shadow
Lessons
Learned
03
Alerts are
not precious!
- apk
P
R
O
D
STAFF
SWE
Sr SRE
Findings
Alerts are not
precious
Findings
Active listening is
tricky
Alerts are not
precious
Findings
Learning is a worthy
goal
Active listening is
tricky
Alerts are not
precious
Findings
Learning is a worthy
goal
Alert triaging is not an
innate skill
Active listening is
tricky
Alerts are not
precious
Findings
Learning is a worthy
goal
Production can
surprise all levels
Alert triaging is not an
innate skill
Active listening is
tricky
Alerts are not
precious
Cognitive
Apprenticeship
04
!mentorship
Apprentice -> Expert
Modeling Coaching Scaffolding Articulation Reflection Exploration
CREDITS: This presentation template was created by Slidesgo,
including icons by Flaticon and infographics & images by Freepik
Modeling
CREDITS: This presentation template was created by Slidesgo,
including icons by Flaticon and infographics & images by Freepik
Coaching
CREDITS: This presentation template was created by Slidesgo,
including icons by Flaticon and infographics & images by Freepik
Scaffolding
CREDITS: This presentation template was created by Slidesgo,
including icons by Flaticon and infographics & images by Freepik
Articulation
CREDITS: This presentation template was created by Slidesgo,
including icons by Flaticon and infographics & images by Freepik
Reflection
CREDITS: This presentation template was created by Slidesgo,
including icons by Flaticon and infographics & images by Freepik
Exploration
Alert Triage
Modeling
Alert Triage
Modeling Coaching
Alert Triage
Modeling Coaching Scaffolding
Alert Triage
Modeling Coaching Scaffolding Articulation
Alert Triage
Modeling Coaching Scaffolding Articulation Reflection
Alert Triage
Modeling Coaching Scaffolding Articulation Reflection Exploration
Outcomes
05
Real Results*
runbook coverage
reliability
unactionable alerts
uptime
100%
-50%
∞
1000%
*jk
Real Results*
runbook coverage
reliability
unactionable alerts
uptime
100%
-50%
∞
1000%
*jk
IC meets VP: Explaining Incident Mgmt
Mr. VP Mr. IC
Mr. VP Mr. IC
We invested a lot in
Incident Management
last quarter and
incidents went up... Is
that expected?
Mr. VP Mr. IC
It’s not that we’re finding
more incidents, it’s really
more that we’re trying to
handle them better and the
investment really helped the
team to do that!
Mr. VP Mr. IC
Really? Here it says
that not only did we
have more incidents
but they also lasted
longer on average
What’s the ROI of Alert Triage Hour of Power?
Really Real Results
Regular attendees
Expanded to PM & Designers
Years strong
15-20
EPD
3
Reduction in spammy alerts
0%
CREDITS: This presentation template was created by Slidesgo,
including icons by Flaticon and infographics & images by Freepik
Thanks!
@paigerduty@hachyderm.io
paigerduty
paigerduty.com
CREDITS: This presentation template was created by Slidesgo,
including icons by Flaticon and infographics & images by Freepik
Booth
#214 ⚡
⚡
⚡

More Related Content

Similar to SRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of Power

Mastering Microsoft 365: The Winning Trio Of Automation, Governance & Adoption
Mastering Microsoft 365: The Winning Trio Of Automation, Governance & AdoptionMastering Microsoft 365: The Winning Trio Of Automation, Governance & Adoption
Mastering Microsoft 365: The Winning Trio Of Automation, Governance & Adoption
Richard Harbridge
 

Similar to SRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of Power (20)

Jira Service Desk for Internal Developer Support: It’s Not Just for IT Anymore!
Jira Service Desk for Internal Developer Support: It’s Not Just for IT Anymore!Jira Service Desk for Internal Developer Support: It’s Not Just for IT Anymore!
Jira Service Desk for Internal Developer Support: It’s Not Just for IT Anymore!
 
Agile is Dead :: Aginext London 2018
Agile is Dead :: Aginext London 2018Agile is Dead :: Aginext London 2018
Agile is Dead :: Aginext London 2018
 
Learnmystuff - Training Catalog
Learnmystuff - Training CatalogLearnmystuff - Training Catalog
Learnmystuff - Training Catalog
 
Tech Due Diligence from CTO's perspective - Talk at code.talks commerce
Tech Due Diligence from CTO's perspective - Talk at code.talks commerceTech Due Diligence from CTO's perspective - Talk at code.talks commerce
Tech Due Diligence from CTO's perspective - Talk at code.talks commerce
 
O'Reilly SACon 2019 - (Continuous) Threat Modeling - What works?
O'Reilly SACon 2019 - (Continuous) Threat Modeling - What works?O'Reilly SACon 2019 - (Continuous) Threat Modeling - What works?
O'Reilly SACon 2019 - (Continuous) Threat Modeling - What works?
 
Charting a Career in Information Security - August 2020
Charting a Career in Information Security - August 2020Charting a Career in Information Security - August 2020
Charting a Career in Information Security - August 2020
 
Agile Development From A Developers Perspective
Agile Development From A Developers PerspectiveAgile Development From A Developers Perspective
Agile Development From A Developers Perspective
 
S360 2015 dev_secops_program
S360 2015 dev_secops_programS360 2015 dev_secops_program
S360 2015 dev_secops_program
 
SplunkLive! Munich 2018: Intro to Security Analytics Methods
SplunkLive! Munich 2018: Intro to Security Analytics MethodsSplunkLive! Munich 2018: Intro to Security Analytics Methods
SplunkLive! Munich 2018: Intro to Security Analytics Methods
 
Are you Ready to Rumble? Let's Migrate Some Jira Data
Are you Ready to Rumble? Let's Migrate Some Jira DataAre you Ready to Rumble? Let's Migrate Some Jira Data
Are you Ready to Rumble? Let's Migrate Some Jira Data
 
SDLC & DevSecOps
SDLC & DevSecOpsSDLC & DevSecOps
SDLC & DevSecOps
 
Agile is Dead :: Pixels Camp 2017
Agile is Dead :: Pixels Camp 2017Agile is Dead :: Pixels Camp 2017
Agile is Dead :: Pixels Camp 2017
 
Microservices, Microfrontends and Feature Teams
Microservices, Microfrontends and Feature TeamsMicroservices, Microfrontends and Feature Teams
Microservices, Microfrontends and Feature Teams
 
Data models pivot with splunk break out session
Data models pivot with splunk break out sessionData models pivot with splunk break out session
Data models pivot with splunk break out session
 
(SEC402) Enterprise Cloud Security via DevSecOps 2.0
(SEC402) Enterprise Cloud Security via DevSecOps 2.0(SEC402) Enterprise Cloud Security via DevSecOps 2.0
(SEC402) Enterprise Cloud Security via DevSecOps 2.0
 
Mastering Microsoft 365: The Winning Trio Of Automation, Governance & Adoption
Mastering Microsoft 365: The Winning Trio Of Automation, Governance & AdoptionMastering Microsoft 365: The Winning Trio Of Automation, Governance & Adoption
Mastering Microsoft 365: The Winning Trio Of Automation, Governance & Adoption
 
Lead Time: What We Know About It...
Lead Time: What We Know About It...Lead Time: What We Know About It...
Lead Time: What We Know About It...
 
Dont wait what 300 ld leaders have learned about building data fluency
 Dont wait what 300 ld leaders have learned about building data fluency Dont wait what 300 ld leaders have learned about building data fluency
Dont wait what 300 ld leaders have learned about building data fluency
 
Beyond the Hack
Beyond the HackBeyond the Hack
Beyond the Hack
 
"Threat Model Every Story": Practical Continuous Threat Modeling Work for You...
"Threat Model Every Story": Practical Continuous Threat Modeling Work for You..."Threat Model Every Story": Practical Continuous Threat Modeling Work for You...
"Threat Model Every Story": Practical Continuous Threat Modeling Work for You...
 

More from Paige Cruz

More from Paige Cruz (14)

Power Up with Podman - Kubernetes Community Day LA
Power Up with Podman - Kubernetes Community Day LAPower Up with Podman - Kubernetes Community Day LA
Power Up with Podman - Kubernetes Community Day LA
 
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
 
OTel Orientation: How to Train Teams (OTel in Practice)
OTel Orientation: How to Train Teams (OTel in Practice)OTel Orientation: How to Train Teams (OTel in Practice)
OTel Orientation: How to Train Teams (OTel in Practice)
 
Avoiding Alert Bankruptcy and Burnout
 Avoiding Alert Bankruptcy and Burnout Avoiding Alert Bankruptcy and Burnout
Avoiding Alert Bankruptcy and Burnout
 
Tracing Adventures from PR - Production
Tracing Adventures from PR - ProductionTracing Adventures from PR - Production
Tracing Adventures from PR - Production
 
Threat Modeling in the Cloud
Threat Modeling in the CloudThreat Modeling in the Cloud
Threat Modeling in the Cloud
 
There's No Place Like Production
There's No Place Like ProductionThere's No Place Like Production
There's No Place Like Production
 
Taming Feral DevOps
Taming Feral DevOps Taming Feral DevOps
Taming Feral DevOps
 
Pushing Observability Uphill - The Single “Pain” of Glass
Pushing Observability Uphill - The Single “Pain” of GlassPushing Observability Uphill - The Single “Pain” of Glass
Pushing Observability Uphill - The Single “Pain” of Glass
 
Power Up with Podman
Power Up with PodmanPower Up with Podman
Power Up with Podman
 
Intro to Instrumentation
Intro to InstrumentationIntro to Instrumentation
Intro to Instrumentation
 
From Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation
From Cardinal(ity) Sins to Cost-Efficient Metrics AggregationFrom Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation
From Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation
 
99.9% of Your Traces are Trash
99.9% of Your Traces are Trash99.9% of Your Traces are Trash
99.9% of Your Traces are Trash
 
3rd Wave Observability: Open or Bust
3rd Wave Observability: Open or Bust 3rd Wave Observability: Open or Bust
3rd Wave Observability: Open or Bust
 

Recently uploaded

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
Earley Information Science
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 

Recently uploaded (20)

Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 

SRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of Power