Incident Management Meets DevOps - Capital One | DevOps Enterprise Summit: Las Vegas 2019

1. Public
Incident Management meets DevOps
Bhavik Gudka
Director, Software Engineering
https://www.linkedin.com/in/bhavikgudka/
@BhavikGudka
Surya Avirneni
Lead Software Engineer
https://www.linkedin.com/in/suryaavirneni/
@SuryaAvirneni

2. Public
WhatisDevOps
DEFINITIONOF
DEVOPS
DevOpsisasetofsoftwaredevelopmentpracticesthatcombinessoftware
development(Dev)andinformationtechnologyoperations(Ops)toshortenthesystem’s
developmentlifecyclewhiledeliveringfeatures,fixes,andupdatesfrequentlyinclose
alignmentwithbusinessobjectives

3. Public
DevOps-What’sinitforyou?
ROLEOF
DEVOPS
✓ContinuousIntegration
✓ContinuousDeployment
IjustwanttoCode.Everythingelseshouldbe
automatic
Developer

4. Public
DevOps-What’sinitforyou?
ROLEOF
DEVOPS
✓Fastertimetomarket
✓Beatthecompetitors
✓Wowthecustomers
Whenwillourchangegolive?
Product
Manager

5. Public
WhataboutIncidentManagement?
ROLEOF
DEVOPS
HowdoesDevOpsapplytoproductionoutages?

6. Public
Whatarewegoingtotalkabout?
SCOPEOFTHIS
TALK
IncidentManagement
✓Definition
✓Lifecycle
✓Goals
✓Evolution
✓RoleofDevOps
Letstalk

7. Public
Definition
INCIDENT
MANAGEMENT
IloveincidentsbutonlywhenIamonvacation
Incidentmanagementisaprocessofreturningaservice
tonormalasquicklyaspossibleafteradisruption,ina
waythataimstocreateaslittlenegativeimpactonthe
businessaspossible
Generalthoughts
Boring,labour,stress,Ihateit,thanklessjob,
escalations,blameothers,wasteofdeveloper’stime,
Nighttimecalls,nolife,
IloveitwhenIamonvacation

8. Public
Lifecycle
INCIDENT
MANAGEMENT
0
41
2
3
Healthy
State
Incident
Start
Detect
Problem
Mobilize
Resources
Restore
Service
TimetoDetect
(TTD)
TimetoMobilize
(TTM)
TimetoResolve
(TTR)
MeanTime
BetweenFailures
(MTBF)
Raceagainsttime

9. Public
Goals
INCIDENT
MANAGEMENT
Returnaservicetonormalasquicklyaspossible
afteradisruption
✓Reducetimetorecover
-ReduceTTD,TTM,TTR
✓ReduceIncidentCount
Incident
Counts
Low
High
LowHigh
RecoveryTime
High
Severity
Incidents
Low
Severity
Incidents
LowIncident
Counts
HighIncident
Counts
GOAL
AVOID
TOLERATE
TOLERATE

10. Public
Stage1-Reactmanuallyonthego
EVOLUTIONOFINCIDENTMANAGEMENT
Weneedyouonbridgetoresolve
anissue
Iamatmoviehall,willjoinin30mins.
OperationsDevelopment
Process
✓Manualmonitoring
✓ManualMobilization
✓OnthegoRemediation

11. Public
Stage1-Reactmanuallyonthego
EVOLUTIONOFINCIDENTMANAGEMENT
Whereareyou?
20moremins.Almostthere
Result
✓SlowDetection
✓Slowmobilization
✓Slowremediation
30minslater

12. Public
Stage2-ReactmanuallyusingsomeRunbook
EVOLUTIONOFINCIDENTMANAGEMENT
Ihaveadoubtwiththis
runbook
Iamatapub,canthiswait?
OperationsDevelopment
Process
✓Automatedmonitoring
✓ManualMobilization
✓PlannedRemediation

13. Public
Stage2-ReactmanuallyusingsomeRunbook
EVOLUTIONOFINCIDENTMANAGEMENT
Thatfixworked.Whywas
runbookoutdated?
Noidea.Iamnewtothisteam.
Result
✓FastDetection
✓Slow/fastmobilization
✓Slow/fastremediation
Fewminslater

14. Public
Stage3-Reactusingascript(automatedrunbook)
EVOLUTIONOFINCIDENTMANAGEMENT
Icannotdownloadthescript
What?Why?
OperationsDevelopment
Process
✓Automatedmonitoring
✓ManualMobilization
✓AutomatedRemediation

15. Public
Stage3-Reactusingascript(automatedrunbook)
EVOLUTIONOFINCIDENTMANAGEMENT
Ididnothavenewurlofthe
repo
Ok
[Inthehead-Ihateyou]
Result
✓FastDetection
✓Slow/fastmobilization
✓Slow/fastremediation
Fewminslater

16. Public
DevOpswaytodoIncidentManagement
EVOLUTIONOFINCIDENTMANAGEMENT
Approach
✓Automatetheendtoendprocess(similartoCICD)connecting
the3stagesofaproductionoutage
✓Solveknownproblemsusingknownsolutionsautomatically
✓Pagetherelevantteamsautomaticallyforunknownproblems
Shortentheincidentmanagementlife
cyclewhiledeliveringfeatures,fixes,andupdates
frequentlyinclosealignmentwithoperationalobjectives

17. Public
Stage4-EventbasedRemediation/RunbookasaService
EVOLUTIONOFINCIDENT
MANAGEMENT
Process
✓AutomaticDetection
✓AutomaticMobilization(locaterunbookforknownissues,devteamforunknownissues)
✓AutomaticRemediationusingrunbook(forknownissues)
Monitoring
Tools
Runbook
asa
Service
platform
Event
Application
RemediationAction
Runbook
Rulematch-
Triggerrunbook
Event
Norulematch-PageOnCall

18. Public
Stage4-EventbasedRemediation/RunbookasaService
EVOLUTIONOFINCIDENTMANAGEMENT
Result
✓FastDetection
✓Fastmobilizationofresources(script/teams)
✓Fastremediationofknownissuesusingknownactions

19. Public
Whydidwebuildthisplatform?
RUNBOOKASASERVICEPLATFORMATCAPITALONE
Runbookasa
Service
Diagnoseandremediate
Monitoring
Tools
EnterpriseServices
(Change,Incident,Esaclation,Notifications)
Learn
✓Engineerscanfocusoncode(applicationlogic)
intherunbookratherthanavailability/resiliency
oftherunbook
✓Integrationwithvariousmonitoringtools
✓Provideswrapperstoconnectwithenterprise
functionslikechangemanagement,incident
managementandalertingservices
✓Runbooklifecyclemanagement
✓Hubofenterpriserunbookstopromotere-
usability.Thisallowsengineerstolearnfromthe
mistakeofothersandusetheirrunbook(if
applicable)

20. Public
Someusecaseswehaveseensofar
RUNBOOKASASERVICEPLATFORMATCAPITALONE
✓ApplicationmultiregionfailoverofdatabaseslikeMySql,Postgress,etc.
✓AutomateDisasterRecovery(TREx)
✓Diskspacecleanup
✓Restartoffailedinstances
✓SupportsbothAutoremediationandOn-demandremediation
✓AutomatedDiagnosistosavetimefortroubleshooting

21. Public
KeepingyourEnterpriseAlwaysOn!
ADVANCINGINCIDENTMANAGEMENTANDOPERATIONALEXCELLENCE
ALWAYS
ON!
Automation+AI+People
BlamelessProblemManagement AutomateeveryRunbook&
FailoverPlans
UnifiedOperationalStandards
forDevOpsPractices
EmbedAutomatedRunbooksinto
SoftwareDeliveryPipelines