The document discusses workflows as sequences of processes for efficiently managing incidents and automations in IT environments. It highlights the benefits of using workflows, including consistency, speed, and connectivity between disparate systems, as well as various automation use cases such as security checks, incident resolution, and troubleshooting. Additionally, it outlines workflow design patterns and emphasizes the importance of diagnostic and remediation workflows for effective incident management.

1. Patrick Hoolboom
September 22, 2016
Incident Management
with Workflows
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY

2. What is a Workflow?
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY

3. What Is a Workflow?
• A sequence of processes through which a piece of work passes from
initiation to completion
• Process as Code
• Living Documentation
– Document your process in an easily human readable, executable format
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY 3

4. Event Driven Automation 2.0
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 4
FBAR (saving 1532 hours/day)
Naoru
Nurse
Winston (powered by StackStorm)
Azure Automation
Mistral workflow service
StackStorm automation platform
ACT
OBSERVE
ORIENT
DECIDE

5. When to use a workflow
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY

6. When to use a workflow
• Clearly defined process
• When multiple systems or services need to be touched
• Frequently performed tasks
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY 6

7. Why Use Workflows?
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY

8. Why Use Workflows?
• Consistency
– Trust that your automations will perform the same tasks every time
for a given event
• Speed
– Reduce time to resolution for an incident
Audit
– Creates a clear audit trail of what was done when
• Connect Disparate Systems
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY 8

9. Tools…
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 9

10. What Can Be Automated?
• Security checks
– On malware detection in a VM, isolate
network port on a switch
• Blue-green app deployment
– On Jenkins tests passed, bring new vm
claster, deploy and configure app, set
loadbalancer to send % of traffic to new app,
monitor, roll forward, or back out
• Networking
– On BGP peer goes down: collect
troubleshooting data, post on slack & create
JIRA ticket
– On Link aggregation member error, check
load, if capacity of rest of LAG bundle
enough, disable link with error
• Restart a down service
– On monitoring event, bounce a service
• OpenStack orphan VM clean-up
– On orphans detected, shut down, email owner,
keep for few days, delete
• NFV:
– Nokia, AT&T, with Mistral and OpenStack
• OpenStack VM evacuation on
hardware failures
– On host RAID failure, get list of impacted VMs,
email VM owners, evacuate VMs, create JIRA
ticket for hardware replacement.
• Cassandra “node down”
recovery
– Replace a node on alert
• Clean up disk space
– On monitoring event, clean up disk space
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 10

11. StackStorm
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY

12. Architecture
12
Web GUI CLI Chatops
Sensor Containers Action Runners
Sensor Plugins
(inbound integrations)
Master
Content Repo
to Audit…
Action Plugins
(outbound integrations)
PLATFORMCLIENTSPLUGINS
AMQP message busAMQP message bus
Workflow
Engine
REST API
{*}
Rules
Engine
IFTTT.yml
KV Store
k[v]

13. ●
Diagnostic Workflows
●
Remediation Workflows
Workflow Design
Patterns
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY

14. Workflow Design Patterns
Diagnostic Workflows
• Troubleshooting and data gathering steps
• No remediations or changes to the system
• Good way to “get your feet wet” with workflows
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY 14

15. Workflow Design Patterns
Remediation Workflows
• Fix the issue!
• Should be triggered after diagnostic workflows if applicable
•
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY 15

16. ●
Facilitated Troubleshooting
●
Auto-Remediation
Workflow Use Cases
During an Incident
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY

17. Workflow Use Cases
Facilitated Troubleshooting
• Useful if you don’t quite trust the automation
– Gain confidence in your workflows
• Faster Time to Resolution
• Consistent Data Collection
• Diagnostic workflow with notifications
– Send data to user via
• Email
• Chat
• Ticketing System
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY 17

18. Workflow Use Cases
Auto-Remediation
• Trusted Automation
– Will make automated changes to the system
• Much Faster Time to Resolution
• Consistent Solutions
• Less Pager Fatigue
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY 18

19. ●
Low Disk Space Event
Example
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY

20. Automation Example
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 20
Automation
EngineerService
Monitoring Incident
Management
Event: “low disk
on web301”
Web301 is
“low disk”
Resolve known
cases, fast. Is it
/var/log?
Clean up!
Unknown
problem, need a
human
Wake up, buddy.
Something real
is going on…

21. 21

22. ●
Email: phoolboo@brocade.com
●
Twitter: @DoriftoShoes
Thank You!
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY