1. StackStorm
If-This-Than-That for DevOps
automation
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.

3. @StackStorm@Stack_Storm

4. Agenda
What is StackStorm
Why we made it
Use cases
Why is it better than …

5. StackStorm: What is it?
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.

7. StackStorm is like …
7
for DevOps

8. IFTTT,
for DevOps
8
cat /opt/stackstorm/packs/st2-demos/rules/demo.yaml
---
name: "sensu_crit_to_slack"
pack: "st2-demos"
description: "Post all critical alerts to the demo
enabled: true
trigger:
type: "sensu.event_handler"
criteria:
trigger.check.status:
pattern: 2
type: "equals"
trigger.check.name:
pattern: "demo_.*"
type: "matchregex"
action:
ref: "slack.post_message"
parameters:
message: >
[ALERT]{{trigger.client.name}}
{{trigger.check.output}}
channel: "#demos"

9. 9

10. 10
Trigger Action
Rule

11. Ingredients
11
IT Domains
Config mgmtStorageNetworking ContainersCloud InfraMonitoring
ActionsSensors
WorkflowsRules
Ops Support
Trigger Call

12. Automation Example
12
Automation
EngineerService
Monitoring Incident
Management
Event: “low disk
on web301”
Web301 is
“low disk”
Resolve known
cases, fast. Is it
/var/log?
Clean up!
Unknown
problem, need
a human
Wake up, buddy.
Something real
is going on…

13. StackStorm: Why we made it?
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.

14. Opalis, now Microsoft SC Orchestrator
2004-2008
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.
Event-driven automation with workflows,
for Enterprise IT

15. VS
VMware, 2008 - 2013

16. VS
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 16
OpenStack VMWare

17. DevOps Tools
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 17
VS
Enterprise Suites

18. VS
18
Event-driven automation with workflows,
for Cloud and DevOps
DevOps Tools Enterprise Suites

19. 19
What can be it automated?

20. 20
Got StackStorm !
Every automation looks like a nail

21. What SHOULD be automated?
From: Practice of Cloud System Administration, by Thomas Limoncelli

22. What HAS BEEN automated with StackStorm
• Security checks
– On malware detection in a VM, isolate
network port on a switch
• App blue-green deployment
– On Jenkins tests passed, bring new vm
claster, deploy and configure app, set
loadbalancer to send % of traffic to new
app, monitor, roll forward, or back out
• Networking
– On BGP peer goes down: collect
troubleshooting data, post on slack &
create JIRA ticket
– On Link aggregation member error,
check load, if capacity of rest of LAG
bundle enough, disable link with error
• OpenStack
– orphan VM clean-up: On orphans
detected, shut down, email owner, keep for
few days, delete
– VM evacuation on HW failures: On host
RAID failure, get list of impacted VMs,
email VM owners, evacuate VMs, create
JIRA ticket for hardware replacement.
• Service remediation:
– Cassandra “node down” recovery: On ring
node dying, deploy new node, configure,
add to the ring.
– Remediating RabbitMQ, Galera cluster,
MySQL, and more…
22

23. 23
Auto-Remediation
FB auto-remediates 98% alarms,
can you?

24. “Auto-Remediation & Automation at Facebook” @ Auto-Remediation Meetup SF
https://www.meetup.com/Auto-Remediation-and-Event-Driven-Automation/events/236704012/
Facebook FBAR:
43 % Problem Fixed
51 % False positives
94 % Automated

25. “Auto-Remediation & Automation at Facebook” @ Auto-Remediation Meetup SF
https://www.meetup.com/Auto-Remediation-and-Event-Driven-Automation/events/236704012/
FBAR & StackStorm are friends
StackStorm is inspired by FBAR
StackStorm and FB FBAR collaborating since 2014

26. “Sleep Better at Night: OpenStack Cloud Auto-Healing” @ OpenStack Summit Barcelona
Mirantis: Auto-remediating 2,000 node OpenStack cluster at Symantec with StackStorm

27. “Sleep Better at Night: OpenStack Cloud Auto-Healing” @ OpenStack Summit Barcelona
Mirantis: Auto-remediating 2,000 node OpenStack cluster at Symantec with StackStorm
User: Symantec (Mirantis)
Use case: OpenStack cluster remediation
Presented by Mirantis at OpenStack Barcelona
StackStorm at Symantec

28. Engineer
Wakes up
Logs in
and ACK
Checks
runbook
Studies
the alert
Fixes the
problem
Runs
diagnostics
PagerDuty
Alert
2:02 AM 2:07 AM 2:15 AM2:10 AM 2:30 AM2:20 AM2:00 AM
On-call, Without Automation
Source: “Winston: Helping Netflix Engineers Sleep at Night” @ Qcon ‘16 SF
https://goo.gl/lHzq4r

29. False
Positive
Winston
2:00
AM
2:05
AM
2:05
AM
2:15
AM
Assisted
Diagnostics
Fixed
the
problem
On-call With Winston
Source: “Winston: Helping Netflix Engineers Sleep at Night” @ Qcon ‘16 SF
https://goo.gl/lHzq4r

30. Benefits
• Reduce MTR (Mean Time to Resolution)
• Avoid failures (fixing on computer time, not human time)
• Reduce risk of human error (no fat fingers)
• Positive team impact
– Avoid pager fatigue and team burn-out
– Turn from reactive to proactive (break reactive vicious cycle)
– Capture operational knowledge – as code
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 30

31. 31
Network Automation
Now supporting Multiple Vendors

33. Device support
Proven approach from cloud compute space
33
NAPALM
AUTOMATION
Packs
PYTHON
BINDINGS
INTEGRATION
PACKS
DEVICE / OS
INTERFACES
Some workflows [1] leverage unique device functions, so they
call actions of the device’s integration pack.
Other workflows [2] need to be abstracted and treat devices
alike (e.g. IXP provision on mixture of SLX and MLX). So they
use “unified” Napalm pack.
device drivers
Integration packs expose device configurations and operations
as st2 actions. VDX and SLX packs will expose most/all of
device functionality. MLX is “best effort”. Napalm integration
pack provide “unified” device actions, just like libcloud does in
“compute” domain.
While integration packs can call device interfaces directly,
python bindings provide reuse, and abstract device OS
versions. PyNOS binds VDX via netconf. SLX and MLX don’t
exist yet.
Napalm is Open source project that exposes a unified python
API to interact with different network devices via device drivers.
Devices expose various interfaces: RESTCONF, NETCONF,
CLI/TELNET, SNMP.
VDX MLX SLX
[1] [2]
Other vendors

34. 34
Including legacy, business apps …
Integration

35. “Innovation at Dimension Data: Optimizing Operations with Event Driven Automation”
https://stackstorm.com/2016/12/15/dimension-data-devops-beyond-deployment/
Dimension Data (SP, part of NTT)

36. “Innovation at Dimension Data: Optimizing Operations with Event Driven Automation”
https://stackstorm.com/2016/12/15/dimension-data-devops-beyond-deployment/
• Integrate IT systems & tools
• Security automation
• Legacy Run-book replacement
• Automation-aaS to end-users
• Top st2 contributors
Dimension Data (SP, part of NTT)

38. 38
IoT
Fun stuff

40. 40
Serverless
Grab StackStorm & DIY

41. StackStorm is like …
41
AWS Lambda AWS Step Functions
OpenSource, for DIY Serverless

42. StackStorm is like …
42
ActionsSensors
WorkflowsRules
IT Domains
Config mgmtStorageNetworking ContainersCloud InfraMonitoring Ops Support
Step Functions
AWS Lambda

43. Serverless with Swarm
for Genomic Annotation Computing
Dmitri Zimine
http://github.com/dzimine/serverle
@dzimine
Image by Miki Yoshihito, Creative Commons license

44. Many Use Cases – One Platform
44
StackStorm automation platform
Network
Automation
Assisted
Troubleshooting
Auto
Remediation
ITprocess
integration
IoT
InternetofThings
Serverless
CI/CD
ContinuousDeployment
NFV
Security
Orchestration
ChatOps

45. Why not…
45

46. Why not Scripts?
46

47. Why not Scripts?
47
• Simple to define, reason, visualize
• Transparent
– state is clear, execution is trackable: running, complete, failed steps
•
–
–
–

48. © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.48

49. Workflows Better in Operations
• Simple to define, reason, visualize
• Transparent
– state is clear, execution is trackable: running, complete, failed steps
• Reliable
– Workflows are long-running
– Crash tolerance
– “Restart from point of failure”
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 49

50. 50
Why not legacy Runbook Automation?
• Microsoft System Center Orchestrator
• HP Operation Orchestrator
• Cisco Process Orchestrator (CPO)
• VMWare vCO / vRealize
They do not DevOps

51. Infrastructure As Code
Leverage social coding and collaboration
OpenSource
Designed for Devops

52. Infrastructure As Code
Leverage social coding and collaboration
OpenSource
Designed for Devops

53. 53
Infrastructure as code
Case Study
• Service Catalog backed up by workflow
• Automate provisioning on VMW/OpenStack, 4 Data centers
• Before: CPO, operator updates via GUI, click and pray, x4
• After: StackStorm, dev -> code review -> staging -> QA-> prod

54. Infrastructure As Code
Leverage social coding and collaboration
OpenSource
Designed for Devops

55. 265 Contributors
Source: https://openhub.net/p/st2

56. 256 = 100,000,0002
Contributors

57. StackStorm & BWC Usage
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 57
0
500
1000
1500
2000
2500
3000
Sep Oct Nov Dec Jan Feb
Installations/month
- StackStorm
- BWC

58. StackStorm Exchange
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 58

59. StackStorm Exchange
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 59

60. © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 60

61. © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 61

62. Take away:
* Try it
* Use it
* Contribute to it
62

63. 63

64. • Use StackStorm.
Try it, find automation, nail POC. Let us know, good & bad.
curl -sSL https://stackstorm.com/packages/install.sh | bash -s
docs.stackstorm.com/install
• Commit code. Become a “community maintainer”
It is not hard (2 days?). We help & support.
• Spread the word
Blog. Tweet. Talk. Mention. Bug. Github Star!
64
Contribute! Everything counts

65. Thank You!
@Stack_Storm
http://github.com/StackStorm/st2 Star 1,869
Dmitri Zimine
@dzimine

66. OpenSource Apache 2.0
• Github: github.com/StackStorm/st2
• Twitter: Stack_Storm
• IRC: #stackstorm on FreeNode
• stackstorm.slack.com on Slack
• www.stackstorm.com
© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 66
StackStorm Brocade Workflow Composer
Commercial Edition
• Enterprise features
• Priority support
• brocade.com/bwc
• docs: bwc-docs.brocade.com
• Network lifecycle automation suite