Emergent behavior in complex systems is our reality. Root Cause Analysis, as a framework, is insufficient to properly manage the patterns present in our continuously changing world.

4. bit.ly/addo-slack
Find me on slack, right now!

5. Complex systems exhibit unique
behavioral characteristics
Change velocity adds complexity
Cynefin is the best framework for
managing complex patterns
Three Takeaways

6. Developer Advocate
VictorOps
Technology Strategist
Dryas.io
18 years (dev)Ops
@matthewboeckman
Matthew Boeckman

7. Incident ingestion, scheduling, routing, escalations, chatops, transformation, reporting
Incident Management for DevOps teams
VictorOps

8. Root Cause Analysis
What went wrong?
Why did that happen?
Who was
responsible?
How can we
prevent this
from recurring?

9. It’s like a tree, but sideways
Fishbone!
People Process Pipeline
Code Systems Data
Something
Bad
Happened!

10. “3 tiers should be enough tiers for anybody” - some guy, probably
Simple Systems

11. “I guess there’s more tiers?” - that guy
Simple Systems

12. Let’s change once a year, then it will be easier to point fingers at Dev.
Deployment Schedules

13. ● It took a long time to create requirements
● It took a long time to write software
● It took a very long time to deploy applications
● It took a really, really long time to test software
● Testing patches was hard
● Deploying patches was all or nothing
● Managing Hardware was an entire departments job
● Software and Hardware changes often required orchestration (that was hard)
Playing the long game
There were some good reasons

14. ● It took a long time to create requirements
● It took a long time to write software
● It took a very long time to deploy applications
● It took a really, really long time to test software
● Testing patches was hard
● Deploying patches was all or nothing
● Managing Hardware was an entire departments job
● Software and Hardware changes often required orchestration (that was hard)
Playing the long game
There aren’t anymore

15. Static model, Binary Thinking
GOOD
Working
Expected
Certain
Understood
Responsible
Uptime
BAD
Broken
Problem
Disaster
Confused
Wrong
FAILURE

16. “3 tiers should be enough tiers for anybody” - some guy, probably
Simple Systems

17. So many tiers
“I thought we agreed on 3 tiers?”

18. What’s traceability, precious?

20. “... refers to the existence or formation of collective
behaviors — what parts of a system do together that they
would not do alone.”1
Emergence and Complex Systems
1 Bar-Yam Concepts: Emergence
Properties and behaviors of systems arise from both the
fine structures that compose those systems, and the
interrelationships between the systems’ discrete parts.

21. RCA: Seeking Blame

22. Emergence Language

23. Subtlety and Nuance
Our shared reality
High Complexity + Dramatic Change Vectors
=
Emergent Behavior

24. It’s not a tree...

25. … it’s a forest

26. Cynefin

27. Cynefin
● Created by Dave Snowden @snowded
● Originally for managing IBM Intellectual
Capital
● Draws on research in systems, complexity,
network and learning theories

29. Simple
Sense
Categorize
Respond
*known

30. Simple
Sense
Alarm: Disk 95%
Categorize
Volume full
Respond
Archive and
Delete Logs

31. Complicated
Sense
Analyze
Respond
*probable

32. Complicated
Sense
Increase in % of
requests
returning 500
Analyze
Aggressive cache
evictions
Respond
Modify cache
Enlarge database

33. Complex
Probe
Sense
Respond
*emergent

34. Complex
“Some users report modal throwing errors”
A/B test for new modal activated
Recent network changes
Application utilizes a healthcheck
All monitoring systems are clear

35. Probe
Confirm Network path LB->Service
Investigate Service Logs
Confirm Network path Service->DB (failing)

36. Sense
Validate “A path” works
Review Changelogs (discover network change)
Observe healthcheck code

37. Respond
Revert A/B test
Refactor healthcheck to include all datasources
Request network rollback

38. Chaotic
Act
Sense
Respond
*buckle up

39. Chaotic
Alert Flood
Some hosts, services down hard, others
intermittent
AWS status clear
Unable to reach any hosts in AZ

40. Chaotic
Act
Relaunch
instances in a
different AZ
Sense
Alerts clear?
Respond
No, new A/Z
failing also

41. Disorder
Reduce
Analyze
Iterate
*culture

42. Disorder
Reduce
What are we
sure of?
Analyze
Build
consensus
Iterate
Move to a
quadrant,
continue

43. Analysis changes the game
Knowledge and
practice move
patterns
towards more
favorable
quadrants.

44. Complacency erodes progress
Slacking off
walks back
progress

45. Adopting Cynefin
In the moment:
What Quadrant does this map to?
In the PIR:
How did we manage the pattern?
In your sprint planning:
What patterns can we manage clockwise?

46. Root Cause Analysis Cynefin
Simple Causality
Static Model
Binary Thinking
After-Action
Focus on Blame
Dynamic
Expects Change
Embraces Emergence
Present in the Moment
Call to Action

47. “Uncertainty is an
uncomfortable position.
But certainty is an
absurd one.”
-Voltaire

48. Thank you!
@matthewboeckman
slideshare.net/matthewboeckman

49. Complex systems exhibit unique
behavioral characteristics
Change velocity adds complexity
Cynefin is a better framework for
successful management of these
complex patterns
Three Takeaways