I’ll discuss how my experience of approaching DevOps not as another siloed effort but instead as a discipline by embedding engineers within cross-functional teams who are dedicated to continuously improving the quality of automation across the entire SDLC.

1. A Practical Guide to NOT building a DevOps Silo
(using Game Theory and Observability)

2. @notsureifkevin
• Docker Captain
• Docker Meetup Organizer
• DevOpsDays Nashville Organizer
• 15+ years in software development
• 5+ years Docker experience
Kevin Crawley
Developer Evangelist @ Instana
Principle Architect / SRE @ Single
Social Media @notsureifkevin
(twitter, github, Instagram, linkedin, etc)
Random Facts:
• Endurance Athletics (training for 100
mile bike ride)
• SciFi Nerd (The Expanse, Stargate,
Firefly)
• Video Game Nerd (RPGs mostly,
Octopath Traveller current fav)

3. @notsureifkevin
Discussion Points
• How can Game Theory tear down Silos
• Characteristics of High Performance Organizations
• DevOps and Site Reliability Engineers
• What do SREs live and breath? o11y and SLI/Os ... LOL?

4. @notsureifkevin
Let’s talk about some Game Theory
(p.s. I’m terrible at math)

5. This is … GAME THEORY
Bad Equilibrium
It’s a strategy that all players in the game can
adopt and converge on, but it won’t produce a
desirable outcome for anyone.
https://pdfs.semanticscholar.org/30d1/a03db196384a17fed3247407fb5859f7c76b.pdf

6. @notsureifkevin
Transformation: Focusing on Automation
https://devops-research.com/

7. @notsureifkevin
Definition of a Silo (in IT)
Silos can be defined as the
contention which exist
between functional barriers
within an organization.
This contention usually exists
between teams where change
management policy
requirements and risks are
high.

8. This is … GAME THEORY
Nash Equilibrium
(Prisoners Dilemma)
A concept of game theory where the optimal
outcome of a game is one where no player has
an incentive to deviate from her chosen strategy
after considering an opponents choice.
https://en.wikipedia.org/wiki/Nash_equilibrium

10. Nash Equilibrium

11. Pareto Efficiency
Is a state of allocation of resources in which it is
impossible to make any one individual better off
without making at least one individual worse off.
aka ZERO SUM
https://en.wikipedia.org/wiki/Pareto_efficiency

13. Pareto Inefficiency
A situation is inefficient if someone can be made
better off even after compensating those made
worse off.

14. Pareto Inefficient Nash Equilibrium
A Bad Equilibrium

16. Don’t like the game?

17. Change the Game

18. New Nash Equilibrium

19. Pareto Inefficient Nash Equilibrium
Gives you permission and proof to
change the game

20. Change the Game

21. SILOS ARE EASY
TO DEFEND
AND DIFFICULT
TO ATTACK

22. @notsureifkevin
Defining your Culture – The Westrum Model
Information is actively sought
Failures are learning opportunities, and messengers of them are not punished
Responsibilities are shared
Cross-functional collaboration is encouraged and rewarded
Failure causes enquiry
New ideas are welcomed
https://langerman.co.za/cultural-types/

23. @notsureifkevin
Percentage of Work Done Manually
ELITE
PERFORMERS
HIGH
PERFORMERS
LOW
PERFORMERS
Configuration
Management
5% 10% 30%
Testing 10% 20% 30%
Deployments 5% 10% 30%
Change
approval
process
10% 30% 40%
https://devops-research.com/

24. @notsureifkevin
High Performance vs Low Performance Organizations
High Performers
• Deployments:
> 1 hour and < 1 day
• Lead Time for Changes:
> 1 day and < 1 week
• MTTR:
< 1 day
• Change Failure Rate:
0-15%
Low Performers
• Deployments:
Once per week/month
• Lead Time for Changes:
> 1 month and <6 months
• MTTR:
> 1 week and < 1 month
• Change Failure Rate:
46-60%
https://devops-research.com/

25. @notsureifkevin
Transformation: Focusing on Automation
https://devops-research.com/

26. @notsureifkevin
What happens when we tear down the silos and become a
DevOps organization?
• We ship software more often, complexity increases and
reliability starts to decline
• We naturally shift our focus to solve the scalability and
reliability issues
• Rise of the Site Reliability Engineers

28. What are some tools / processes that organizations can put
in place to change our equilibrium and communicate?
• Communication & Collaboration Tools
• Slack, Git, Pagerduty
• Observability (SRE) Tooling
• Custom Dashboards / Metrics
• Log Analytics
• Distributed Tracing

29. @notsureifkevin
Observability
What does that word mean?

30. @notsureifkevin
• Not just tooling
• Similar to how DevOps is a mindset
• No longer treating services like
Schrödinger's cat
• (A lot) more context around events
and transactions
• Monitoring
• Instrumentation
• Structured Logging (Tracing)
• Alerting
• Dashboards
Observability

31. @notsureifkevin
Observability
Why does my organization need it?

32. @notsureifkevin
How many of you are running
staging environments?

33. @notsureifkevin
Now, how many of you actually
trust your staging environments?

34. This is your staging environment

35. … and this is your
production
environment

36. @notsureifkevin
Observability enables Developers, Operations, and
SREs to debug production systems via:
•Context
•Status
•Expectations
•Contributing Factor Analysis

37. @notsureifkevin
In order to observe a system, we must emit
signals from within it. Those signals can answer
the following questions (and more):
• Number of Retries / Backoffs
• Request parameters / Query Statements
• Failing back to defaults
• Top-Level Exceptions / Log Messages

38. @notsureifkevin
How do we make our systems more observable?
• We must wrap communication functionality within our
applications utilizing Distributed Tracing implementations:
• Community Libraries (OpenTracing, OpenCensus, Zipkin)
• Vendor Libraries (Instana, SignalFX, Dynatrace, Datadog)
Each of these solutions support many of the same languages and
frameworks, some more than others. They each have trade-offs.
They each have their own level of efforts at implementation.

39. @notsureifkevin
Distributed Tracing
• Also known as Distributed Structured Logging
• Larger Payloads
• Rich Contextual Data
https://w3c.github.io/trace-context

40. @notsureifkevin
How does Observability help enable a DevOps
culture?
Let’s take a look at a production microservice
application which has been instrumented by a
distributed tracing solution

41. @notsureifkevin
• Operated by 3 engineers (1 FE/1 BE/1 SRE)
• We built Single to give artists and labels the
ability to easily distribute their music in their
own stores
• Soft launched in 2018 with 15 microservices
– has since expanded to over 28
microservices with zero additional
engineering personnel

42. @notsureifkevin
“DevOps allows small teams to
achieve great things.” – Gene Kim
Observability enables those small teams to scale
effectively while doing great things.

43. @notsureifkevin

44. @notsureifkevin

45. @notsureifkevin
When we begin emitting signals from every
transaction in the system
• We enable powerful insights into production applications
• Dependency mapping becomes trivial
• SREs and Engineers can track golden signals on EVERY
operational perspective on their applications

46. @notsureifkevin

47. @notsureifkevin
Exponential
Backoff

48. @notsureifkevin
Slow Death of a
Service

49. @notsureifkevin
And Feedback when
we’ve fixed it

50. @notsureifkevin
• Enables your organization to understand the behavior of your
system
• Empowers your engineers to find and fix problems
• Enables you to build more reliable systems and ship software faster
• Promotes empathy through understanding, transparency, and
communication.
Focusing on Observability

51. @notsureifkevin
Follow Me on Twitter
@notsureifkevin
• Use Game Theory to change the game – put all
stakeholders on-call, everyone should feel the
pain of an outage.
• Observability compliments and more
importantly enables a DevOps culture.