Detangling complex systems with compassion & production excellence

•

0 likes•206 views

Taming the complex distributed systems we're responsible for requires changing not just the tools and technical approaches we use; it also requires changing who is involved in production, how they collaborate, and how we measure success. In this talk, you'll learn about several practices core to production excellence: giving everyone a stake in production, collaborating to ensure observability, measuring with Service Level Objectives, and prioritizing improvements using risk analysis.

Technology

Detangling complex systems
Liz Fong-Jones
@lizthegrey
#DevOpsDays DFW
August 20, 2019
with compassion & production excellence
1w/ illustrations by @emilywithcurls!

@lizthegrey at #DevOpsDays DFW
Production is increasingly complex.
2

@lizthegrey at #DevOpsDays DFW
Especially for hybrid systems.
3

@lizthegrey at #DevOpsDays DFW
What does uptime mean?
4

@lizthegrey at #DevOpsDays DFW
Is it measured in servers?
5

@lizthegrey at #DevOpsDays DFW
Is it measured in complaints?
6

@lizthegrey at #DevOpsDays DFW
How about juggling everything else?
7

@lizthegrey at #DevOpsDays DFW
Our strategies need to evolve.
8

@lizthegrey at #DevOpsDays DFW
Don't "buy" DevOps.
9

@lizthegrey at #DevOpsDays DFW
When we order the alphabet soup...
10

@lizthegrey at #DevOpsDays DFW
Noisy alerts. Grumpy engineers.
11

@lizthegrey at #DevOpsDays DFW
Walls of meaningless dashboards.
12

@lizthegrey at #DevOpsDays DFW
Incidents take forever to ﬁx.
13

@lizthegrey at #DevOpsDays DFW
Everyone bugs the "expert".
14

@lizthegrey at #DevOpsDays DFW
Deploys are unpredictable.
15

@lizthegrey at #DevOpsDays DFW
There's no time to do projects...
16

@lizthegrey at #DevOpsDays DFW
and when there's time, there's no plan.
17

@lizthegrey at #DevOpsDays DFW
The team is struggling to hold on.
18

@lizthegrey at #DevOpsDays DFW
What are we missing?
19

@lizthegrey at #DevOpsDays DFW
We forgot who operates systems.
20

@lizthegrey at #DevOpsDays DFW
Tools aren't magical.
21

@lizthegrey at #DevOpsDays DFW
Invest in people, culture, & process.
22

@lizthegrey at #DevOpsDays DFW
Enter the art of
Production Excellence.
23

@lizthegrey at #DevOpsDays DFW
Make systems more reliable & friendly.
24

@lizthegrey at #DevOpsDays DFW
ProdEx takes planning.
25

@lizthegrey at #DevOpsDays DFW
Measure and act on what matters.
26

@lizthegrey at #DevOpsDays DFW
Involve everyone.
27

@lizthegrey at #DevOpsDays DFW
Build everyone's conﬁdence.
Encourage asking questions.
28

@lizthegrey at #DevOpsDays DFW
How do we get started?
29

@lizthegrey at #DevOpsDays DFW
Know when it's too broken.
30

@lizthegrey at #DevOpsDays DFW
& be able to debug, together when it is.
31

@lizthegrey at #DevOpsDays DFW
Eliminate (unnecessary) complexity.
32

@lizthegrey at #DevOpsDays DFW
Our systems are always failing.
33

@lizthegrey at #DevOpsDays DFW
What if we measure too broken?
34

@lizthegrey at #DevOpsDays DFW
We need
Service Level Indicators
35

@lizthegrey at #DevOpsDays DFW
SLIs and SLOs are common language.
36

@lizthegrey at #DevOpsDays DFW
Think in terms of events in context.
37

@lizthegrey at #DevOpsDays DFW
Is this event good or bad?
38

@lizthegrey at #DevOpsDays DFW
Are users grumpy? Ask your PM.
39

@lizthegrey at #DevOpsDays DFW
What threshold buckets events?
40

@lizthegrey at #DevOpsDays DFW
HTTP Code 200? Latency < 300ms?
41

@lizthegrey at #DevOpsDays DFW
How many eligible events did we see?
42

@lizthegrey at #DevOpsDays DFW
Availability: Good / Eligible Events
43

@lizthegrey at #DevOpsDays DFW
Set a target Service Level Objective.
44

@lizthegrey at #DevOpsDays DFW
Use a window and target percentage.
45

@lizthegrey at #DevOpsDays DFW
99.9% of events good in past 30 days.
46

@lizthegrey at #DevOpsDays DFW
A good SLO barely keeps users happy.
47

@lizthegrey at #DevOpsDays DFW
Drive alerting with SLOs.
48

@lizthegrey at #DevOpsDays DFW
Error budget: allowed unavailability
49

@lizthegrey at #DevOpsDays DFW
How long until I run out?
50

@lizthegrey at #DevOpsDays DFW
Page if it's hours.
51
Ticket if it's days.

@lizthegrey at #DevOpsDays DFW
Data-driven business decisions.
52

@lizthegrey at #DevOpsDays DFW
Is it safe to do this risky experiment?
53

@lizthegrey at #DevOpsDays DFW
Should we invest in more reliability?
54

@lizthegrey at #DevOpsDays DFW
Perfect SLO > Good SLO >>> No SLO
55

@lizthegrey at #DevOpsDays DFW
Measure what you can today.
56

@lizthegrey at #DevOpsDays DFW
Iterate to meet user needs.
57

@lizthegrey at #DevOpsDays DFW
Only alert on what matters.
58

@lizthegrey at #DevOpsDays DFW
SLIs & SLOs are
only half the picture...
59

@lizthegrey at #DevOpsDays DFW
Our outages are never identical.
60

@lizthegrey at #DevOpsDays DFW
Failure modes can't be predicted.
61

@lizthegrey at #DevOpsDays DFW
Support debugging novel cases.
In production.
62

@lizthegrey at #DevOpsDays DFW
Allow forming & testing hypotheses.
63

@lizthegrey at #DevOpsDays DFW
Dive into data to ask new questions.
64

@lizthegrey at #DevOpsDays DFW
Our services must be observable.
65

@lizthegrey at #DevOpsDays DFW
Can you examine events in context?
66

@lizthegrey at #DevOpsDays DFW
Can you explain the variance?
67

@lizthegrey at #DevOpsDays DFW
Can you mitigate impact & debug later?
68

@lizthegrey at #DevOpsDays DFW
SLOs and Observability go together.
69

@lizthegrey at #DevOpsDays DFW
But they alone don't
create collaboration.
70

@lizthegrey at #DevOpsDays DFW
Heroism isn't sustainable.
71

@lizthegrey at #DevOpsDays DFW
Debugging is not a solo activity.
72

@lizthegrey at #DevOpsDays DFW
Debugging is for everyone.
73

@lizthegrey at #DevOpsDays DFW
Collaboration is interpersonal.
74

@lizthegrey at #DevOpsDays DFW
Lean on your team.
75

@lizthegrey at #DevOpsDays DFW
We learn better when we document.
76

@lizthegrey at #DevOpsDays DFW
Fix hero culture. Share knowledge.
77

@lizthegrey at #DevOpsDays DFW
Reward curiosity and teamwork.
78

@lizthegrey at #DevOpsDays DFW
Learn from the past.
Reward your future self.
79

@lizthegrey at #DevOpsDays DFW
Outages don't repeat, but they rhyme.
80

@lizthegrey at #DevOpsDays DFW
Risk analysis
helps us plan.
81

@lizthegrey at #DevOpsDays DFW
Quantify risks by frequency & impact.
82

@lizthegrey at #DevOpsDays DFW
Which risks are most signiﬁcant?
83

@lizthegrey at #DevOpsDays DFW
Address risks that threaten the SLO.
84

@lizthegrey at #DevOpsDays DFW
Make the business case to ﬁx them.
85

@lizthegrey at #DevOpsDays DFW
And prioritize completing the work.
86

@lizthegrey at #DevOpsDays DFW
Don't waste time chrome polishing.
87

@lizthegrey at #DevOpsDays DFW
Lack of observability is systemic risk.
88

@lizthegrey at #DevOpsDays DFW
So is lack of collaboration.
89

@lizthegrey at #DevOpsDays DFW
90
Success doesn't demand heroism.

@lizthegrey at #DevOpsDays DFW
Season the alphabet soup with ProdEx
91

@lizthegrey at #DevOpsDays DFW
Production Excellence
brings teams closer together.
Measure. Debug. Collaborate. Fix.
92
lizthegrey.com; @lizthegrey

No matter what profession you’re in, our jobs can contain a variety of stress factors, which are unknown to individuals looking from the outside. “TECH-LIVES MATTER: HANDS UP, DON’T REBOOT” is a lecture based on research that recommends corporations should devote in IT specific employee assistance agendas. Working as a Developer, Specialist, Designer, Engineer, Expert, Manager, and Technician demands a high level of precision over the extended period, and any minute lapse in one’s job could be disastrous. MY METHOD OF HUMOR AND AFFECTION TITLED THE “DR. DRE METHOD.” ATTENDEES WILL LEARN TO: D. DOWNLOAD the cause of your Stress. R. Use a mental ROUTER to direct the stress to a secure site in your mind. D. Learn to DELETE future Stress elements. R. REBOOT yourself and focus on positive aspects. E. ENCRYPT your mind to secure the positive parts. The pressure of working in the field of computer technology can be a dream for the observers and the nightmare for the workers. Working in the world of technology can be a great experience, which DEVELOPERS, SPECIALISTS, DESIGNERS, ENGINEERS, EXPERTS, MANAGERS AND TECHNICIANS create and dive each time they start working. “TECH-LIVES MATTER, HANDS UP, DON’T REBOOT” offers solutions to support individuals who are afflicted by stress within the IT community is employee input, better task content, amplified job control, equal production values, career expansion, enhanced peer socialization, and more excellent workplace ergonomics. Key Takeaways: Overall Alertness- Regarding the Onset to Stress Stress at one’s place of employment Mentally Supporting Yourself Recognize the best method in a tense setting

Software Carpentry and the Hydrological Sciences @ AGU 2013

Aron Ahmadia

This document discusses bringing computational skills training to hydrologists through Software Carpentry workshops. It notes that while many hydrologists are focused on their research, computational methods are now essential. Software Carpentry teaches practical skills like the Unix shell, version control with Git, Python and R programming, and databases. These intensive, short workshops have been effective at training graduate students. The document encourages hydrologists to host their own workshops and support computational literacy by discussing code and practices in their papers.

Test Driven Development: Blueprint, Toolbox, and Master Craft

Stephen Ritchie

Many say that the practice of test driven development (TDD) has the ability to transform software engineering. Let’s break it down from a skeptical pragmatist’s viewpoint. This presentation focuses on three key ideas; the blueprint, the toolbox, and the master craft. Let’s look at TDD as a blueprint for writing code and discuss the implications and benefit. Let’s go beyond the theoretical application of TDD and focus on the toolbox, with examples of what developers need to know to get going. Finally, TDD is a master craft of Agile engineering. To be effective with TDD, let’s look at the progression of skills needed to achieve the promised results of TDD and to maximize its value. Stephen is the organizer of DC Alt.Net (http://bit.ly/ZRDxUj). He frequent speaker and trainer on many .NET topics, including automated testing, continuous integration, code analysis, and .NET best practices.Stephen is the author of the book "Pro .NET Best Practices" (http://amzn.to/v8TD8E).

Continuous Integration DCAEC12

Stephen Ritchie

The document discusses continuous integration, describing it as a blueprint and toolbox for master craftspeople. It defines continuous integration as automatically building and testing code changes frequently on a centralized server. This catches issues early by running unit tests, code analysis, and other checks on every commit. The document recommends tools like Jenkins, CruiseControl, and TeamCity for implementing continuous integration and emphasizes practices like unit testing with coverage, code quality analysis, reporting, and deployment automation for a complete continuous integration system.

Dev Conf LK 2019 - Announcement Flyer

Nithin Mohan T K

The document summarizes the devConf LK 2019 conference to be held on February 23rd, 2019 in Letterkenny, Ireland. The one-day conference will run from 9:00 AM to 5:00 PM at LYIT, Port Road, Letterkenny and feature talks on topics such as building online multiplayer games with Azure App Services and Socket.io, hosting and scaling service-based applications on Azure, using akka.net actor systems, performance testing with JMeter, automated Node.js API deployment with GitHub and Azure Pipelines, storage choices in Azure, Azure analytics, .NET Core with EF Core and generics. The code of conduct for the conference is also outlined.

2018-05-09_CRAFTConf_FirstStepsMovingToTheCloud

Abigail Bangser

Rip DevOps

Ryan Lockard

Ryan Lockard argues that DevOps is not dead, but there are three main failures that have prevented its widespread adoption: 1) Over-indexing on technology without a shared understanding, 2) Widespread misconceptions about what DevOps is, and 3) Conflicting interests between development and operations that DevOps aims to resolve. While tools can help, DevOps is primarily a cultural shift that requires leadership, empowering teams, and focusing on business outcomes over individual roles and technologies.

Accidental DataOps

Steve Ross

Usually, DataOps means applying DevOps principles to existing data analytics projects. We accidentally reversed it, taking a DevOps initiative and catalyzing adoption of data-driven practices across our company. What started as a practical initiative to bring better reliability and visibility to our software product had the unexpected effect of catalyzing a transformation that helped our organization become more data-driven across the company. What we learned in the process was how and why DevOps principles can naturally expand the role of a traditional operations team and bring wider culture change to the organization.

Is DevOps dead? Have we seen the end of the original intent of the movement that set a path to re-energize the engineering community? With the emergence of a pointed focus on tools and now massive organizations selling 2-day DevOps certifications, have we lost our way? This talk re-centers the focus of the engineering community to why we started this journey originally; to focus on learning, culture and feedback loops within complex systems and organizations. We will use data and stories from the trenches to weave a message about the anti-patterns that are emerging, but shine a focus on the stories of success and what good outcomes look like.

Trusting Your Ingredients - What Building Software And Cheesecake Have In Common

Leon Stigter

The document discusses the similarities between making cheesecake and building software applications. It notes that both processes require trusting where ingredients/libraries come from and having transparency in the overall process. This includes knowing the ingredients/libraries, who uses them, and where they are stored. The document advocates integrating security practices like DevSecOps earlier in the development lifecycle to help build more secure applications faster.

Devoxx 2016 Using Jenkins, Gerrit and Spark for Continuous Delivery Analytics

Luca Milanesio

devopsdays Kiel 2018 - Can the AI hype & ML algorithms harm your devops initi...

Thiago de Faria

This document discusses how machine learning and artificial intelligence hype can potentially harm DevOps initiatives. It notes examples of company executives making unrealistic claims about AI capabilities. It also summarizes key aspects of machine learning like supervised vs unsupervised learning. The document emphasizes that the data pipeline is crucial for machine learning and emphasizes applying software engineering best practices to the data pipeline. It also stresses the importance of continuous delivery for machine learning models and monitoring for issues after models are put into production.

Why Tooling (Only) Isn’t The Answer

Arnold Van Wijnbergen

The increasing importance of our tooling, the rely on their ability to automate and potential damage when they don’t work properly should make us think about repositioning your tooling strategy. Nowadays tools are an essential part and getting more important every day for delivering high quality productivity, but why are we still handle them as non critical parts? We can all agree that it’s seems such simple thing selecting your favourite tools, but that it’s actually a complex undertaking, by avoiding pitfalls and select your Swiss knife. Most companies are still struggling to find their best fitted tools for their organisation, processes and technology stack, caused by various legitimate reasons, but end up with no beneficial use or even not used at all. Think that we all recognize this situation? During this presentation I will take you on the journey of defining and refining your end goal, effectively learn from the things that matter and define your Enterprise Tooling strategy. We will explore some typical pitfalls and practices that help you to identify their strengths and opportunities for your organisation to leverage from. After all we want to get productive as early as possible.

New Farming Methods in the Epistemological Wasteland of Application Security

James Wickett

Over the years, application security (appsec) has made progress, but it has also made some considerable mis-steps. Appsec focuses almost solely on developer awareness and secure development training as remediation. This isn't sustainable and arguably does little good. There is a better way, but we have to separate ourselves from the core assumptions we have made that got us here. Lets journey together to find old truths and better approaches. We will explore ways to make a change for the better across all levels of the development lifecycle, but we will focus on security testing early on in the development process. From this session, you will learn pragmatic approaches and tooling that will affect your development processes and delivery pipelines. You will walk away with code examples and tools that you can put into practice right away for security and rugged testing. http://lascon.org http://lascon2015.sched.org/event/175e3c828095386b2fa0fc660b2502a3

DevOps in an IoT World - Brighton Web Development - 29-10-20

Peter Gallagher

Agile Intro to DevOps

Gil Zilberfeld

Gil Zilberfeld introduces himself and provides an overview of DevOps. DevOps emphasizes collaboration between software developers and IT professionals while automating the process of software delivery and infrastructure changes. The document discusses new challenges with cloud, serverless environments, and governance. It also covers DevOps practices like source control, continuous integration, testing, deployment, monitoring, and product validation.

The Death and Rise of Enterprise DevOps

DevOps Indonesia

The document discusses the implementation of DevOps in small, medium, and large enterprises. In small and medium enterprises, DevOps allows for more agile delivery but often faces challenges of manual deployments and lack of engineering practices. As enterprises grow larger, they struggle to implement DevOps effectively when viewing it simply as tools or separate from Agile. The document argues that DevOps succeeds in large enterprises by adopting a holistic delivery model, dual operating systems, and viewing IT as engineering with a focus on flow and scale. It is a journey that requires experimenting with frameworks and improving processes continuously.

FoMO-driven culture. What we can do to stay productive

Daiany Palacios

FoMO: Acronym for the “Fear of Missing Out” What’s the first thing you do in the morning at the office? You check your email. You can’t start working in peace unless you’ve reached Inbox zero. And still you need to keep Outlook, Lync and Slack open “just in case”. Have you ever wondered why it is like this? In principle, your company doesn’t expect you to have all these apps open. You should be able to design your working day as you want, as long as you deliver what you have to. But de-facto you are expected to be available. Even more when we talk about DevOps and so much collaboration is needed between teams that might even be geographically distributed. There seems not to be any other way! Unfortunately, such a “FoMO-driven culture”, is not healthy for your organization. It starts killing productivity when let out of control. In this talk we will be discussing some ways how to live with it, how we can keep it under control so that we can stay productive and not let it influence the DevOps culture in the wrong way.

[KubeCon NA 2018] Effective Kubernetes Develop: Turbocharge Your Dev Loop - P...

Ambassador Labs

Every software development cycle is rife with inefficiency. Seasoned devs know the pain of getting access to essential remote systems, waiting for tests to run (and then fail), or debugging with only log files. This talk teaches you how to best leverage Kubernetes, remote infrastructure and related tooling to create a dev cycle that maximizes velocity and minimizes developer friction and frustration. Using tools such as Kubernetes, Docker and Telepresence, I will walk attendees through several advanced techniques that can be used to produce an effective developer experience and optimized dev loop. The goal of this is to eliminate many sources of frustrating inefficiency and reduce cycle time between releases. I will demonstrate how to incrementally adopt some of these techniques and how to approach introducing new and unfamiliar technology and techniques to skeptical dev teams.

Leadership in Open Source and Why Companies Care

Dmitry Vinnik

In this talk, we will discuss the importance of open source for companies and how to become a leader in the open source space. By contributing to open source projects, companies can establish themselves as thought leaders, build credibility with customers, and increase productivity. Join us to learn how open source can help drive innovation, improve customer satisfaction, and keep companies ahead of the competition.

How to Effect Change in the Epistemological Wasteland of Application Security

James Wickett

From GOTO London 2015 Over the years, application security (appsec) has made progress, but it has also made some considerable mis-steps. Appsec focuses almost solely on developer awareness and secure development training as remediation. This isn't sustainable and arguably does little good. There is a better way, but we have to separate ourselves from the core assumptions we have made that got us here. Lets journey together to find old truths and better approaches. We will explore ways to make a change for the better across all levels of the development lifecycle, but we will focus on security testing early on in the development process. From this session, you will learn pragmatic approaches and tooling that will affect your development processes and delivery pipelines. You will walk away with code examples and tools that you can put into practice right away for security and rugged testing.

Devops & Agility - Build the Culture, Get the Tools, Win the Day - Dundee Tec...

David Walker

DevOps involves development and operations engineers collaborating throughout the entire service lifecycle from design to deployment and support. It builds upon agile principles by applying them throughout the entire workflow. Implementing DevOps requires cultural changes across the entire organization where teams work together towards shared goals. Automation, lean principles, continuous monitoring and improvements are key aspects of DevOps.

The Latest in DevOps: Elite Performance, Productivity, and Scaling - Google

Marilyne Huret

Application Security Epistemology in a Continuous Delivery World

James Wickett

CD Summit - Austin, from DevOps Connect Desc: Over the years, application security (appsec) has made progress, but it has also made some considerable mis-steps. Appsec focuses almost solely on developer awareness and secure development training as remediation. This isn’t sustainable and arguably does little good. There is a better way, but we have to separate ourselves from the core assumptions we have made that got us here. http://www.devopsconnect.com/events/cd-summit-austin/

DevOpsGuys FutureDecoded 2016 - is DevOps the Answer

DevOpsGroup

This document provides information about DevOps and digital transformation. It discusses how DevOps can help organizations transform by moving from traditional command and control models to more collaborative and iterative approaches. A DevOps operating model is proposed that uses multi-disciplinary product delivery teams and platform teams to continuously deliver value to customers. Microsoft tools that can support a DevOps transformation are also highlighted. The document concludes by demonstrating how to deploy a Docker container cluster on Azure using DevOps techniques.

AgileLIVE Webinar: Build a DevOps Culture & Infrastructure for Success Part 1

VersionOne

Michael Coté - The Eternal Recurrence of DevOps

Detangling complex systems with compassion & production excellence

Recommended

Recommended

More Related Content

Similar to Detangling complex systems with compassion & production excellence

Similar to Detangling complex systems with compassion & production excellence (20)

More from DevOpsDays DFW

More from DevOpsDays DFW (20)

Recently uploaded

Recently uploaded (20)

Detangling complex systems with compassion & production excellence