Embracing Failure

•

0 likes•393 views

The document discusses embracing failures in software and systems. It recommends monitoring for failures, conducting blameless postmortem analyses to determine root causes, simulating failures through "gameday" exercises before they occur, and designing systems using a "SafeMachine" approach to make failures safe rather than unsafe. The key ideas are to understand failures will happen, learn from them through root cause analysis and prevention practices, and design systems that can fail safely.

Technology

Setting the Mood
•Understand that they WILL
happen
•Failures are not binary
•Impact determines importance
•deadlines for fixes are variable

Terminology
•Website
•Production
•Downtime

What is Monitoring?
•Graphs. Everywhere.
•Alerts on failures
•phone calls
•texts
•Answers: Are we failing?

healthcare.gov
•Know when you’re down
before CNN

Postmortems
(fool me once. shame on you.
fool me twice. shame on me.)

Postmortems
1. Reconstruct the factual
timeline
2. Root cause analysis
3. Remediation items

Postmortems
•Why did we fail?
•Blameless
•Moderated

Gamedays
(You wouldn’t wing a talk.
Don’t wing a hot fix)

Gameday
•Best defense is a good
offense
•Simulate possible failures
•Do it in production

kill -9
1. Draw a block
diagram
2. Cut every connection
3. Watch the fireworks

SafeMachine
(like a state machine … but safer)

Try, Try, Try again
•What if we could just retry
failures?
•Side effects are the root of all
evil
•Safe failures vs Unsafe failures

What’s in a SafeMachine
•Actions
•States
START
Computed
File
Uploaded
File
END
compute upload
record
successful

initialize_succeeded
initialize_failed
initialize_inprogress
computed_succeeded

START
a1
a1
a2
a2
a2
a3
a3
a3
END
The Pipeline

The Pipeline
START
Computed
File
Uploaded
File
END
Safe Unsafe Safe

Embracing Failure
•Monitor
•Postmortems
•Gamedays - you wouldn’t
wing a talk?
•SafeMachine

Additional resources
• Postmortems https://codeascraft.com/2012/05/22/blameless-
postmortems/
• Gamedays - https://stripe.com/blog/game-day-exercises-at-stripe
• links at the bottom of this post are also great
• Error Tracking - https://getsentry.com/welcome/

Physical security controls have been found lacking in assessments against targets ranging from financial institutions to health care organizations, and from critical infrastructure and governments- city, state, and federal alike. While complex security programs address complex security problems, successful attacks often result from a cascade of minor security failures being leveraged in a damaging manner. In this session, walk in the shoes of an attacker as organizations are profiled, vulnerabilities cataloged, and attacks launched to gain unauthorized access to restricted areas and/or sensitive data. This 40 minute discussion will conclude with 10 minutes for Q&A on strategies to strengthen the existing physical security posture of an organization without overhauling all the guards, guns, and gates.

Building Secure By Default Nodejs Applications

Yolonda Smith

DOES SFO 2016 - Kevina Finn-Braun & J. Paul Reed - Beyond the Retrospective: ...

Gene Kim

At DOES15, we presented the work we'd done at Salesforce to take their SRE teams to the "blameless cloud." We worked with various roles in the SRE teams so they could start asking the right questions about failure, and through the postmortem and retrospective process, begin to make lasting changes in _how_ Salesforce worked with and remediated identified failures. But DevOps espouses less siloed thinking and more shared responsibilities, so we found postmortems within the SRE organization weren't enough. As Salesforce was moving toward a model of "service ownership," teams along the entire software delivery value stream needed to start to understand their roadblocks to remediation and what aspects of the complex system they worked in were impeding their ability to "own their service." We'll discuss the second phase of our work in helping these operations _and product_ teams gain a deeper understanding of service ownership, and why just "DevOps'ing it up" wasn't quite enough on its own to help. plus we'll introduce an expanded model from last year's talk that incorporates human factors and complexity theory. These additions helped prime the teams to more effectively grapple with the challenges facing them on the road to true service ownership.

Postmortem toxicologyBruno Mmassy

The Most Important Thing: How Mozilla Does Security and What You Can Steal

mozilla.presentations

How To Run a 5 Whys (With Humans, Not Robots)

Dan Milstein

Monitoring Complex Systems - Chicago Erlang, 2014

Brian Troutwine

Gamification of Chaos Testing

Bram Vogelaar

Athletes, Firemen and Doctors train everyday to be the best at their chosen profession. As engineers we spend much of our time getting stuff to production and making sure our infrastructure doesn’t burn down out right. We however spend very little time learning to understand and respond to outages. Things like Infrastructure as Code, Service Discovery and Config Management can and have helped us to quickly build and rebuild infrastructure but we haven’t nearly spend enough time to train our self to review, monitor and respond to outages. Does our platform degrade in a graceful way or what does a high cpu load really mean? What can we learn from level 1 outages to be able to run our platforms more reliably. In this talk we ll discuss the need for and the options of creating a game day culture. Where we as engineers not only write, maintain and operate our software platforms but actively pursue ways to learn and predict its (non-functional) behavior. We ll look at tools like toxiproxy and the simian army for ways to prepare teams to tweak their testing and monitoring setup and work instructions to quickly observe, react to and resolve problems.

Athletes, Firemen and Doctors train everyday to be the best at their chosen profession. As engineers, we spend much of our time getting stuff to production and making sure our infrastructure doesn’t burn down out right. In this talk, we'll discuss the need for and the options of creating a game day culture. Where we as engineers not only write, maintain and operate our software platforms but actively pursue ways to learn and predict its (non-functional) behavior. We'll look at tools like toxiproxy and the simian army for ways to prepare teams to tweak their testing and monitoring setup and work instructions to quickly observe, react to and resolve problems.

How to Run a Post-Mortem (With Humans, Not Robots), Velocity 2013

Dan Milstein

Jason Yee - Chaos! - Codemotion Rome 2019

Codemotion

Are We Secure? Answering the Unanswerable

Justin Berman

2016-04-28 - VU Amsterdam - testing safety critical systems

Jaap van Ekris

Win#23 it

Olivier Marechal

debugging.ppt

jerlinS1

The Two Generals Problem

VoltDB

Jeff Atwood, Michael Krakovskiy "The role of catastrophic failure in software...

Fwdays

Everyone knows success is boring! Anyone can succeed, and they won't shut up about it after they do. Happy families are all alike, as the saying goes. Let Michael and Jeff take you on a grand tour of what really matters in software development: failure. Take a glorious, life affirming tour through the variaous ways they've failed at building major software projects, from Stack Overflow to Fast Company, from Discourse to Inc Magazine. You too, can fail spectacularly, and look good doing it.

Deja vu security Adam Cecchetti - Security is a Snapshot in Time BSidesPDX ...

adamdeja

As the air gap between our daily lives and the Internet continues to shrink the security of our personal data and devices grows in importance. We are facing the daily threat of putting 2000s era computers bolted to toasters online while expecting them to defend against 2017 capable attackers. This talk will explore the continuing trend of IoT, discuss how we’ve been here before, and layout strategies for keeping pace with attackers in the future. This talk will focus on enumerating this risk, discuss the challenges involved, and explore solutions. First, we will examine this history of how we got here, and what it means to say “security is a snapshot in time.” We then introduce the idea of shared ken – the range of one’s knowledge or sight – and how it impacts security. Third, we discuss the influence of data as code, the meta game, and secrecy as a way of mastering impact and ken. This talk will allow attendees to walk away with A holistic view of the history of computer security and how it impacts them today The importance of extending the range of collective vision to reduce blind spots Practical advice for BSiders to grow their mindset and improve their impact Adam is a founding partner and Chief Executive Officer at Deja vu Security. He is dedicated to the leadership and relentless innovation in Deja’s products and services. Previously he has lead teams conducting application and hardware penetration tests for the Fortune 500 technology firms. Adam is a contributing author to multiple security books, benchmarks, tools, and DARPA research projects. Adam holds a degree in Computer Science and a Masters from Carnegie Mellon University in Information Networking.

Quality Assurance 1: Why Quality Matters

Marc Miquel

Enabling effective hunt teaming and incident response

jeffmcjunkin

Coderetreat @Turku 2012 10 20 Adi Bolboaca

Ops Meta-Metrics: The Currency You Pay For ChangeJohn Allspaw

GameDay: Creating Resiliency Through Destruction - LISA11

Jesse Robbins

Human Error Prevention

Toru Nakata

Coderetreat @AgileFinland Tampere 2014 11 12Adi Bolboaca

More Aim, Less Blame: How to use postmortems to turn failures into something ...

Daniel Kanchev

Mistakes and failure are inevitable. Instead of being afraid of them, we should use them as lessons that help identify weak points in our organisations and systems. One way to do this is by writing blameless postmortems. Daniel details exactly how postmortems can help organizations and teams focus on improvement, and how that boosts work morale, makes products better, and strengthens your relationship with customers.

Testing As A Bottleneck - How Testing Slows Down Modern Development Processes...

TEST Huddle

We often claim the purpose of testing is to verify that software meets a desired level of quality. Frequently, the term “testing” is associated with checking for functional correctness. However, in large, complex software systems with an established user-base, it is also important to verify system constraints such as backward compatibility, reliability, security, accessibility, usability. Kim Herzig from Microsoft explores these issues with the latest webinar on test Huddle.

Quantum Computing

Owen Wang

IBME

Owen Wang

Similar to Embracing Failure

Gamification of Chaos Testing

Bram Vogelaar

How to Run a Post-Mortem (With Humans, Not Robots), Velocity 2013

Dan Milstein

Jason Yee - Chaos! - Codemotion Rome 2019

Codemotion

Are We Secure? Answering the Unanswerable

Justin Berman

2016-04-28 - VU Amsterdam - testing safety critical systems

Jaap van Ekris

Win#23 it

Olivier Marechal

debugging.ppt

jerlinS1

The Two Generals Problem

VoltDB

Jeff Atwood, Michael Krakovskiy "The role of catastrophic failure in software...

Fwdays

Deja vu security Adam Cecchetti - Security is a Snapshot in Time BSidesPDX ...

adamdeja

Quality Assurance 1: Why Quality Matters

Marc Miquel

Enabling effective hunt teaming and incident response

jeffmcjunkin

Coderetreat @Turku 2012 10 20 Adi Bolboaca

Ops Meta-Metrics: The Currency You Pay For ChangeJohn Allspaw

GameDay: Creating Resiliency Through Destruction - LISA11

Jesse Robbins

Human Error Prevention

Toru Nakata

Coderetreat @AgileFinland Tampere 2014 11 12Adi Bolboaca

More Aim, Less Blame: How to use postmortems to turn failures into something ...

Daniel Kanchev

Testing As A Bottleneck - How Testing Slows Down Modern Development Processes...

TEST Huddle

Similar to Embracing Failure (20)

Gamification of Chaos Testing

How to Run a Post-Mortem (With Humans, Not Robots), Velocity 2013

Jason Yee - Chaos! - Codemotion Rome 2019

Are We Secure? Answering the Unanswerable

2016-04-28 - VU Amsterdam - testing safety critical systems

Win#23 it

debugging.ppt

The Two Generals Problem

Jeff Atwood, Michael Krakovskiy "The role of catastrophic failure in software...

Deja vu security Adam Cecchetti - Security is a Snapshot in Time BSidesPDX ...

Quality Assurance 1: Why Quality Matters

Enabling effective hunt teaming and incident response

Coderetreat @Turku 2012 10 20

Ops Meta-Metrics: The Currency You Pay For Change

GameDay: Creating Resiliency Through Destruction - LISA11

Human Error Prevention

Coderetreat @AgileFinland Tampere 2014 11 12

More Aim, Less Blame: How to use postmortems to turn failures into something ...

Testing As A Bottleneck - How Testing Slows Down Modern Development Processes...

Recently uploaded

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality

Inflectra

In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring. Learn about: • The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks. • Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective. • Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification. • Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process. Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Product School

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

Product School

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Sri Ambati

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Product School

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

91mobiles

When stars align: studies in data quality, knowledge graphs, and machine lear...

Elena Simperl

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

Product School

How world-class product teams are winning in the AI era by CEO and Founder, P...

Product School

PHP Frameworks: I want to break free (IPC Berlin 2024)

Ralf Eggert

In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development. This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

Bits & Pixels using AI for Good.........

Alison B. Lowndes

UiPath Test Automation using UiPath Test Suite series, part 3

DianaGray10

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams. Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Product School

Recently uploaded (20)

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

When stars align: studies in data quality, knowledge graphs, and machine lear...

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

How world-class product teams are winning in the AI era by CEO and Founder, P...

PHP Frameworks: I want to break free (IPC Berlin 2024)

GraphRAG is All You need? LLM & Knowledge Graph

The Art of the Pitch: WordPress Relationships and Sales

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

Bits & Pixels using AI for Good.........

UiPath Test Automation using UiPath Test Suite series, part 3

Neuro-symbolic is not enough, we need neuro-*semantic*

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Embracing Failure

1. Embracing Failure (not my life story)

4. Setting the Mood •Understand that they WILL happen •Failures are not binary •Impact determines importance •deadlines for fixes are variable

5. Terminology •Website •Production •Downtime

6. Monitor Failures

7. What is Monitoring? •Graphs. Everywhere. •Alerts on failures •phone calls •texts •Answers: Are we failing?

9. healthcare.gov •Know when you’re down before CNN

10.

11. Postmortems (fool me once. shame on you. fool me twice. shame on me.)

12. Postmortems 1. Reconstruct the factual timeline 2. Root cause analysis 3. Remediation items

13. Postmortems •Why did we fail? •Blameless •Moderated

14. Gamedays (You wouldn’t wing a talk. Don’t wing a hot fix)

15. Gameday •Best defense is a good offense •Simulate possible failures •Do it in production

16. kill -9 1. Draw a block diagram 2. Cut every connection 3. Watch the fireworks

17. SafeMachine (like a state machine … but safer)

18. Try, Try, Try again •What if we could just retry failures? •Side effects are the root of all evil •Safe failures vs Unsafe failures

19. What’s in a SafeMachine •Actions •States START Computed File Uploaded File END compute upload record successful

20. initialize_succeeded initialize_failed initialize_inprogress computed_succeeded

21. START a1 a1 a2 a2 a2 a3 a3 a3 END The Pipeline

22. The Pipeline START Computed File Uploaded File END Safe Unsafe Safe

23. Embracing Failure •Monitor •Postmortems •Gamedays - you wouldn’t wing a talk? •SafeMachine

24. @chriswu_

25. Additional resources • Postmortems https://codeascraft.com/2012/05/22/blameless- postmortems/ • Gamedays - https://stripe.com/blog/game-day-exercises-at-stripe • links at the bottom of this post are also great • Error Tracking - https://getsentry.com/welcome/

Embracing Failure

Recommended

Recommended

More Related Content

Similar to Embracing Failure

Similar to Embracing Failure (20)

More from Owen Wang

More from Owen Wang (19)

Recently uploaded

Recently uploaded (20)

Embracing Failure