SlideShare a Scribd company logo
Mistakes were made
                             Selena Deckelmann
                         selena@primeradiant.com
                         Twitter/IRC: @selenamarie
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c
Failure
LC
 A
 20
   12
“Prevention”
         “Risk management”
          “Risk mitigation”
           “MTBF, MTTR”
        “Success Engineering”
LC
 A
 20
   12
Plan for the worst.
        Minimize risk.
        Fail.
        Recover, gracefully.
LC
 A
 20
   12
“We don’t need a risk
      management plan,” he
      emphatically stated, “because this
      project can’t be allowed to fail.”
                                                   - Jim Hightower,
     http://jimhighsmith.com/2012/01/09/can-do-thinking-makes-risk-
                                           management-impossible/
LC
 A
 20
   12
x
           2
       210
        01
       E
  CAAL
SLC
Failure is an option.
LC
 A
 20
   12
SCIENCE
LC
 A
 20
   12
Dr. Jerker Denrell 
LC
 A
 20
   12
x
           2
       210
        01
       E
  CAAL
SLC
x
           2
       210
        01
       E
  CAAL
SLC
x
           2
       210
        01
       E
  CAAL
SLC
"I think getting two accidents
        of this type at the same time
            is a freak occurrence."
             -David Cunliffe, NZ Communications Minister
LC
 A
 20
   12
x
           2
       210
        01
       E
  CAAL
SLC
“Further damage was incurred
            on Tuesday afternoon and our
            engineers returned to repair
            the damage,” said Virgin Media.
SLC
  CAAL
     01E
       2
     10
        2
        x
Plan for when things fail.
LC
 A
 20
   12
x
           2
       210
        01
       E
  CAAL
SLC
x
           2
       210
        01
       E
  CAAL
SLC
Tales of failure to...
                      Document
                      Test
                      Verify
                      Imagine
                      Implement
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c
Failure to document.
SLC
  CAAL
     01E
       2
     10
        2
        x
Moving Day




                    Thanks, David Prior!
SLC
  CAAL
     01E
       2
     10
        2
        x
Prevent documentation
                             failures.
                      • Write documentation.
                      • Update documentation.
                      • Make documenting a step in your written
                        process.
                      • Assign a fixed amount of time to that step.
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c
Documentation tools

                      • Graphic designers. (Pretty wikis. Pretty
                        docs. (Sphinx?) Diagrams.)
                      • Timelines.
                      • Bug tracking.
                      • Ordered todo lists.
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c
Failure to test.
SLC
  CAAL
     01E
       2
     10
        2
        x
“My first day posing as a sysadmin
        (~1990, no previous training....) I
        deleted all zero length files on a Sun
        workstation.”
LC
 A
 20
   12
Prevent testing failures.

                      • Verify success criteria.
                      • Write tests.
                      • Test with a buddy.
                      • Have a plan.
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c
Testing tools

                      • Your favorite test framework
                      • Repeatable shell scripts
                      • Staging environments
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c
Failure to verify.
SLC
  CAAL
     01E
       2
     10
        2
        x
“What does ‘-d’ actually do?”
LC
 A
 20
   12
Prevent verification
                              failures.

                      • Have a plan for things going wrong.
                      • Have a staging environment.
                      • Test your rollback plan, not just your
                        implementation plan.
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c
Verification tools


                      • Staging environments
                      • Your buddy
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c
Failure to imagine.
LC
 A
 20
   12
For my group the
          bottom line was
        "don't trust anyone".

                     Thanks, Maggie!
LC
 A
 20
   12
Recover from failures
                          to imagine.
                      • Share your stories of failure.
                      • Talk with people who are different from
                        you.
                      • Act out implementation scenarios.
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c
Failure to implement.
LC
 A
 20
   12
Re-implement.


                      • Learn from mistakes.
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c
Reflection.
        (or, the Post-Mortem)
LC
 A
 20
   12
Before

                      • Plan to do a post-mortem.
                      • Document the plan with numbered steps
                        and a timeline.
                      • Test the plan and the rollback plan.
                      • Identify a “point of no return”.
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c
During

                      • Screen sharing: UNIX screen,VNC, etc.
                      • Chatroom: IRC, AIM, Campfire (scrollback!)
                      • Voice: Campfire, Skype,VOIP, POTS call line
                      • Headsets!
                      • Designated time-keeper.
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c
After

                      • Documentation updates
                      • Post-mortem to identify areas of success
                        and areas for improvement.
                      • Limit improvements to 1-2 things.
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c
Plan for the worst.
        Minimize risk.
        Fail.
        Recover, gracefully.
LC
 A
 20
   12
Thanks!                  xn
                        0e
                       1r2
                            ce
                   ne
                   2 f1
                   E0
                 Ao
                CL
              CA
             SeC
            mL
          So
Mistakes were made
                             Selena Deckelmann
                         selena@primeradiant.com
                         Twitter/IRC: @selenamarie
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c
Photo credits


                      • Flickr: sheepguardingllama
So
  mL
   SeC
    CA
      CL
       Ao
         E0
         2 f1
         ne
             1r2
              0e
               xn
                e c

More Related Content

Viewers also liked (10)

Twitter User Hype Cycle
Twitter User Hype CycleTwitter User Hype Cycle
Twitter User Hype Cycle
 
Saxion 7 januari 2008
Saxion 7 januari 2008Saxion 7 januari 2008
Saxion 7 januari 2008
 
Adobe LiveCycle Data Services
Adobe LiveCycle Data ServicesAdobe LiveCycle Data Services
Adobe LiveCycle Data Services
 
Fm Mc Presentation Ria2008
Fm Mc   Presentation Ria2008Fm Mc   Presentation Ria2008
Fm Mc Presentation Ria2008
 
Assertiveness
AssertivenessAssertiveness
Assertiveness
 
Create folder in start Menu in Windows 7
Create folder in start Menu in Windows 7Create folder in start Menu in Windows 7
Create folder in start Menu in Windows 7
 
лезин
лезинлезин
лезин
 
Max2013 rejected apps presentation
Max2013   rejected apps presentationMax2013   rejected apps presentation
Max2013 rejected apps presentation
 
クロスブラウザ拡張ライブラリExtension.js
クロスブラウザ拡張ライブラリExtension.js クロスブラウザ拡張ライブラリExtension.js
クロスブラウザ拡張ライブラリExtension.js
 
Australia PowerPoint Content
Australia PowerPoint Content Australia PowerPoint Content
Australia PowerPoint Content
 

More from Selena Deckelmann

While we're here, let's fix computer science education
While we're here, let's fix computer science educationWhile we're here, let's fix computer science education
While we're here, let's fix computer science education
Selena Deckelmann
 
Harder, better, faster, stronger: PostgreSQL 9.1
Harder, better, faster, stronger: PostgreSQL 9.1Harder, better, faster, stronger: PostgreSQL 9.1
Harder, better, faster, stronger: PostgreSQL 9.1
Selena Deckelmann
 
Own it: working with a changing open source community
Own it: working with a changing open source communityOwn it: working with a changing open source community
Own it: working with a changing open source community
Selena Deckelmann
 
Managing terabytes: When Postgres gets big
Managing terabytes: When Postgres gets bigManaging terabytes: When Postgres gets big
Managing terabytes: When Postgres gets big
Selena Deckelmann
 

More from Selena Deckelmann (20)

While we're here, let's fix computer science education
While we're here, let's fix computer science educationWhile we're here, let's fix computer science education
While we're here, let's fix computer science education
 
Algorithms are Recipes
Algorithms are RecipesAlgorithms are Recipes
Algorithms are Recipes
 
Hire the right way
Hire the right wayHire the right way
Hire the right way
 
Pg92 HA, LCA 2012, Ballarat
Pg92 HA, LCA 2012, BallaratPg92 HA, LCA 2012, Ballarat
Pg92 HA, LCA 2012, Ballarat
 
Managing terabytes
Managing terabytesManaging terabytes
Managing terabytes
 
Mistakes were made
Mistakes were madeMistakes were made
Mistakes were made
 
Postgres needs an aircraft carrier
Postgres needs an aircraft carrierPostgres needs an aircraft carrier
Postgres needs an aircraft carrier
 
Mistakes were made
Mistakes were madeMistakes were made
Mistakes were made
 
Harder, better, faster, stronger: PostgreSQL 9.1
Harder, better, faster, stronger: PostgreSQL 9.1Harder, better, faster, stronger: PostgreSQL 9.1
Harder, better, faster, stronger: PostgreSQL 9.1
 
How to ask for money
How to ask for moneyHow to ask for money
How to ask for money
 
Letters from the open source trenches - Postgres community
Letters from the open source trenches - Postgres communityLetters from the open source trenches - Postgres community
Letters from the open source trenches - Postgres community
 
Own it: working with a changing open source community
Own it: working with a changing open source communityOwn it: working with a changing open source community
Own it: working with a changing open source community
 
Managing terabytes: When Postgres gets big
Managing terabytes: When Postgres gets bigManaging terabytes: When Postgres gets big
Managing terabytes: When Postgres gets big
 
Managing terabytes: When PostgreSQL gets big
Managing terabytes: When PostgreSQL gets bigManaging terabytes: When PostgreSQL gets big
Managing terabytes: When PostgreSQL gets big
 
Pdxpugday2010 pg90
Pdxpugday2010 pg90Pdxpugday2010 pg90
Pdxpugday2010 pg90
 
Making Software Communities
Making Software CommunitiesMaking Software Communities
Making Software Communities
 
Illustrated buffer cache
Illustrated buffer cacheIllustrated buffer cache
Illustrated buffer cache
 
Bucardo
BucardoBucardo
Bucardo
 
How a bunch of normal people Used Technology To Repair a Rigged Election
How a bunch of normal people Used Technology To Repair a Rigged ElectionHow a bunch of normal people Used Technology To Repair a Rigged Election
How a bunch of normal people Used Technology To Repair a Rigged Election
 
Open Source Bridge Opening Day
Open Source Bridge Opening DayOpen Source Bridge Opening Day
Open Source Bridge Opening Day
 

Recently uploaded

Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
Search and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical FuturesSearch and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical Futures
Bhaskar Mitra
 

Recently uploaded (20)

AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří Karpíšek
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
Search and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical FuturesSearch and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical Futures
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
 
In-Depth Performance Testing Guide for IT Professionals
In-Depth Performance Testing Guide for IT ProfessionalsIn-Depth Performance Testing Guide for IT Professionals
In-Depth Performance Testing Guide for IT Professionals
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
 
Exploring UiPath Orchestrator API: updates and limits in 2024 🚀
Exploring UiPath Orchestrator API: updates and limits in 2024 🚀Exploring UiPath Orchestrator API: updates and limits in 2024 🚀
Exploring UiPath Orchestrator API: updates and limits in 2024 🚀
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 

Mistakes were made - LCA 2012

  • 1. Mistakes were made Selena Deckelmann selena@primeradiant.com Twitter/IRC: @selenamarie So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  • 3. “Prevention” “Risk management” “Risk mitigation” “MTBF, MTTR” “Success Engineering” LC A 20 12
  • 4. Plan for the worst. Minimize risk. Fail. Recover, gracefully. LC A 20 12
  • 5. “We don’t need a risk management plan,” he emphatically stated, “because this project can’t be allowed to fail.” - Jim Hightower, http://jimhighsmith.com/2012/01/09/can-do-thinking-makes-risk- management-impossible/ LC A 20 12
  • 6. x 2 210 01 E CAAL SLC
  • 7. Failure is an option. LC A 20 12
  • 10. x 2 210 01 E CAAL SLC
  • 11. x 2 210 01 E CAAL SLC
  • 12. x 2 210 01 E CAAL SLC
  • 13. "I think getting two accidents of this type at the same time is a freak occurrence." -David Cunliffe, NZ Communications Minister LC A 20 12
  • 14. x 2 210 01 E CAAL SLC
  • 15. “Further damage was incurred on Tuesday afternoon and our engineers returned to repair the damage,” said Virgin Media. SLC CAAL 01E 2 10 2 x
  • 16. Plan for when things fail. LC A 20 12
  • 17. x 2 210 01 E CAAL SLC
  • 18. x 2 210 01 E CAAL SLC
  • 19. Tales of failure to... Document Test Verify Imagine Implement So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  • 20. Failure to document. SLC CAAL 01E 2 10 2 x
  • 21. Moving Day Thanks, David Prior! SLC CAAL 01E 2 10 2 x
  • 22. Prevent documentation failures. • Write documentation. • Update documentation. • Make documenting a step in your written process. • Assign a fixed amount of time to that step. So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  • 23. Documentation tools • Graphic designers. (Pretty wikis. Pretty docs. (Sphinx?) Diagrams.) • Timelines. • Bug tracking. • Ordered todo lists. So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  • 24. Failure to test. SLC CAAL 01E 2 10 2 x
  • 25. “My first day posing as a sysadmin (~1990, no previous training....) I deleted all zero length files on a Sun workstation.” LC A 20 12
  • 26. Prevent testing failures. • Verify success criteria. • Write tests. • Test with a buddy. • Have a plan. So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  • 27. Testing tools • Your favorite test framework • Repeatable shell scripts • Staging environments So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  • 28. Failure to verify. SLC CAAL 01E 2 10 2 x
  • 29. “What does ‘-d’ actually do?” LC A 20 12
  • 30. Prevent verification failures. • Have a plan for things going wrong. • Have a staging environment. • Test your rollback plan, not just your implementation plan. So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  • 31. Verification tools • Staging environments • Your buddy So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  • 33. For my group the bottom line was "don't trust anyone". Thanks, Maggie! LC A 20 12
  • 34. Recover from failures to imagine. • Share your stories of failure. • Talk with people who are different from you. • Act out implementation scenarios. So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  • 36. Re-implement. • Learn from mistakes. So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  • 37. Reflection. (or, the Post-Mortem) LC A 20 12
  • 38. Before • Plan to do a post-mortem. • Document the plan with numbered steps and a timeline. • Test the plan and the rollback plan. • Identify a “point of no return”. So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  • 39. During • Screen sharing: UNIX screen,VNC, etc. • Chatroom: IRC, AIM, Campfire (scrollback!) • Voice: Campfire, Skype,VOIP, POTS call line • Headsets! • Designated time-keeper. So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  • 40. After • Documentation updates • Post-mortem to identify areas of success and areas for improvement. • Limit improvements to 1-2 things. So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  • 41. Plan for the worst. Minimize risk. Fail. Recover, gracefully. LC A 20 12
  • 42. Thanks! xn 0e 1r2 ce ne 2 f1 E0 Ao CL CA SeC mL So
  • 43. Mistakes were made Selena Deckelmann selena@primeradiant.com Twitter/IRC: @selenamarie So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  • 44. Photo credits • Flickr: sheepguardingllama So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c