Mistakes were made - LCA 2012

1,566 views
1,526 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,566
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Mistakes were made - LCA 2012

  1. 1. Mistakes were made Selena Deckelmann selena@primeradiant.com Twitter/IRC: @selenamarieSo mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  2. 2. FailureLC A 20 12
  3. 3. “Prevention” “Risk management” “Risk mitigation” “MTBF, MTTR” “Success Engineering”LC A 20 12
  4. 4. Plan for the worst. Minimize risk. Fail. Recover, gracefully.LC A 20 12
  5. 5. “We don’t need a risk management plan,” he emphatically stated, “because this project can’t be allowed to fail.” - Jim Hightower, http://jimhighsmith.com/2012/01/09/can-do-thinking-makes-risk- management-impossible/LC A 20 12
  6. 6. x 2 210 01 E CAALSLC
  7. 7. Failure is an option.LC A 20 12
  8. 8. SCIENCELC A 20 12
  9. 9. Dr. Jerker Denrell LC A 20 12
  10. 10. x 2 210 01 E CAALSLC
  11. 11. x 2 210 01 E CAALSLC
  12. 12. x 2 210 01 E CAALSLC
  13. 13. "I think getting two accidents of this type at the same time is a freak occurrence." -David Cunliffe, NZ Communications MinisterLC A 20 12
  14. 14. x 2 210 01 E CAALSLC
  15. 15. “Further damage was incurred on Tuesday afternoon and our engineers returned to repair the damage,” said Virgin Media.SLC CAAL 01E 2 10 2 x
  16. 16. Plan for when things fail.LC A 20 12
  17. 17. x 2 210 01 E CAALSLC
  18. 18. x 2 210 01 E CAALSLC
  19. 19. Tales of failure to... Document Test Verify Imagine ImplementSo mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  20. 20. Failure to document.SLC CAAL 01E 2 10 2 x
  21. 21. Moving Day Thanks, David Prior!SLC CAAL 01E 2 10 2 x
  22. 22. Prevent documentation failures. • Write documentation. • Update documentation. • Make documenting a step in your written process. • Assign a fixed amount of time to that step.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  23. 23. Documentation tools • Graphic designers. (Pretty wikis. Pretty docs. (Sphinx?) Diagrams.) • Timelines. • Bug tracking. • Ordered todo lists.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  24. 24. Failure to test.SLC CAAL 01E 2 10 2 x
  25. 25. “My first day posing as a sysadmin (~1990, no previous training....) I deleted all zero length files on a Sun workstation.”LC A 20 12
  26. 26. Prevent testing failures. • Verify success criteria. • Write tests. • Test with a buddy. • Have a plan.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  27. 27. Testing tools • Your favorite test framework • Repeatable shell scripts • Staging environmentsSo mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  28. 28. Failure to verify.SLC CAAL 01E 2 10 2 x
  29. 29. “What does ‘-d’ actually do?”LC A 20 12
  30. 30. Prevent verification failures. • Have a plan for things going wrong. • Have a staging environment. • Test your rollback plan, not just your implementation plan.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  31. 31. Verification tools • Staging environments • Your buddySo mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  32. 32. Failure to imagine.LC A 20 12
  33. 33. For my group the bottom line was "dont trust anyone". Thanks, Maggie!LC A 20 12
  34. 34. Recover from failures to imagine. • Share your stories of failure. • Talk with people who are different from you. • Act out implementation scenarios.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  35. 35. Failure to implement.LC A 20 12
  36. 36. Re-implement. • Learn from mistakes.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  37. 37. Reflection. (or, the Post-Mortem)LC A 20 12
  38. 38. Before • Plan to do a post-mortem. • Document the plan with numbered steps and a timeline. • Test the plan and the rollback plan. • Identify a “point of no return”.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  39. 39. During • Screen sharing: UNIX screen,VNC, etc. • Chatroom: IRC, AIM, Campfire (scrollback!) • Voice: Campfire, Skype,VOIP, POTS call line • Headsets! • Designated time-keeper.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  40. 40. After • Documentation updates • Post-mortem to identify areas of success and areas for improvement. • Limit improvements to 1-2 things.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  41. 41. Plan for the worst. Minimize risk. Fail. Recover, gracefully.LC A 20 12
  42. 42. Thanks! xn 0e 1r2 ce ne 2 f1 E0 Ao CL CA SeC mL So
  43. 43. Mistakes were made Selena Deckelmann selena@primeradiant.com Twitter/IRC: @selenamarieSo mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c
  44. 44. Photo credits • Flickr: sheepguardingllamaSo mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

×