Mistakes were made - LCA 2012

Mistakes were made Selena Deckelmann selena@primeradiant.com Twitter/IRC: @selenamarieSo mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

FailureLC A 20 12

“Prevention” “Risk management” “Risk mitigation” “MTBF, MTTR” “Success Engineering”LC A 20 12

Plan for the worst. Minimize risk. Fail. Recover, gracefully.LC A 20 12

“We don’t need a risk management plan,” he emphatically stated, “because this project can’t be allowed to fail.” - Jim Hightower, http://jimhighsmith.com/2012/01/09/can-do-thinking-makes-risk- management-impossible/LC A 20 12

x 2 210 01 E CAALSLC

Failure is an option.LC A 20 12

SCIENCELC A 20 12

Dr. Jerker Denrell LC A 20 12

"I think getting two accidents of this type at the same time is a freak occurrence." -David Cunliffe, NZ Communications MinisterLC A 20 12

“Further damage was incurred on Tuesday afternoon and our engineers returned to repair the damage,” said Virgin Media.SLC CAAL 01E 2 10 2 x

Plan for when things fail.LC A 20 12

Tales of failure to... Document Test Verify Imagine ImplementSo mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

Failure to document.SLC CAAL 01E 2 10 2 x

Moving Day Thanks, David Prior!SLC CAAL 01E 2 10 2 x

Prevent documentation failures. • Write documentation. • Update documentation. • Make documenting a step in your written process. • Assign a ﬁxed amount of time to that step.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

Documentation tools • Graphic designers. (Pretty wikis. Pretty docs. (Sphinx?) Diagrams.) • Timelines. • Bug tracking. • Ordered todo lists.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

Failure to test.SLC CAAL 01E 2 10 2 x

“My ﬁrst day posing as a sysadmin (~1990, no previous training....) I deleted all zero length ﬁles on a Sun workstation.”LC A 20 12

Prevent testing failures. • Verify success criteria. • Write tests. • Test with a buddy. • Have a plan.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

Testing tools • Your favorite test framework • Repeatable shell scripts • Staging environmentsSo mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

Failure to verify.SLC CAAL 01E 2 10 2 x

“What does ‘-d’ actually do?”LC A 20 12

Prevent veriﬁcation failures. • Have a plan for things going wrong. • Have a staging environment. • Test your rollback plan, not just your implementation plan.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

Veriﬁcation tools • Staging environments • Your buddySo mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

Failure to imagine.LC A 20 12

For my group the bottom line was "dont trust anyone". Thanks, Maggie!LC A 20 12

Recover from failures to imagine. • Share your stories of failure. • Talk with people who are different from you. • Act out implementation scenarios.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

Failure to implement.LC A 20 12

Re-implement. • Learn from mistakes.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

Reﬂection. (or, the Post-Mortem)LC A 20 12

Before • Plan to do a post-mortem. • Document the plan with numbered steps and a timeline. • Test the plan and the rollback plan. • Identify a “point of no return”.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

During • Screen sharing: UNIX screen,VNC, etc. • Chatroom: IRC, AIM, Campﬁre (scrollback!) • Voice: Campﬁre, Skype,VOIP, POTS call line • Headsets! • Designated time-keeper.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

After • Documentation updates • Post-mortem to identify areas of success and areas for improvement. • Limit improvements to 1-2 things.So mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

Plan for the worst. Minimize risk. Fail. Recover, gracefully.LC A 20 12

Thanks! xn 0e 1r2 ce ne 2 f1 E0 Ao CL CA SeC mL So

Mistakes were made Selena Deckelmann selena@primeradiant.com Twitter/IRC: @selenamarieSo mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

Photo credits • Flickr: sheepguardingllamaSo mL SeC CA CL Ao E0 2 f1 ne 1r2 0e xn e c

Mistakes were made - LCA 2012

by Selena Deckelmann on Jan 17, 2012

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 23

Statistics

Mistakes were made - LCA 2012 — Presentation Transcript

http://lanyrd.com	22
http://a0.twimg.com	1