What can a real world disaster teach us about running better projects, changing our attitudes and putting our best foot forward as professionals and technologists? Quite a lot actually. In this very interactive presentation, I discuss attitudes and actions that lead to disaster, how we can find them in ourselves and some tools and techniques for avoiding them.

1. Iceberg, Dead Ahead! Lessons from real life disasters for the database professional 5/11/2011 – Mike Walsh

2. About Mike SQL Server MVP SQL Server Practice Lead at Winxnet Seacoast SQL Server Users Group (NH) http://seacoastsql.sqlpass.org Blog: http://www.StraightPathSQL.com Former Firefighter/EMT-I Paranoid? Worry-Wart? Well Qualified to Talk About Failing asfTitle 2

3. Real Disasters Will Be Discussed Professional Development No T-SQL & No Demos Obviously! Iceberg, Dead Ahead! 3 You Can Leave

4. The Main Point “I have yet to be in a game where luck was involved. Well-prepared players make plays. I have yet to be in a game where the most prepared team didn't win.” – Urban Meyer

5. Goals Understand Disaster Causing Attitudes Leave Prepared Iceberg, Dead Ahead! 5

6. Our Approach Some Accidents One Success Plagiarism Iceberg, Dead Ahead! 6

7. Avianca 52 Bogota to JFK in 1990 Boeing 707 Missed One Approach Impacted Hill – 73 of 158 Died Why? Iceberg, Dead Ahead! 7

8. Contributors To Crash Ultimately: Flame Out Poor Planning/Dispatch No updated weather info, no follow up Failure To Declare Emergency Culture? Disobedience? Language? “Avianca fifty two, climb and maintain three thousand.” “Ah, negative sir. We just running out of fuel. We okay three thousand. Now okay.” Failure to Divert Earlier Aborted First Landing Attempt Iceberg, Dead Ahead! 8

9. Our Lessons From Avianca 52 Plan For The Worst Rollbacks Go/No-Go Meetings Scenario Based Communication Phraseology Ask For Help In Spite Of Barriers Pride? Politeness? Iceberg, Dead Ahead! 9

10. United 173 DC8 From Denver to Portland, 12/28/78 Happened On Approach Landing Gear Issue Loud thud and yaw when lowered Indicator light failed to indicate Eventual Crash Landing, 10 of 189 died Why? Iceberg, Dead Ahead! 10

11. What Went Wrong? Gear Check Light Troubleshooting That took much longer than it should have Evacuation Discussion/Preparation That took much longer than it should have Captain Paid No Attention To Fuel Remaining Crew’s Fuel Attitude Concerned but not direct/forceful Corrosion On Landing Gear Indicator Wires Iceberg, Dead Ahead! 11

12. Our Lessons From UAL 173 Troubleshooting Skills Don’t “Stick” On One Data Point Don’t Forget To Fly While Troubleshooting Communication Having “A Team of Equal Weight” Team Oriented Leadership Status Checks Routine Maintenance Iceberg, Dead Ahead! 12

13. Air Canada 621 DC8 From Montreal in 1970 Well Acquainted & Experienced Flight Crew Plane Bounced Off Runway, Lost an Engine Broke Apart as Pilot Tried to Salvage Landing All 109 On Board Killed Anyone know why? Iceberg, Dead Ahead! 13

14. What Happened? Some History Air Canada’s rule – Arm on approach Capt./FO Disagreed & Compromised Capt to FO: “When you fly we arm on approach” FO to Capt: “When you fly we arm/deploy on ground” Captain Changed Mind During Approach “Give them to me on the flare. I have given up. I’m tired of fighting it” First Officer Had Routine Broken Twice No External Factors Iceberg, Dead Ahead! 14

15. Our Lessons From Air Canada 621 Policies and Routines Can Be Good Exceptions Can Be Dangerous Don’t Ignore Non-Compliance of Standards Plan & Document Approach Before Doing “Peer Pressure” Pitfalls Iceberg, Dead Ahead! 15

16. Complacency “Those masks give you headaches, you can’t talk with them on and the statistics are pretty low of sudden depressurization anyway. Have you ever seen them?” – A Learjet pilot Payne Stewart’s Crash – Masks Part of Cause? Monitoring/Warning Systems Qantas Near Miss? Sterile Cockpit Recent Controller “Issues” Iceberg, Dead Ahead! 16

17. Tenerife Island – Los Rodeos Airport Two Young & “Clean” 747s Experienced Pilots Deadliest Aviation Disaster In History Iceberg, Dead Ahead! 17

18. What Happened At Tenerife? Additive Factors Airport Diversion Sudden Heavy Fog Understaffed Tower Clogged Runway Back Taxiing 12 Feet 700 yards Communication, Attitude, Teamwork Iceberg, Dead Ahead! 18

19. Two Cockpits – Two Conversations KLM Changed Routine ATC Clearance Late Impatience “We’re Going” “We are Now At uh Take-off” “Is He Not Clear, That Pan American?” “Oh Yes” Pan-Am Confusion On Turnoff Disagreement w/ Tower? “And We’re Still On The Runway”

20. Tenerife Lessons For Us Departures From Routine Don’t End Well Having “A Team of Equal Weight” Team Oriented Leadership Communication Phraseology & Language Clear & Concise Confirmation Make Calm Decisions Iceberg, Dead Ahead! 20

21. US Airways Flight 1549 Source: commons.wikimedia.com Iceberg, Dead Ahead! 21

22. Why Did 1549 Turn Out Like That? “I have yet to be in a game where luck was involved. Well-prepared players make plays. I have yet to be in a game where the most prepared team didn't win.” – Urban Meyer Iceberg, Dead Ahead! 22

23. Or Maybe? “Luck Favors The Prepared” - Louis Pasteur

24. 1549’s CVR Highlights “My aircraft”“Your aircraft” “Get the QRH” “If fuel remaining, engine mode selector..” “This is the Captain, brace for impact” “Got any ideas?” “Actually not” “We’re gonna brace” During This Time – Calm ATC contact, checklists, multiple attempts to relight/power engines. Iceberg, Dead Ahead! 24

25. We Can Land On The Hudson, Too Troubleshooter Wanted – Leave Shotgun Home Checklists (“QRH”) Teamwork (“Got any ideas?”) Notification (“Brace for impact”, “We’re gonna be in the Hudson”) Clear, Concise Communication (“Unable”) Training & Muscle Memory Iceberg, Dead Ahead! 25

26. Checklists Routine and Emergency Think About What Can Go Wrong Test Checklist Keep Best Practices In Mind Iceberg, Dead Ahead! 26

27. Iceberg, Dead Ahead! 27

28. Best Practices & Standards Don’t Just Write Them Follow Them Understand Them Explain The “Why” Discuss, Plan and Document Variances We Need To Move Away from “Do as I Say” Iceberg, Dead Ahead! 28

29. Simulators Can Help Who Has Practiced a Restore? With log backups? With the tail backed up? What Happens When _____ ? Test It – You Can’t Know Unless/Until You Do Iceberg, Dead Ahead! 29

30. Lessons Learned The Point Isn’t Blame Identify Lapses Prevent Future Occurrences Some Failures/Near Misses Can be Blessings Iceberg, Dead Ahead! 30

31. Plan For The Worst, Expect The Best Level 1 text is 28 pt Arial Level 2 text is 24 pt Arial Level 3 text is 20 pt Arial Level 4 text is 20 pt Arial Level 5 text is 20 pt Arial Iceberg, Dead Ahead! 31

32. Your Homework Test Something (Restore Plans?) Create a Checklist (Disaster Response?) Get Paranoid (But only a little) Ask for a Postmortem (Sedation Helps First) Adopt and Write Standards (Sensible) Find your flaws (Review the accidents for tips) Iceberg, Dead Ahead! 32

33. We Saw Failures In Communication Teamwork Planning Standards/Policies Routine Attitude Iceberg, Dead Ahead! 33

34. We Saw Failures In Troubleshooting Narrowly Appreciating Need for Help Assertiveness Listening Iceberg, Dead Ahead! 34

35. Slides, Notes and Links www.straightpathsql.com/disaster mwalsh@winxnet.com THANKS! Iceberg, Dead Ahead! 35

36. Please Complete the Evaluation Form Pick up your evaluation form: In each presentation room Drop off your completed form Near the exit of each presentation room At the registration area Presented by Dell Session Title 36

37. THANK YOU! For attending this session and PASS SQLRally Orlando, Florida Presented by Dell Session Code | Session Title 37