Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
11The ValidationThe ValidationAttitudeAttitudeBob ColwellBob ColwellApril 2010April 2010
22AttitudeAttitudeI could talk about techniques, tools,I could talk about techniques, tools,FVFVEnvironments, algorithms...
334/4/07 Bob ColwellNo Perfect DesignsNo Perfect Designs Nothing is perfect, everything has bugsNothing is perfect, every...
44Prescription: SW visualization, tools to localizebugs, diagnose problems, and instrument behavior
554/4/07 Bob ColwellAccidents Are InevitableAccidents Are Inevitable– Its the nature of engineeringIts the nature of engin...
664/4/07 Bob ColwellIsn’t that justIsn’t that just ??Close. But Murphy is notClose. But Murphy is notquite right.quite rig...
774/4/07 Bob ColwellFailures Getting WorseFailures Getting Worse Mechanical things usually fail predictably due to physic...
88The Steely-Eyed Missile ValidatorThe Steely-Eyed Missile ValidatorApollo 12Apollo 12 22ndndtry to land on moon, launch...
99Complexity Implies SurprisesComplexity Implies Surprises……and surprises areand surprises are badbadChaos effects in co...
1010Thinking about validationThinking about validationAbility to think in analogies is highestAbility to think in analogi...
1111Listen to Your Inner ValidatorListen to Your Inner ValidatorYouYou knewknew it wouldn’t be 3, didn’t you?it wouldn’t ...
1212Lesson: Trust NothingLesson: Trust NothingHyatt RegencyHyatt Regencyhotel, Missouri,hotel, Missouri,19801980Catwalks...
1313What Happened?What Happened?Spec was marginalSpec was marginal40’ threaded rods40’ threaded rods“too hard”, changed“...
1414Therac-25Therac-25Medical particleMedical particleacceleratoracceleratorElectrons,Electrons,protons, X-raysprotons, ...
1515Question EverythingQuestion EverythingTestTest assumptionsassumptions as well as designas well as design– If assumpti...
1616Fight Urge to Relax RequirementsFight Urge to Relax RequirementsChallengerChallenger– Not ok to slip design assumptio...
1717If You Didn’t Test It,If You Didn’t Test It,It Doesn’t WorkIt Doesn’t WorkMir: fire extinguishersMir: fire extinguish...
18184/4/07 Bob ColwellComplexity Makes Everything WorseComplexity Makes Everything Worse Some things must be complicated ...
1919What To DoWhat To Do Get the spec rightGet the spec right Design for correctness but…Design for correctness but… de...
20204/4/07 Bob ColwellRolesRolesEngineers must standEngineers must standtheir groundtheir ground– There are always doubts,...
2121Roles cont.Roles cont.ManagementManagement– wants to assume a product is safewants to assume a product is safe– knows...
2222Future Directions:Future Directions:Public ExpectationsPublic ExpectationsAndy Grove’s FDIV epiphanyAndy Grove’s FDIV ...
2323Future of ValidationFuture of ValidationMultiple Culture Changes NeededMultiple Culture Changes NeededPublic needs to ...
2424The EndThe End
Upcoming SlideShare
Loading in …5
×

The Validation Attitude

0 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

The Validation Attitude

  1. 1. 11The ValidationThe ValidationAttitudeAttitudeBob ColwellBob ColwellApril 2010April 2010
  2. 2. 22AttitudeAttitudeI could talk about techniques, tools,I could talk about techniques, tools,FVFVEnvironments, algorithms, machineryEnvironments, algorithms, machineryLanguages, suites, trainingLanguages, suites, trainingbut I thinkbut I think attitudeattitude is more importantis more importantthan any of thosethan any of those
  3. 3. 334/4/07 Bob ColwellNo Perfect DesignsNo Perfect Designs Nothing is perfect, everything has bugsNothing is perfect, everything has bugs– Shortcomings, compromises, defects, design errata, gaffes, goofs,Shortcomings, compromises, defects, design errata, gaffes, goofs,fumbles, errors, boneheaded mistakes, bobbles, bungles, boo-boosfumbles, errors, boneheaded mistakes, bobbles, bungles, boo-boos– But not all bugs are equal!But not all bugs are equal! Can’t test to saturation: schedule matters tooCan’t test to saturation: schedule matters too Why is everything always so darned buggy?Why is everything always so darned buggy?– Software…need say no more…Software…need say no more…– Why did Titanic not have waterproof compartments?Why did Titanic not have waterproof compartments?– Why did Ford Pinto have gas tank in back?Why did Ford Pinto have gas tank in back?– Why did Challenger fly with leaky O-rings?Why did Challenger fly with leaky O-rings?– Why did torpedoes not explode in WWII?Why did torpedoes not explode in WWII?Entropy has a preferred directionEntropy has a preferred directionOnly genius could paint Mona Lisa,Only genius could paint Mona Lisa,but any small child can destroy it quicklybut any small child can destroy it quickly1000 ways to do things wrong, 1 or 2 that work1000 ways to do things wrong, 1 or 2 that work
  4. 4. 44Prescription: SW visualization, tools to localizebugs, diagnose problems, and instrument behavior
  5. 5. 554/4/07 Bob ColwellAccidents Are InevitableAccidents Are Inevitable– Its the nature of engineeringIts the nature of engineeringto push designs to edge ofto push designs to edge offailure (schedule, reliability,failure (schedule, reliability,thermals, materials, tools,thermals, materials, tools,judgment of unknowns)judgment of unknowns)– P(accident) =P(accident) = εε , for, for εε ≠≠ 00– World rewards this behaviorWorld rewards this behaviorCool new features + first toCool new features + first tomarket often preferred tomarket often preferred todependabilitydependabilityOther markets (life-support)Other markets (life-support)make (or should make) thismake (or should make) thistrade-off differently!trade-off differently!
  6. 6. 664/4/07 Bob ColwellIsn’t that justIsn’t that just ??Close. But Murphy is notClose. But Murphy is notquite right.quite right.1.1. #Near-misses >> #disasters#Near-misses >> #disasters2.2. Competent design/test findsCompetent design/test findssimple errorssimple errors3.3. Complex sequences & unlikelyComplex sequences & unlikelyevent cascades survive to prod’nevent cascades survive to prod’n
  7. 7. 774/4/07 Bob ColwellFailures Getting WorseFailures Getting Worse Mechanical things usually fail predictably due to physicsMechanical things usually fail predictably due to physics– Wings bend, bridges groan, engines rattle, knees acheWings bend, bridges groan, engines rattle, knees ache– By contrast, computer-based things fail “all over the place”By contrast, computer-based things fail “all over the place”Helpful Engineering Attitude:Helpful Engineering Attitude:1.1. Nature does not want yourNature does not want yourengineered system to work; willengineered system to work; willactively work against youactively work against you2.2. Your design will do only whatYour design will do only whatyou’ve constrained it to do, onlyyou’ve constrained it to do, onlyas long as it has toas long as it has to3.3. Watch out for…Watch out for…Normalization of devianceNormalization of deviance(Challenger O-rings, Apollo(Challenger O-rings, Apollo1 fire)1 fire)
  8. 8. 88The Steely-Eyed Missile ValidatorThe Steely-Eyed Missile ValidatorApollo 12Apollo 12 22ndndtry to land on moon, launched 11/14/69try to land on moon, launched 11/14/69 36 seconds after liftoff, spacecraft struck by lightning => power36 seconds after liftoff, spacecraft struck by lightning => powersurgesurge– All telemetry went haywire; book said to abort liftoffAll telemetry went haywire; book said to abort liftoff– Both spacecraft pilot and mission controller were furiously considering that optionBoth spacecraft pilot and mission controller were furiously considering that option– But John Aaron was on shift, and thought he’d seen this malfunction beforeBut John Aaron was on shift, and thought he’d seen this malfunction before During testing 1 year earlier, Aaron observed test that went off into weedsDuring testing 1 year earlier, Aaron observed test that went off into weeds– Aaron took it on himself to investigate this – led him to obscure SCE subsystemAaron took it on himself to investigate this – led him to obscure SCE subsystem In critical “abort or not” few seconds, with lives on line, Aaron made one ofIn critical “abort or not” few seconds, with lives on line, Aaron made one ofmost famous calls in NASA historymost famous calls in NASA history– ““Flight, try SCE to ‘Aux’”Flight, try SCE to ‘Aux’”– Neither Flight nor spacecraft pilot Conrad knew what that even meant, but Alan Bean tried itNeither Flight nor spacecraft pilot Conrad knew what that even meant, but Alan Bean tried it– Telemetry came right back, vaulted Aaron into validation stardomTelemetry came right back, vaulted Aaron into validation stardom He could have blown off earlier test, butHe could have blown off earlier test, but he didn’the didn’t His inner validator wanted to know “what just happened?”His inner validator wanted to know “what just happened?”Isaac Asimov once said 3 most importantwords in science are “What was THAT?”
  9. 9. 99Complexity Implies SurprisesComplexity Implies Surprises……and surprises areand surprises are badbadChaos effects in complexChaos effects in complex µµ P’sP’s– Decomposability is a fundamental tenet ofDecomposability is a fundamental tenet ofcomplex system designcomplex system design– Butterfly wings ruin decomposabilityButterfly wings ruin decomposability– ““Improve design, get slower performance” notImprove design, get slower performance” notat all uncommonat all uncommonWe must stop designing largeWe must stop designing largesystems as though small ones simplysystems as though small ones simplyscale upscale up– lesson from comm engineers:lesson from comm engineers: assumeassume errorserrors
  10. 10. 1010Thinking about validationThinking about validationAbility to think in analogies is highestAbility to think in analogies is highestform of intelligenceform of intelligence– IQ tests like “a:b :: c:d”IQ tests like “a:b :: c:d”– Hofstadters book: numerical sequencesHofstadters book: numerical sequencesAnalogies may illuminate a subject inAnalogies may illuminate a subject ina way that direct introspection cannota way that direct introspection cannot– They drive our minds to their creative limitsThey drive our minds to their creative limits
  11. 11. 1111Listen to Your Inner ValidatorListen to Your Inner ValidatorYouYou knewknew it wouldn’t be 3, didn’t you?it wouldn’t be 3, didn’t you?– You sensed something’s not quite as it seemsYou sensed something’s not quite as it seemsAnswer: 0, 1, 2, 720!, …Answer: 0, 1, 2, 720!, …= 0, 1, 2, 6!!= 0, 1, 2, 6!!= 0, 1!, 2!!, 3!!!, …= 0, 1!, 2!!, 3!!!, …That was the voice of your innerThat was the voice of your innervalidator that you were hearingvalidator that you were hearingD. Hofstadter, Fluid Concepts and Creative Analogies0, 1, 2, …?0, 1, 2, …?
  12. 12. 1212Lesson: Trust NothingLesson: Trust NothingHyatt RegencyHyatt Regencyhotel, Missouri,hotel, Missouri,19801980Catwalks on rodsCatwalks on rods40’ threaded rods40’ threaded rodswith nuts halfwaywith nuts halfwayKilled 114,Killed 114,injured 200injured 200
  13. 13. 1313What Happened?What Happened?Spec was marginalSpec was marginal40’ threaded rods40’ threaded rods“too hard”, changed“too hard”, changedto 2x20’ by contractorto 2x20’ by contractorNo simulation, no testNo simulation, no testWho goofed?Who goofed?Engineer, contractor,Engineer, contractor,inspector…everyoneinspector…everyone
  14. 14. 1414Therac-25Therac-25Medical particleMedical particleacceleratoracceleratorElectrons,Electrons,protons, X-raysprotons, X-raysSix fatalitiesSix fatalitiesfrom poorfrom poorsystem/SWsystem/SWdesigndesign– And blind naïveAnd blind naïvefaith in computers!faith in computers!
  15. 15. 1515Question EverythingQuestion EverythingTestTest assumptionsassumptions as well as designas well as design– If assumptions are broken, design surely is tooIf assumptions are broken, design surely is too– Try to “catch the field goals”Try to “catch the field goals”
  16. 16. 1616Fight Urge to Relax RequirementsFight Urge to Relax RequirementsChallengerChallenger– Not ok to slip design assumptions (launch temp,Not ok to slip design assumptions (launch temp,# of unburnt O-rings) to suit desires# of unburnt O-rings) to suit desiresAirbusAirbus– Blaming pilot not reasonable explanation; pilotBlaming pilot not reasonable explanation; pilotis part of system designis part of system designRunway “incursions” up 71% since ‘93Runway “incursions” up 71% since ‘93– Near-misses are trying toNear-misses are trying to tell us somethingtell us somethingDiane Vaughan, The Challenger Launch Decision, Chicago PressDiane Vaughan, The Challenger Launch Decision, Chicago Press1996; Nancy Leveson, Safeware, Addison-Wesley 19951996; Nancy Leveson, Safeware, Addison-Wesley 1995
  17. 17. 1717If You Didn’t Test It,If You Didn’t Test It,It Doesn’t WorkIt Doesn’t WorkMir: fire extinguishersMir: fire extinguishers boltedbolted to wallto wall– Still had strong metal launch strapsStill had strong metal launch straps– Had never been needed before, so never testedHad never been needed before, so never tested– Discovered with a roaring fire several feet awayDiscovered with a roaring fire several feet away
  18. 18. 18184/4/07 Bob ColwellComplexity Makes Everything WorseComplexity Makes Everything Worse Some things must be complicated to do their jobSome things must be complicated to do their job– Our brains, for exampleOur brains, for example But complex sequences are root of most disastersBut complex sequences are root of most disasters– Challenger, Bhopal, Chernobyl, FDIV, Exxon ValdezChallenger, Bhopal, Chernobyl, FDIV, Exxon Valdez Where does complexity come from? Why does itWhere does complexity come from? Why does itkeep increasing? Where are the limits?keep increasing? Where are the limits?– Pentium 4Pentium 4 ““in the small” vs “in the large” design (micros vsin the small” vs “in the large” design (micros vscomm systems)comm systems) What to do? Vigilance, testing, awareness…we areWhat to do? Vigilance, testing, awareness…we areall validatorsall validators
  19. 19. 1919What To DoWhat To Do Get the spec rightGet the spec right Design for correctness but…Design for correctness but… design knowing perfection is unattainabledesign knowing perfection is unattainable Users are part of the systemUsers are part of the system Formal methodsFormal methods Pre-production testing and validationPre-production testing and validation Post-production testing and verificationPost-production testing and verification Education of the publicEducation of the public
  20. 20. 20204/4/07 Bob ColwellRolesRolesEngineers must standEngineers must standtheir groundtheir ground– There are always doubts,There are always doubts,incomplete data; don’t letincomplete data; don’t let‘em use those against you‘em use those against youJudgment is cruciallyJudgment is cruciallyneeded --needed -- YOURSYOURS–Remember the ChallengerRemember the Challenger““My God, Thiokol, when do you want me to launch? Next April?”My God, Thiokol, when do you want me to launch? Next April?”–Be careful with “data”Be careful with “data”““Risk assessment data is like a captured spy; if you torture it long enough, it will tell youRisk assessment data is like a captured spy; if you torture it long enough, it will tell youanything you want to know…”anything you want to know…” (Wm. Ruckelshaus)(Wm. Ruckelshaus)–Crushing, conflicting demands are normCrushing, conflicting demands are normDesignDesign must push the envelope w/o ceding responsibilitymust push the envelope w/o ceding responsibilityValidationValidation establishes whether theyve pushed it too farestablishes whether theyve pushed it too farManagementManagement must beware overriding tech judgmentmust beware overriding tech judgmentPublicPublic must understand limits of human design processmust understand limits of human design processAll players must value roles of others!All players must value roles of others!engineermgt HR
  21. 21. 2121Roles cont.Roles cont.ManagementManagement– wants to assume a product is safewants to assume a product is safe– knows nothing’s ever perfect,knows nothing’s ever perfect,comes a time to “shoot the engineers” or they’ll nevercomes a time to “shoot the engineers” or they’ll neverstop tinkeringstop tinkeringValidatorsValidators– want to prove a product is safewant to prove a product is safe– assume it is not by defaultassume it is not by default– only informed arbiters of when product is readyonly informed arbiters of when product is readydon’t fall for “might as well sign, we’re
  22. 22. 2222Future Directions:Future Directions:Public ExpectationsPublic ExpectationsAndy Grove’s FDIV epiphanyAndy Grove’s FDIV epiphanyParadoxically, the more high tech, the more public expects of productParadoxically, the more high tech, the more public expects of productUsers caused Chernobyl, TMI by going “off book”, but prevented manyUsers caused Chernobyl, TMI by going “off book”, but prevented manyother disasters with real-time creativity…lessons are subtleother disasters with real-time creativity…lessons are subtleTakes exquisite understanding & judgment to discernTakes exquisite understanding & judgment to discernaccidents from reasonable risk-taking andaccidents from reasonable risk-taking andbonehead errors or incompetencebonehead errors or incompetenceThis is what a jury must do.This is what a jury must do.How?How?Can’t keep trending this wayCan’t keep trending this way
  23. 23. 2323Future of ValidationFuture of ValidationMultiple Culture Changes NeededMultiple Culture Changes NeededPublic needs to stop expecting perfectionPublic needs to stop expecting perfectionDesign teams must explicitly limit complexityDesign teams must explicitly limit complexityand avoid auto-scale-up assumptionsand avoid auto-scale-up assumptionsCompanies must mature past point of viewingCompanies must mature past point of viewingvalidation as an unpleasant overheadvalidation as an unpleasant overheaddoes your company have “Validation Fellows?”does your company have “Validation Fellows?”Validation is a profession of its own.Validation is a profession of its own.Cultivate the Validation Attitude!Cultivate the Validation Attitude!
  24. 24. 2424The EndThe End

×