Presents an example of a referencing document completely missing the conclusion in the original reference

1. An Example of “Lessons Learned” Not Being Learned
NASA wasextremelyinterested inthe lessonslearnedduringthe Space Shuttle programdevelopment,
operations,andretirementbe carefullycaptured. The Original Source quote below isfromapaper I co-
authoredduringthe lastmonthsof the operational phase of the Space Shuttle program. Recently,in
lookingforreference material,Ionce againcame across NancyLeveson’sSoftwareandthe Challengeof
FlightControl chapter. Ina quickscan of the material,myeye caughtthe passage below labeled Used
As Reference. While the situationisquotedalmostverbatim, the “lessonlearned”(explanation) is
totallyincorrect.
The statementbyNancyLevesonthat“nervousnessaboutthe patchingledtothe use of much more
extensive verificationforthe patchesthanfor the high-levellanguage changes”is100 % incorrect. All
Space Shuttle flightsoftwareverificationusedexactlythe same processregardlessif the change wasvia
machine level patchesorhighorderlanguage source updates. Infact,many of the errors foundonthe
STS-2 systemwere foundbythe identical testcase ranagainstthe machine language patchesonSTS-1.
The “lessonlearned”(explanation) wasthatforthe STS-1 machine language patches,the process
requiredthe software verificationanalysttoparticipate inthe pre-release formal inspectionof the
patch. Duringthissame period,source updatedforSTS-2requiredaformal inspectionbutparticipation
was limitedtodevelopmentpersonnelonly. Verificationdidanindependentcode inspectionusingits
ownformal processafterthe source solutionwastestedbydevelopmentandpromotedtothe baseline
source library.
Once this“lessonlearned”wasobservedbyIBMmanagement,there wasanimmediate actionto
developanewhighorderlanguage source inspectionprocessusedtothe endof the programwere the
verificationanalystwasarequiredparticipantinthe pre-buildjointdevelopment/verificationformal
software inspection. Iwasone of two people assignedthisactionsince Iwasa software verifieratthe
time. Withinaweek,we combinedthe well documentdevelopmentinspectionprocess(previously pre-
build) withthe well documentedverificationinspectionprocess(previously postbuild)intothe
organization’sonlyformal inspectionprocesswhichwasperformedpre-build(nominallypriorto
developmenttesting)whose inspectionparticipants includedmoderator,developer,developmentpeer,
requirementanalyst,andverificationanalyst.
The resultingprocessservedthe programwell overthe nextquartercentury.
Original Source

2. J. Christopher. Hickey, James B. Loveall, James K. Orr, and Andres L. Klausman, “The Legacy of Space
Shuttle Flight Software,” AIAA Space 2011 Conference, Sept. 2-29, 2011, Long Beach California, 2011.
Also available for free at http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20110014946.pdf
“For overone year priorto STS-1, the compiledandlinkedsystemwasfrozenandall mandatorychanges
were made usingmachine language patches. Inparallel,thesesame mandatorychangeswere
implementedonthe STS-2systemas highorderlanguage source updates. In an analysisof the quality
of STS-1versusSTS-2, itwas determinedthatthe qualityof the machine languagespatchesforSTS-1
were higherthanthe correspondingsource changesonSTS-2. Causal analysisdeterminedthatthe
significantdifferencewasthatVerificationAnalystswere addedtothe patchinspectionsdue tothe
perceivedhighriskof makingmachine language patches. Thisimmediatelyledtoaprocesschange
where the VerificationAnalystwasarequiredDesign/Code inspectionparticipantonall changes
effectivewiththe release forSTS-5andsubsequent.”
Used As Reference
Software andthe Challenge of FlightControl byNancyLeveson.Toappearas a chapterinSpace Shuttle
Legacy: How We Did It/What We Learned edited by Roger Launius, James Craig, and John Krige and to
be published in AIAA in 2013. Note: Book has been published.
“For a year priorto STS-1, the software wasfrozenandall mandatorychangeswere made usingmachine
language patches.Inparallel,the same changes were made inthe STS-2software.Lateritwas
determinedthatthe qualityof the machine language patchesforSTS-1wasbetterthanthe
correspondinghigh-level language(HAL/S) changesinSTS-2.23
Thisresultseemedtodefycommon
beliefsaboutthe dangerof patchingsoftware.Laterthe difference wasexplainedbythe factthat
nervousnessaboutthe patchingledtothe use of muchmore extensive verificationforthe patchesthan
for the high-levellanguage changes.”