1. Historical Data
PASS Annual Failure Rate Versus
Residual Product Defect Density
James K. Orr
Independent Consultant
jkorr@gatech.edu
Copyright 2015 By James K. Orr 17/14/2015

2. Space Shuttle PASS Project
• Data is presented for the Space Shuttle Primary Avionics Software
System (PASS).
• The organization that developed and maintained the PASS has
existed in four companies: (1) IBM Corporation from March 10,
1973 through December 31, 1993, (2) Loral Corporation from
January 1, 1994 to April 21, 1996, (3) Lockheed Martin from April
22, 1996 to July 3, 1998, and (4) United Space Alliance from July 4,
1998 until August 12, 2011. (Reference 1, page 13)
• Extensive data on release product defects and failures was collected
by the project starting with STS-1 over the 30 year operational flight
life of the program. (Reference 1, page 19 – 24)
Copyright 2015 By James K. Orr 27/14/2015

3. Focus Of This Presentation
• This presentation focuses on a subset of Product
DRs.
– Product DRs are defined as found from Software
Readiness Review immediately prior to first flight of
that software version until end of program
• Failures in this presentation are Product DRs that
occur in either operation flight, in training, or
testing in realistic scenarios
– A Product DR found in a testing scenario with an
unrealistic number of failures that would result in loss
of vehicle would not be counted as a failure
Copyright 2015 By James K. Orr 37/14/2015

4. Annual Failure Rate Versus
Product Defect Density
• The following chart (Page 7) combines data in a
format that provide potentially significant insight to
future flight systems.
– The number of KNOWN Product DRs (Page 8)
• These are Product DRs that were unknown at
the time of a specific Space Shuttle flight, but
were later detected. In analysis of the Product
DR, it was determined what was the earliest
flight system in which that Product DR existed
Copyright 2015 By James K. Orr 47/14/2015

5. Annual Failure Rate Versus
Product Defect Density
– The number of failures are grouped by the year in
which the failure occurred
– On Page 7, annual number of failures (Y-axis) are
plotted versus Known Product DRs by year (X-axis)
from page 8.
– Also shown is the predicted values and statistical
confidence intervals using the Software Reliability
Model developed uniquely for PASS project.
• PASS Software Reliability Model Prediction of all
Product DRs (failures plus non-failures) per year shown
on Page 9
Copyright 2015 By James K. Orr 57/14/2015

6. Observations
– Operational PASS software scenarios were executed at least
50 times on the ground for each flight flown
• Commercial Software may have millions of users daily
– Key Observations:
• 6 percent of existing errors were found each year as
failures in operational use of the software
• The more undiscovered errors in the flight system, the
more failures are expected
– For future space flight software systems, similar execution
failure rates are likely
– For PASS software, Product DRs introduced prior to STS-1 (to
1981) were found as late as 2009 (Page 10)
Copyright 2015 By James K. Orr 67/14/2015

7. Copyright 2015 By James K. Orr
Late 1980’sLate 2000’s
STS-26
STS-114
Defects Removed
Over Time
Example: 1988
290 Latent Defects
16 Failures During Year
All PASS Failures
Most Found In Development, Testing Of New Capabilities or Training
Total of 19 Failures Occurred In Flight or Terminal Count Down Over Life Of Program
See Page 9 For More Information
7/14/2015 7
IBMUnited Space
Alliance

8. Reference 1, Page 20 and Reference 2, Page 5
Copyright 2015 By James K. Orr 87/14/2015

9. Reference 2, Page 11
Copyright 2015 By James K. Orr
Mean Time Between Failures in Calendar Days Between Product DRs By Year
97/14/2015

10. Reference 1, Page 23
Copyright 2015 By James K. Orr 107/14/2015
Previously Unknown Errors Are
Discovered Over Time.
As Errors Are Found, The Initial Quality
Of Initial Release Is Better Known.
Current Quality Is Better Because Fewer
Errors Remain Undetected.

11. References
1. Christopher J. Hickey, Andrew L. Klausman, James B. Loveall, and James K. Orr, The
Legacy of Space Shuttle Flight Software, August, 2011, NASA JSC-CN-24428, JSC-
CN-24683, presented at AIAA Space 2011 Conference & Exposition, 27 – 29
September, 2011, Long Beach California
– Free at
• http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20110014946.pdf
– Also available through AIAA as AIAA 2011-7307
• http://arc.aiaa.org/doi/abs/10.2514/6.2011-7307
2. James K. Orr, Daryl Peltier, Space Shuttle Program Primary Avionics Software
System (PASS) Success Legacy – Quality & Reliability Data, Page 5, August 24,
2010, NASA JSC-CN-21317, presented at NASA-Contractors Chief Engineers
Council 3-day meeting August 24-26, 2010, in Montreal, Canada
– Free at
• http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20100029536.pdf
Copyright 2015 By James K. Orr 117/14/2015