The document outlines the Software Assurance Research Program at NASA's IV&V Facility in West Virginia, detailing its definition, goals, and relevance to software reliability and safety. It highlights a series of research projects and initiatives aimed at improving software assurance practices across various NASA projects, acknowledging the challenges of data collection and transition. The document also discusses future directions, emphasizing the need for collaborative research and enhanced data management strategies.

1. IV&V Facility
Research Heaven,
West Virginia
1
SA @ WV
(software assurance
research at West Virginia)
Kenneth McGill
NASA IV&V Facility Research Lead
304.367.8300
Kenneth.McGill@ivv.nasa.gov
Dr. Tim Menzies Ph.D. (WVU)
Software Engineering Research Chair
tim@menzies,com

2. IV&V Facility
Research Heaven,
West Virginia
2
Why, what is software
assurance?
• Definition:
– Planned and systematic set of
activities
– Ensures that software
processes and products conform
to requirements, standards, and
procedures.
• Goals:
– Confidence that SW will do what is
needed when it’s needed.
Before bad software After bad software
• Why software assurance?
–bad software can kill good
hardware.
–E.g. ARIANE 5: (and many others)
•Software errors in inertial
reference system
•Floating point conversion overflow
Ariane 5

3. IV&V Facility
Research Heaven,
West Virginia
3
OSMA Software Assurance
Research Program
• Office of Safety & Mission Assurance (Code Q- OSMA)
• Five million per year
• Applied software assurance research
• Focus:
– Software, not hardware
– SW Assurance
– NASA-wide applicability
• Externally valid results; i.e. useful for MANY projects
• Organization:
– Managed from IV&V Facility
– Delegated Program Manager: Dr. Linda Rosenberg, GSFC

4. IV&V Facility
Research Heaven,
West Virginia
4
Many projects
• Mega: highest-level perspective
– e.g. project planning tools like ASK-PETE
[Kurtz]
• Macro:
– e.g. understanding faults [Sigal, Lutz &
Mikulski]
• Micro:
– e.g. source code browsing [Suder]
• Applied to basic:
– Applied:
• (e.g.) MATT/RATT [Henry]: support large
scale runs of MATLAB
– Basic (not many of these)
• e.g. Fractal analysis of time series data
[Shereshevsky]
• Many, many more
– Too numerous to list
– Samples follow
– See rest of SAS!
Horn of
plenty

5. IV&V Facility
Research Heaven,
West Virginia
5
Many more projects!
0
7
11
12
6
5
1 1
3
1
6
2
7
27
10
12
4
0 0
5
26
22
0
5
10
15
20
25
30
ARC
GRC
GSFC
IV&V
JPL
JSC
KSC
LaRC
MSFC
Industry
University
2002
2003
Total proposals: 2.2
NASA centers: 1.5
Industry: 26
University: 3.7
Ratio
FY02/FY01
Good news!
• More good proposals
than we can fund
Bad news!
• same as the good news

6. IV&V Facility
Research Heaven,
West Virginia
6
A survey of 44 FY01 CSIPs
project 1 2 3 4 5 6 7 8 9 10 11 12 13 14 to 44
AATT 2
ISS 2
Space Shuttle 2
ST5 2
Aura 1
CHIPS 1
CLCS 1
CM2 1
CMMI 1
DSMS 1
EOSDIS 1
FAMS 1
GLAST 1
HSM4 1
HST 1
Mars 07 1
Mars 08 1
PCS 1
Space Station 1
Starlight 1
Stereo 1
SWIFT 1
X-38 1
5 4 3 2 2 2 2 2 1 1 1 1 1 0
Need more
transitions!
(but don’t
forget the
theory)
75% with no
claim for
project
connections

7. IV&V Facility
Research Heaven,
West Virginia
7
Action plan- restructure
CSIPS: more transitions!
• New (year 1)
– Fund many
• Renewed (year 2)
– Continue funding the promising new
projects
– Recommended: letter of endorsement
from NASA project manager
• Transition (year 3)
– Select a few projects
– Aim: tools in the hands of project folks
– Required: project manager involvement
• Reality check:
– Transition needs time
– Data drought

8. IV&V Facility
Research Heaven,
West Virginia
8
Long transition cycles
CO2 + 2H2 —> CH4 + O2
Mars
atmosphere
oxidizerfuel
on-board
(no photo)
Carmen
Mikulski
JPL
Robyn Lutz
JPL, CS-Iowa
State
• Pecheur &
practical formal methods
– In-Situ Propellant Production project
– Taught developers:
• Livingstone model-based
diagnosis
• model-checking tool tools
• developed by Reid Simmons,
(CMU)
– Technology to be applied to the
Intelligent Vehicle Health Maintenance
(IVMS) for 2nd generation shuttles
• Lutz, Mikulski &
ODC-based analysis of defects
– Deep-space NASA missions
– Found 8 clusters of recurring defects
– Proposed and validated 5
explanations of the clusters
– Explanations  changes to NASA
practices
– ODC being evaluated by JPL’s defect
management tool team
Charles
Pecheur
RIACS, ASE,
ARC

9. IV&V Facility
Research Heaven,
West Virginia
9
The data
drought
Gasp…
need
data…

10. IV&V Facility
Research Heaven,
West Virginia
10
End the drought:
bootstrap off other systems
• Find the
enterprise-wide
management
information
system
• Insert data
collection hooks
– E.g. JPL adding
ODC to their defect
tracking system
– WVU SIAT sanitizer

11. IV&V Facility
Research Heaven,
West Virginia
11
End the drought:
Contractors as researchers
active data
repository
• Buy N licenses of a defect
tracking tool (e.g. Clearquest)
• Give away to projects
– In exchange for their data
• Build and maintain a central
repository for that data
– With a web-based query
interface
• Data for all
take me to
your data

12. IV&V Facility
Research Heaven,
West Virginia
12
End the drought:
Contractors as researchers (2)
abstractionabstraction
actionaction
reflectionreflection
experienceexperience 1
2
3
4
Mark Suder
Titan, IV&V
Hypertext power browser for source code4 SIAT-1}
high-severity errors, recall what SIAT queries
d to finding those errors
4’
2’
Assess each such “power queries”
Reject the less useful ones
3’
Procedures manual for super SIAT or
new search options in interface
SIAT2
}
1’ Use it.
See also:
• Titan’s new
ROI project
• Any
contractor
proposing an
NRA
• Galaxy
Global’s
metric
project
See also:
• Titan’s new
ROI project
• Any
contractor
proposing an
NRA
• Galaxy
Global’s
metric
project

13. IV&V Facility
Research Heaven,
West Virginia
13
End the drought:
raid old/existing projects
• Cancelled projects with
public-domain software
– E.g. X-34
• Or other open source NASA
projects
– E.g. GSFC’s ITOS:
– real-time control and
monitoring system during
development, test, and on-orbit
operations,
– UNIX, Solaris, FreeBSD,
Linux, PC
– Free!!
– NASA project connections:
• Triana,
• Swift,
• HESSI,
• ULDB,
• SMEX,
• Formation Flying Testbed,
• Spartan

14. IV&V Facility
Research Heaven,
West Virginia
14
End the drought:
synergy groups
• N researchers
– Same task
– Different
technologies
• Share found data
• E.g. IV&V business
case workers
• E.g. monthly fault
teleconferences
– JPL:
• Lutz, Nikora
– Uni. Kentucky:
• Hayes
– Uni. Maryland:
• Smidts
– WV:
• Chapman
(Galaxy Global) &
Menzies (WVU)

15. IV&V Facility
Research Heaven,
West Virginia
15
End the drought:
Tandem experiments
• “Technique X finds errors”
– So?
• Industrial defect detection
capability rates:
– TR(min,mean,max)
– TR(0.35, 0.50, 0.65)
– Assumes manual
“Fagan inspections”
• Is “X” better than a
manual 1976
technique?
• Need “tandem
experiments”
to check
• I.e. do it twice
– Once by the researchers
– Once by IV&V
contractors (baseline)
0
20
40
60
80
100
120
defects
found
analysis design code test
baseline FM Fagan
fictional
data
0
20
40
60
80
100
120
cost
analysis design code test

16. IV&V Facility
Research Heaven,
West Virginia
16
Alternatively:
End your own drought
• Our duty, our goal:
– Work the data problem (e.g. see above)
– Goal of CI project year1: build bridges
– But the more workers, the better
• Myth: there is a “data truck” parked at IV&V
– full of goodies, just for you
• Reality: Access negotiation takes time
– With contractors, within NASA
• We actively assist:
– Each connection is a joy to behold,
an occasion to celebration
– We don’t celebrate much
• Bottom line:
– We chase data for dozens of projects
– Researchers have more time, more focus on
their particular data needs
• Ken’s law:
– $$$ chases researchers who chase projects
– CI year2, year3: needs a project connection

17. IV&V Facility
Research Heaven,
West Virginia
17
Alternatively (2), accept the
drought and sieve the dust
• The DUST project:
– Assumes a few key options control the rest
• Methodology:
– Simulate across range of options
– Data dust clouds
– Too many options: what leads to what?
– Summarize via machine learning
– Condense dust cloud
– Improve mean, reduce variance
• Case studies:
– JPL requirements engineering:
• Feather/JPL [Re02]
– Project planning:
• DART- Raque/ IVV; Chaing/UBC;
• IV&V costing: Marinaro/IVV, Smith/WVU
• general: Raffo, et.al/PSU [Ase02]
– An analysis of pair programming: Smith/WVU
– Better predictors for:
• testability: Cukic/WVU, Owen/WVU [Issre02, Ase02]
• faults: diStefano/WVU, McGill/IVV; Chapman/GG
• reuse : diStefano/WVU [ToolsWithAI02]
Figure 2. Initial (scattered black points)
and Final (dense white points)
0
50
100
150
200
250
300
0 300000 600000 900000 1200000
Cost
Benefit
Each dot =
1 random
project plan
The answer my
friend, is blowin’
in the wind
But wait: the
times they
are changing

18. IV&V Facility
Research Heaven,
West Virginia
18
Katerina Goseva Popstojanova
Other WVU SA research
Architectural
descriptions
Fault,
failure
data on
components,
connectors
Software
Specs & design
(early life cycle)
Code analysis
(iv&v,operational
usage)
Metrics(complexity,coupling,entropy )
Failure data from testing
Severity of failures
UML (sequence
diagrams,
state charts)
UML simulations
Static (SIAT,
Mccabe, entrophy)
Dynamic (testing,
runtime monitoring)
 Testing & formal methods
 Bayesian approach to reliability
 Architectural metrics
Risk assessment & dynamic UML
 Reliability &
operational profile errors
Hany Ammar
Bojan Cukic
collaborator
Goal: accurate,
stable, risk
assessment
early in the
lifecycle
Goal: accurate,
stable, risk
assessment
early in the
lifecycle

19. IV&V Facility
Research Heaven,
West Virginia
19
More WVU research
(FY02 UIs)
Architectural metrics
Risk assessment & dynamic UML
Intelligent flight controllers
Testing & formal methods
Bayesian approach to reliability
Fractal study of resource dynamics
Reliability & operational profile errors
SE research chair
interns
DUST
Ammar
Cukic
Goseva-
Popstojanova
Menzies
new
renewed
c = conference
w = workshop
j = journal
ISS hub controller,
“Dryden application”
F15
“JPL deep space mission”
DART
“KC-2”
IVV cost models
SIAT
X34
ITOS
X38
jj
j, ccccccc, w
c
cccccc
jc
c
w
FY03 proposals = 2.2*FY02

20. IV&V Facility
Research Heaven,
West Virginia
20
Function Point Metrics for
Safety-Critical Software
• Thesis:
– Traditional function-point
cost estimation
– Incorrect for safety-critical
software
• > 1 way to skin a cat
– >1 way to realize a safety
critical function:
– NCP=
N-copy programming
– NVP=
N-Version
Programming ,
– NSCP=
N Self-Checking
Programming,
– …
– With, without redundancy,
• Method:
– explore them all!
1.3000
1.4000
1.5000
1.6000
1.7000
1.8000
1.9000
2.0000
0 0.033 0.1 0.33 1
Algorithm Complexity
H2/H1,C2/C1
NCP
NVP,NSCP
RFCS
CRB
RB,NRB
DRB,EDRB
NCP
NVP,NSCP
RFCS
CRB
RB,NRB
DRB,EDRB
Design Diversity, add eight
more
Design Diversity, add one
more
Data Diversity
H2 and C2 : effort & cost, redundant system
H1 and C1: effort & cost, non-redundant
system Afzel Noore

21. IV&V Facility
Research Heaven,
West Virginia
21
Pre-disaster warnings
[Cukic, Shereshevsky]
Can we defer a maintenance cycle and keep doing science for a while longer?
Mark
Shereshevsky
CrashEarly warning
}
Time for graceful
shutdown
Bojan Cukic
ARTS II

22. IV&V Facility
Research Heaven,
West Virginia
22
Intelligent flight controllers
[Napolitano, Cukic] (and menzies)
Marcello Napolitano
(Mechanical and
Aerospace)
Bojan Cukic
(CSEE)
Lifecycle opportunities for
V&V of neural network based
adaptive control systems.

23. IV&V Facility
Research Heaven,
West Virginia
23
The road ahead: applied &
theoretical research
CSIPs: applied
research
USIPs: applied +
theoretical
research
Need both
To boldly go…