1. IV&V Facility
Research Heaven,
West Virginia
1
SA @ WV
(software assurance
research at West Virginia)
Kenneth McGill
NASA IV&V Facility Research Lead
304.367.8300
Kenneth.McGill@ivv.nasa.gov
Dr. Tim Menzies Ph.D. (WVU)
Software Engineering Research Chair
tim@menzies,com

2. IV&V Facility
Research Heaven,
West Virginia
2
Why, what is software
assurance?
• Definition:
– Planned and systematic set of
activities
– Ensures that software
processes and products conform
to requirements, standards, and
procedures.
• Goals:
– Confidence that SW will do what is
needed when it’s needed.
Before bad software After bad software
• Why software assurance?
–bad software can kill good
hardware.
–E.g. ARIANE 5: (and many others)
•Software errors in inertial
reference system
•Floating point conversion overflow
Ariane 5

3. IV&V Facility
Research Heaven,
West Virginia
3
OSMA Software Assurance
Research Program
• Office of Safety & Mission Assurance (Code Q- OSMA)
• Five million per year
• Applied software assurance research
• Focus:
– Software, not hardware
– SW Assurance
– NASA-wide applicability
• Externally valid results; i.e. useful for MANY projects
• Organization:
– Managed from IV&V Facility
– Delegated Program Manager: Dr. Linda Rosenberg, GSFC

4. IV&V Facility
Research Heaven,
West Virginia
4
Many projects
• Mega: highest-level perspective
– e.g. project planning tools like ASK-PETE
[Kurtz]
• Macro:
– e.g. understanding faults [Sigal, Lutz &
Mikulski]
• Micro:
– e.g. source code browsing [Suder]
• Applied to basic:
– Applied:
• (e.g.) MATT/RATT [Henry]: support large
scale runs of MATLAB
– Basic (not many of these)
• e.g. Fractal analysis of time series data
[Shereshevsky]
• Many, many more
– Too numerous to list
– Samples follow
– See rest of SAS!
Horn of
plenty

5. IV&V Facility
Research Heaven,
West Virginia
5
Many more projects!
0
7
11
12
6
5
1 1
3
1
6
2
7
27
10
12
4
0 0
5
26
22
0
5
10
15
20
25
30
ARC
GRC
GSFC
IV&V
JPL
JSC
KSC
LaRC
MSFC
Industry
University
2002
2003
Total proposals: 2.2
NASA centers: 1.5
Industry: 26
University: 3.7
Ratio
FY02/FY01
Good news!
• More good proposals
than we can fund
Bad news!
• same as the good news

6. IV&V Facility
Research Heaven,
West Virginia
6
A survey of 44 FY01 CSIPs
project 1 2 3 4 5 6 7 8 9 10 11 12 13 14 to 44
AATT 2
ISS 2
Space Shuttle 2
ST5 2
Aura 1
CHIPS 1
CLCS 1
CM2 1
CMMI 1
DSMS 1
EOSDIS 1
FAMS 1
GLAST 1
HSM4 1
HST 1
Mars 07 1
Mars 08 1
PCS 1
Space Station 1
Starlight 1
Stereo 1
SWIFT 1
X-38 1
5 4 3 2 2 2 2 2 1 1 1 1 1 0
Need more
transitions!
(but don’t
forget the
theory)
75% with no
claim for
project
connections

7. IV&V Facility
Research Heaven,
West Virginia
7
Action plan- restructure
CSIPS: more transitions!
• New (year 1)
– Fund many
• Renewed (year 2)
– Continue funding the promising new
projects
– Recommended: letter of endorsement
from NASA project manager
• Transition (year 3)
– Select a few projects
– Aim: tools in the hands of project folks
– Required: project manager involvement
• Reality check:
– Transition needs time
– Data drought

8. IV&V Facility
Research Heaven,
West Virginia
8
Long transition cycles
CO2 + 2H2 —> CH4 + O2
Mars
atmosphere
oxidizerfuel
on-board
(no photo)
Carmen
Mikulski
JPL
Robyn Lutz
JPL, CS-Iowa
State
• Pecheur &
practical formal methods
– In-Situ Propellant Production project
– Taught developers:
• Livingstone model-based
diagnosis
• model-checking tool tools
• developed by Reid Simmons,
(CMU)
– Technology to be applied to the
Intelligent Vehicle Health Maintenance
(IVMS) for 2nd generation shuttles
• Lutz, Mikulski &
ODC-based analysis of defects
– Deep-space NASA missions
– Found 8 clusters of recurring defects
– Proposed and validated 5
explanations of the clusters
– Explanations  changes to NASA
practices
– ODC being evaluated by JPL’s defect
management tool team
Charles
Pecheur
RIACS, ASE,
ARC

9. IV&V Facility
Research Heaven,
West Virginia
9
The data
drought
Gasp…
need
data…

10. IV&V Facility
Research Heaven,
West Virginia
10
End the drought:
bootstrap off other systems
• Find the
enterprise-wide
management
information
system
• Insert data
collection hooks
– E.g. JPL adding
ODC to their defect
tracking system
– WVU SIAT sanitizer

11. IV&V Facility
Research Heaven,
West Virginia
11
End the drought:
Contractors as researchers
active data
repository
• Buy N licenses of a defect
tracking tool (e.g. Clearquest)
• Give away to projects
– In exchange for their data
• Build and maintain a central
repository for that data
– With a web-based query
interface
• Data for all
take me to
your data

12. IV&V Facility
Research Heaven,
West Virginia
12
End the drought:
Contractors as researchers (2)
abstractionabstraction
actionaction
reflectionreflection
experienceexperience 1
2
3
4
Mark Suder
Titan, IV&V
Hypertext power browser for source code4 SIAT-1}
high-severity errors, recall what SIAT queries
d to finding those errors
4’
2’
Assess each such “power queries”
Reject the less useful ones
3’
Procedures manual for super SIAT or
new search options in interface
SIAT2
}
1’ Use it.
See also:
• Titan’s new
ROI project
• Any
contractor
proposing an
NRA
• Galaxy
Global’s
metric
project
See also:
• Titan’s new
ROI project
• Any
contractor
proposing an
NRA
• Galaxy
Global’s
metric
project

13. IV&V Facility
Research Heaven,
West Virginia
13
End the drought:
raid old/existing projects
• Cancelled projects with
public-domain software
– E.g. X-34
• Or other open source NASA
projects
– E.g. GSFC’s ITOS:
– real-time control and
monitoring system during
development, test, and on-orbit
operations,
– UNIX, Solaris, FreeBSD,
Linux, PC
– Free!!
– NASA project connections:
• Triana,
• Swift,
• HESSI,
• ULDB,
• SMEX,
• Formation Flying Testbed,
• Spartan

14. IV&V Facility
Research Heaven,
West Virginia
14
End the drought:
synergy groups
• N researchers
– Same task
– Different
technologies
• Share found data
• E.g. IV&V business
case workers
• E.g. monthly fault
teleconferences
– JPL:
• Lutz, Nikora
– Uni. Kentucky:
• Hayes
– Uni. Maryland:
• Smidts
– WV:
• Chapman
(Galaxy Global) &
Menzies (WVU)

15. IV&V Facility
Research Heaven,
West Virginia
15
End the drought:
Tandem experiments
• “Technique X finds errors”
– So?
• Industrial defect detection
capability rates:
– TR(min,mean,max)
– TR(0.35, 0.50, 0.65)
– Assumes manual
“Fagan inspections”
• Is “X” better than a
manual 1976
technique?
• Need “tandem
experiments”
to check
• I.e. do it twice
– Once by the researchers
– Once by IV&V
contractors (baseline)
0
20
40
60
80
100
120
defects
found
analysis design code test
baseline FM Fagan
fictional
data
0
20
40
60
80
100
120
cost
analysis design code test

16. IV&V Facility
Research Heaven,
West Virginia
16
Alternatively:
End your own drought
• Our duty, our goal:
– Work the data problem (e.g. see above)
– Goal of CI project year1: build bridges
– But the more workers, the better
• Myth: there is a “data truck” parked at IV&V
– full of goodies, just for you
• Reality: Access negotiation takes time
– With contractors, within NASA
• We actively assist:
– Each connection is a joy to behold,
an occasion to celebration
– We don’t celebrate much
• Bottom line:
– We chase data for dozens of projects
– Researchers have more time, more focus on
their particular data needs
• Ken’s law:
– $$$ chases researchers who chase projects
– CI year2, year3: needs a project connection

17. IV&V Facility
Research Heaven,
West Virginia
17
Alternatively (2), accept the
drought and sieve the dust
• The DUST project:
– Assumes a few key options control the rest
• Methodology:
– Simulate across range of options
– Data dust clouds
– Too many options: what leads to what?
– Summarize via machine learning
– Condense dust cloud
– Improve mean, reduce variance
• Case studies:
– JPL requirements engineering:
• Feather/JPL [Re02]
– Project planning:
• DART- Raque/ IVV; Chaing/UBC;
• IV&V costing: Marinaro/IVV, Smith/WVU
• general: Raffo, et.al/PSU [Ase02]
– An analysis of pair programming: Smith/WVU
– Better predictors for:
• testability: Cukic/WVU, Owen/WVU [Issre02, Ase02]
• faults: diStefano/WVU, McGill/IVV; Chapman/GG
• reuse : diStefano/WVU [ToolsWithAI02]
Figure 2. Initial (scattered black points)
and Final (dense white points)
0
50
100
150
200
250
300
0 300000 600000 900000 1200000
Cost
Benefit
Each dot =
1 random
project plan
The answer my
friend, is blowin’
in the wind
But wait: the
times they
are changing

18. IV&V Facility
Research Heaven,
West Virginia
18
Katerina Goseva Popstojanova
Other WVU SA research
Architectural
descriptions
Fault,
failure
data on
components,
connectors
Software
Specs & design
(early life cycle)
Code analysis
(iv&v,operational
usage)
Metrics(complexity,coupling,entropy )
Failure data from testing
Severity of failures
UML (sequence
diagrams,
state charts)
UML simulations
Static (SIAT,
Mccabe, entrophy)
Dynamic (testing,
runtime monitoring)
 Testing & formal methods
 Bayesian approach to reliability
 Architectural metrics
Risk assessment & dynamic UML
 Reliability &
operational profile errors
Hany Ammar
Bojan Cukic
collaborator
Goal: accurate,
stable, risk
assessment
early in the
lifecycle
Goal: accurate,
stable, risk
assessment
early in the
lifecycle

19. IV&V Facility
Research Heaven,
West Virginia
19
More WVU research
(FY02 UIs)
Architectural metrics
Risk assessment & dynamic UML
Intelligent flight controllers
Testing & formal methods
Bayesian approach to reliability
Fractal study of resource dynamics
Reliability & operational profile errors
SE research chair
interns
DUST
Ammar
Cukic
Goseva-
Popstojanova
Menzies
new
renewed
c = conference
w = workshop
j = journal
ISS hub controller,
“Dryden application”
F15
“JPL deep space mission”
DART
“KC-2”
IVV cost models
SIAT
X34
ITOS
X38
jj
j, ccccccc, w
c
cccccc
jc
c
w
FY03 proposals = 2.2*FY02

20. IV&V Facility
Research Heaven,
West Virginia
20
Function Point Metrics for
Safety-Critical Software
• Thesis:
– Traditional function-point
cost estimation
– Incorrect for safety-critical
software
• > 1 way to skin a cat
– >1 way to realize a safety
critical function:
– NCP=
N-copy programming
– NVP=
N-Version
Programming ,
– NSCP=
N Self-Checking
Programming,
– …
– With, without redundancy,
• Method:
– explore them all!
1.3000
1.4000
1.5000
1.6000
1.7000
1.8000
1.9000
2.0000
0 0.033 0.1 0.33 1
Algorithm Complexity
H2/H1,C2/C1
NCP
NVP,NSCP
RFCS
CRB
RB,NRB
DRB,EDRB
NCP
NVP,NSCP
RFCS
CRB
RB,NRB
DRB,EDRB
Design Diversity, add eight
more
Design Diversity, add one
more
Data Diversity
H2 and C2 : effort & cost, redundant system
H1 and C1: effort & cost, non-redundant
system Afzel Noore

21. IV&V Facility
Research Heaven,
West Virginia
21
Pre-disaster warnings
[Cukic, Shereshevsky]
Can we defer a maintenance cycle and keep doing science for a while longer?
Mark
Shereshevsky
CrashEarly warning
}
Time for graceful
shutdown
Bojan Cukic
ARTS II

22. IV&V Facility
Research Heaven,
West Virginia
22
Intelligent flight controllers
[Napolitano, Cukic] (and menzies)
Marcello Napolitano
(Mechanical and
Aerospace)
Bojan Cukic
(CSEE)
Lifecycle opportunities for
V&V of neural network based
adaptive control systems.

23. IV&V Facility
Research Heaven,
West Virginia
23
The road ahead: applied &
theoretical research
CSIPs: applied
research
USIPs: applied +
theoretical
research
Need both
To boldly go…