Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
IV&V Facility
Research Heaven,
West Virginia
1
SA @ WV
(software assurance
research at West Virginia)
Kenneth McGill
NASA ...
IV&V Facility
Research Heaven,
West Virginia
2
Why, what is software
assurance?
• Definition:
– Planned and systematic set...
IV&V Facility
Research Heaven,
West Virginia
3
OSMA Software Assurance
Research Program
• Office of Safety & Mission Assur...
IV&V Facility
Research Heaven,
West Virginia
4
Many projects
• Mega: highest-level perspective
– e.g. project planning too...
IV&V Facility
Research Heaven,
West Virginia
5
Many more projects!
0
7
11
12
6
5
1 1
3
1
6
2
7
27
10
12
4
0 0
5
26
22
0
5
...
IV&V Facility
Research Heaven,
West Virginia
6
A survey of 44 FY01 CSIPs
project 1 2 3 4 5 6 7 8 9 10 11 12 13 14 to 44
AA...
IV&V Facility
Research Heaven,
West Virginia
7
Action plan- restructure
CSIPS: more transitions!
• New (year 1)
– Fund man...
IV&V Facility
Research Heaven,
West Virginia
8
Long transition cycles
CO2 + 2H2 —> CH4 + O2
Mars
atmosphere
oxidizerfuel
o...
IV&V Facility
Research Heaven,
West Virginia
9
The data
drought
Gasp…
need
data…
IV&V Facility
Research Heaven,
West Virginia
10
End the drought:
bootstrap off other systems
• Find the
enterprise-wide
ma...
IV&V Facility
Research Heaven,
West Virginia
11
End the drought:
Contractors as researchers
active data
repository
• Buy N...
IV&V Facility
Research Heaven,
West Virginia
12
End the drought:
Contractors as researchers (2)
abstractionabstraction
act...
IV&V Facility
Research Heaven,
West Virginia
13
End the drought:
raid old/existing projects
• Cancelled projects with
publ...
IV&V Facility
Research Heaven,
West Virginia
14
End the drought:
synergy groups
• N researchers
– Same task
– Different
te...
IV&V Facility
Research Heaven,
West Virginia
15
End the drought:
Tandem experiments
• “Technique X finds errors”
– So?
• I...
IV&V Facility
Research Heaven,
West Virginia
16
Alternatively:
End your own drought
• Our duty, our goal:
– Work the data ...
IV&V Facility
Research Heaven,
West Virginia
17
Alternatively (2), accept the
drought and sieve the dust
• The DUST projec...
IV&V Facility
Research Heaven,
West Virginia
18
Katerina Goseva Popstojanova
Other WVU SA research
Architectural
descripti...
IV&V Facility
Research Heaven,
West Virginia
19
More WVU research
(FY02 UIs)
Architectural metrics
Risk assessment & dynam...
IV&V Facility
Research Heaven,
West Virginia
20
Function Point Metrics for
Safety-Critical Software
• Thesis:
– Traditiona...
IV&V Facility
Research Heaven,
West Virginia
21
Pre-disaster warnings
[Cukic, Shereshevsky]
Can we defer a maintenance cyc...
IV&V Facility
Research Heaven,
West Virginia
22
Intelligent flight controllers
[Napolitano, Cukic] (and menzies)
Marcello ...
IV&V Facility
Research Heaven,
West Virginia
23
The road ahead: applied &
theoretical research
CSIPs: applied
research
USI...
Upcoming SlideShare
Loading in …5
×

172529main ken and_tim_software_assurance_research_at_west_virginia

1,395 views

Published on

SA @ WV (software assurance research at West Virginia)

Kenneth McGill
NASA IV&V Facility Research Lead
304.367.8300
Kenneth.McGill@ivv.nasa.gov

Dr. Tim Menzies Ph.D. (WVU)
Software Engineering Research Chair
tim@menzies.us

  • Be the first to comment

  • Be the first to like this

172529main ken and_tim_software_assurance_research_at_west_virginia

  1. 1. IV&V Facility Research Heaven, West Virginia 1 SA @ WV (software assurance research at West Virginia) Kenneth McGill NASA IV&V Facility Research Lead 304.367.8300 Kenneth.McGill@ivv.nasa.gov Dr. Tim Menzies Ph.D. (WVU) Software Engineering Research Chair tim@menzies,com
  2. 2. IV&V Facility Research Heaven, West Virginia 2 Why, what is software assurance? • Definition: – Planned and systematic set of activities – Ensures that software processes and products conform to requirements, standards, and procedures. • Goals: – Confidence that SW will do what is needed when it’s needed. Before bad software After bad software • Why software assurance? –bad software can kill good hardware. –E.g. ARIANE 5: (and many others) •Software errors in inertial reference system •Floating point conversion overflow Ariane 5
  3. 3. IV&V Facility Research Heaven, West Virginia 3 OSMA Software Assurance Research Program • Office of Safety & Mission Assurance (Code Q- OSMA) • Five million per year • Applied software assurance research • Focus: – Software, not hardware – SW Assurance – NASA-wide applicability • Externally valid results; i.e. useful for MANY projects • Organization: – Managed from IV&V Facility – Delegated Program Manager: Dr. Linda Rosenberg, GSFC
  4. 4. IV&V Facility Research Heaven, West Virginia 4 Many projects • Mega: highest-level perspective – e.g. project planning tools like ASK-PETE [Kurtz] • Macro: – e.g. understanding faults [Sigal, Lutz & Mikulski] • Micro: – e.g. source code browsing [Suder] • Applied to basic: – Applied: • (e.g.) MATT/RATT [Henry]: support large scale runs of MATLAB – Basic (not many of these) • e.g. Fractal analysis of time series data [Shereshevsky] • Many, many more – Too numerous to list – Samples follow – See rest of SAS! Horn of plenty
  5. 5. IV&V Facility Research Heaven, West Virginia 5 Many more projects! 0 7 11 12 6 5 1 1 3 1 6 2 7 27 10 12 4 0 0 5 26 22 0 5 10 15 20 25 30 ARC GRC GSFC IV&V JPL JSC KSC LaRC MSFC Industry University 2002 2003 Total proposals: 2.2 NASA centers: 1.5 Industry: 26 University: 3.7 Ratio FY02/FY01 Good news! • More good proposals than we can fund Bad news! • same as the good news
  6. 6. IV&V Facility Research Heaven, West Virginia 6 A survey of 44 FY01 CSIPs project 1 2 3 4 5 6 7 8 9 10 11 12 13 14 to 44 AATT 2 ISS 2 Space Shuttle 2 ST5 2 Aura 1 CHIPS 1 CLCS 1 CM2 1 CMMI 1 DSMS 1 EOSDIS 1 FAMS 1 GLAST 1 HSM4 1 HST 1 Mars 07 1 Mars 08 1 PCS 1 Space Station 1 Starlight 1 Stereo 1 SWIFT 1 X-38 1 5 4 3 2 2 2 2 2 1 1 1 1 1 0 Need more transitions! (but don’t forget the theory) 75% with no claim for project connections
  7. 7. IV&V Facility Research Heaven, West Virginia 7 Action plan- restructure CSIPS: more transitions! • New (year 1) – Fund many • Renewed (year 2) – Continue funding the promising new projects – Recommended: letter of endorsement from NASA project manager • Transition (year 3) – Select a few projects – Aim: tools in the hands of project folks – Required: project manager involvement • Reality check: – Transition needs time – Data drought
  8. 8. IV&V Facility Research Heaven, West Virginia 8 Long transition cycles CO2 + 2H2 —> CH4 + O2 Mars atmosphere oxidizerfuel on-board (no photo) Carmen Mikulski JPL Robyn Lutz JPL, CS-Iowa State • Pecheur & practical formal methods – In-Situ Propellant Production project – Taught developers: • Livingstone model-based diagnosis • model-checking tool tools • developed by Reid Simmons, (CMU) – Technology to be applied to the Intelligent Vehicle Health Maintenance (IVMS) for 2nd generation shuttles • Lutz, Mikulski & ODC-based analysis of defects – Deep-space NASA missions – Found 8 clusters of recurring defects – Proposed and validated 5 explanations of the clusters – Explanations  changes to NASA practices – ODC being evaluated by JPL’s defect management tool team Charles Pecheur RIACS, ASE, ARC
  9. 9. IV&V Facility Research Heaven, West Virginia 9 The data drought Gasp… need data…
  10. 10. IV&V Facility Research Heaven, West Virginia 10 End the drought: bootstrap off other systems • Find the enterprise-wide management information system • Insert data collection hooks – E.g. JPL adding ODC to their defect tracking system – WVU SIAT sanitizer
  11. 11. IV&V Facility Research Heaven, West Virginia 11 End the drought: Contractors as researchers active data repository • Buy N licenses of a defect tracking tool (e.g. Clearquest) • Give away to projects – In exchange for their data • Build and maintain a central repository for that data – With a web-based query interface • Data for all take me to your data
  12. 12. IV&V Facility Research Heaven, West Virginia 12 End the drought: Contractors as researchers (2) abstractionabstraction actionaction reflectionreflection experienceexperience 1 2 3 4 Mark Suder Titan, IV&V Hypertext power browser for source code4 SIAT-1} high-severity errors, recall what SIAT queries d to finding those errors 4’ 2’ Assess each such “power queries” Reject the less useful ones 3’ Procedures manual for super SIAT or new search options in interface SIAT2 } 1’ Use it. See also: • Titan’s new ROI project • Any contractor proposing an NRA • Galaxy Global’s metric project See also: • Titan’s new ROI project • Any contractor proposing an NRA • Galaxy Global’s metric project
  13. 13. IV&V Facility Research Heaven, West Virginia 13 End the drought: raid old/existing projects • Cancelled projects with public-domain software – E.g. X-34 • Or other open source NASA projects – E.g. GSFC’s ITOS: – real-time control and monitoring system during development, test, and on-orbit operations, – UNIX, Solaris, FreeBSD, Linux, PC – Free!! – NASA project connections: • Triana, • Swift, • HESSI, • ULDB, • SMEX, • Formation Flying Testbed, • Spartan
  14. 14. IV&V Facility Research Heaven, West Virginia 14 End the drought: synergy groups • N researchers – Same task – Different technologies • Share found data • E.g. IV&V business case workers • E.g. monthly fault teleconferences – JPL: • Lutz, Nikora – Uni. Kentucky: • Hayes – Uni. Maryland: • Smidts – WV: • Chapman (Galaxy Global) & Menzies (WVU)
  15. 15. IV&V Facility Research Heaven, West Virginia 15 End the drought: Tandem experiments • “Technique X finds errors” – So? • Industrial defect detection capability rates: – TR(min,mean,max) – TR(0.35, 0.50, 0.65) – Assumes manual “Fagan inspections” • Is “X” better than a manual 1976 technique? • Need “tandem experiments” to check • I.e. do it twice – Once by the researchers – Once by IV&V contractors (baseline) 0 20 40 60 80 100 120 defects found analysis design code test baseline FM Fagan fictional data 0 20 40 60 80 100 120 cost analysis design code test
  16. 16. IV&V Facility Research Heaven, West Virginia 16 Alternatively: End your own drought • Our duty, our goal: – Work the data problem (e.g. see above) – Goal of CI project year1: build bridges – But the more workers, the better • Myth: there is a “data truck” parked at IV&V – full of goodies, just for you • Reality: Access negotiation takes time – With contractors, within NASA • We actively assist: – Each connection is a joy to behold, an occasion to celebration – We don’t celebrate much • Bottom line: – We chase data for dozens of projects – Researchers have more time, more focus on their particular data needs • Ken’s law: – $$$ chases researchers who chase projects – CI year2, year3: needs a project connection
  17. 17. IV&V Facility Research Heaven, West Virginia 17 Alternatively (2), accept the drought and sieve the dust • The DUST project: – Assumes a few key options control the rest • Methodology: – Simulate across range of options – Data dust clouds – Too many options: what leads to what? – Summarize via machine learning – Condense dust cloud – Improve mean, reduce variance • Case studies: – JPL requirements engineering: • Feather/JPL [Re02] – Project planning: • DART- Raque/ IVV; Chaing/UBC; • IV&V costing: Marinaro/IVV, Smith/WVU • general: Raffo, et.al/PSU [Ase02] – An analysis of pair programming: Smith/WVU – Better predictors for: • testability: Cukic/WVU, Owen/WVU [Issre02, Ase02] • faults: diStefano/WVU, McGill/IVV; Chapman/GG • reuse : diStefano/WVU [ToolsWithAI02] Figure 2. Initial (scattered black points) and Final (dense white points) 0 50 100 150 200 250 300 0 300000 600000 900000 1200000 Cost Benefit Each dot = 1 random project plan The answer my friend, is blowin’ in the wind But wait: the times they are changing
  18. 18. IV&V Facility Research Heaven, West Virginia 18 Katerina Goseva Popstojanova Other WVU SA research Architectural descriptions Fault, failure data on components, connectors Software Specs & design (early life cycle) Code analysis (iv&v,operational usage) Metrics(complexity,coupling,entropy ) Failure data from testing Severity of failures UML (sequence diagrams, state charts) UML simulations Static (SIAT, Mccabe, entrophy) Dynamic (testing, runtime monitoring)  Testing & formal methods  Bayesian approach to reliability  Architectural metrics Risk assessment & dynamic UML  Reliability & operational profile errors Hany Ammar Bojan Cukic collaborator Goal: accurate, stable, risk assessment early in the lifecycle Goal: accurate, stable, risk assessment early in the lifecycle
  19. 19. IV&V Facility Research Heaven, West Virginia 19 More WVU research (FY02 UIs) Architectural metrics Risk assessment & dynamic UML Intelligent flight controllers Testing & formal methods Bayesian approach to reliability Fractal study of resource dynamics Reliability & operational profile errors SE research chair interns DUST Ammar Cukic Goseva- Popstojanova Menzies new renewed c = conference w = workshop j = journal ISS hub controller, “Dryden application” F15 “JPL deep space mission” DART “KC-2” IVV cost models SIAT X34 ITOS X38 jj j, ccccccc, w c cccccc jc c w FY03 proposals = 2.2*FY02
  20. 20. IV&V Facility Research Heaven, West Virginia 20 Function Point Metrics for Safety-Critical Software • Thesis: – Traditional function-point cost estimation – Incorrect for safety-critical software • > 1 way to skin a cat – >1 way to realize a safety critical function: – NCP= N-copy programming – NVP= N-Version Programming , – NSCP= N Self-Checking Programming, – … – With, without redundancy, • Method: – explore them all! 1.3000 1.4000 1.5000 1.6000 1.7000 1.8000 1.9000 2.0000 0 0.033 0.1 0.33 1 Algorithm Complexity H2/H1,C2/C1 NCP NVP,NSCP RFCS CRB RB,NRB DRB,EDRB NCP NVP,NSCP RFCS CRB RB,NRB DRB,EDRB Design Diversity, add eight more Design Diversity, add one more Data Diversity H2 and C2 : effort & cost, redundant system H1 and C1: effort & cost, non-redundant system Afzel Noore
  21. 21. IV&V Facility Research Heaven, West Virginia 21 Pre-disaster warnings [Cukic, Shereshevsky] Can we defer a maintenance cycle and keep doing science for a while longer? Mark Shereshevsky CrashEarly warning } Time for graceful shutdown Bojan Cukic ARTS II
  22. 22. IV&V Facility Research Heaven, West Virginia 22 Intelligent flight controllers [Napolitano, Cukic] (and menzies) Marcello Napolitano (Mechanical and Aerospace) Bojan Cukic (CSEE) Lifecycle opportunities for V&V of neural network based adaptive control systems.
  23. 23. IV&V Facility Research Heaven, West Virginia 23 The road ahead: applied & theoretical research CSIPs: applied research USIPs: applied + theoretical research Need both To boldly go…

×