SA @ WV(software assurance research at West Virginia)
Kenneth McGill
NASA IV&V Facility Research Lead
304.367.8300
Kenneth.McGill@ivv.nasa.gov
Dr. Tim Menzies Ph.D. (WVU)
Software Engineering Research Chair
tim@menzies.us
Dr. Michael Kendra presents an overview of his program, Test and Evaluation, at the AFOSR 2013 Spring Review. At this review, Program Officers from AFOSR Technical Divisions will present briefings that highlight basic research programs beneficial to the Air Force.
At the Intersection of NAAQS Modeling, Permitting, and ComplianceShirley Rivera
This presentation provides perspectives for the project developer and the air agency regarding the identification and selection of the NO2/NOx In-Stack Ratios (ISRs). The Federal EPA recognizes this is important and has been in the process of collecting information for a publicly available database; however, this database is still evolving. Choosing a representative ISR can mean the difference between operational flexibility, constrained operations, and/or more timely permit processing. But unlike emission benchmarks established for BACT requirements, there are not yet ISR benchmarks. A "call to action" to populate the EPA database is included.
The Royal Society of Chemistry hosts large scale data collections and provides access to the data to the chemistry community. The largest RSC data set of wide scale interest to the community offers access to tens of millions of compounds. The host platform, ChemSpider, is limited as it is a structure centric hub only. A new architecture, the RSC data repository, has been developed that extends support to reactions, spectral data, crystallography data and related property data. It is also the architecture underlying a series of exemplar projects for managing data for a number of diverse laboratories. The adoption of data standards for the integration and distribution of data has been essential. Specific standards include molecular structure formats such as molfiles and InChIs, and spectral data formats such as JCAMP. This presentation will report on our development of the data repository, the importance of utilizing standards for data integration, the flexible nature of the architecture to deliver solutions for various laboratories and our efforts to develop new large data collections. This includes text-mining efforts to extract large spectrum-structure collections from large corpuses.
Dr. Michael Kendra presents an overview of his program, Test and Evaluation, at the AFOSR 2013 Spring Review. At this review, Program Officers from AFOSR Technical Divisions will present briefings that highlight basic research programs beneficial to the Air Force.
At the Intersection of NAAQS Modeling, Permitting, and ComplianceShirley Rivera
This presentation provides perspectives for the project developer and the air agency regarding the identification and selection of the NO2/NOx In-Stack Ratios (ISRs). The Federal EPA recognizes this is important and has been in the process of collecting information for a publicly available database; however, this database is still evolving. Choosing a representative ISR can mean the difference between operational flexibility, constrained operations, and/or more timely permit processing. But unlike emission benchmarks established for BACT requirements, there are not yet ISR benchmarks. A "call to action" to populate the EPA database is included.
The Royal Society of Chemistry hosts large scale data collections and provides access to the data to the chemistry community. The largest RSC data set of wide scale interest to the community offers access to tens of millions of compounds. The host platform, ChemSpider, is limited as it is a structure centric hub only. A new architecture, the RSC data repository, has been developed that extends support to reactions, spectral data, crystallography data and related property data. It is also the architecture underlying a series of exemplar projects for managing data for a number of diverse laboratories. The adoption of data standards for the integration and distribution of data has been essential. Specific standards include molecular structure formats such as molfiles and InChIs, and spectral data formats such as JCAMP. This presentation will report on our development of the data repository, the importance of utilizing standards for data integration, the flexible nature of the architecture to deliver solutions for various laboratories and our efforts to develop new large data collections. This includes text-mining efforts to extract large spectrum-structure collections from large corpuses.
• “Detecting radio-astronomical "Fast Radio Transient Events" via an OODT-based metadata processing pipeline”, Chris Mattmann, Andrew Hart , Luca Cinquini, David Thompson, Kiri Wagstaff, Shakeh Khudikyan. ApacheCon NA 2013, Februrary 2013
Climate Science presents several data intensive challenges that are the intersection of software architecture and data science. This includes developing approaches for scaling the analysis of highly distributed data across institutional and system boundaries. JPL has been developing approaches for quantitatively evaluating software architectures to consider different topologies in the deployment of computing capabilities and methodologies in order to support the analysis of distributed climate data. This talk will cover those approaches and also needed research in new methodologies as remote sensing and climate model output data continue to increase in their size and distribution.
Swimming upstream: OPNFV Doctor project case studyOPNFV
Based on the lifecycle of the OPNFV Doctor project, this case study shows how operator requirements “on paper” have successfully been realized step-by-step and in close cooperation with upstream community projects into a mature fault management framework. A demo of the solution had been presented in a keynote at the last OpenStack Summit. The talk will describe how we have worked in the OPNFV Doctor project and will provide some lessons learned on this journey. With significant experience now of working OPNFV requirements upstream to OpenStack, we’ll share best practices for submitting contributions upstream, how to best communicate, and how to overcome the primary challenges.
M4M 2 the Rescue of M2M - Eclipse DemoCamps Fall 2013Werner Keil
M4M or Measure 4 Measure, ever since Shakespeare's play with the same name we know, people can be mistaken for one another. A Duke (like the beloved Java mascot) claims to be a monk, the head of a dead pirate is presented to be that of the young hero. So can important information like Units of Measurement be misinterpreted. While humans reading 10°C, 10 C or 10 Degree Celsius, each of those could be interpreted and understood well enough. For M2M communication, unless a program is provided with a large glossary of alternate terms, only ONE of these would be acceptable.
This is where the Unified Code for Units of Measurement (UCUM) among similar approaches like UnitsML, SensorML or a few others are vital for error-free M2M transactions, not just between sensors or measurement devices, but also and especially vehicles or distributed devices.
OSGi Measurement has been around for some time (R3) but never gained as much momentum, as many other bundles of OSGi did. Except for very few use cases in the Embedded or Automotive sector it is practically unused and based on statements by its contributors in the OSGi Alliance to be considered legacy with no plans continue development.
Engineers play critical roles in astronomy, from building telescopes, to designing scientific instruments, to operating observatories. Working together, engineers and scientists answer fundamental questions about our universe. In this session, you'll hear from women engineers making contributions to astronomy by developing a new high resolution optical spectrograph, adapting telescope control software for remote operations, architecting document management and managing critical systems for the next generation of telescopes. You will learn about the different engineering disciplines involved in astronomy, key concepts and technologies shaping astronomy today, and how to find job opportunities in astronomy as an engineer.
Consequences of Mispredictions of Software ReliabilityRAKESH RANA
Consequences of Mispredictions of Software Reliability
Presented at:
International Conference on Software Measurement, IWSM-Mensura, Rotterdam, Netherland, 2014
Get full text of publication at:
http://rakeshrana.website/index.php/work/publications/
Statistical Analysis of New Product Development (NPD) Cycle-time DataSteven Pratt
Alpha Technologies recently concluded a comprehensive study of historical NPD performance data (cost, cycle-time, resource requirements, program type and complexity, plans versus actuals, etc.) and has successfully applied the results of this study to make measureable improvements to their NPD process. Probability-based summary data is used to aide in planning and budgeting, set statistically valid continuous improvement targets for performance scorecards, and develop a visual resource management tool for allocation of human resources to present and future NPD programs.
The title of this talk is a crass attempt to be catchy and topical, by referring to the recent victory of Watson in Jeopardy.
My point (perhaps confusingly) is not that new computer capabilities are a bad thing. On the contrary, these capabilities represent a tremendous opportunity for science. The challenge that I speak to is how we leverage these capabilities without computers and computation overwhelming the research community in terms of both human and financial resources. The solution, I suggest, is to get computation out of the lab—to outsource it to third party providers.
Abstract follows:
We have made much progress over the past decade toward effective distributed cyberinfrastructure. In big-science fields such as high energy physics, astronomy, and climate, thousands benefit daily from tools that enable the distributed management and analysis of vast quantities of data. But we now face a far greater challenge. Exploding data volumes and new research methodologies mean that many more--ultimately most?--researchers will soon require similar capabilities. How can we possible supply information technology (IT) at this scale, given constrained budgets? Must every lab become filled with computers, and every researcher an IT specialist?
I propose that the answer is to take a leaf from industry, which is slashing both the costs and complexity of consumer and business IT by moving it out of homes and offices to so-called cloud providers. I suggest that by similarly moving research IT out of the lab, we can realize comparable economies of scale and reductions in complexity, empowering investigators with new capabilities and freeing them to focus on their research.
I describe work we are doing to realize this approach, focusing initially on research data lifecycle management. I present promising results obtained to date, and suggest a path towards large-scale delivery of these capabilities. I also suggest that these developments are part of a larger "revolution in scientific affairs," as profound in its implications as the much-discussed "revolution in military affairs" resulting from more capable, low-cost IT. I conclude with some thoughts on how researchers, educators, and institutions may want to prepare for this revolution.
Presentation by Dr David Kelsey at Networkshop50 in June 2022.
Shows the work of the UK GridPP particle physics group in migrating the large volumes of science traffic for the CERN experiments to use IPv6.
Big data visualization frameworks and applications at Kitwarebigdataviz_bay
Big data visualization frameworks and applications at Kitware
Marcus Hanwell, Technical Leader at Kitware, Inc.
March 27th 2014
Kitware develops permissively licensed open source frameworks and applications for scientific data applications, and related areas. Some of the frameworks developed by our High Performance Computing and Visualization group address current challenges in big data visualization and analysis in a number of application domains including geospatial visualization, social media, finance, chemistry, biological (phylogenetics), and climate. The frameworks used to develop solutions in these areas will be described, along with the applications and the nature of the underlying data. These solutions focus on shared frameworks providing data storage, indexing, retrieval, client-server delivery models, server-side serial and parallel data reduction, analysis, and diagnostics. Additionally, they provide mechanisms that enable server-side or client-side rendering based on the capabilities and configuration of the system.
Big Data Visualization Meetup - South Bay
http://www.meetup.com/Big-Data-Visualisation-South-Bay/
M4M 2 the Rescue of M2M (Eclipse DemoCamp Trondheim)Werner Keil
M4M or Measure 4 Measure, ever since Shakespeare's play with the same name we know, people can be mistaken for one another. A Duke (like the beloved Java mascot) claims to be a monk, the head of a dead pirate is presented to be that of the young hero. So can important information like Units of Measurement be misinterpreted. While humans reading 10°C, 10 C or 10 Degree Celsius, each of those could be interpreted and understood well enough. For M2M communication, unless a program is provided with a large glossary of alternate terms, only ONE of these would be acceptable.
This is where the Unified Code for Units of Measurement (UCUM) among similar approaches like UnitsML, SensorML or a few others are vital for error-free M2M transactions, not just between sensors or measurement devices, but also and especially vehicles or distributed devices.
OSGi Measurement has been around for some time (R3) but never gained as much momentum, as many other bundles of OSGi did. Except for very few use cases in the Embedded or Automotive sector it is practically unused and based on statements by its contributors in the OSGi Alliance to be considered legacy with no plans continue development.
After a brief overview of common M2M errors from Gimli to Mars, This session provides an overview of OSGi Measurement, Eclipse OUMo, what they have in common and where the differences lie. Although most of today's OSGi containers are capable of dealing with units or measurement better and more reliable with UOMo, both can where necessary also exchange information and collaborate. E.g. if legacy devices and code cannot be easily replaced. For this We'll take a look at interoperability between different systems or with other unit technologies and languages like F#, Fantom, Python or Lua.
Science has escaped the lab and is roaming free in the world. People use software to understand the world . What tools are needed to support that work?
More Related Content
Similar to 172529main ken and_tim_software_assurance_research_at_west_virginia
• “Detecting radio-astronomical "Fast Radio Transient Events" via an OODT-based metadata processing pipeline”, Chris Mattmann, Andrew Hart , Luca Cinquini, David Thompson, Kiri Wagstaff, Shakeh Khudikyan. ApacheCon NA 2013, Februrary 2013
Climate Science presents several data intensive challenges that are the intersection of software architecture and data science. This includes developing approaches for scaling the analysis of highly distributed data across institutional and system boundaries. JPL has been developing approaches for quantitatively evaluating software architectures to consider different topologies in the deployment of computing capabilities and methodologies in order to support the analysis of distributed climate data. This talk will cover those approaches and also needed research in new methodologies as remote sensing and climate model output data continue to increase in their size and distribution.
Swimming upstream: OPNFV Doctor project case studyOPNFV
Based on the lifecycle of the OPNFV Doctor project, this case study shows how operator requirements “on paper” have successfully been realized step-by-step and in close cooperation with upstream community projects into a mature fault management framework. A demo of the solution had been presented in a keynote at the last OpenStack Summit. The talk will describe how we have worked in the OPNFV Doctor project and will provide some lessons learned on this journey. With significant experience now of working OPNFV requirements upstream to OpenStack, we’ll share best practices for submitting contributions upstream, how to best communicate, and how to overcome the primary challenges.
M4M 2 the Rescue of M2M - Eclipse DemoCamps Fall 2013Werner Keil
M4M or Measure 4 Measure, ever since Shakespeare's play with the same name we know, people can be mistaken for one another. A Duke (like the beloved Java mascot) claims to be a monk, the head of a dead pirate is presented to be that of the young hero. So can important information like Units of Measurement be misinterpreted. While humans reading 10°C, 10 C or 10 Degree Celsius, each of those could be interpreted and understood well enough. For M2M communication, unless a program is provided with a large glossary of alternate terms, only ONE of these would be acceptable.
This is where the Unified Code for Units of Measurement (UCUM) among similar approaches like UnitsML, SensorML or a few others are vital for error-free M2M transactions, not just between sensors or measurement devices, but also and especially vehicles or distributed devices.
OSGi Measurement has been around for some time (R3) but never gained as much momentum, as many other bundles of OSGi did. Except for very few use cases in the Embedded or Automotive sector it is practically unused and based on statements by its contributors in the OSGi Alliance to be considered legacy with no plans continue development.
Engineers play critical roles in astronomy, from building telescopes, to designing scientific instruments, to operating observatories. Working together, engineers and scientists answer fundamental questions about our universe. In this session, you'll hear from women engineers making contributions to astronomy by developing a new high resolution optical spectrograph, adapting telescope control software for remote operations, architecting document management and managing critical systems for the next generation of telescopes. You will learn about the different engineering disciplines involved in astronomy, key concepts and technologies shaping astronomy today, and how to find job opportunities in astronomy as an engineer.
Consequences of Mispredictions of Software ReliabilityRAKESH RANA
Consequences of Mispredictions of Software Reliability
Presented at:
International Conference on Software Measurement, IWSM-Mensura, Rotterdam, Netherland, 2014
Get full text of publication at:
http://rakeshrana.website/index.php/work/publications/
Statistical Analysis of New Product Development (NPD) Cycle-time DataSteven Pratt
Alpha Technologies recently concluded a comprehensive study of historical NPD performance data (cost, cycle-time, resource requirements, program type and complexity, plans versus actuals, etc.) and has successfully applied the results of this study to make measureable improvements to their NPD process. Probability-based summary data is used to aide in planning and budgeting, set statistically valid continuous improvement targets for performance scorecards, and develop a visual resource management tool for allocation of human resources to present and future NPD programs.
The title of this talk is a crass attempt to be catchy and topical, by referring to the recent victory of Watson in Jeopardy.
My point (perhaps confusingly) is not that new computer capabilities are a bad thing. On the contrary, these capabilities represent a tremendous opportunity for science. The challenge that I speak to is how we leverage these capabilities without computers and computation overwhelming the research community in terms of both human and financial resources. The solution, I suggest, is to get computation out of the lab—to outsource it to third party providers.
Abstract follows:
We have made much progress over the past decade toward effective distributed cyberinfrastructure. In big-science fields such as high energy physics, astronomy, and climate, thousands benefit daily from tools that enable the distributed management and analysis of vast quantities of data. But we now face a far greater challenge. Exploding data volumes and new research methodologies mean that many more--ultimately most?--researchers will soon require similar capabilities. How can we possible supply information technology (IT) at this scale, given constrained budgets? Must every lab become filled with computers, and every researcher an IT specialist?
I propose that the answer is to take a leaf from industry, which is slashing both the costs and complexity of consumer and business IT by moving it out of homes and offices to so-called cloud providers. I suggest that by similarly moving research IT out of the lab, we can realize comparable economies of scale and reductions in complexity, empowering investigators with new capabilities and freeing them to focus on their research.
I describe work we are doing to realize this approach, focusing initially on research data lifecycle management. I present promising results obtained to date, and suggest a path towards large-scale delivery of these capabilities. I also suggest that these developments are part of a larger "revolution in scientific affairs," as profound in its implications as the much-discussed "revolution in military affairs" resulting from more capable, low-cost IT. I conclude with some thoughts on how researchers, educators, and institutions may want to prepare for this revolution.
Presentation by Dr David Kelsey at Networkshop50 in June 2022.
Shows the work of the UK GridPP particle physics group in migrating the large volumes of science traffic for the CERN experiments to use IPv6.
Big data visualization frameworks and applications at Kitwarebigdataviz_bay
Big data visualization frameworks and applications at Kitware
Marcus Hanwell, Technical Leader at Kitware, Inc.
March 27th 2014
Kitware develops permissively licensed open source frameworks and applications for scientific data applications, and related areas. Some of the frameworks developed by our High Performance Computing and Visualization group address current challenges in big data visualization and analysis in a number of application domains including geospatial visualization, social media, finance, chemistry, biological (phylogenetics), and climate. The frameworks used to develop solutions in these areas will be described, along with the applications and the nature of the underlying data. These solutions focus on shared frameworks providing data storage, indexing, retrieval, client-server delivery models, server-side serial and parallel data reduction, analysis, and diagnostics. Additionally, they provide mechanisms that enable server-side or client-side rendering based on the capabilities and configuration of the system.
Big Data Visualization Meetup - South Bay
http://www.meetup.com/Big-Data-Visualisation-South-Bay/
M4M 2 the Rescue of M2M (Eclipse DemoCamp Trondheim)Werner Keil
M4M or Measure 4 Measure, ever since Shakespeare's play with the same name we know, people can be mistaken for one another. A Duke (like the beloved Java mascot) claims to be a monk, the head of a dead pirate is presented to be that of the young hero. So can important information like Units of Measurement be misinterpreted. While humans reading 10°C, 10 C or 10 Degree Celsius, each of those could be interpreted and understood well enough. For M2M communication, unless a program is provided with a large glossary of alternate terms, only ONE of these would be acceptable.
This is where the Unified Code for Units of Measurement (UCUM) among similar approaches like UnitsML, SensorML or a few others are vital for error-free M2M transactions, not just between sensors or measurement devices, but also and especially vehicles or distributed devices.
OSGi Measurement has been around for some time (R3) but never gained as much momentum, as many other bundles of OSGi did. Except for very few use cases in the Embedded or Automotive sector it is practically unused and based on statements by its contributors in the OSGi Alliance to be considered legacy with no plans continue development.
After a brief overview of common M2M errors from Gimli to Mars, This session provides an overview of OSGi Measurement, Eclipse OUMo, what they have in common and where the differences lie. Although most of today's OSGi containers are capable of dealing with units or measurement better and more reliable with UOMo, both can where necessary also exchange information and collaborate. E.g. if legacy devices and code cannot be easily replaced. For this We'll take a look at interoperability between different systems or with other unit technologies and languages like F#, Fantom, Python or Lua.
Science has escaped the lab and is roaming free in the world. People use software to understand the world . What tools are needed to support that work?
GALE: Geometric active learning for Search-Based Software EngineeringCS, NcState
Multi-objective evolutionary algorithms (MOEAs) help software engineers find novel solutions to complex problems. When automatic tools explore too many options, they are slow to use and hard to comprehend. GALE is a near-linear time MOEA that builds a piecewise approximation to the surface of best solutions along the Pareto frontier. For each piece, GALE mutates solutions towards the better end. In numerous case studies, GALE finds comparable solutions to standard methods (NSGA-II, SPEA2) using far fewer evaluations (e.g. 20 evaluations, not 1,000). GALE is recommended when a model is expensive to evaluate, or when some audience needs to browse and understand how an MOEA has made its conclusions.
Three Laws of Trusted Data Sharing:(Building a Better Business Case for Dat...CS, NcState
Discussions about sharing
- Too much fear
- Not enough about benefits
Can we learn more from sharing that hoarding ?
- Yes (results from SE)
Three laws of trusted data sharing:
- For SE quality prediction..
- Better models from shared privatized data that from all raw data
Q: does this work for other kinds of data?
A: don’t know… yet
Next Generation “Treatment Learning” (finding the diamonds in the dust)CS, NcState
Q: How have dummies (like me) managed to gain (some) control over a (seemingly) complex world?
A:The world is simpler than we think.
◆ Models contain clumps
◆ A few collar variables decide which clumps to use.
ICSE’14 Workshop Keynote Address: Emerging Trends in Software Metrics (WeTSOM’14).
Data about software projects is not stored in metrc1, metric2,…,
but is shared between them in some shared, underlying,shape.
Not every project has thesame underlying simple shape; many projects have different,
albeit simple, shapes.
We can exploit that shape, to great effect: for better local predictions; for transferring
lessons learned; for privacy-preserving data mining/
172529main ken and_tim_software_assurance_research_at_west_virginia
1. IV&V Facility
Research Heaven,
West Virginia
1
SA @ WV
(software assurance
research at West Virginia)
Kenneth McGill
NASA IV&V Facility Research Lead
304.367.8300
Kenneth.McGill@ivv.nasa.gov
Dr. Tim Menzies Ph.D. (WVU)
Software Engineering Research Chair
tim@menzies,com
2. IV&V Facility
Research Heaven,
West Virginia
2
Why, what is software
assurance?
• Definition:
– Planned and systematic set of
activities
– Ensures that software
processes and products conform
to requirements, standards, and
procedures.
• Goals:
– Confidence that SW will do what is
needed when it’s needed.
Before bad software After bad software
• Why software assurance?
–bad software can kill good
hardware.
–E.g. ARIANE 5: (and many others)
•Software errors in inertial
reference system
•Floating point conversion overflow
Ariane 5
3. IV&V Facility
Research Heaven,
West Virginia
3
OSMA Software Assurance
Research Program
• Office of Safety & Mission Assurance (Code Q- OSMA)
• Five million per year
• Applied software assurance research
• Focus:
– Software, not hardware
– SW Assurance
– NASA-wide applicability
• Externally valid results; i.e. useful for MANY projects
• Organization:
– Managed from IV&V Facility
– Delegated Program Manager: Dr. Linda Rosenberg, GSFC
4. IV&V Facility
Research Heaven,
West Virginia
4
Many projects
• Mega: highest-level perspective
– e.g. project planning tools like ASK-PETE
[Kurtz]
• Macro:
– e.g. understanding faults [Sigal, Lutz &
Mikulski]
• Micro:
– e.g. source code browsing [Suder]
• Applied to basic:
– Applied:
• (e.g.) MATT/RATT [Henry]: support large
scale runs of MATLAB
– Basic (not many of these)
• e.g. Fractal analysis of time series data
[Shereshevsky]
• Many, many more
– Too numerous to list
– Samples follow
– See rest of SAS!
Horn of
plenty
5. IV&V Facility
Research Heaven,
West Virginia
5
Many more projects!
0
7
11
12
6
5
1 1
3
1
6
2
7
27
10
12
4
0 0
5
26
22
0
5
10
15
20
25
30
ARC
GRC
GSFC
IV&V
JPL
JSC
KSC
LaRC
MSFC
Industry
University
2002
2003
Total proposals: 2.2
NASA centers: 1.5
Industry: 26
University: 3.7
Ratio
FY02/FY01
Good news!
• More good proposals
than we can fund
Bad news!
• same as the good news
6. IV&V Facility
Research Heaven,
West Virginia
6
A survey of 44 FY01 CSIPs
project 1 2 3 4 5 6 7 8 9 10 11 12 13 14 to 44
AATT 2
ISS 2
Space Shuttle 2
ST5 2
Aura 1
CHIPS 1
CLCS 1
CM2 1
CMMI 1
DSMS 1
EOSDIS 1
FAMS 1
GLAST 1
HSM4 1
HST 1
Mars 07 1
Mars 08 1
PCS 1
Space Station 1
Starlight 1
Stereo 1
SWIFT 1
X-38 1
5 4 3 2 2 2 2 2 1 1 1 1 1 0
Need more
transitions!
(but don’t
forget the
theory)
75% with no
claim for
project
connections
7. IV&V Facility
Research Heaven,
West Virginia
7
Action plan- restructure
CSIPS: more transitions!
• New (year 1)
– Fund many
• Renewed (year 2)
– Continue funding the promising new
projects
– Recommended: letter of endorsement
from NASA project manager
• Transition (year 3)
– Select a few projects
– Aim: tools in the hands of project folks
– Required: project manager involvement
• Reality check:
– Transition needs time
– Data drought
8. IV&V Facility
Research Heaven,
West Virginia
8
Long transition cycles
CO2 + 2H2 —> CH4 + O2
Mars
atmosphere
oxidizerfuel
on-board
(no photo)
Carmen
Mikulski
JPL
Robyn Lutz
JPL, CS-Iowa
State
• Pecheur &
practical formal methods
– In-Situ Propellant Production project
– Taught developers:
• Livingstone model-based
diagnosis
• model-checking tool tools
• developed by Reid Simmons,
(CMU)
– Technology to be applied to the
Intelligent Vehicle Health Maintenance
(IVMS) for 2nd generation shuttles
• Lutz, Mikulski &
ODC-based analysis of defects
– Deep-space NASA missions
– Found 8 clusters of recurring defects
– Proposed and validated 5
explanations of the clusters
– Explanations changes to NASA
practices
– ODC being evaluated by JPL’s defect
management tool team
Charles
Pecheur
RIACS, ASE,
ARC
10. IV&V Facility
Research Heaven,
West Virginia
10
End the drought:
bootstrap off other systems
• Find the
enterprise-wide
management
information
system
• Insert data
collection hooks
– E.g. JPL adding
ODC to their defect
tracking system
– WVU SIAT sanitizer
11. IV&V Facility
Research Heaven,
West Virginia
11
End the drought:
Contractors as researchers
active data
repository
• Buy N licenses of a defect
tracking tool (e.g. Clearquest)
• Give away to projects
– In exchange for their data
• Build and maintain a central
repository for that data
– With a web-based query
interface
• Data for all
take me to
your data
12. IV&V Facility
Research Heaven,
West Virginia
12
End the drought:
Contractors as researchers (2)
abstractionabstraction
actionaction
reflectionreflection
experienceexperience 1
2
3
4
Mark Suder
Titan, IV&V
Hypertext power browser for source code4 SIAT-1}
high-severity errors, recall what SIAT queries
d to finding those errors
4’
2’
Assess each such “power queries”
Reject the less useful ones
3’
Procedures manual for super SIAT or
new search options in interface
SIAT2
}
1’ Use it.
See also:
• Titan’s new
ROI project
• Any
contractor
proposing an
NRA
• Galaxy
Global’s
metric
project
See also:
• Titan’s new
ROI project
• Any
contractor
proposing an
NRA
• Galaxy
Global’s
metric
project
13. IV&V Facility
Research Heaven,
West Virginia
13
End the drought:
raid old/existing projects
• Cancelled projects with
public-domain software
– E.g. X-34
• Or other open source NASA
projects
– E.g. GSFC’s ITOS:
– real-time control and
monitoring system during
development, test, and on-orbit
operations,
– UNIX, Solaris, FreeBSD,
Linux, PC
– Free!!
– NASA project connections:
• Triana,
• Swift,
• HESSI,
• ULDB,
• SMEX,
• Formation Flying Testbed,
• Spartan
14. IV&V Facility
Research Heaven,
West Virginia
14
End the drought:
synergy groups
• N researchers
– Same task
– Different
technologies
• Share found data
• E.g. IV&V business
case workers
• E.g. monthly fault
teleconferences
– JPL:
• Lutz, Nikora
– Uni. Kentucky:
• Hayes
– Uni. Maryland:
• Smidts
– WV:
• Chapman
(Galaxy Global) &
Menzies (WVU)
15. IV&V Facility
Research Heaven,
West Virginia
15
End the drought:
Tandem experiments
• “Technique X finds errors”
– So?
• Industrial defect detection
capability rates:
– TR(min,mean,max)
– TR(0.35, 0.50, 0.65)
– Assumes manual
“Fagan inspections”
• Is “X” better than a
manual 1976
technique?
• Need “tandem
experiments”
to check
• I.e. do it twice
– Once by the researchers
– Once by IV&V
contractors (baseline)
0
20
40
60
80
100
120
defects
found
analysis design code test
baseline FM Fagan
fictional
data
0
20
40
60
80
100
120
cost
analysis design code test
16. IV&V Facility
Research Heaven,
West Virginia
16
Alternatively:
End your own drought
• Our duty, our goal:
– Work the data problem (e.g. see above)
– Goal of CI project year1: build bridges
– But the more workers, the better
• Myth: there is a “data truck” parked at IV&V
– full of goodies, just for you
• Reality: Access negotiation takes time
– With contractors, within NASA
• We actively assist:
– Each connection is a joy to behold,
an occasion to celebration
– We don’t celebrate much
• Bottom line:
– We chase data for dozens of projects
– Researchers have more time, more focus on
their particular data needs
• Ken’s law:
– $$$ chases researchers who chase projects
– CI year2, year3: needs a project connection
17. IV&V Facility
Research Heaven,
West Virginia
17
Alternatively (2), accept the
drought and sieve the dust
• The DUST project:
– Assumes a few key options control the rest
• Methodology:
– Simulate across range of options
– Data dust clouds
– Too many options: what leads to what?
– Summarize via machine learning
– Condense dust cloud
– Improve mean, reduce variance
• Case studies:
– JPL requirements engineering:
• Feather/JPL [Re02]
– Project planning:
• DART- Raque/ IVV; Chaing/UBC;
• IV&V costing: Marinaro/IVV, Smith/WVU
• general: Raffo, et.al/PSU [Ase02]
– An analysis of pair programming: Smith/WVU
– Better predictors for:
• testability: Cukic/WVU, Owen/WVU [Issre02, Ase02]
• faults: diStefano/WVU, McGill/IVV; Chapman/GG
• reuse : diStefano/WVU [ToolsWithAI02]
Figure 2. Initial (scattered black points)
and Final (dense white points)
0
50
100
150
200
250
300
0 300000 600000 900000 1200000
Cost
Benefit
Each dot =
1 random
project plan
The answer my
friend, is blowin’
in the wind
But wait: the
times they
are changing
18. IV&V Facility
Research Heaven,
West Virginia
18
Katerina Goseva Popstojanova
Other WVU SA research
Architectural
descriptions
Fault,
failure
data on
components,
connectors
Software
Specs & design
(early life cycle)
Code analysis
(iv&v,operational
usage)
Metrics(complexity,coupling,entropy )
Failure data from testing
Severity of failures
UML (sequence
diagrams,
state charts)
UML simulations
Static (SIAT,
Mccabe, entrophy)
Dynamic (testing,
runtime monitoring)
Testing & formal methods
Bayesian approach to reliability
Architectural metrics
Risk assessment & dynamic UML
Reliability &
operational profile errors
Hany Ammar
Bojan Cukic
collaborator
Goal: accurate,
stable, risk
assessment
early in the
lifecycle
Goal: accurate,
stable, risk
assessment
early in the
lifecycle
19. IV&V Facility
Research Heaven,
West Virginia
19
More WVU research
(FY02 UIs)
Architectural metrics
Risk assessment & dynamic UML
Intelligent flight controllers
Testing & formal methods
Bayesian approach to reliability
Fractal study of resource dynamics
Reliability & operational profile errors
SE research chair
interns
DUST
Ammar
Cukic
Goseva-
Popstojanova
Menzies
new
renewed
c = conference
w = workshop
j = journal
ISS hub controller,
“Dryden application”
F15
“JPL deep space mission”
DART
“KC-2”
IVV cost models
SIAT
X34
ITOS
X38
jj
j, ccccccc, w
c
cccccc
jc
c
w
FY03 proposals = 2.2*FY02
20. IV&V Facility
Research Heaven,
West Virginia
20
Function Point Metrics for
Safety-Critical Software
• Thesis:
– Traditional function-point
cost estimation
– Incorrect for safety-critical
software
• > 1 way to skin a cat
– >1 way to realize a safety
critical function:
– NCP=
N-copy programming
– NVP=
N-Version
Programming ,
– NSCP=
N Self-Checking
Programming,
– …
– With, without redundancy,
• Method:
– explore them all!
1.3000
1.4000
1.5000
1.6000
1.7000
1.8000
1.9000
2.0000
0 0.033 0.1 0.33 1
Algorithm Complexity
H2/H1,C2/C1
NCP
NVP,NSCP
RFCS
CRB
RB,NRB
DRB,EDRB
NCP
NVP,NSCP
RFCS
CRB
RB,NRB
DRB,EDRB
Design Diversity, add eight
more
Design Diversity, add one
more
Data Diversity
H2 and C2 : effort & cost, redundant system
H1 and C1: effort & cost, non-redundant
system Afzel Noore
21. IV&V Facility
Research Heaven,
West Virginia
21
Pre-disaster warnings
[Cukic, Shereshevsky]
Can we defer a maintenance cycle and keep doing science for a while longer?
Mark
Shereshevsky
CrashEarly warning
}
Time for graceful
shutdown
Bojan Cukic
ARTS II
22. IV&V Facility
Research Heaven,
West Virginia
22
Intelligent flight controllers
[Napolitano, Cukic] (and menzies)
Marcello Napolitano
(Mechanical and
Aerospace)
Bojan Cukic
(CSEE)
Lifecycle opportunities for
V&V of neural network based
adaptive control systems.
23. IV&V Facility
Research Heaven,
West Virginia
23
The road ahead: applied &
theoretical research
CSIPs: applied
research
USIPs: applied +
theoretical
research
Need both
To boldly go…
Editor's Notes
IV&V proposals include those by government PI only. University PIs are included in the University category.
WVU proposals are not included.