This document summarizes an approach to automated testing of large, multi-language software systems using cloud computing. It discusses applying a lightweight, model-based test generation and execution approach to several NASA projects, including GMSEC, Core Flight Software, Space Network, and Mars Science Laboratory. Models of system APIs and interfaces are developed and used to automatically generate test cases, finding bugs. The approach has been successfully transferred to other teams and is being applied to additional complex NASA systems.

1. CESE
Automated Testing of Large Multi-
language SW Systems using Cloud
Computing
Executive Status Report
Principal Investigator (PI): Dr. Mikael Lindvall, CESE
NASA POC: Markland Benson, White Sands
Team members:
Dharma Ganesan, Dr. Chris Ackermann(CESE)
GMSEC, CFS, Space Network, MSL
© 2011 Fraunhofer USA, Inc. 1
Center for Experimental Software Engineering

2. CESE
Motivation
• Bugs can lead to deaths, injuries, or financial
loss
• Software testing is necessary to find the
presence of bugs
• Software testing consumes 50% - 75% of the
development effort
• Many NASA projects could benefit from
automated test generation
• In fact, several skeptical engineers noted that
the proposed work is illuminating!
© 2011 Fraunhofer USA, Inc.
Center for Experimental Software Engineering

3. CESE
Technical Context
• The system(s) under test (SUT) are complex
• Multiple programming languages
– Ada, C, C++, Fortran, Java
• Distributed legacy systems
– More than 20 yrs old
– Runs on multiple machines and involves various
OSes,e.g. OpenVMS
• GUI intensive
– Web, Java Swing, AWT, X Windows, Motif
© 2011 Fraunhofer USA, Inc.
3
Center for Experimental Software Engineering

4. CESE
Problems
• Test cases are developed manually
• Some test execution is automated (e.g., JUnit)
• Test cases miss valid “corner” cases
• Test cases are also programs: not easy for non-
technical stakeholders to understand
• Difficult to summarize what was tested
 Approach: Lightweight Model-based Test
Generation and Execution
© 2011 Fraunhofer USA, Inc.
4
Center for Experimental Software Engineering

5. CESE
Approach
• Develop a model of the system under test
• The model contains actions and expected results
• Automatically derive test cases from the model
• Execute the test cases
• Decide when to stop testing based on model
coverage!
© 2011 Fraunhofer USA, Inc.
Center for Experimental Software Engineering

6. CESE
An Example Model
APIs of the module under test
public interface IConnection
{
public Status Connect();
public Status Disconnect();
© 2011 Fraunhofer USA, Inc.
}
Center for Experimental Software Engineering

7. CESE
Currently Targeted Projects
• GMSEC – Reusable framework for ground systems
– Modeled the Core API and Applications
– Generated executable test cases from the model
– Confirmed defects/violations reported and fixed
• Core Flight Software – Reusable framework for
flight systems
– Modeled the OS abstraction layer (OSAL)
– Generated executable test cases from the model
– Confirmed defects/violations reported and fixed
– Test cases delivered to the CFS team
© 2011 Fraunhofer USA, Inc.
7
Center for Experimental Software Engineering

8. CESE
Currently Targeted Projects
• Space Network – White Sands
– Evaluating different test execution tools suitable
for Open VMS environment
– E.g. Image-based test execution tools: EggPlant
and Sikuli
– Challenges in this project:
• SUT not possible to run at Fraunhofer
• Must visit Goddard with an escort (secured building)
• Not possible to connect to Internet
© 2011 Fraunhofer USA, Inc.
8
Center for Experimental Software Engineering

9. CESE
Currently Targeted Projects
• Mars Science Laboratory (MSL) – JPL
– Can current test cases be modeled and used to
generate new test cases?
• New set cover scenarios not previously covered?
• New approach more cost effective?
– Challenges in this project:
• SUT not possible to run at Fraunhofer
• Software is sensitive (ITAR)
© 2011 Fraunhofer USA, Inc.
9
Center for Experimental Software Engineering

10. CESE
Current Results
• An end-to-end approach for test generation
– Successfully used on the GMSEC and the
CFS and detected bugs
• Approach works well on different levels:
– API (Module interface) level testing
– GUI testing
• Approach transferred to an intern at GSFC
– The intern found new bugs using the approach
– Demonstrates the approach is lightweight
© 2011 Fraunhofer USA, Inc.
10
Center for Experimental Software Engineering

11. CESE
Summary and Next Steps
• We’re building a new approach that
– Helps in test automation using lightweight
models
• Prototype approach successfully applied
to GMSEC and CFS/cFE
• Next steps are to apply to the Space
Network and the MSL projects
• Examples and technical details in
technical presentation!
© 2011 Fraunhofer USA, Inc.
11
Center for Experimental Software Engineering