Software Testing and Reliability Testing Real-Time Systems Aditya P. Mathur Purdue University May 19-23, 2003 @ Guidant Co...
Learning objectives <ul><li>Issues in testing real-time systems </li></ul><ul><li>Tools for testing real-time systems </li...
References <ul><li>Testing Embedded Systems, Do You have the GuTs for it? , Vincent Encontre, Rational June 2001,  http://...
The RTCA DO-178B Guidelines <ul><li>The guidelines in DO-178B impose constraints on the software development process so th...
Levels of Criticality-A, B <ul><li>Level A:  Software with anomalous behavior, as shown by the system safety assessment pr...
Levels of Criticality-C, D, E <ul><li>Level C:  Software with anomalous behavior, as shown by the system safety assessment...
Guidelines Related to Testing <ul><li>Test Procedures are correct </li></ul><ul><li>Compliance with Software Verification ...
Issues in Testing Real-Time Systems   <ul><li>Separation of development and execution platforms </li></ul><ul><li>Variety ...
Test Methodology: Unit Testing   <ul><li>Identify the module to be tested. This module becomes MuT. </li></ul><ul><li>Prep...
Issues in Unit Testing   <ul><li>How to generate tests? </li></ul><ul><li>What to observe inside MuT?  </li></ul><ul><li>R...
Test Methodology: Integration Testing   <ul><li>Combine multiple MuT into a larger software component. </li></ul><ul><li>B...
Test Methodology: System (Unit) Testing   <ul><li>The MuT is now a complete software application. </li></ul><ul><li>Test i...
Test Methodology: System Integration Testing   <ul><li>Integrate other software components of the end-application. These m...
Test Methodology: System Validation Testing   <ul><li>For overall embedded system, perform the following: </li></ul><ul><l...
Run-time Analysis Source WindRiver and Agilent offer real-time trace tools that work with Agilent’s logic analyzers. Compi...
Applicability of Test Techniques to Real-Time Systems   <ul><li>All techniques for test data generation and coverage analy...
Tools for Testing Real Time Systems   <ul><li>Rational: Test RealTime:  code coverage; execution sequence, unit testing; i...
Summary <ul><li>Issues in testing real-time systems </li></ul><ul><li>Test tool architecture </li></ul><ul><li>Test method...
Upcoming SlideShare
Loading in...5
×

Software Testing and Reliability Testing Real-Time Systems

1,496

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,496
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
37
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Software Testing and Reliability Testing Real-Time Systems

  1. 1. Software Testing and Reliability Testing Real-Time Systems Aditya P. Mathur Purdue University May 19-23, 2003 @ Guidant Corporation Minneapolis/St Paul, MN Last update: April 17, 2003 Graduate Assistants : Ramkumar Natarajan Baskar Sridharan
  2. 2. Learning objectives <ul><li>Issues in testing real-time systems </li></ul><ul><li>Tools for testing real-time systems </li></ul><ul><li>Methodology for testing real-time systems </li></ul>
  3. 3. References <ul><li>Testing Embedded Systems, Do You have the GuTs for it? , Vincent Encontre, Rational June 2001, http://www.rational.com/products/whitepapers/final_encontre_testing.jsp </li></ul><ul><li>CodeTest Vision for Embedded Software , Applied Microsystems Corporation, http://www.amc.com/products/codetest/. </li></ul><ul><li>G-Cover Object Code Analyzer , Green Hills Software Inc., http://www.ghs.com/products/safety_critical/gcover.html </li></ul><ul><li>DO-178B , http://www.rtca.org and http://www.windriver.com/products/html/do178b.html </li></ul>
  4. 4. The RTCA DO-178B Guidelines <ul><li>The guidelines in DO-178B impose constraints on the software development process so that the resulting system is safe. </li></ul><ul><li>The FAA’s DO-178B offers guidelines for the development of airborne systems equipment software. </li></ul><ul><li>Most RTOS tool vendors have accepted the guidelines in DO-178B and begun to offer tool support. </li></ul>
  5. 5. Levels of Criticality-A, B <ul><li>Level A: Software with anomalous behavior, as shown by the system safety assessment process, would cause or contribute to a failure of system function resulting in a catastrophic failure condition for the aircraft. </li></ul><ul><li>Level B: Software with anomalous behavior, as shown by the system safety assessment process, would cause or contribute to a failure of system function resulting in a hazardous/severe- major failure condition for the aircraft. </li></ul>Guidant products: Level A or B?
  6. 6. Levels of Criticality-C, D, E <ul><li>Level C: Software with anomalous behavior, as shown by the system safety assessment process, would cause or contribute to a failure of system function resulting in a major failure condition for the aircraft. </li></ul><ul><li>Level D: Software with anomalous behavior, as shown by the system safety assessment process, would cause or contribute to a failure of system function resulting in a minor failure condition for the aircraft. </li></ul><ul><li>Level E: Software with anomalous behavior, as shown by the system safety assessment process, would cause or contribute to a failure of system function with no effect on aircraft operational capability or pilot workload. </li></ul>
  7. 7. Guidelines Related to Testing <ul><li>Test Procedures are correct </li></ul><ul><li>Compliance with Software Verification Plan </li></ul><ul><li>Results are correct </li></ul><ul><li>Test coverage of High-Level requirements </li></ul><ul><li>Test coverage of Low-Level requirements </li></ul><ul><li>Test coverage of object code </li></ul><ul><li>Test coverage - Statement, Decision, and Modified condition/decision </li></ul><ul><li>Test coverage - Data coupling and control coupling </li></ul>
  8. 8. Issues in Testing Real-Time Systems <ul><li>Separation of development and execution platforms </li></ul><ul><li>Variety of execution platforms (combinations of hardware and OS, e.g. Z80/QNX, Tornado/PowerPC) </li></ul><ul><li>Tight resources and timing constraints </li></ul><ul><li>Emerging quality and certification standards </li></ul>
  9. 9. Test Methodology: Unit Testing <ul><li>Identify the module to be tested. This module becomes MuT. </li></ul><ul><li>Prepare MuT for testing. </li></ul><ul><li>Isolate the MuT from its environment. </li></ul><ul><li>Make MuT an independently executable module by adding a stub and a test driver . </li></ul>Replaces the rest of the application Module under test Generate tests stub MuT Test Driver Test harness
  10. 10. Issues in Unit Testing <ul><li>How to generate tests? </li></ul><ul><li>What to observe inside MuT? </li></ul><ul><li>Returned parameters </li></ul><ul><li>Internal state such as values of global variables, coverage, control flow for verification against a UML sequence diagram, etc. </li></ul><ul><li>How to observe? </li></ul><ul><li>Use coverage tools or debuggers </li></ul><ul><li>Insert probes </li></ul><ul><li>When to stop testing? </li></ul>
  11. 11. Test Methodology: Integration Testing <ul><li>Combine multiple MuT into a larger software component. </li></ul><ul><li>Build a test harness. </li></ul><ul><li>Look for the correctness of interactions amongst the various MuTs in the component. Again, UML sequence diagrams can be useful in validating the interactions against design. </li></ul>
  12. 12. Test Methodology: System (Unit) Testing <ul><li>The MuT is now a complete software application. </li></ul><ul><li>Test is executed under the RTOS. </li></ul><ul><li>Communication amongst individual modules might be via the RTOS. </li></ul><ul><li>The test driver could now be a simulator. The system needs to be brought to a desired initial state and then generate and send to the system a sequence of test messages. </li></ul><ul><li>Observation is done using probes. The data so collected is analyzed on the host platform. </li></ul><ul><li>Usually one does grey-box testing at this stage but more could be done. </li></ul>
  13. 13. Test Methodology: System Integration Testing <ul><li>Integrate other software components of the end-application. These might be off-the-shelf components. </li></ul><ul><li>Test is integrated system </li></ul>
  14. 14. Test Methodology: System Validation Testing <ul><li>For overall embedded system, perform the following: </li></ul><ul><li>Check if the system meets the end-user functional requirements. </li></ul><ul><li>Perform non-functional testing. This includes load testing, robustness testing, and performance testing. </li></ul><ul><li>Check for conformance with any inter-operability requirements. </li></ul>
  15. 15. Run-time Analysis Source WindRiver and Agilent offer real-time trace tools that work with Agilent’s logic analyzers. Compiler Assembler/Linker Executable Target platform RTA server GUI Analysis tools
  16. 16. Applicability of Test Techniques to Real-Time Systems <ul><li>All techniques for test data generation and coverage analysis covered in this course are applicable when testing real-time applications. </li></ul><ul><li>Due to the embedded nature of such applications, one generally needs to use special tools that operate in a client-server mode; the server running on the host machine and the client on the target environment. </li></ul><ul><li>The client provides data gathered during the execution of the embedded application and the server analyses and presents it to the tester. </li></ul>
  17. 17. Tools for Testing Real Time Systems <ul><li>Rational: Test RealTime: code coverage; execution sequence, unit testing; integration testing, checking assertions for C++ classes; testing C threads, tasks, processes; evaluatin the UML model; requirement to test traceability; plug-in for mathworks Simulink. </li></ul><ul><li>Applied Microsystems: CodeTest: A suite of several tools; performance measurement; tracing of execution history; finds memory leaks; code coverage measurement; supports QNX . </li></ul><ul><li>Green Hills Software: G-Cover: Coverage analysis at the object code level; indirect source code coverage information. </li></ul><ul><li>WindRiver: RTA: Detects memory leaks; identifies hot spots; per task profiling. </li></ul>
  18. 18. Summary <ul><li>Issues in testing real-time systems </li></ul><ul><li>Test tool architecture </li></ul><ul><li>Test methodology </li></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×