A System Testing Framework


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

A System Testing Framework

  1. 1. A System Testing Framework RTECC, 6. May 2003 Dr. Ferdinand Gramsamer gramsamer@bbv.ch www.bbv.ch
  2. 2. System Testing Framework RTECC, May 6 2003 Abstract While a large variety of test frameworks for unit testing of software systems exists, similar frameworks for testing hardware systems and distributed components are hardly found. We introduce ‘TestClamp’, a scalable, modular, and customizable system-testing framework provided with a single common user-interface ‘clamping’ on the hardware I/Os. Its benefits are automated regression testing, administering test suites and test cases, and reporting. Summary TestClamp, developed as an open source application by bbv Software Services Corp. for a customer in the building technology arena, is a framework for system testing. It is not only a workbench to easily write test cases, but also • Middleware to integrate standard or proprietary interfaces, such as RS232, CAN, and BACNet. • Interface to simulation platforms such as MATLAB • Test-manager to administer tests and generate test reports • Tool box for often commonly required functionality, such as logging, schedulability, timing control, etc. TestClamp realizes these design goals: 1. Scalable. TestClamp is covered by the LGPL 1.2 or higher licence, which opens the door to self-written add-ons and changes to the main application. It is therefore scalable by the needs of the user or the requirements of the system under test. The scalability feature is guaranteed by the underlying software architecture, which rigorously implements common design patterns [Gamma97]. 2. Modular. TestClamp is modular. Middleware provides functionality to the main application. The main application brings all the required functionality together and delivers the framework for the test case developer. All modules, middleware, the application and test cases are written in one language, which is Java to provide platform-independence. This opens the possibility to utilize open-source development tools. TestClamp has been designed for someone not knowledgeable in Java or the application, to be able to write test cases after two days of coaching. 3. Customizable. TestClamp is customizable by its command-line options and configuration capabilities. Whereas command-line options set up the application in the desired mode of operation, configuration files contain set-up information for attached subsystems. Furthermore, TestClamp can run in master, configuration, or server mode, which allows TestClamp to be used in cooperation with third-party tools. All in all, TestClamp is a framework to perform system testing. It not only has its strengths in its design, but has also matured through the close cooperation of customers with bbv Software Services Corp. bbv Software Services Corp. 2
  3. 3. System Testing Framework RTECC, May 6 2003 A System Testing Framework RTECC, Zurich, May 6 Dr. Ferdinand Gramsamer bbv Software Services Corp. 1 Situation You have successfully implemented your real-time embedded application. It is running on a piece of hardware. You now want to test it. bbv Software Services Corp. 2 bbv Software Services Corp. 3
  4. 4. System Testing Framework RTECC, May 6 2003 Agenda • Motivation • Problem Description • Solutions - Examples • State of the Art • Summary bbv Software Services Corp. 3 Motivation • Systems grow larger and are more complex • Too many different tools - expensive • Proprietary interfaces supported? • Increased need for simulation • Amount of test data needs to be dealt with • Repetitive tasks for testing Need of framework for system testing bbv Software Services Corp. 4 bbv Software Services Corp. 4
  5. 5. System Testing Framework RTECC, May 6 2003 Typical Test Environment • Components – Subsystems (adapter cards, Simulation uP subsystem), peer- systems Component Instrument • Instruments PC – Logic Analyzer, Oscilloscope, BERT, etc. SUT • Simulation Component Instrument – MatLab/Simulink, Filter • SUT: System Under Test bbv Software Services Corp. 5 Goal Develop endurable system testing framework: – Platform independent (OS) – I/O adaptable (new standards) – Configurable (ever more complex cards, and boards) – Test administration: register and execute test cases, store results, replay, schedulable – Open source (no license fee, customizable) – One programming language – Flexible to integrate third party tools bbv Software Services Corp. 6 bbv Software Services Corp. 5
  6. 6. System Testing Framework RTECC, May 6 2003 Focus • PC • Software architecture? Simulation Component Instrument PC SUT Component Instrument bbv Software Services Corp. 7 Logical View Testclamp Testhilfsmittel Testscript- Schnittstellen Generator Middle- Testscript Testhilfsmittel ware Testscript- RS232 Editor Testhilfsmittel IO-Emulator Ablauf- Testscript- steuerung Debugger Testhilfsmittel (Applikation) …. Graphisches User Interface Testhilfsmittel IO-System Test- Testresultat resultate Analyser bbv Software Services Corp. 8 bbv Software Services Corp. 6
  7. 7. System Testing Framework RTECC, May 6 2003 Distributed Architecture CommandLineMgr ConfigurationMgr Application TestMgr Tools RepositoryMgr Execution ComponentMgr bbv Software Services Corp. 9 Elements • CommandLineMgr Parses command line: CommandLineMgr • Database • Directory ConfigurationMgr • Tests Application • Configuration file TestMgr • others • ConfigurationMgr Tools RepositoryMgr key-value pairs • TestMgr Execution Registered tests to execute ComponentMgr bbv Software Services Corp. 10 bbv Software Services Corp. 7
  8. 8. System Testing Framework RTECC, May 6 2003 Elements cont’d • RepositoryMgr Create directory structure CommandLineMgr • ComponentMgr Download file, or ConfigurationMgr configuration to subsystem(s) Application • Execution TestMgr Run tests Tools • Tools RepositoryMgr – Matlab (simulation) Execution – I/Os, interfaces ComponentMgr – Source control/database bbv Software Services Corp. 11 Functionality • Timer: – measure elapsed time; delays – Example: Timer t = new Timer(); t.start(); t.stop(); t.getElapsedTime(); • Scheduler: – execute tests at a specific time of the day – Example: addTestSuite(“TestSuite1”, “2003:05:06:10:12:00”); • Comparator: – test event in a configurable time window – Example: Comparator c = new Comparator(x, minusdelta, plusdelta); boolean result = c.isInLimits(y); • Logger: – progress of test execution (view/file) – Example: getDefaultLogger().addInformativeLog(“Hello”, 3); bbv Software Services Corp. 12 bbv Software Services Corp. 8
  9. 9. System Testing Framework RTECC, May 6 2003 TestClamp Design • Creation, Start-up, Shutdown – Basic services: i.e. kernel, memory management – Creation: create parts – Initialization: set-up relationship between elements – Startup: put system in full operational mode – Shut-down: stop system before leaving application • Post-Office – One element requests service from another-one – Central object rather than hard-coded in elements – Expl: DemoAppl a=(DemoAppl)Application.getDefault(); a.getScheduler().getSuiteResultFile().writeln(“Hello”); • Composite for tests (next slides) bbv Software Services Corp. 13 Test Hierarchy Common management TestItem capabilities Aggregation of TestItems TestContainer with parameters TestProject TestCampaign TestSuite TestCase Entity holding all All tests for a sub- All tests for A scenario tests and direct- System a unit run(); ory addTestSuite(); addTestCase(); bbv Software Services Corp. 14 bbv Software Services Corp. 9
  10. 10. System Testing Framework RTECC, May 6 2003 TestItem • Name • Author • Version • Modification Date • Description • General administrative information • Example: author("Gramsamer (bbv Software Services Corp.)"); modificationDate("May 6th 2003"); version("1.0"); bbv Software Services Corp. 15 Test Composition 1 TestManager optional Application 0 .. 1 1 .. n For each subsystem 1 TestProject A TestProject TestContainer 1 1 .. n 1 .. n 1 .. n TestCampaign TestSuite TestCase 1 1 0 .. n 0 .. n TestSuite TestCase 1 0 .. n TestCase bbv Software Services Corp. 16 bbv Software Services Corp. 10
  11. 11. System Testing Framework RTECC, May 6 2003 TestCase Example bbv Software Services Corp. 17 Commercial Tools • LabView/Windows (NI) – Focus: system tests, standard interfaces • Rational Test RealTime (IBM) – Focus: rt-tests of code as it executes on target • TestDirector (Mercury) – Focus: test management – Execution of tests through third party tools, scripts. WinRunner designed for GUI-testing bbv Software Services Corp. 18 bbv Software Services Corp. 11
  12. 12. System Testing Framework RTECC, May 6 2003 Summary • Flexibility by design • One language from low to high-level programming • Test management • Automated test execution • Open source bbv Software Services Corp. 19 12 May 200312 May 2003 References • www.ni.com • www.rational.com • www.mercury.com • www.bbv.ch • E. Gamma et al. “Design Patterns – Elements of Reusable Object-oriented Software,” Addison-Wesley, 1997 bbv Software Services Corp. 20 bbv Software Services Corp. 12
  13. 13. System Testing Framework RTECC, May 6 2003 Thank You! See you at the bbv booth for a demonstration! bbv Software Services Corp. 21 bbv Software Services Corp. 13