Automated Regression Testing for Embedded Systems in Action


Published on

This presentation shows a real world example of streamlining the software development for a medical device system, using continuous integration, Behavior Driven Development, and even robotics!

These ideas may be applied to any software project, regardless of budget or technologies.

Published in: Technology, Business

Automated Regression Testing for Embedded Systems in Action

  1. 1. Automated Regression Testing for Embedded Systems in Action by Steve Peterson AAND Tech, Inc
  2. 2. Situation <ul><li>Client manufactures a Class II* medical device </li></ul><ul><li>Produces reliable software. </li></ul><ul><li>Productivity is OK </li></ul><ul><li>All the T s are crossed and I s are dotted in terms of process and documentation. </li></ul><ul><li>However … </li></ul><ul><li>* - See this site for an explanation of the FDA’s medical classifications: </li></ul>
  3. 3. Inefficiencies <ul><li>Efficiency-wise we were getting killed by: </li></ul><ul><li>Requirements (traceability, maintaining) </li></ul><ul><li>Test Plans (writing, maintaining, executing) </li></ul>
  4. 4. Old Way <ul><li>Requirements written in Word documents. Manual Updates. Brute Force Traceability. </li></ul><ul><li>Test Plans written in Word documents. Execution by personnel. </li></ul>
  5. 5. Let’s Step Back in order to Solve This <ul><li>What are the “requirements” for the process? </li></ul><ul><li>We started with defining “project complete” </li></ul><ul><ul><li>All Requirements are written. </li></ul></ul><ul><ul><li>All Requirements are (successfully) tested. </li></ul></ul><ul><ul><li>All Requirements are documented. </li></ul></ul>
  6. 6. Next, Define the Deliverables* <ul><li>Software/Firmware </li></ul><ul><li>Requirements Documents </li></ul><ul><li>Test Plans </li></ul><ul><li>Test Results </li></ul><ul><li>*- This is far from a complete list , and doesn’t include items such as reviews and project plans. </li></ul>
  7. 7. Define Some Characteristics <ul><li>Software and its process are </li></ul><ul><li>Testable </li></ul><ul><li>Changeable </li></ul><ul><li>Traceable </li></ul>
  8. 8. Now something we would really like <ul><li>For the 4 deliverables from a couple slides ago: </li></ul><ul><ul><li>Software/Firmware </li></ul></ul><ul><ul><li>Requirements Documents </li></ul></ul><ul><ul><li>Test Plans </li></ul></ul><ul><ul><li>Test Results </li></ul></ul><ul><li>We would like this to be in sync for the entire duration of the project! </li></ul>
  9. 9. Our Plan <ul><li>Previously we did all the basics with revision controlling the documents, source code, binary images, etc. </li></ul><ul><li>We added: </li></ul><ul><li>A method of automatically linking requirements to tests/implementation </li></ul><ul><li>A framework that will build the code (from revision control) deploy it to the target, and perform automated regression testing. </li></ul>
  10. 10. Our Implementation* <ul><li>These items are covered in detail later. </li></ul><ul><li>Requirements remain in Word documents. However now we have automated requirements traceability (to tests/implementation) </li></ul><ul><li>Automated Regression Tests </li></ul><ul><li>Continuous Integration** </li></ul><ul><li>* - Many 3 rd party solutions for requirements tracking and testing exist. Due to cost and limits of customization, we chose to implement our own. </li></ul><ul><li>** - True Continuous Integration runs the entire regression test suite after each check in. In our case we stretch the term “continuous” to mean nightly. This is due to the abbreviated test suite requiring 3 hours to run and several days for the formal V&V test suite. </li></ul>
  11. 11. Requirements Tracking Tool <ul><li>Dynamic and Static Real Time generation of requirements traceability*. </li></ul><ul><li>Requirements based Testing. </li></ul><ul><li>Finds untested requirements, orphaned tests. </li></ul><ul><li>PDF report generation </li></ul><ul><li>Coming soon, changed requirements, triggers for changed code. </li></ul><ul><li>PDF report generated by the tool tracing the requirements to test/implementation. </li></ul><ul><li>Static coverage occurs from parsing the requirements documents and using Ruby’s introspection on the classes to indentify coverage </li></ul><ul><li>Dynamic coverage occurs from running the regression test. Each assert statement lists the requirements that it’s testing. We generate coverage matrices both for assert successes and failures. For you UML fans, the relationship between asserts and requirements is many-many (an assert can test more than one requirement and a requirement can be tested by more than one assert). </li></ul>
  12. 12. Tri-State Coverage <ul><li>Green Light – Test item passes </li></ul><ul><li>Red Light – Test item fails </li></ul><ul><li>Yellow Light – No test coverage for this item. </li></ul><ul><li>Red Light items get immediate attention. </li></ul><ul><li>Yellow Light items are given some slack (but not too much!) </li></ul><ul><li>The idea is to continuously add features and keep the code, tests, and documents in sync. </li></ul><ul><li>No “big bang” testing or documentation effort at the end! </li></ul>
  13. 13. Automated Testing Framework <ul><li>Components </li></ul><ul><li>Revision Control System (we used Subversion) </li></ul><ul><li>The Scheduler </li></ul><ul><li>The Build engine </li></ul><ul><li>The “Deployer” – installs the correct software under test on the device under test, whether a web or embedded device. </li></ul><ul><li>Peripheral Measurement Devices. </li></ul>
  14. 14. Embedded Device <ul><li>Want as much of the testing to be as non-invasive as possible. </li></ul><ul><li>Next revision will have robot button presses and a camera to read the display </li></ul><ul><li>Until then, the firmware allows simulated button presses and reading the LCD using commands via the serial port. Also a couple generate commands to read values. </li></ul><ul><li>Software safeguards exist to mitigate against these commands interfering with device operation. </li></ul>Chassis of the DUT (Device Under Test) with the serial port clip used to program the DUT, Issue commands and read status
  15. 15. Build Engine <ul><li>Performs a complete source code check out from the SVN* (Subversion) repository. The default is the trunk, but any branch, tag or SVN ID may be used. Launched from a cron job** or manual initiation. </li></ul><ul><li>The build may be production code base, a special developer code base, or for the embedded code, unit test code. </li></ul><ul><li>Applicable to both the embedded and web based technologies. </li></ul><ul><li>* - Subversion (SVN) is an open source revision control system: http:// / </li></ul><ul><li>** - A Cron job executes commands or scripts (groups of commands) automatically at a specified time/date. </li></ul>
  16. 16. The Scheduler (Page 1) <ul><li>UML diagram showing the relationship between a Job, Test Runs, a DUT, Peripheral Test Modules and installed Software version. </li></ul>
  17. 17. The Scheduler (Page 2) <ul><li>A DUT (Device Under Test) may be a single embedded device, a web component or a system DUT consisting of both. </li></ul><ul><li>A test module consists of peripheral devices under script control such as a programmable power supply, breathing simulator, manometer. </li></ul><ul><li>A run consists of a DUT (see above), and 0 or more peripheral test modules running a single script </li></ul>Web Interface for monitoring and controlling the testing “jobs”
  18. 18. The Scheduler (Page 3) <ul><li>A job consists of 1 or more runs. May span numerous DUTs </li></ul><ul><ul><li>Noteworthy jobs are the formal V&V job (may require up to 3 days to run), the nightly (the full V&V run trimmed down to about 6 hours), and developer submitted jobs. </li></ul></ul><ul><li>The job “manifest” specifies either a specific version of code to test (as produced by the build engine or a developer produced version). </li></ul><ul><li>The scheduler also acts as a traffic cop. A database tracks the attributes of various DUT units and queues up job/run requests based on run requirements. </li></ul>
  19. 19. Flash Memory Programmer (The Deployer) <ul><li>Under script control, this programs the embedded device’s flash memory with the specified binary image. </li></ul>
  20. 20. Peripheral Test Modules (Page 1) <ul><li>MacGyver Manometer* </li></ul>* The term MacGyver Manometer is a tongue in cheek reference to the 80/s TV show, where MacGyver could create anything using a paper clip and rubber band (see photo above). In this case the DUT, which contains a pressure sensor more accurate than most handheld manometers, is outfitted with special firmware that displays the current pressure, and reports the pressure up to 16 samples/second over the serial connector. Programmable Power Supply
  21. 21. Peripheral Test Modules (Page 2) <ul><li>Hans Rudolph Breathing Simulator </li></ul>
  22. 22. A couple more pictures <ul><li>More testing power per sq foot of space anywhere! </li></ul><ul><li>X10 Power Control </li></ul>
  23. 23. More Pictures <ul><li>Robotic Arm </li></ul><ul><li>Pressure Control Valve </li></ul>
  24. 24. Benefits <ul><li>Able to give embedded devices the same automated regression testing benefits available to web and PC applications </li></ul><ul><li>Process accommodates change. It is still not free, but much less painful. </li></ul><ul><li>Sanity check. No more being 90% done for 50% of the project. </li></ul><ul><li>Much less Stress! Bugs are fixed immediately. </li></ul><ul><li>Formal V&V testing took just a couple days, instead of weeks. </li></ul><ul><li>Over 97% of testing was automated! </li></ul>
  25. 25. Additional Benefits <ul><li>Because we used the real devices (day after day, night after night), this doubled as life testing. Non-software items were validated as well. </li></ul><ul><li>Due to the easy command set for controlling the DUT and capturing information from the measuring equipment, this was used for research applications as well. </li></ul><ul><li>Ruby has a rich set of Microsoft Office plugins so automated formatting of data in Excel and Word is straight forward. </li></ul>
  26. 26. Challenges <ul><li>Automated Testing is not the same as a user </li></ul><ul><li>We found that in actuality it is (or very close). We tested the Behavior (read button presses tangible user responses). In addition we constantly had real people performing unstructured testing. If this located a problem, additional items were added to the automated testing. </li></ul><ul><li>How can you be sure that the tests work (how do you test the tests?) </li></ul><ul><li>The test suites incorporate counter test cases – inject out of bounds conditions and verify that the test correctly catches it. </li></ul>
  27. 27. Contact <ul><li>Steve Peterson </li></ul><ul><li>AAND Tech, Inc. </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li>[email_address] </li></ul>