Wind River Framework for Automated Software Testing (FAST)


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Wind River Framework for Automated Software Testing (FAST)

  1. 1. Wind River Framework for Automated Software Testing (FAST) Wind River Framework for Automated complicate the test and validation developer or quality assurance (QA) Software Testing (FAST) was developed process. To properly validate the engineer needs to adapt tests to run on to help customers overcome the distribution, tests for the following are each device. Each target may have a challenges of testing large software needed: unique hardware configuration, distributions for mobile devices. FAST • Requirements peripherals, file storage systems, and so provides a system for combining • Performance on. Tests that were developed for one multiple test frameworks into a single • Stress and stability device frequently need to be adapted meta-framework that can be run with “a • Device driver to work on different devices. single button press.” It helps automate, • Middleware User interfaces (UI) that come with execute, and manage thousands of tests • Application different devices are often changing. and their results. • User interface • Multidevice Even if the functionality behind the UI is Wind River leverages as many open • Peripheral the same, the developer or QA source and existing test frameworks as • Board support package (BSP) engineer must adapt tests to the possible so testing is comprehensive, • Version checking different layouts and widget styles. without sacrificing time-to-market. • Telephony With FAST, each individual test Many open source test frameworks • Network framework or application is encapsulated contain thousands of tests and many Each of these types of tests needs to be within a “wrapper” script executed by packages that come with their own test performed as often as possible on each the FAST system. The results are all suites. FAST makes executing tests new release, and the results of these stored in a consistent manner in a much simpler because they are all tests need to be reported and managed. MySQL database. This provides a flexible wrapped in FAST scripts and run from and comprehensive system for handling the FAST graphical user interface (GUI) Because test programs are pulled from large amounts of test results from and an available command-line so many different sources, and they all heterogeneous test frameworks, and interface (CLI). have diverse ways of being run and of vastly simplifies the problem of wading generating and reporting results, The Challenge of Testing through and comparing the various executing the tests and tracking and individual test reports. Testing very large open source software evaluating the test results can especially distributions for mobile devices is a complicate the testing process. The The database contains all test artifacts tremendous challenge. There are amount of information can be and logs from the original test programs typically dozens of subsystems staggering, particularly when test cases and additional metrics collected by containing thousands of packages have different reporting formats. FAST. FAST can be thought of as a included in a distribution, with many meta-test framework that can run other As new original device manufacturers applications built using those packages. test frameworks and individual tests. (ODMs) develop hardware and wish to Differences in device types, peripherals have their products tested with a From the FAST database, Wind River used, and user interface variations certain distribution of the software, the can create a variety of test reports. Customizable formal reports (for QA managers), test run comparison reports Table of Contents Test Execution .................................... 3 (for developers), and interactive Wind River Test Library ...................... 4 regression analysis reports (for The Challenge of Testing.................... 1 Reporting ............................................ 5 engineering managers) are all provided. The FAST Solution .............................. 2 Appendix ............................................ 6 Additional reports can easily be created System Testing Plug-In ....................... 2 using the reporting tools of your choice.
  2. 2. The FAST Solution ods that perform common operations Results from other test tracking that are repeated in many tests (i.e., set- software (such as manual test tracking FAST offers significant advantages to ting up network interfaces). When a new software) can be imported into the both developers and QA engineers: target is introduced, Wind River only FAST database so FAST can track the • Integrates testing efforts in one appli- needs to modify the board class. Only test effort of the entire test cation: Once the different test frame- the communication and interface meth- organization. works and applications are “wrapped” ods are rewritten. The actual test scripts by a FAST script, they are accessible and wrappers need not change and can System Testing Plug-In from the testing GUI. Users may select be used to test both target types. Once Wind River System Testing Plug-in is an which tests to execute, and with one a script is written, it most likely will work button execute them all. There is no on multiple targets. Occasionally minor application test mechanism that can need to know how the test applications changes are needed if the functional- simulate user interactions with user work or what report formats they use; ity of an application/package changes interfaces (UIs). Along with FAST, Wind everything is handled by the scripts. from one target to another; but for the River developed this technology to test • Integrates other testing frameworks: most part, Wind River test libraries do large software distributions. System Any existing test framework or test not require many changes. Testing Plug-in enables the automated application can be integrated into • Leverages advanced reporting ca- testing of UIs and the user experience. FAST using a FAST script. FAST uses pabilities: Because all the results and This kind of testing has traditionally advanced parsing routines to decode test artifacts go into one database, it been manual because by its very nature the test results from any log file or is relatively easy to generate reports. it involves human interaction with the program output. No matter where the These reports can serve the following system UI and interaction among tests come from and how they present functions: multiple applications on the device or test results, FAST scripts can integrate - Provide customers with a list of tests even multiple devices. them into the system. performed and their results. - Compare the current test run to a System Testing Plug-in is a unique, • Creates custom test scripts: Often previous test run. This is useful to the powerful, and exciting technology that there are no open source or existing test assets available. Wind River or the software design team because they makes UI elements available for customer can create custom scripts can see what new bugs were intro- interrogation and manipulation from in FAST that can execute commands duced and what bugs were fixed since automated test scripts. It can easily on both a test server and targets and the last time tests were run. repurpose existing tests for use with a analyze their output. With this capabil- - Illustrate performance results new UI. For example, a calendar ity, a whole suite of tests that did not graphically so that managers can supports the same features across previously exist can be created. quickly see if identified metrics are implementations, such as entering an degrading or improving over time. • Applies identical tests to different - Track requirements, last test dates, appointment, listing today’s targets with minimal changes: FAST and results. This makes it easy to see appointments, setting an appointment is structured so that test scripts/wrap- where more tests need to be added alarm, and so on. With System Testing pers never directly communicate with and which requirements need to be Plug-in, tests for these common the targets. They always go through a debugged. scenarios need only be written once for “board” class that abstracts the test - Summarize test history to show the one UI and can be easily ported to a framework from the specific require- results and artifacts of a test over the different UI by using simple abstraction ments of a board. The board class can past several runs. This is useful for layer code. send commands to the target, read the developers to debug issues that have output, install a payload, and reboot recently been introduced in the System Testing Plug-in also supports the target. There are also special meth- software. parallelizing tests, such as testing across multiple targets and automating use case and scenario-based testing. FAST System Testing Plug-in offers significant Test Script advantages for application testing. It is able to simulate almost any user interaction with the target, including clicking widgets in the UI, dragging Board Class Board Class graphical elements, reading the for for “Target Type A” “Target Type B” contents of the UI elements, typing entries on a physical or virtual keyboard, and so forth. Target Target System Testing Plug-in scripts are Type A Type B integrated into FAST. FAST has a Figure 1: FAST test script configuration for multiple targets using board class standard wrapper that can be applied to 2 | Wind River Framework for Automated Software Testing (FAST)
  3. 3. Test Execution STP Scripts Configuring a target for testing can be easy and straightforward. FAST uses a board class to specify communication and control for the target. With a board Button Button class and network connection to the Mapping for Mapping for target, the tester is ready to begin testing. Target A Target B The FAST user interface offers significant flexibility to automate the testing process: Target A Target B • Software may be loaded onto the Application Application target before testing begins, and these Figure 2: System Testing Plug-in script GUI mapping uploaded files may be removed once testing is completed. any System Testing Plug-in test script. This allows these scripts to be executed, and their results can be tracked in the FAST database and reports. System Testing Plug-in scripts do not rely on the position of UI elements in an application. Instead, System Testing Plug-in uses the accessibility features of widget frameworks to know the names of the widgets to interact with. The widget creates a map of the buttons so the test scripts can refer to the buttons by name rather than screen location. Most GUI testing applications specify actions such as “Click at position 10,10.” A statement by System Testing Plug-in would be more like “Click the button mapped as ‘OK.’” This means that no matter how the application changes its layout, the System Testing Plug-in script can still execute the test. The same System Figure 3: FAST user interface Testing Plug-in script works on multiple implementations of the same programs (i.e., calculators, calendars, dialers, browsers) with different interfaces. However, the same script can be executed on all of them, with only minor changes to the button name maps. System Testing Plug-in scripts can be combined into one. This enables the testing of multiple applications running at the same time, for example, when a user is listening to music and reading email and a call comes in. Interleaving existing test scripts allows Wind River to easily create new test scenarios. Figure 4: FAST test lists 3 | Wind River Framework for Automated Software Testing (FAST)
  4. 4. • Special arguments may be executed whether performance is gradually • Compliance: FAST includes the latest on the command line for boards that dropping or improving over time. This version of the compliance suites for require them. prevents surprises at the end of a release Android Compatibility Test Suite (CTS), • The target may be configured to power testing cycle. Moblin (OSV, ISV, and Application down or reboot during testing or be- Checker), and GENIVI. tween tests. In the web GUI, the user can specify the • System framework: There is a complete • Baseline results may be identified for test environment and select the desired set of system framework tests specifi- comparing test results in the test report. tests and, instead of running them, save cally for Wind River Platform for Moblin. an .ini file that contains all tests planned • Test RPMs: Many of the open source The web-based graphical user interface for execution. The .ini file may be packages used in Mobile Linux include makes test selection and execution a their own tests. These are usually dif- loaded into the system at a later date quick process. The user navigates the ficult and procedurally cumbersome to with settings specific to that file to pass test tree at the left and selects run frequently. The Wind River Linux it to FAST for execution. This executes individual tests for running by right- Distribution Assembly Tool (LDAT) build the tests in the same fashion as if they clicking them. Each test may then be system for Mobile Linux extracts these were executed from the GUI. configured, including default time-out, into test RPMs that can be executed by count, parameters, and reboot options The command-line interface also has the FAST, providing an ongoing regression (reboot before, after, or after when the capability to stop a test run, get its status, test of these components, thus making and download its results and artifacts. it much easier to detect failures caused test does not pass). Once the test runs by changes in other areas of the system. are configured, each test’s progress is FAST also offers scripts that can • Packages: Wind River has created tests displayed and a test’s details are automatically analyze the FAST to assure that packages that are part available, including the start and end database to find the best tests to of the Mobile Linux system that do not dates, any test cases it ran, their status execute, based on the time passed since have their own tests are installed and (passed/failed), and any artifacts, such their last execution and their last operational. as log files, or key values (performance execution status (giving higher priority • Open source: Linux Test Project (LTP), metrics) that are associated with the test to those tests that have not run recently IOzone, iPerf, and Moblin Compliance case. Tool Suite are all integrated. Addi- and those that last failed). Using these tional third-party test frameworks are FAST supports a command-line scripts, FAST can run tests for a specific straightforward to add. interface that allows execution of tests amount of time (e.g., eight hours). The • Performance: These include automated from the Linux command line. This is system will find the best tests to run that tests for graphics, networking, and very useful for running tests regularly will approximately fill the time (tests boot-time performance. from a cron job. Running tests regularly typically take the same amount of time • Interoperability (IOT): IOT tests include is an excellent tool for catching bugs to run, but if a bug is introduced, certain both automated and manual tests for early. A report emailed to the tests may take a longer or shorter time Bluetooth and Wi-Fi interoperability. developers every day can show what has to execute). These scripts automatically • Telephony: Wind River offers auto- changed in the build since the last time create an .ini file containing the best mated tests for its telephony stacks for tests to run. Moblin. it was tested and what bugs were • BSPs: Tests include IOzone for board introduced, while recent changes to the Wind River Test Library support packages (BSPs). software are still fresh in the developers’ • Stress/stability/soak: FAST provides an minds. Because the changes are recent, Wind River has a large suite of tests easy mechanism for combining multiple it’s much easier to identify the cause of created for our mobile platforms, which tests that stress the system under the bugs rather than getting feedback are run regularly. A dedicated team unusual loads. Most commonly, FAST weeks or even months later on hundreds continues to add more tests to this accomplishes stress tests with LTP, but of changes made to the software. This library, so it is constantly expanding. any automated test can be used into frequent, even daily, feedback makes it Tests are grouped into the following this kind of test scenario. much faster to identify a change that categories, with some overlap: • UI: Automated test suites for Wind River Platform for Moblin UIs are created us- introduced a bug. • Requirements: Wind River has defined ing the System Testing Plug-in frame- more than 1,000 requirements across By storing performance metrics and work. System Testing Plug-in tests are our mobile and automotive commercial running tests regularly, design managers run by FAST in the same manner as any platforms. All requirements are tested can see the impact of changes to the with a combination of manual and auto- other test framework. performance of the system. They can see mated tests developed by Wind River. whether performance dropped off Additional tests can easily be added to radically after a certain check-in or cover customer-specific requirements. 4 | Wind River Framework for Automated Software Testing (FAST)
  5. 5. Figure 5: FAST test report window Reporting (e.g., showing the last 30 test runs and highlighting the current test run). This Generating test reports from FAST is allows developers and managers to see easy. Report generation can be whether a change in the software on a controlled from the FAST GUI or can be certain date caused a degradation (or executed from the CLI. improvement) of a particular perfor- There are four different types of reports mance metric. See Appendix Figure 5 available in the standard FAST system: for a sample report. • Requirements tracking: The require- • Final: Final reports contain a list of all ments tracking report shows all the the test cases executed and their sta- requirements in the FAST database, the tus. See Appendix Figures 1 and 2 for last time they were tested, and the last samples. test result. This is primarily to help QA • Comparative: Comparative reports managers decide whether to ship a re- compare two test runs against each lease because they can quickly see the other to show what has changed status of all the requirements testing. between the two. This report sum- See Appendix Figure 6 for a sample marizes the pass/fail rates of each and report. shows tests with different results, tests that are new, and tests that have been Adding new reports is relatively easy. removed from the test run. This report Since all the information is stored in a is especially useful as a nightly report MySQL database, you can use your for designers because it shows only preferred reporting engine to generate what has changed since the last time a report or engage Wind River to create the software was tested. See Appendix new reports that meet your needs. FAST Figures 3 and 4 for samples. uses the open source JasperReports to • Performance: Performance reports create and generate the reports as part take all of the performance metrics of the standard delivered product. and display them as graphs over time 5 | Wind River Framework for Automated Software Testing (FAST)
  6. 6. Appendix The following images are samples of the types of reports that can be generated with the FAST system. In addition to these reports, others can be created. Appendix Figure 1: Sample final report—summary Appendix Figure 2: Sample final report—detail 6 | Wind River Framework for Automated Software Testing (FAST)
  7. 7. Appendix Figure 3: Sample comparative report—summary New Test Cases Pass/Fail Detail Removed Test Cases Appendix Figure 4: Sample comparative report—new test cases, pass/fail, and removed test cases 7 | Wind River Framework for Automated Software Testing (FAST)
  8. 8. Appendix Figure 5: FAST performance report 8 | Wind River Framework for Automated Software Testing (FAST)
  9. 9. Appendix Figure 6: Sample requirements tracking report Wind River is a world leader in embedded and mobile software. We enable companies to develop, run, and manage device software faster, better, at lower cost, and more reliably. © 2010 Wind River Systems, Inc. The Wind River logo is a trademark of Wind River Systems, Inc., and Wind River and VxWorks are registered trademarks of Wind River Systems, Inc. Other marks used herein are the property of their respective owners. For more information, see Rev. 03/2010