Wind River Framework for Automated Software Testing (FAST)
Wind River Framework for Automated
Software Testing (FAST)
Wind River Framework for Automated complicate the test and validation developer or quality assurance (QA)
Software Testing (FAST) was developed process. To properly validate the engineer needs to adapt tests to run on
to help customers overcome the distribution, tests for the following are each device. Each target may have a
challenges of testing large software needed: unique hardware configuration,
distributions for mobile devices. FAST • Requirements peripherals, file storage systems, and so
provides a system for combining • Performance on. Tests that were developed for one
multiple test frameworks into a single • Stress and stability device frequently need to be adapted
meta-framework that can be run with “a • Device driver to work on different devices.
single button press.” It helps automate, • Middleware
User interfaces (UI) that come with
execute, and manage thousands of tests • Application
different devices are often changing.
and their results. • User interface
• Multidevice Even if the functionality behind the UI is
Wind River leverages as many open • Peripheral the same, the developer or QA
source and existing test frameworks as • Board support package (BSP) engineer must adapt tests to the
possible so testing is comprehensive, • Version checking different layouts and widget styles.
without sacrificing time-to-market. • Telephony
With FAST, each individual test
Many open source test frameworks • Network
framework or application is encapsulated
contain thousands of tests and many
Each of these types of tests needs to be within a “wrapper” script executed by
packages that come with their own test
performed as often as possible on each the FAST system. The results are all
suites. FAST makes executing tests
new release, and the results of these stored in a consistent manner in a
much simpler because they are all
tests need to be reported and managed. MySQL database. This provides a flexible
wrapped in FAST scripts and run from
and comprehensive system for handling
the FAST graphical user interface (GUI) Because test programs are pulled from
large amounts of test results from
and an available command-line so many different sources, and they all
heterogeneous test frameworks, and
interface (CLI). have diverse ways of being run and of
vastly simplifies the problem of wading
generating and reporting results,
The Challenge of Testing through and comparing the various
executing the tests and tracking and
individual test reports.
Testing very large open source software evaluating the test results can especially
distributions for mobile devices is a complicate the testing process. The The database contains all test artifacts
tremendous challenge. There are amount of information can be and logs from the original test programs
typically dozens of subsystems staggering, particularly when test cases and additional metrics collected by
containing thousands of packages have different reporting formats. FAST. FAST can be thought of as a
included in a distribution, with many meta-test framework that can run other
As new original device manufacturers
applications built using those packages. test frameworks and individual tests.
(ODMs) develop hardware and wish to
Differences in device types, peripherals have their products tested with a From the FAST database, Wind River
used, and user interface variations certain distribution of the software, the can create a variety of test reports.
Customizable formal reports (for QA
managers), test run comparison reports
Table of Contents Test Execution .................................... 3
(for developers), and interactive
Wind River Test Library ...................... 4
regression analysis reports (for
The Challenge of Testing.................... 1 Reporting ............................................ 5 engineering managers) are all provided.
The FAST Solution .............................. 2 Appendix ............................................ 6 Additional reports can easily be created
System Testing Plug-In ....................... 2 using the reporting tools of your choice.
The FAST Solution ods that perform common operations Results from other test tracking
that are repeated in many tests (i.e., set- software (such as manual test tracking
FAST offers significant advantages to
ting up network interfaces). When a new software) can be imported into the
both developers and QA engineers:
target is introduced, Wind River only FAST database so FAST can track the
• Integrates testing efforts in one appli- needs to modify the board class. Only test effort of the entire test
cation: Once the different test frame- the communication and interface meth-
works and applications are “wrapped” ods are rewritten. The actual test scripts
by a FAST script, they are accessible and wrappers need not change and can System Testing Plug-In
from the testing GUI. Users may select be used to test both target types. Once
Wind River System Testing Plug-in is an
which tests to execute, and with one a script is written, it most likely will work
button execute them all. There is no on multiple targets. Occasionally minor application test mechanism that can
need to know how the test applications changes are needed if the functional- simulate user interactions with user
work or what report formats they use; ity of an application/package changes interfaces (UIs). Along with FAST, Wind
everything is handled by the scripts. from one target to another; but for the River developed this technology to test
• Integrates other testing frameworks: most part, Wind River test libraries do large software distributions. System
Any existing test framework or test not require many changes. Testing Plug-in enables the automated
application can be integrated into • Leverages advanced reporting ca- testing of UIs and the user experience.
FAST using a FAST script. FAST uses pabilities: Because all the results and This kind of testing has traditionally
advanced parsing routines to decode test artifacts go into one database, it been manual because by its very nature
the test results from any log file or is relatively easy to generate reports. it involves human interaction with the
program output. No matter where the These reports can serve the following system UI and interaction among
tests come from and how they present functions: multiple applications on the device or
test results, FAST scripts can integrate - Provide customers with a list of tests even multiple devices.
them into the system. performed and their results.
- Compare the current test run to a System Testing Plug-in is a unique,
• Creates custom test scripts: Often
previous test run. This is useful to the powerful, and exciting technology that
there are no open source or existing
test assets available. Wind River or the software design team because they makes UI elements available for
customer can create custom scripts can see what new bugs were intro- interrogation and manipulation from
in FAST that can execute commands duced and what bugs were fixed since automated test scripts. It can easily
on both a test server and targets and the last time tests were run. repurpose existing tests for use with a
analyze their output. With this capabil- - Illustrate performance results
new UI. For example, a calendar
ity, a whole suite of tests that did not graphically so that managers can
supports the same features across
previously exist can be created. quickly see if identified metrics are
implementations, such as entering an
degrading or improving over time.
• Applies identical tests to different - Track requirements, last test dates, appointment, listing today’s
targets with minimal changes: FAST and results. This makes it easy to see appointments, setting an appointment
is structured so that test scripts/wrap- where more tests need to be added alarm, and so on. With System Testing
pers never directly communicate with and which requirements need to be Plug-in, tests for these common
the targets. They always go through a debugged. scenarios need only be written once for
“board” class that abstracts the test - Summarize test history to show the one UI and can be easily ported to a
framework from the specific require- results and artifacts of a test over the different UI by using simple abstraction
ments of a board. The board class can past several runs. This is useful for layer code.
send commands to the target, read the developers to debug issues that have
output, install a payload, and reboot recently been introduced in the System Testing Plug-in also supports
the target. There are also special meth- software. parallelizing tests, such as testing
across multiple targets and automating
use case and scenario-based testing.
FAST System Testing Plug-in offers significant
Test Script advantages for application testing. It is
able to simulate almost any user
interaction with the target, including
clicking widgets in the UI, dragging
Board Class Board Class graphical elements, reading the
“Target Type A” “Target Type B”
contents of the UI elements, typing
entries on a physical or virtual keyboard,
and so forth.
System Testing Plug-in scripts are
Type A Type B
integrated into FAST. FAST has a
Figure 1: FAST test script configuration for multiple targets using board class standard wrapper that can be applied to
2 | Wind River Framework for Automated Software Testing (FAST)
Configuring a target for testing can be
easy and straightforward. FAST uses a
board class to specify communication
and control for the target. With a board
class and network connection to the
Mapping for Mapping for target, the tester is ready to begin testing.
Target A Target B
The FAST user interface offers significant
flexibility to automate the testing process:
Target A Target B • Software may be loaded onto the
Application Application target before testing begins, and these
Figure 2: System Testing Plug-in script GUI mapping
uploaded files may be removed once
testing is completed.
any System Testing Plug-in test script.
This allows these scripts to be executed,
and their results can be tracked in the
FAST database and reports. System
Testing Plug-in scripts do not rely on the
position of UI elements in an application.
Instead, System Testing Plug-in uses
the accessibility features of widget
frameworks to know the names of the
widgets to interact with. The widget
creates a map of the buttons so the test
scripts can refer to the buttons by name
rather than screen location. Most GUI
testing applications specify actions such
as “Click at position 10,10.” A statement
by System Testing Plug-in would be more
like “Click the button mapped as ‘OK.’”
This means that no matter how the
application changes its layout, the
System Testing Plug-in script can still
execute the test. The same System
Figure 3: FAST user interface
Testing Plug-in script works on multiple
implementations of the same programs
(i.e., calculators, calendars, dialers,
browsers) with different interfaces.
However, the same script can be
executed on all of them, with only minor
changes to the button name maps.
System Testing Plug-in scripts can be
combined into one. This enables the
testing of multiple applications running
at the same time, for example, when a
user is listening to music and reading
email and a call comes in. Interleaving
existing test scripts allows Wind River to
easily create new test scenarios.
Figure 4: FAST test lists
3 | Wind River Framework for Automated Software Testing (FAST)
• Special arguments may be executed whether performance is gradually • Compliance: FAST includes the latest
on the command line for boards that dropping or improving over time. This version of the compliance suites for
require them. prevents surprises at the end of a release Android Compatibility Test Suite (CTS),
• The target may be configured to power testing cycle. Moblin (OSV, ISV, and Application
down or reboot during testing or be- Checker), and GENIVI.
tween tests. In the web GUI, the user can specify the • System framework: There is a complete
• Baseline results may be identified for test environment and select the desired set of system framework tests specifi-
comparing test results in the test report. tests and, instead of running them, save cally for Wind River Platform for Moblin.
an .ini file that contains all tests planned • Test RPMs: Many of the open source
The web-based graphical user interface
for execution. The .ini file may be packages used in Mobile Linux include
makes test selection and execution a their own tests. These are usually dif-
loaded into the system at a later date
quick process. The user navigates the ficult and procedurally cumbersome to
with settings specific to that file to pass
test tree at the left and selects run frequently. The Wind River Linux
it to FAST for execution. This executes
individual tests for running by right- Distribution Assembly Tool (LDAT) build
the tests in the same fashion as if they
clicking them. Each test may then be system for Mobile Linux extracts these
were executed from the GUI.
configured, including default time-out, into test RPMs that can be executed by
count, parameters, and reboot options The command-line interface also has the FAST, providing an ongoing regression
(reboot before, after, or after when the capability to stop a test run, get its status, test of these components, thus making
and download its results and artifacts. it much easier to detect failures caused
test does not pass). Once the test runs
by changes in other areas of the system.
are configured, each test’s progress is FAST also offers scripts that can • Packages: Wind River has created tests
displayed and a test’s details are automatically analyze the FAST to assure that packages that are part
available, including the start and end database to find the best tests to of the Mobile Linux system that do not
dates, any test cases it ran, their status execute, based on the time passed since have their own tests are installed and
(passed/failed), and any artifacts, such their last execution and their last operational.
as log files, or key values (performance execution status (giving higher priority • Open source: Linux Test Project (LTP),
metrics) that are associated with the test to those tests that have not run recently IOzone, iPerf, and Moblin Compliance
case. Tool Suite are all integrated. Addi-
and those that last failed). Using these
tional third-party test frameworks are
FAST supports a command-line scripts, FAST can run tests for a specific
straightforward to add.
interface that allows execution of tests amount of time (e.g., eight hours). The
• Performance: These include automated
from the Linux command line. This is system will find the best tests to run that tests for graphics, networking, and
very useful for running tests regularly will approximately fill the time (tests boot-time performance.
from a cron job. Running tests regularly typically take the same amount of time • Interoperability (IOT): IOT tests include
is an excellent tool for catching bugs to run, but if a bug is introduced, certain both automated and manual tests for
early. A report emailed to the tests may take a longer or shorter time Bluetooth and Wi-Fi interoperability.
developers every day can show what has to execute). These scripts automatically • Telephony: Wind River offers auto-
changed in the build since the last time create an .ini file containing the best mated tests for its telephony stacks for
tests to run. Moblin.
it was tested and what bugs were
• BSPs: Tests include IOzone for board
introduced, while recent changes to the
Wind River Test Library support packages (BSPs).
software are still fresh in the developers’ • Stress/stability/soak: FAST provides an
minds. Because the changes are recent, Wind River has a large suite of tests
easy mechanism for combining multiple
it’s much easier to identify the cause of created for our mobile platforms, which
tests that stress the system under
the bugs rather than getting feedback are run regularly. A dedicated team
unusual loads. Most commonly, FAST
weeks or even months later on hundreds continues to add more tests to this accomplishes stress tests with LTP, but
of changes made to the software. This library, so it is constantly expanding. any automated test can be used into
frequent, even daily, feedback makes it Tests are grouped into the following this kind of test scenario.
much faster to identify a change that categories, with some overlap: • UI: Automated test suites for Wind River
Platform for Moblin UIs are created us-
introduced a bug. • Requirements: Wind River has defined
ing the System Testing Plug-in frame-
more than 1,000 requirements across
By storing performance metrics and work. System Testing Plug-in tests are
our mobile and automotive commercial
running tests regularly, design managers run by FAST in the same manner as any
platforms. All requirements are tested
can see the impact of changes to the with a combination of manual and auto- other test framework.
performance of the system. They can see mated tests developed by Wind River.
whether performance dropped off Additional tests can easily be added to
radically after a certain check-in or cover customer-specific requirements.
4 | Wind River Framework for Automated Software Testing (FAST)
Figure 5: FAST test report window
Reporting (e.g., showing the last 30 test runs and
highlighting the current test run). This
Generating test reports from FAST is
allows developers and managers to see
easy. Report generation can be
whether a change in the software on a
controlled from the FAST GUI or can be certain date caused a degradation (or
executed from the CLI. improvement) of a particular perfor-
There are four different types of reports mance metric. See Appendix Figure 5
available in the standard FAST system: for a sample report.
• Requirements tracking: The require-
• Final: Final reports contain a list of all ments tracking report shows all the
the test cases executed and their sta- requirements in the FAST database, the
tus. See Appendix Figures 1 and 2 for last time they were tested, and the last
samples. test result. This is primarily to help QA
• Comparative: Comparative reports managers decide whether to ship a re-
compare two test runs against each lease because they can quickly see the
other to show what has changed status of all the requirements testing.
between the two. This report sum- See Appendix Figure 6 for a sample
marizes the pass/fail rates of each and report.
shows tests with different results, tests
that are new, and tests that have been Adding new reports is relatively easy.
removed from the test run. This report Since all the information is stored in a
is especially useful as a nightly report MySQL database, you can use your
for designers because it shows only preferred reporting engine to generate
what has changed since the last time a report or engage Wind River to create
the software was tested. See Appendix new reports that meet your needs. FAST
Figures 3 and 4 for samples.
uses the open source JasperReports to
• Performance: Performance reports
create and generate the reports as part
take all of the performance metrics
of the standard delivered product.
and display them as graphs over time
5 | Wind River Framework for Automated Software Testing (FAST)
The following images are samples of the types of reports that can be generated with the FAST system. In addition to these
reports, others can be created.
Appendix Figure 1: Sample final report—summary
Appendix Figure 2: Sample final report—detail
6 | Wind River Framework for Automated Software Testing (FAST)
Appendix Figure 3: Sample comparative report—summary
New Test Cases
Removed Test Cases
Appendix Figure 4: Sample comparative report—new test cases, pass/fail, and
removed test cases
7 | Wind River Framework for Automated Software Testing (FAST)
Appendix Figure 5: FAST performance report
8 | Wind River Framework for Automated Software Testing (FAST)