This document discusses research into automatic test case generation for train control systems. It describes a tool called CompleteTest that uses model checking to generate test cases from function block diagram programs that satisfy various logic coverage criteria. The tool was evaluated in a case study with Bombardier Transportation where it generated tests for some programs, but failed to terminate within 10 minutes for larger programs. Ongoing work involves addressing state space explosions, complementing model checking with other techniques, and measuring test effectiveness at finding faults.

1. Automatic
Test Case Generation
for Train Control Systems
Adnan Causevic
TOCSYC Industrial Workshop, 2014-11-13, Västerås

2. 2
Today’s talk is about…

3. ● Software Testing Laboratory @ MDH
● Research:
● Test Automation
● Quality of Tests
● Developers Testing
● AGENTS - KKS funded project
● Bombardier Transportation
● Maximatecc
● Involved in the TOCSYC
research environment
● Intelligent Embedded Systems
Master Program Coordinator
3
About me

4. ● ATAC research project
● 2011-2014
● Funded by ITEA2/Vinnova
● Advanced Test Automation for Complex Software-
Intensive System
4
Cooperation with Bombardier
“The project has brought test automation
on the agenda of European industry.”
- ITEA3 Final Review Report

5. ● Train Control Management System (TCMS)
● a high capacity, infrastructure backbone
● the center of the distributed system
● Functions controlled by TCMS include
● collecting line voltage,
● controlling the train engines,
● opening and closing the train doors,
● upload of diagnostic data.
5
About Bombardier

6. ● Vehicle
● Vehicle Integration testing
● System
● System Integration testing
● Software
● Modules integration testing
● Modules testing
● “Get to the driver’s perspective as soon as possible”
6
About Bombardier
… and their testing
Responsibility
of a Developer!

7. ● IEC 61131-3
● Function Block Diagram
● Graphical programming language
● Based on element composition
● Various predefined elements
7
Development Environment

8. ● Coverage
● Strict measurements mandated by safety standards
● Tooling
● No tool support to measure coverage on FBD
● Time
● Creating tests just to reach the coverage leaves very little
time for functional testing
8
Current challenges
… with module testing

9. 9
CompleteTest

10. 10
Research approach
FBD Program
Transformation 1 Annotation2
UPPAAL timed
automata
...
Reachability
properties
∃
♢
β
,
∃
♢
pi
,
∃
♢
c0
and
c1
Test traces Test cases
(Step) (1)
(Time) (20s)
(Inputs) (0 1 23)
(Outputs) (1 2 543,9)
UPPAAL
model checker
Test Generation
3
Logic-­‐‑based
coverage criteria
DC, CC,
MC/DC
”Automated Test Generation using Model-Checking: An Industrial Evaluation”, Eduard Paul Enoiu, Adnan
Causevic, Thomas J. Ostrand, Elaine J. Weyuker, Daniel Sundmark, Paul Pettersson. International Journal
on Software Tools for Technology Transfer, 2014, Springer.

11. ● Time required to generate tests
● satisfying the DC, CC and MC/DC logic coverage criteria
11
Case Study at BT

12. ● For 34 of the 157 programs, the tool did not terminate
after running for a substantial period of time.
● Cut-off time was set to 10 minutes.
● There is an indication that as the number of decisions
increases, the performance deteriorates and the cost of
using the tool may become prohibitive.
● But, if you have so many decisions, how would you test
it manually?
12
Case Study at BT (2)

13. ● Should a developer generate the tests from scratch?
● Should a developer provide manual tests first?
● How many (or how much in terms of coverage)?
● When should a developer use the Validate Test button?
● Should developers use this tool at all or should we
automate that as well (nightly build, etc.) ?
13
Best Practices?

14. ● Internal behavior of each function block has to be
modeled
● Currently, blocks from the standard language are modeled
● and a few custom made by Bombardier which are often used
● State-space explosion
● a known problem of model-checking
● large number of Boolean or Integer input values
● but, again, how would you do it manually ?
14
Limitations

15. ● Complementing model-checking with:
● Static analysis of FBD’s
● Search-based software testing
● Trying some new approaches:
● Different model-checkers, SMT solvers, …
● Measuring effectiveness of generated tests
● In terms of how good they are in finding faults
● Experimenting with FBD programs from
other organizations
15
On-going and Future work

16. ● Adnan Causevic
adnan.causevic@mdh.se
● Eduard Enoiu
eduard.paul.enoiu@mdh.se
● www.completetest.org
16
Questions?