Often, high-level (functional) tests are carried out manually, implementing a tailored solution, e.g, via shell scripts or launch files, for a specific system setup. Besides the effort of manual execution and supervision, current tests mostly do not take timing and orchestration, i.e., required process start-up sequence, into account. Furthermore, successful execution of components is not verified, which might lead to subsequent errors during the execution chain. Most importantly, all this knowledge about the test and its environment is implicit, often hidden in the actual implementation of the tailored test suite. To overcome these issues, this contribution introduces a generic and configurable state-machine based process.

2. From System Modeling to
Automated System Testing
An overview of the CITK Toolchain
BOSCH — February 12th 2015. Stuttgart, Germany | Florian Lier [flier@techfak.uni-bielefeld.de]

3. Systems in Robotics | Current State
Research on autonomous robots achieved considerable progress over the
recent years.
Reported results are typically validated through experimental evaluation or
demonstrated live at robotics competitions. [ DARPA Challenge, RoboCup, …]
Amongst other reasons, this is due to: „off-the-shelf“ robots [ iCub, NAO, … ]
and vivid open source communities providing reusable building blocks, e.g.
ROS, Orocos, OPRoS, Yarp …
Publicly available data sets are used to improve benchmarking procedures,
i.e., Rawseeds Project.
3

4. Systems in Robotics | Software Foundation
4
!
Most current robotics systems are realized by implementing a component-based architecture
These components do not necessarily share the same ecosystem, build, deploy and
testing environment or integration model …

5. AR Toolkit

6. Systems in Robotics | Software Foundation

7. Systems in Robotics | Problem Statement
System artifacts are often distributed over different locations, thus already the
discovery, identification and aggregation of all required artifacts is difficult.
Which specific versions (master or v0.1) of software components were in use
for a particular system or ideally system version?
How can automated integration testing, evaluation and preservation of
results of these heterogeneous components be achieved?
7

8. Systems in Robotics | Aspect Overview
8
Before you can test

9. Systems in Experimental Robotics | Goal
To tackle these issues we introduced an approach based on an
integrated software toolchain for system developers (testers)
and experiment designers.
9

10. Systems in Experimental Robotics | System Model
10
Test!

11. n SCM’s
dev dev
iCub-nightly
auto-gen
auto deploy
/somewhere/$USER/
Recipes: Java, C/C++, Python, Experiments System Composite
Run Software & Experiment in Lab
direct use
..or in office/home
Local & Global CI
11
sync
Web-based catalog (Systems and related artifacts)
Systems in Robotics | Tool Chain

12. n SCM’s
dev dev
iCub-nightly
auto-gen
auto deploy
/somewhere/
Recipes: Java, C/C++, Python System Composite
Lab
direct use
Local & Global CI
12
Systems in Robotics | Technical Realization Recipes
..or in office/home
sync
Web-based catalog (Systems and related artifacts)

13. 13
Systems in Robotics | Catalog

14. 14
Systems in Robotics | Catalog

15. Local & Global CI
15
Systems in Robotics | Bootstrapping
download pre-packaged CI Server

16. Local & Global CI
16
Systems in Robotics | Bootstrapping
extract & simply start it

17. 17
Systems in Robotics | Jenkins
find ready to use local (or global) CI Server

18. 18
clone distribution/recipe repository
Systems in Robotics | Job Generator

19. 19
invoke job-configurator with desired distribution including tests
Systems in Robotics | Job Generator

20. 20
Systems in Robotics | Job Generator

21. Systems in Robotics | Testing
21
!
What are these “tests” actually?

22. Systems in Robotics | Test Framework Goals
22
Not a “tailored” solution for a specific system or even language (component-
based architecture may even prevent this in the first place)
Minimize the effort, let the machine do the work (CI) and enable developers
to design well-structured functional tests
Explicit environment setup and configuration, transparent orchestration,
repeatable and comparable tests
Serialize building and testing automatically
Enable and verify system start-up and tear-down

23. 23
SM
$ENVIONMENT_VARS_DEF
$COMPONENT_A(CHECK_CRITERIA)
$COMPONENT_B(CHECK_CRITERIA)
$COMPONENT_N(CHECK_CRITERIA)
DM
EXEC
RUN
CLEAN
EVAL
CLEAN / REPORT
exec(COMPONENT_A): if "OK" B
exec(COMPONENT_B): if "OK" C
exec(COMPONENT_C): if "OK" D
exec(COMPONENT_Y): if "OK" Z
inject()
exec(COMPONENT_Z): if "OK" E
EXIT
ERROR
export ROS_ROOT=/opt/ros/indigo
export DISPLAY=:0
rosbag record -j /some/topic
java -jar slow_in_java.jar
sh overly_complicated.sh
must start after 5 seconds
monitor PID continuously
must not print “segfaulting()”
must print “conn. established”
lockfile.file must be present
Component Success Criteria

24. 24
Systems in Robotics | Test State Machine Concept
SM
$ENVIONMENT_VARS_DEF
$COMPONENT_A(CHECK_CRITERIA)
$COMPONENT_B(CHECK_CRITERIA)
$COMPONENT_N(CHECK_CRITERIA)
DM
EXEC
RUN
CLEAN
EVAL
CLEAN / REPORT
exec(COMPONENT_A): if "OK" B
exec(COMPONENT_B): if "OK" C
exec(COMPONENT_C): if "OK" D
exec(COMPONENT_Y): if "OK" Z
inject()
exec(COMPONENT_Z): if "OK" E
EXIT
ERROR
stop processes in RUN.invert()
e.g., plot.sh < $FSMT_RUN/logs/*
e.g., read_and_clean.logs
stop processes in EVAL.invert()

25. 25
SM
$ENVIONMENT_VARS_DEF
$COMPONENT_A(CHECK_CRITERIA)
$COMPONENT_B(CHECK_CRITERIA)
$COMPONENT_N(CHECK_CRITERIA)
DM
EXEC
RUN
CLEAN
EVAL
CLEAN / REPORT
exec(COMPONENT_A): if "OK" B
exec(COMPONENT_B): if "OK" C
exec(COMPONENT_C): if "OK" D
exec(COMPONENT_Y): if "OK" Z
inject()
exec(COMPONENT_Z): if "OK" E
EXIT
ERROR
Close all component loggers
Report exit status sys.exit(n)
Create zip archive
Create xUnit output
Systems in Robotics | Test State Machine Concept

26. 26
Systems in Robotics | Test State Machine Implementation
Framework called FSMT (Finite State Machine Based Testing)*
Set of tools written in Python
Implementation of executable “test” state machines including all features as defined in the concept
Simple configuration using *.ini files
* http://opensource.cit-ec.de/projects/fsmt/

27. 27
Systems in Robotics | Test State Machine Implementation

28. 28
Systems in Robotics | Test State Machine Implementation

29. 29
Systems Robotics | FSMT Invocation
fsmt_iniparser my_test.ini -o my_test.scxml

30. 30
Systems Robotics | DEMO
DEMO
https://vimeo.com/112005754

31. 31
Systems Robotics | Provided Benefits
Not a “tailored” solution for a specific system or even language (component-
based architecture may even prevent this in the first place)
Minimize the effort, let the machine do the work (CI) and enable developers
to design well-structured functional tests
Explicit environment setup and configuration, transparent orchestration,
repeatable and comparable tests
Serialize building and testing automatically
Enable and verify system start-up and tear-down

32. 32
Systems Robotics | Thank you!
Links
Web Catalog: https://toolkit.cit-ec.de
Distribution Project: https://opensource.cit-ec.de/citk
Video Material: http://vimeo.com/groups/citk
http://opensource.cit-ec.de/projects/fsmt/
Affiliation
Florian Lier is with Cognitive Interaction Technology — Center of Excellence, Bielefeld University
Supervisor: Dr.-Ing. habil Sven Wachsmuth

33. 33
Systems Robotics | Literature
Publications (2014)
Modeling Software Systems in Experimental Robotics for Improved Reproducibility -- A Case Study with the iCub Humanoid Robot
http://pub.uni-bielefeld.de/publication/2705677
The Cognitive Interaction Toolkit – Improving Reproducibility of Robotic Systems Experiments
http://pub.uni-bielefeld.de/publication/2702526
Towards Automated Execution and Evaluation of Simulated Prototype HRI Experiments
http://pub.uni-bielefeld.de/publication/2645922
Simulation and HRI Recent Perspectives with the MORSE Simulator
http://pub.uni-bielefeld.de/publication/2685211

34. 34
https://vimeo.com/112005754
Systems in Robotics | Static Demo