Robot simulators, like the MORSE project, provide a safe and readily available environment for robot system testing, reducing the effort for testing drastically. In principle, simulation testing is automatable, and thus a good target for Continuous Integration (CI) testing. However, so far, high-level scenario tests still require complex component setup and configuration before they can be run in the simulator. An added complication is, that there is no standard for starting, configuring, or monitoring software components on todays robots. Often, high-level tests are carried out manually, implementing a tailored solution, e.g, via shell scripts or launch files, for a specic system setup. Besides the eort of manual execution and supervision, current tests mostly do not take timing and orchestration, i.e., required process start-up sequence, into account. Furthermore, successful execution of components is not verfied, which might lead to subsequent errors during the execution chain. Most importantly, all this knowledge about the test and its environment is implicit, often hidden in the actual implementation of the tailored test suite. To overcome these issues, this contribution introduces a generic and congfiurable state-machine based process to automate a) environment setup, b) system bootstrapping, c) system tests, d) result assessment, and e) exit and clean-up strategy.

1. 1

2. State Machine Based Testing
Florian Lier, Norman Köster, Ingo Lütkebohle, and Sven Wachsmuth
2

3. 3 talks mentioned CI already ;)
3

4. (my) Motivation
Besides working in other research projects as a PhD student...
4

5. (my) Motivation
...my motivation is deeply grounded in the...
5

6. ROBOCUP
6

7. Source: http://www.youtube.com/watch?v=YpjeNa8BAYg
7

8. (my) Motivation
Robots operating in domestic environments
... rich interaction with humans
... object detection and recognition
... cluttered “real world” environment
... speech recognition and synthesis
... autonomous navigation and localization
... competitive event, which is “thrilling”
8

9. (my) Motivation
Source: http://www.cit-ec.de/sites/www.cit-ec.de/files/TDP_ToBI_2011_0.pdf
X

10. (my) Motivation
The team “ToBI” consists of
... 6-10 BA/MA students, they earn CPs!
... 1-3 PhDs, Postdocs supervision
... new students, every year...
... constantly improving/evolving system
... hackathons, during regular term
... “agile” on-site all around the world
9

11. Coding at actual event
10

12. Which is quite stressful, sometimes
11

13. Agile methods are applied
X

14. Problem I
Approximately 10 developers but
... only 1 Robot to test with
... > 10 source code repos to “watch”
... frequent changes everywhere
... local file system vs. robot file system
12

15. Problem I
“The task starts in 5 minutes”
13

16. Problem I
“Oh wait, I didn’t commit yet...”
14

17. Problem I
“Ok, it’s checked-in send the robot to the arena!”
15

18. Problem I
“Waaaait, I just realized I broke the task!!!”
(RoboCup classic)
16

19. Solution I
17

20. Solution I
Yes! A Continuous Integration Server
... monitors all build statuses (red, green)
... checks robot vs. local file system sync
... delivers instant feedback
... automated build process, static checks
... de-stresses situation, best-practice, etc
18

21. Problem II
“I tested the task in the area everything ..., but the robot doesn’t move!”
19

22. Solution II
Yes! Simulation for everyone!
... but,
... > 15 components to start, manually
... operate the simulator, manually
... based on subjective measurements
... first test prob. != second test
... timing, sequence, operator, machine
... you name it
20

23. “Simulation?”
21

24. Proposal
Yes! Combine CI and simulation testing
... serialize building an testing automatically
... start the simulation and all required components
... deliver feedback and results
... frequent checks (per check-in)
... works also outside RoboCup ;)
22

25. Approach
Create a state-machine for simulation tests!
23

26. Goals of SM based testing
No “tailored” solution for a specific system
Minimize the effort, let the machine do the work
Enable developers to design a well-structured test
Explicit environment setup and configuration
Transparent orchestration, repeatable and comparable
Verify system start-up and tear-down
24

27. 25

28. 26

29. 27

30. Benefits
Well-structured, explicit definition of env. and comp.
Fully automatable (CI enabled and CLOUD enabled)
Repeatable, exchangeable, composable and comparable
Execution checks, early exit (error states)
Timing and sequence is taken into account
Language, tool, and framework independent (!= roslaunch)
(OS independent, not tested on windows yet)
Distribution (STRANDS?)
28

31. Benefits // MORSE
Morse setup and execution
Fully automatable (CI enabled and CLOUD enabled)
Repeatable, exchangeable, composable and comparable
Execution checks, early exit (error states)
Timing and sequence is taken into account
Language, tool, and framework independent (!= roslaunch)
(OS independent, not tested on windows yet)
X

32. Morse Demo: ATRV + ROS, rosbag, rostopic, rxplot
29

33. 30Source: https://vimeo.com/67803733

34. Thank you!
31

35. Discussion Topics
Useful?
Problems: SCXML is pain!
MORSE integration
Metrics
UnitT integration
32

36. 33