A Novel Approach to Automated Systems Engineering on a Multi-Agent Robotics Platform using Enterprise Conﬁguration Testing Software Stephen Cossell Engineer, ScriptRock Inc. firstname.lastname@example.org Abstract systems are being given more responsibility in society and as such must be robust and able to function with- This paper presents a case study of applying out fault. All modern industrial robotics projects are an enterprise grade systems conﬁguration man- more than likely to have extensive systems engineering agement platform to a set of unmanned ground processes tied into their development life-cycles. This vehicles and a ground control station. Much process typically involves a thorough set of unit and like large scale enterprise infrastructure, mod- integration tests to validate each software component ern robotics systems are comprised of many of a system, from a functional point of view. It is diﬀerent machines communicating over a va- however diﬃcult to ﬁnd evidence in published literature riety of media, let alone the large number of of development and testing practices that validate the modules and applications running on each ma- underlying environment and conﬁguration of software chine. Each module typically has its own con- modules themselves. ﬁguration settings, with each individual piece of conﬁguration information being crucial to the overall working state of the robotic sys- Software conﬁguration management has been a point tem. When one conﬁguration item is changed of interest for computer science researchers over the inadvertently, or otherwise, without an opera- last three decades. In the 1980s, research by [Bersoﬀ, tor’s knowledge, a manual and lengthy expedi- 1984] concentrated on identifying and understanding tion through a series of conﬁguration ﬁles and the problem and promoted the need for conﬁguration command output is usually used to diagnose management in large enterprise systems. By the the cause of the problem. This situation is ex- 1990s, researchers had begun proposing solutions to acerbated when the platform is used by a range particular niche problem areas. Work by [Hall et al., of people for diﬀerent scenarios on a regular ba- 1997] presented a possible solution to conﬁguration sis. The ScriptRock platform is used by large management of wireless local area networks. As part enterprise software and infrastructure teams to of his job maintaining university workstations, Mark encode system conﬁguration requirements into Burgess developed CFEngine[Burgess, 1995], a cross executable documentation so the underlying platform package for automated system conﬁguration environment their applications run on can be management. While CFEngine provides a useful tool validated immediately. The application of the for conﬁguration management, it’s potential usage is ScriptRock platform to a multi-agent robotic limited to developers, as conﬁguration promises must be system has shown improved re-conﬁguration encoded in scripts. This is in contrast to the ScriptRock times between diﬀerent use cases as well as a platform, which provides an interface that abstracts signiﬁcantly simpliﬁed troubleshooting and di- underlying code required to validate in individual agnosis process when the system is found not conﬁguration item[Sharp-Paul, 2012]. to be in a working state. In the last decade companies with large dynamic cloud infrastructures such as Netﬂix [Hoﬀ, 2010], have1 Introduction applied custom in-house solutions to test and correctAs more robotic systems are entering mainstream oper- conﬁguration state of their own systems and infrastruc-ation they are becoming more complex in order to be ture via deliberate and regular breaking of conﬁgurationable to complete more intricate tasks. In addition, these items. This implementation is in line with IBM’s
manifesto release in [Horn, 2001] requesting a push for 2.1 The Unmanned Ground Vehicle andthe widespread adoption of Autonomic Computing. Ground Control Station Platform In the last six years there has been a push in robotics This section gives a high level software module andresearch circles to adopt a common platform for software network layout outline for the Unmanned Groundmodules. Platforms such as aRD [Hirzinger and Bauml, Vehicle (UGV) and Ground Control Station (GCS)2006], OpenRDK [Quigley et al., 2009] and the Robot setup discussed in this paper. For a more detailedOperating System (ROS) [Calisi et al., 2008] attempt description of the platform as a whole, see [Whittyto ﬁnd this common standard, with ROS gaining much et al., 2010] and [Guivant et al., 2012]. For a descriptionadoption in the global robotics community. While these of speciﬁc individual software components, please referplatforms allow rapid prototyping and application of to [Robledo et al., 2010], [Guivant, 2008] and [Guivant,common robotics algorithms, they are still diﬃcult to 2012].conﬁgure correctly and ensure critical dependencies aresatisﬁed. Although this paper focusses on applyingthe ScriptRock platform to a custom, in-house robotic 2.2 The ScriptRock Platformsystem developed within the mechatronics researchgroup at UNSW, future work will attempt to apply At a high level the ScriptRock platform allows a persona ScriptRock test suite to common ROS modules. to submit system and conﬁguration requirements into anThe aim is to be able to validate the installation and online cloud based platform. Requirement informationinclusion of a ROS module by running a ScriptRock test is submitted using templates that abstract away thethat is distributed with the module. technical knowledge required to actually perform a requirement validation. For example, a requirement This paper uses a multi-node system ﬁrst presented might be to ensure that a particular program is locatedin [Whitty et al., 2010] and later from a more software in the correct location on a target machine so thatand networking perspective in [Guivant et al., 2012]. Al- the automated start-up scripts can ﬁnd it properly.though these papers describe multi-vehicle systems, a Although there are commands that can be run on anysimpliﬁed single unmanned ground vehicle (UGV) setup mainstream operating system (OS) to check that ais used in this paper in various common use case sce- particular ﬁle or application exists in a given folder,narios. As presented in Section 3, these scenarios range each command is diﬀerent on each OS and must befrom direct on board tele-operation to semi-autonomous manually encoded into a test script. The respectiveoperation, used commonly for research purposes. ScriptRock template used for the this type of validation only requires a full path to be entered with a human1.1 Outline readable description, with the actual testing process itself abstracted away.The application presented in this paper demonstratesa simple and eﬃcient method for deﬁning and manag- Once a set of requirements have been submitted toing system conﬁguration information as well as enabling the website they are then downloaded as a zip ﬁle con-fast troubleshooting when a conﬁguration item, among taining all the requirements translated into executablehundreds, has been misconﬁgured. Section 2 begins by tests. The zip ﬁle contains a shell or batch scriptgiving a high level background on both the UGV and that runs the complete set of tests, which generatesground control station (GCS) system as well as the high either plain text or an HTML report containing thelevel capabilities of the ScriptRock platform. Section 3 results of each test run. If a test fails, a remediationoutlines how the ScriptRock platform has been applied step or set of steps is presented to the tester so theyto the UGV and GCS system. Sections 4 and 5 respec- can correct the state of the system and thereforetively discuss the details and results of an experiment subsequently satisfy the test requirement. As softwareused to gauge the increased beneﬁt of using the Scrip- modules get updated, the online tests validating thesetRock platform on the UGV and GCS system over an modules can be modiﬁed and re-downloaded to reﬂectexisting manual method. the change in conﬁguration state. Since the UGVs and GCS are used in diﬀerent scenarios on a regular2 Platform Background basis their underlying conﬁgurations change often. A separate ScriptRock test project was set up for each ofThis section gives a high level overview of both the these scenarios, so that an particular test script can beUGV/GCS and ScriptRock platforms in terms of soft- used to validate the state of a particular desired scenario.ware and network system conﬁguration.
3 Method correct values.The UGV and GCS system is ﬂexible enough to be con-ﬁgured in a number of diﬀerent ways, but there are three The third major category of tests applicable to allmain scenarios that are used most often and are usu- scenarios are the requirements that folder structure,ally required at short notice. These scenarios are direct conﬁguration ﬁles for custom modules, as well as thetele-operation, remote tele-operation and remote point- modules themselves, existed and were located in theand-click semi-autonomous operation. Each scenario is correct locations. A number of “directory exists”outlined in the following sections, along with the mod- and “file exists” tests were used to validate theules and infrastructure required and an overview of the complete folder structure was constructed correctly.types of ScriptRock tests that have been applied to trackconﬁguration. It should be noted that although this paper focuses on validating that a system is in the correct state as it3.1 All Scenarios changes between use cases, these tests have also greatlyAlthough the modules and devices used in each of assisted in rolling out software and network settings tothe following scenarios diﬀer, there are a number of a fresh on board laptop. By running these tests on acommon conﬁguration items that apply to all scenarios. regular basis as modules were added, it was immedi-Each scenario relies on a set of dynamic link libraries ately apparent that a module or setting was missing(DLLs) as well as certain software interpreters to be without needing to actually start the software up andinstalled and be accessible in the operating system’s then manually detect the problem or missing component.search path. To check a speciﬁc DLL is installed andlocated in a known location, a “file exists” test was In each of the following sections an infrastructure dia-created within ScriptRock. Modules are started on each gram is used to assist in describing each scenario. Figurenode using a Python script. As a result, a test was 1 outlines each of the diagram components.created to check that Python was installed on the sys-tem and that the correct version of Python was installed. Even with the simplest scenario of direct tele-operation, modules are required to communicate dataover a network connection. Direct tele-operationrequires two ethernet connections on board whereasthe other scenarios require multiple hops via wired andwireless connections and over diﬀerent subnets. Asa result, a set of “ping success” tests were used tocheck that the require nodes could communicate withone another. The underlying network conﬁgurations ofthese nodes were also tested by using a combination of“command output” and “file contents” checks. Forexample, the routing table can be checked on Windowsby running the command “route print” and checkingthe output for an expected value. The same can bedone on Unix based platforms using the “route -n”command. Figure 1: Legend used for infrastructure diagrams. In the remote tele-operation and point-and-click au-tonomy scenarios, certain speciﬁc settings were requiredat boot on each of the wireless routers to make sure 3.2 Direct Tele-operationthe UGV’s mesh network settings took precedence over Direct tele-operation involved an operator using an Xboxthe system defaults. A custom script was placed in the 360 controller to drive the UGV for situations such asLinux start-up directory “/etc/init.d” to apply these Open Day demonstrations, where direct deterministicsettings. In addition to checking that the required set- control is required for OH&S purposes, as well as simplytings were actually set using custom “command output” moving the robot from one lab to another. The scenariotests running commands like “ifconfig”, an extensive requires only infrastructure on a single UGV and involvesset of “file contents” tests were also used to validate a minimal set of conﬁguration checks to be created, asthat the custom initialisation script itself contained outlined below.
Infrastructure Layout Conﬁguration ItemsThe infrastructure used in this scenario is contained In addition to the general conﬁguration items applica-within a single UGV. Figure 2 shows how data ﬂows ble to all scenarios, a number of “file contents” basedbetween hardware and software modules in this scenario. checks were created for critical conﬁguration ﬁles of mod- ules required to achieve tele-operation. The most impor- Tele-operated commands are generated by an Xbox tant checks for this scenario center around sensor inter-360 controller, which are received by a software module facing modules. Each of the three laser range ﬁnderson the UGV’s on board laptop. These commands are attached to the UGV communicates data either via apushed by the module into the centralised database wired ethernet connection or via a RS-422 serial connec-system. tion converted to USB. Each of these modules encodes the IP address and port number, or the baud rate and Laser range ﬁnder measurements from the front COM port in a conﬁguration ﬁle. Ensuring that theseand back proximity lasers are also pushed into their values are not only present, but also set correctly, is anrespective queues in the database. A laser proximity important step in validating whether the UGV is conﬁg-module then reads these queues and determines if an ured correctly.object is too close to allow motion in a particular 3.3 Remote Tele-operationdirection. This module pushes data to a queue in the Remote tele-operation is used predominantly for teach-database on a regular basis, but modiﬁes a speciﬁc ﬂag ing purposes during the later stages of undergraduatein each pushed record as to whether it can, or cannot robot design and autonomous systems courses. In addi-detect anything in its proximate range. tion to the on board modules used in the Direct Tele- operation scenario outlined in Section 3.2, this scenario The command arbiter module reads new tele- makes use of a number of machines that replicate dataoperation commands and laser proximity records from over both a wireless and wired network connection. Athis centralised database on a regular basis and allows detailed overview of the infrastructure is given below,the instructions to pass through to the DMC interfacing followed by the additional conﬁguration items requiredmodule if the proximity queues in the database have not for this scenario to run.ﬂagged any objects as being in the immediate vicinity ofthe UGV. The DMC module then takes the ﬁnal ﬁltered Infrastructure Layoutdriving commands from the database and communicates In addition to the modules used on board for laserthem to the on board DMC via the local ethernet con- proximity checks and arbitrating driving instructions tonection. the DMC interfacing module, this scenario relies on a number of network and centralised database replication settings, which are covered in great detail in [Guivant et al., 2012]. This infrastructure is shown in Figure 3. Data from the UGV is replicated to a teacher’s desktop computer over a wireless network connection, using a pair of mesh network routers. The environment the UGV operates in also makes use of a ﬁxed external laser range ﬁnder, which acts as a local positioning system, much like a GPS unit in outdoor environments. This laser data is replicated over an ethernet connection to the teacher’s desktop computer. Student machines are connected to the teacher’s machine via a local subnet over ethernet. Students run a local copy of the centralised database, which listens for broadcasted data from the teacher’s machine. In turn the student runs custom written software to read data from their local database, interpret the data and push driving instructions back into the local database.Figure 2: The software and networking component lay- This local database is then conﬁgured to replicateout used for direct tele-operation. driving instructions back to the teacher’s machine, which are then in turn replicated back to the UGV’s
database. Once on the UGV, these driving instructions students’ machines brought from outside the lab.are handled via the same tele-operation mechanism asin Section 3.2. 3.4 Semi-autonomous Point-and-Click The semi-autonomous point-and-click scenario is the most complex scenario outlined in this paper. It is predominantly used for research purposes and involves a number of modules spread over one or more UGVs and a GCS machine. A typical use case involves fusing sensor data and displaying a virtualised reality on the GCS machine. At a high level, an operator can click a ground location on the virtualised reality interface and immediately see a given UGV travel autonomously to the selected destination in reality. The point-and-click functionality can be interchanged with other higher level planner or voice controlled modules that issue desired UGV way points and destinations in the same manner. However, simple virtualised interface point-and-click is used here for simplicity. Infrastructure Layout This scenario uses the same conﬁguration on the UGV and wireless routers as the Remote Tele-operation scenario discussed in Section 3.3. The main diﬀerence lies in the GCS machine used to remotely control the UGV and the missing laser range ﬁnder previously usedFigure 3: The software and networking component lay- as a local positioning system. Figure 4 shows a highout used for remote tele-operation. level representation of data ﬂow and modules used in this scenario.Conﬁguration ItemsOn board, additional “ping success” tests were added The GCS database receives replicated data thatfor each additional node that data needs to be repli- includes raw laser range values from the top mountedcated to, to ensure the underlying medium is conﬁgured rotating laser on the UGV, a pose estimation andcorrectly. In fact, a test suite of network conﬁguration DMC wheel rotation and steering positions. This datachecks was created to test the connectivity of each node is then read by both a fusion and laser conversionto every other node it is required to communicate with. module to produce a three-dimensional point cloudThe database replication functionality is also based and occupancy grid, which are pushed back into theon an internal code, called an icode, that enables data GCS database. This data is then read by an OpenGLto be received into a correctly conﬁgured queue on a based visualisation module to display the spatial datadestination node. A set of “file contents” checks in real-time. This interface allows an operator towere created for each replicated queue on each node click on a location on the occupancy grid, which pushesto ensure the numeric icodes were consistent. Custom a desired destination into another queue in the database.database queue deﬁnition checks were also included foreach node to ensure the internal data structures in the This desired destination is in turn read by a plannerdatabase were conﬁgured correctly. module, which uses the UGV’s current pose estimation, along with the current occupancy grid, to plan a path to Having a test suite designed for the UGV and teach- that location. This path is pushed to another databaseing workstation greatly reduced the set up time required queue on the GCS machine and replicated to the UGV.to prepare the machines for lab work. In addition, a A module on the UGV reads this path and translatesstudent machine test suite allowed students to ensure it into driving instructions. These instructions aretheir module set was installed correctly without having handled in an identical manner to driving instructionsto take up a lab demonstrator’s time. The student ma- in other scenarios and processed through the arbitrationchine test suite was also used by lab demonstrators to module mentioned in Section 3.2.assist in troubleshooting exotic problems that arose on
data, has the highest precedence in the command arbiter program, but is rarely used in practice. If, for example, the autonomous driving instruction genera- tion module was misconﬁgured to push data to this queue, then all safety protocols will be unintentionally ignored when the UGV is driving in autonomous mode. Again, a series of “file contents” checks were used to enforce correct queue assignments to all modules. A ﬁnal example involves mathematical constants stored in the planned path to driving instructions conﬁguration ﬁle. This module takes a path and current pose and uses control theory to continually issue driving instructions. These calculations rely on correctly conﬁgured control constants as well as knowing the UGV’s maximum forward and backward speeds and steering angle limits. A misconﬁguration to one or more of these values can lead to erratic motion, which can damage the UGV, let alone objects or people in its operating environment. A number of tests were also included to validate each of these constants individually,Figure 4: The software and networking component lay- with certain domain knowledge encoded into the testsout used for point-and-click autonomy. relating to valid ranges of these values for proper operation.Conﬁguration ItemsEach module requires the correct queue conﬁgurationsset so that data passes from one module to the next 4 Experimental Outlinecorrectly. Each database queue must also be replicated To test the added troubleshooting beneﬁt of using thebetween database instances correctly and as such a ScriptRock platform over an otherwise manual process,range of conﬁguration items are required to be validated an experiment consisting of a series of tests was devisedvia a set of tests. that cover the three main scenarios outlined in Section 3. For each scenario, a set of critical conﬁguration items One example of this involves the point cloud gen- were selected at random and encoded into a chaos scripteration module having the correct laser input queue as separate tests. When run, the chaos script randomlyset in its conﬁguration. Since the UGV has three chooses one conﬁguration item and modiﬁes it into alasers on board, the correct laser queue for point cloud broken state. This is based on a similar concept usedgeneration must be set, or an incorrect point cloud is by the Netﬂix Chaos Monkey [Hoﬀ, 2010].generated. This can have a lead on eﬀect to occupancygrid generation and ultimately path planning throughan environment that does not match reality. A separate Each misconﬁguration item encoded into the chaos“file contents” test was created for each database script falls into one of the following eight categories:input and output queue entry on each module. • a missing DLL ﬁle or missing software dependency such as the Python runtime being made inaccessible; Another example of an insidious misconﬁgurationis pushing tele-operated driving instructions, au- • a missing executable ﬁle of a software module;tonomously generated driving instructions or laser • a missing conﬁguration ﬁle for a given software mod-proximity eﬀecting driving instructions to the wrong ule;queue. On board, the command arbiter module uses • an incorrectly set database queue in a module’s con-inbuilt ﬂags and a hierarchy system to give precedence ﬁguration ﬁle for either the queue it reads data fromto certain driving instructions over others. This module or pushes data to;relies on the assumption that a given database queuehas been conﬁgured in another module to push records • an icode misalignment between the two databasefrom the correct location. For example, an emergency instances replicating data between one another formode of tele-operation, that ignores laser proximity a particular database queue;
• an incorrect IP address or port number for database replication settings; Comparison of Problem Solving Time • an underlying network misconﬁguration such as in- 110 correct subnet settings or incorrect IP address as- 100 signment; or 90 • any module speciﬁc conﬁguration ﬁle lines such as 80 incorrect baud rates, invalid IP address or COM 70 port for hardware modules and particular constant Test No. 60 conﬁguration deﬁnitions required for module calcu- lations. 50 40 A total of 111 misconﬁgurations were spread acrossthree scenario chaos scripts. Before each test, a scenario 30is chosen at random and the respective chaos script was 20run on a given node used in that scenario. A working 10state action was then attempted upon the system unsuc- 0cessfully to prove that the chaos script did in fact break a 0 5 10 15 20 25 30crucial conﬁguration item. An experienced engineer that minuteswas involved in the design, development and now mainte-nance of the entire software and networking component Figure 5: Comparison of time take to solve a given sce-set was given access to the system to attempt to diag- nario problem. A blue circle represents an attempt wherenose and correct the introduced problem and therefore the engineer was allowed to use the ScriptRock platform,complete the given working state action. Use of every while a red cross represents attempts without access toexisting piece of software, testing tool and piece of doc- the ScriptRock platform.umentation was allowed in every scenario attempt, butthe engineer was only allowed access to a ScriptRock testsuite for the given scenario on every second attempt. A On each scenario attempt, a timer was started whenscenario attempt is deemed complete if the system is able the engineer was ﬁrst allowed access to the system. Theto carry out the respective working state action. These timer was paused when the engineer believed they hadare deﬁned as: solved the problem. A working state action was then at- tempted immediately to determine if the problem had inDirect Tele-operation Use an Xbox 360 controller to fact been solved. Successful completion of the action lead tele-operate the UGV directly over a distance of to the paused time being recorded as the oﬃcial solve three metres and return to the starting position, time of that test. On a unsuccessful action attempt, the completely demonstrating that it can also steer both timer was resumed from the previously paused state and left and right. the engineer was again given further access to the systemRemote Tele-operation Be able to use a keyboard until they again claimed to have solved the problem. Sec- based application1 on the GCS machine to tele- tion 5 outlines the results of this experimental approach operate the UGV remotely over a distance of three applied multiple times and provides an analysis of the metres, completely demonstrating that it can steer results. both left and right while driving in both directions.Semi-autonomous Point-and-Click Given an open 5 Experimental Results and Analysis area that is traversable by the UGV, issue a UGV The experiment outlined in Section 4 was run a total of desired destination by clicking on a location in a vir- 50 times, 25 of which allowed the use of the ScriptRock tualised reality interface on the GCS and have the platform and the other 25 without. Figure 5 shows UGV then autonomously travel to that destination the recorded times of each attempt. The graph clearly in reality. Then issue another desired destination demonstrates that using the ScriptRock platform to back to the starting location. In the process, the diagnose system misconﬁguration adds an element of path chosen must demonstrate that the UGV can determinism and repeatability to the troubleshooting move forward and backwards, and be able to steer process. both left and right during motion. 1 A keyboard tele-operation program was used to simulate The main observation made from tests allowing thea student’s software module. use of the ScriptRock platform was that the engineer
would immediately initiate the ScriptRock test suite Test Type w/ ScriptRock w/o ScriptRockwithout considering the use of any other approaches. Average Time 1m 47s 13m 16sThe test suites created for each scenario took 1m 25s Std. Dev. 11.1s 9m 40.5s(± 10s) to execute and upon viewing the resulting test Total Time 44m 38s 5h 31m 38sreport2 , the engineer subsequently solved the problemwithin a further 10s to 40s. Table 1: Statistical summary of experimental test times with and without the use of the ScriptRock platform, When not able to use the ScriptRock platform, the each over 25 tests.engineer appeared to begin the diagnosis process bystarting all software start-up scripts to attempt to could be tele-operated but not autonomously driven.observe which modules appeared to function normally This lead the engineer to focus the search predominantlyand which did not. If everything appeared to initialise on the GCS. One of the more successful and uniquecorrectly, the engineer would often then attempt to test attempts in this category involved the UGV neverdrive the UGV via tele-operation or, in autonomy being able to steer left. This was eventually diagnosedrelated tests, set a desired destination to drive to. The as a constant being set to zero that represented one ofengineer also appeared to make extensive use of the two steering angle limits in the DMC interfacing module.queue activity tab of the centralised database instanceon each node to see if certain critical queues were The most diﬃcult group of tests to diagnose involvednot being populated. The database queue activity semi-autonomous operation, with the UGV also notand attempted motion testing methods then lead the responding to remote or local tele-operation attempts.engineer to begin looking through between one and ﬁve Here the engineer resorted back to tracking the virtualconﬁguration ﬁles to check conﬁguration items, line by ﬂow of data from sensors through software modules,line, until the problem was ultimately solved. database replication and resulting actuator modules. More than half of these type of tests involved the The four quickest test attempts in Figure 5 did not engineer consulting a wiki for information about theinvolve any use of the ScriptRock platform. These test usage and conﬁguration of modules.all fell into the misconﬁguration category of moving ormisplacing one of the software module executable ﬁles. A statistical summary of the experimental results isDue to the engineer beginning each non-ScriptRock given in Table 1, which signiﬁcantly highlights the morediagnosis attempt by running a start-up script, the than ﬁve-fold improvement in average troubleshootingoperating system presented a message box when it time using ScriptRock over existing methods. It shouldcould not ﬁnd a particular executable ﬁle to run. This be noted that each test attempt presented in this pa-provided the engineer with the ability to solve this type per involved a single item being misconﬁgured. Fu-of problem consistently within one minute. ture experiments are planned to test the troubleshooting times of multiple misconﬁguration points within a sce- Direct and remote tele-operation scenario tests were nario. The hypothesis here is that multiple misconﬁgu-among the easier misconﬁgurations to diagnose. By ration items may linearly or exponentially make diagno-attempting to drive the UGV and by observing the sis more diﬃcult for existing manual processes, but onlyactivity of database queues, the engineer could solve constantly more diﬃcult for ScriptRock enabled tests.the problem in a number of minutes. If databaseactivity appeared to be normal, then conﬁguration ﬁles 6 Conclusionspeciﬁcally on the UGV were checked to ﬁnd a solution. The application of the ScriptRock testing platform to a multi-node robotics system has demonstrated an in- Autonomy related tests were the next most diﬃ- creased amount of understandability and determinismcult misconﬁguration to diagnose as they involved both in managing the software and networking conﬁguration.checking low level tele-operation modules on the UGV as Undertaking conﬁguration change, or troubleshootingwell as sensor fusion and path planning modules on the misconﬁgured nodes, has been reduced to time framesGCS machine. Some diagnosis attempts were assisted well within ﬁve minutes, as opposed to fractions of anby observing inconsistencies in database queue activity hour, using existing manual techniques. As a naturaland inter-node database queue replication anomalies. progression these tests have also improved the process ofSome tests were also assisted by the fact that the UGV rolling out software and network settings on a new UGV. 2 The default layout of a results report places the results The progress and ultimate success of a fresh roll outof all failed tests at the top, followed by a complete set of all can be measured accurately through the setup process.test results. Making slight conﬁguration changes for use in diﬀering
scenarios can also be validated without having to resort [Hirzinger and Bauml, 2006] Hirzinger, G. and Bauml,to the same previously used manual process. Robotic B. (2006). Agile robot development (ard): A prag-systems, much like large enterprise grade architectures, matic approach to robotic software. In Intelligenthave an extensive range of solutions capable of testing Robots and Systems, 2006 IEEE/RSJ Internationalwhether an application performs correctly, but little is Conference on, Beijing.available to test the underlying environment and mod- [Hoﬀ, 2010] Hoﬀ, T. (2010). Netﬂix: Continually testule conﬁguration. This paper has demonstrated that this by failing servers with chaos monkey.innovative application to enterprise systems can also beapplied to complex robotic systems to greatly simplify [Horn, 2001] Horn, P. (2001). Autonomic computing:the development, maintenance and use of these systems. Ibms perspective on the state of information technol- ogy.Acknowledgments [Quigley et al., 2009] Quigley, M., Gerkey, B., Conley,The author would like to acknowledge, Dr Jos´ Guivant, e K., Faust, J., Foote, T., Leibs, J., Berger, E., Wheeler,Mr Mark Whitty and A/Prof Jayantha Katupitiya for R., and Ng, A. (2009). Ros: an open-source robottheir extensive contributions to the design and develop- operating system.ment of the UGV and GCS system and providing access [Robledo et al., 2010] Robledo, A., Guivant, J., andto the platform. Their assistance in the test creation Cossell, S. (2010). Pseudo priority queues for real-and testing process is also appreciated. time performance on dynamic programming processes applied to path planning. In Proceedings of the 2010 ScriptRock should also be acknowledged for provid- Australasian Conference on Robotics & Automation,ing a custom educational licence to the engineers and Brisbane.researchers in Mechatronics at UNSW. [Sharp-Paul, 2012] Sharp-Paul, A. (2012). Scriptrock - frequently asked questions.References https://www.scriptrock.com/faq.[Bersoﬀ, 1984] Bersoﬀ, E. H. (1984). Elements of soft- [Whitty et al., 2010] Whitty, M., Cossell, S., Dang, ware conﬁguration management. IEEE Transactions K. S., Guivant, J. E., and Katupitiya, J. (2010). on Software Engieneering, SE-10(1). Autonomous navigation using a Real-Time 3D point[Burgess, 1995] Burgess, M. (1995). Cfengine: a site cloud. In Proceedings of the 2010 Australasian Con- conﬁguration engine. USENIX Computing Systems, ference on Robotics & Automation, Brisbane. 8(3).[Calisi et al., 2008] Calisi, D., Censi, A., Iocchi, L., and Nardi, D. (2008). Openrdk: A modular framework for robotic software development. In Intelligent Robots and Systems, 2008. IROS 2008. IEEE/RSJ Interna- tional Conference on, pages 1872–1877, Nice.[Guivant, 2012] Guivant, J. (2012). Possum robot. http://www.possumrobot.com.[Guivant et al., 2012] Guivant, J., Cossell, S., Whitty, M., and Katupitiya, J. (2012). Internet-based oper- ation of autonomous robots: The role of data repli- cation, compression, bandwidth allocation and visual- ization. Journal of Field Robotics, 29(5):793–818.[Guivant, 2008] Guivant, J. E. (2008). Real time synthe- sis of 3d images based on low cost laser scanner on a moving vehicle. In V Jornadas Argentinas de Robtica (JAR’08).[Hall et al., 1997] Hall, R. S., Heimbigner, D., van der Hoek, A., and Wolf, A. L. (1997). An architecture for post-development conﬁguration management in a wide-area network. In Proceedings of the 17th Interna- tional Conference on Distributed Computing Systems, Baltimore, MD.