Welcome Our Robot Overlords By Stephen CossellConfiguration Testing on a Fleet of Unmanned Ground VehiclesUp until now, most of our discussion at ScriptRock has been on configuration testing in anenterprise context. Google’s Driverless CarAnd why not? These systems are complex, touched by many people, and they can be critical inthe day-to-day runnings of a large company. This post, however, is about changing out of oursuits and ties, and putting on a lab coat to look a little closer into the toys current academicsplay with, called robots. Robotics systems are becoming more mainstream, and put into moreand more industrial and public-facing situations (military, mining, space exploration, deep seaexploration, Google’s driverless car), and will continue to become more mainstream for theforeseeable future. The US military have their attack drones. Australia is a world leader inautonomous mining (yes, those big trucks with wheels the size of a human are being driven
around mining sites autonomously). NASA is sending more and more awesome rovers to Mars,and they plan on exploring Jupiter and Saturn’s moons with similar probes.I, for one, welcome our new robotic overlords.However, robotics systems will likely be just as complex, or even more so, in the coming years.Ignoring the materials science, the mechanical design, and electrical hardware and the circuitrythat goes into a robotic system, and while just focusing on the networking and softwarecomponents, robots are complex machines. They will continue to grow in complexity as they aremade to function in more specific environments and adhere to strict safety regulations.When a system goes down in the enterprise world it can cost companies hundreds of thousandsor even millions of dollars, maybe a few customers, and a bit of a bruised ego. When roboticsystems are unleashed into an industrial context and misbehave, they can cause real physicaldestruction of objects, or even cost lives.Industrial robotics engineers are going to have to adhere to very high standards in the comingyears. Unlike current software, any minor mishap with a robotics system could snowball intopublic distrust and set the young industry back a number of years. Imagine the potential publicreaction to the Google car autonomously, but accidentally, running over and killing a person andthe public image of driverless cars everywhere. Compare that to a human accidentally runningover a person and killing them…Public facing robotics systems are going to have be designed, built, and maintained at thehighest standard and one of the most overlooked parts of this quality comes down to knowingthe state of your software environment and being able to validate it, at will.Let’s step into the lab and have a look at a pretty simple robotics platform relative to industrialpublic facing robots out there:The team of engineers and researchers in the Mechatronics group at UNSW use a fleet ofUGVs (Unmanned Ground Vehicles) for teaching and research purposes.
UNSW Unmanned Ground VehicleThe robots are designed to autonomously explore a large urban area and generate a 3Dvirtualized representation of that area back on a base station computer.Looking under the hood, each robot has a number of sensors that enable it to sense the worldaround it. These include webcams, laser range finders, an IMU (Inertial Measurement Unit),wheel turn and speed readers on the onboard DMC (Digital Motion Controller), and an XboxKinect. In addition the robot has a number of actuators that allow it to move and interact with itssurroundings. These cover the drive and steering motors for the wheels as well as other motorsthat assist some of the sensors in being able to sense with a greater field of view. A prettystandard rig for ground vehicle applications.Here’s where we get to the complexity in the system: each sensor and each actuator has aspecific software module designed to interface with each piece of hardware. The system alsohas many other small software modules, which handle data processing and decision making. Atthe center of it all is a circular database system called Possum, which allows data to flowbetween modules and auto-replicates data between robots and the base station computer.Each of these software modules on each of the robots and base station computer have one ormore configuration files associated with them. Each machine has a set of startup scripts thatmake sure the correct programs are started for a particular scenario, and that these programsare started in the correct order.
Each hardware and non-hardware related software module relies on certain system settings tobe in place such as correct IP addresses, subnets, and gateways being set. Some of the oldersensors still make use of COM ports, so these need to be validated between the operatingsystem and configuration files.For the team of engineers who maintain this robotic system, configuration errors are the mostfrequent, ongoing problem. When a piece of software misbehaves, this functional error can beidentified and fixed permanently. Configuration errors, however, re-occur due to the systembeing used in different contexts, and therefore requiring different configurations, on a regularbasis. A constantly changing configuration makes it difficult to confidently know which state thesystem is set up for and manually validating a configuration is a time consuming process.At the recent Australasian Conference on Robotics and Automation we presented work thatshowed time taken to troubleshoot configuration problems on this system. In short, we created achaos script similar to Netflix’s Chaos Monkey, which was used to break a random, but critical,configuration item within the system. An experienced engineer was then asked to diagnose theproblem using any resource they would normally have access to. On every second attempt theengineer was additionally allowed access to a ScriptRock test package that covered the entireconfiguration space of the robotic system. This test was run 50 times over 111 configurationitems. The time taken for the engineer to diagnose and solve the problem is shown by the graphbelow:
Red crosses indicate test attempts without the aid of the ScriptRock test package, while bluecircles indicate tests attempts where ScriptRock was allowed to be used. For those of youplaying at home, a total of around 5.5 hours was taken over 25 attempts under normalconditions, compared to 45 minutes with the use of ScriptRock over the same number ofattempts.Although this post has discussed how ScriptRock can greatly assist in validating a robot’ssoftware configuration for troubleshooting purposes, it is also being used as part of routine pre-experiment checks, and to validate software environments of new robots that join the fleet.Follow ScriptRock on TwitterReferences: J. Guivant, S. Cossell, M. Whitty and J. Katupitiya, “Internet-based operation of autonomousrobots: The role of data replication, compression, bandwidth allocation andvisualization”, Journal of Field Robotics, Vol. 29, No. 5, pp. 793-818, September/October 2012.(DOI: 10.1002/rob.21432) M. Whitty, S. Cossell, K. S. Dang, J.Guivant and J. Katupitiya, “Autonomous Navigationusing a real-time 3D point cloud”, Australasian Conference on Robotics and Automation,Brisbane, Australia, December 2010. J. Guivant, “Possum robot”, http://www.possumrobot.com, 2012. S. Cossell, “A novel approach to automated systems engineering on a multi-agent roboticsplatform using enterprise grade configuration testing software”, Australasian Conference onRobotics and Automation, Wellington, New Zealand, December, 2012.