1. Improving the performance and capabilities of the sample handling
robots at the Australian Synchrotron
Robbie Clarken, Mark Clift, Gautam Manoharan, Jason Price, Tom Caradoc–Davies
Australian Synchrotron, 800 Blackburn Road, Clayton 3168, Victoria Australia
Abstract
The Australian Synchrotron crystallography beamlines use robots for faster and safer sample
handling and to facilitate remote operation of the beamlines. The requirements of operating in
liquid nitrogen, handling non–uniform sample pins and precision placement of the pins create
unique challenges for the robot hardware and software. Since a robot failure can result in
cancellation of an experiment there is pressure on Synchrotron staff to maximise reliability of
the robots and make recovery from faults as fast as possible. To achieve these ends we have
undertaken a major redesign of the software architecture. The new design will deliver higher
reliability, extended capabilities, better logging of errors, faster recovery times and better
integration with beamline control software. This will allow the future development of a
screening server and other capabilities not currently available on the Australian Synchrotron MX
beamlines.
Motivation
When the Australian Synchrotron crystallography beamlines were commissioned we adopted a
hardware and software stack developed at the Stanford Synchrotron Radiation Laboratory
(SSRL)[1]. This included:
The Distributed Control System (DCS)[2]
The Blu–Ice graphical user interface
The Stanford Automated Mounting system (SAM)[3]: an EPSON robot and hardware
configuration for mounting and dismounted cryo–cooled samples
This implementation has enabled exceptional results to be delivered on the MX beamlines but
certain aspects of the software stack have also impeded delivering upgrades to the beamlines:
The DCS and Blu–Ice are written primarily in Tcl/Tk - a language that has declined in
popularity, making it difficult to find engineers experienced in programming in this language.
The robot software consisted of an intractable C++ application and a separate program
written in EPSON’s SPEL language. Both programs needed to be running simultaneously
and be aware of the state of the robot. This lead to duplication of code and confusion about
where new robot functionality should be added.
The Australian Synchrotron has adopted EPICS as the preferred control system. This system
has many advantages over SSRL’s DCS, such as a wide range of supporting tools and
infrastructure and expertise within Australian Synchrotron staff.
Redesign
To improve the reliability of the robot system and allow development of new features, a
software architecture redesign has been undertaken. The goals of redesign are:
Move all code that directly controls the robot into SPEL to reduce redundancy and make it
easier to pinpoint the location of bugs.
Expose the state of the robot using EPICS to enable us to utilise the suite of controls tools
supported at the Australian Synchrotron.
Develop a comprehensive test–suite to enable rapid development of new features with
confidence that changes won’t break existing functionality.
Implement a client–server model to allow new user interfaces to be developed with the
long–term goal of deprecating the DCSS and Blu–Ice.
Develop all high–level functionality in the Python programming language to make it easier to
deliver robot upgrades.
New Software Architecture
DCSS + Blu–Ice
Python DHS yaIBEX Web Interface
RobotServer
SPEL Application
Robot
In this new architecture, shown to the left, all
robot operations are implemented in a SPEL
application that runs on the robot controller.
These are initiated by a Python application,
RobotServer, which is also responsible for
preparing the environment for the robot to
perform its operations. This includes moving
hardware like the detector and the cryojet to
safe positions before mounting.
The RobotServer is controlled via remote
procedure calls by RobotClients. One such
RobotClient is a Python implementation of a
DCS Distributed Hardware Server providing
support for the existing Blu–Ice/DCSS system.
Another RobotClient will be yaIBEX, a web
interface for experimental control, written in
Python, that will support new features being
developed at the MX beamlines.
EPICS Communication
To enable an EPICS interface to the robot a network server has been developed in SPEL. This
server, which runs on the robot controller, exchanges messages over a TCP socket with an
EPICS StreamDevice application called RobotEpsonIP. RobotEpsonIP enables developers to
easily create EPICS process variables to monitor the robot’s state and request operations to be
executed inside the SPEL application. The network server has a run loop which updates the
IOC, enabling real–time monitoring of the robot state.
This design led to some challenges due to limitations with the SPEL programming language.
One such limitation is SPEL does not have an efficient method to convert an array of values
into a string. This meant sending large arrays from the network server run loop would have
degraded the performance of the application. To work around this issue we developed a
message exchange protocol that the Python RobotServer application can use to extract the
data it needs on start–up and then receive updates when the arrays in SPEL change.
Python Layer
The Python programming language has been adopted at the MX beamlines due to its intuitive
and expressive syntax, powerful object–oriented features and vast number of supporting libraries
providing features like communication with EPICS supported equipment. For these reasons it
was decided the layers between the robot controller and the user interfaces would be written in
Python. A client–server model was developed in Python to:
Support multiple clients connecting to the robot simultaneously.
Ensure equipment protection logic lives in only one place.
Allow clients to call robot functions without being concerned with the details of the robot
communication protocol.
RobotClient RobotServer
Robot
Detector
Goniometer
Cryojet
Operation requests
State updates
ZeroMQ
EPICS
Communication between the RobotServer and RobotClient is implemented with the ZeroMQ
messaging library. This is a widely used, battle–tested library that enables efficient
implementation of common messaging patterns. For this application we use two ZeroMQ
channels:
Request–Reply: Used by clients to request robot operations such as sample mounts.
Publish–Subscribe: The RobotServer broadcasts robot state information to all connected
clients.
Conclusions
The robot software architecture presented here is expected to increase reliability of the sample
mounting system and facilitate future improvements, including:
Deployment of the yaIBEX experiment control interface.
Enabling the robot to remain in the liquid nitrogen, eliminating the lengthy cooling and
drying times between mounts.
Having the robot prepare the next sample for mounting while data collection is happening to
further speed up mount times.
Sample screening where many samples are automatically mounted and test images collected
to determine which samples should be subjected to more thorough data collection.
These changes will empower users of the MX beamlines to collect more data of higher quality
and ensure the Australian Synchrotron remains a world–class venue for crystallography data
collection.
References
[1] Nathan Philip Cowieson et al. “MX1: a bending-magnet crystallography beamline serving
both chemical and macromolecular crystallography communities at the Australian
Synchrotron”. In: Journal of synchrotron radiation 22.1 (2015), pp. 187–190.
[2] Timothy M McPhillips et al. “Blu-Ice and the Distributed Control System: software for data
acquisition and instrument control at macromolecular crystallography beamlines”. In:
Journal of synchrotron radiation 9.6 (2002), pp. 401–406.
[3] Aina E Cohen et al. “An automated system to mount cryo-cooled protein crystals on a
synchrotron beamline, using compact sample cassettes and a small-scale robot”. In: Journal
of applied crystallography 35.6 (2002), pp. 720–726.