1. High Level Applications for Accelerators
By
Shannon W. Krause
July 31, 2009
I started working in Telescope Operations about 18 years ago at the Very Large Array (VLA),
Radio Astronomy Observatory. At the time I had already been designing and writing software
for a decade, so I was compelled to work on the VLA control system. While learning the
fundamentals of digital control theory and application, I was also the operator responsible for
the Array Operations Interface software. The VLA control computers were MODCOMPs
running the Max32 operating system with VT100 terminals and analog thumbwheels for a user
interface. The requirement of synchronizing the simultaneous operation of 27 radio telescopes
insured a high degree of automation. This concept and vision of how to automate large
complex machines has always be incorporated into my software applications.
Next I moved on to Accelerator Operations at Jefferson Lab (JLAB) during the later part of the
facility commissioning phase. Due to my previous experience, I worked closely with the
Controls Group developing many of the high level applications Operations would use during
the upcoming experiments. At the time, JLAB was transitioning from a TACL control system
running on CAMAC crates, to an EPICS system running on VME crates. The control room
computers were HP workstations running Unix. MEDM was used for basic control screens,
and TCL/TK and Motif for the high level applications. Since only the hardware controls were
in place at the time I arrived at JLAB, there was a great need for high level applications.
One of the first projects in which I was involved was the transition from a paper log book to a
web based log book which also included downtime and failure tracking. JLAB is funded by
the Dept. of Energy (DOE) and they have very strict requirements on log book content.
Running the JLAB accelerator is the responsibility of a “Crew Chief” who directs multiple
operators and insures that DOE operating requirements are fulfilled. To document the
transition of Crew Chiefs there is a “Crew Chief Stamp” for the log book. With the paper log
it was literally a rubber stamp with various fields to be entered, checked off, and signed.
Below is the software version I created for the same purpose.
1

2. Like most of the programs to follow, this application is written in TCL/TK. The colors are the
JLAB GUI standard so raised “cyan” fields are buttons, and sunken “cyan” fields are entries.
Grey is a back ground color and black text are labels. Yellow text on a dark blue back ground
is a read back. There are a large number of parameters that must be verified before the
beginning of a shift. However, this application would read the Operator shift schedule, the
Short Term Schedule and the system clock to automatically determine the value of every entry
field. Therefore, instead of the Crew Chief manually digging through these various documents
and recording the information every shift, they simply had to verify the information was
correct, modify it if required, fill out the check list on the upper right and then hit the “Stamp
2

3. In” button. The program would then make an entry into the electronic log book such as that
below.
This program introduced an important concept to the lab management, which is for software to
make log entries for the operators. Eventually all the high level applications at JLAB were
required to automatically log their actions after manipulating the accelerator. This insured the
log book information was provided at the moment of completion, presented in a complete and
consistent format, and required no additional action by operations. This saved a lot of “paper”
work which is time consuming and interferes with other aspect of operations. This type of
automation follows my philosophy that: “if the computer can do it, then the computer should
do it”.
3

4. JLAB is a CW electron beam accelerator with two SRF linacs and a five pass recirculator
system that runs beam to three simultaneous experiments, in three different target areas, each at
a different beam energy and current. At the time, the DOE maximum integrated operating
envelope was 800 kW but the accelerator could produce in excess of a MW of beam to a single
target. Even the high power beam dumps couldn’t survive a MW for long, and the targets were
typical limited to 100’s of kWs. The Crew Chief was responsible for adhering to these
restrictions but the beam delivery parameters changed frequently and given the high power
beams and pitfalls in the control system, the result was a series of destroyed targets. The
individual beam dumps were protected by hardware interlocks but JLAB needed a soft
interlock for the targets and beam envelope restrictions. The result was my program, Max
Juice.
Not only did Max Juice provide intelligent soft interlocks for the targets, but it also had an
event log, and at the end of each shift it would make an entry to the log book, and to a
database. These recorded beam delivery parameters including kiloWatt-hours, electrons, and
Coulombs delivered to target. This data was useful for tracking radiation production, machine
performance, and for failure analysis. The software was very effective, especially after the
target limits were incorporated into the program, based on the end station target ladder
positions.
JLAB and DOE have a contract that is very specific about beam delivery requirements and
machine performance. Therefore the beam delivery status must to be tracked accurately and
reliably for JLAB to fulfill their obligations. The result was a “Down Time Logging” team
which was comprised of people from Operations and Controls. After defining the software
4

5. requirements, I volunteered to write the software and the result below is BOOM – The Beam
Operations Objective Monitor.
BOOM is a beam status monitor with a couple dozen timers, one for each state, for each beam
area. The accelerator and end stations normally will be in one of four states: UP, TUNE,
DOWN, or OFF. There is an additional state that was manually triggered called, BANU:
Beam Acceptable but Not in Use. Comparing UP and BANU provides a measure of the
experimental efficiency. Reducing the TUNE category is a primary goal for JLAB operations
and it drives the demand for further automation.
At the beginning of each shift, BOOM automatically reads the beam objectives from the Short
Term Schedule and monitors them accordingly. Reliably determining the beam status is a
tricky task accomplished by monitoring states from the safety, system, beam diagnostics,
5

6. magnet status, and conditions in the injector. Redundancy is provided so IOC reboots and
hardware failures do not corrupt the status logic. Even then, all states except the “UP” state
can be manually selected as a logic override. In addition, at the end of each shift a form pops
up automatically with all the shift data about to be logged. The Crew Chief is responsible for
the accuracy of the numbers and can manually modify them at the end of the shift. In general,
that means no action other then pressing the “CREW CHIEF VALIDATION” button which
dumps the shift data to a log book entry and into a database. BOOM also has an event logger
which records every state change, error outputs, and other items of interest.
6

7. BOOM end of shift log book entry.
Developing effective software includes tracking the system resources used by the code and the
ability to terminate rouge processes that are inadvertently created while debugging beta
software. Back in my days of writing software in Unix, process control was an ineffective
command line business. I also have a rule for operations software: “we will not use command
line for any operations or machine control”. Therefore, I wrote Cob Sniper to provide a GUI
based method of process control that any user could easily use. The significance of Cob Sniper
is that it introduced me to the concept of list building. List building is a key component in
beam line applications that I have since developed.
7

8. During the commissioning at JLAB, I was a member of the “Magnet Performance Team” to
address both hardware and software issues related to 2000+ magnets and their effect on
machine performance. One major hardware and software issue was hysteresis. Maintaining
and restoring tunes for running loss free high power beams requires the elimination of all
hysteresis effects from the normal conducting magnets. Hardware and software glitches
frequently cause large sections of the machine to get current spikes and any number of magnets
may require cycling even when the “cycle fault” doesn’t catch the error. The solution to these
problems is my program, “HystArea”.
8

9. HystArea is a perfect example of how list building can be used for magnet management. The
software uses a menu tree to select magnets by machine area, beam line, IOC, or magnet
control rack. In addition to the menu tree, a filter is also available to add or subtract magnets
from the list. Individual list items can be deleted by simply double clicking on the specific list
element. Once the final magnet list is created, the CYCLE button will force the magnets to be
ramped in the prescribed manner to eliminate the hysteresis in the iron cores. Errors in the
cycling process are reported in a separate pop-up window.
A major problem for magnet software maintenance is that new beam line configurations are
implemented so magnets come and go and names change as new magnets are introduced.
Keeping the IOCs, MEDM screens, high level apps and alarm handlers up to date is a tedious
task. The solution is to create a master magnet database and write various scripts to
automatically update the magnet related applications directly from the database when it
changes. For example, the first time it is run each day, HystArea automatically updates it’s
data files accordingly. If the magnet configuration is known to change during day, the “Update
Magnets” button allows a manual update to the master magnet list.
9

10. After JLAB I moved on to the MIT Bates Linear Accelerator. My position was Operations
Group Leader and my primary job was to learn the facility and then commission and run the
electron beam accelerator, recirculator and ring during the BLAST experiment. After the
experiment DOE had already planned that the accelerator would be shut down and
decommissioned. When I arrived there were some MEDM screens for device control and the
control room computers ran Linux, but there were almost no high level applications and much
of the control system was still analog hardware from the past. The plan was to upgrade as
much hardware as possible to EPICS digital control and it was my job to manage the entire
project to insure the machine would run reliably.
My team of software developers included three operators doing screens and high level
applications, and three part time controls people with an odd student doing the low level
EPICS applications. My first priority was getting the EPCIS Alarm Handlers up and running
as we still had to run part of the SAMPLE experiment before BLAST. Getting the BPMs to be
operationally useful was my next priority, followed by the timing generator, source controls,
the RF systems, and a long list of other devices. I was responsible for prioritization, software
design requirements, and filling out the operational parameters in the EPICS database such as
alarm limits, drive high/lows, data flow control, BPM calibration offsets, etc. etc.
The large amount of control system development and frequent power glitches resulted in a lot
of locked up IOCs. Due to the size of the facility hard booting the VME crates was
problematic. Soft boots required crate specific information, often did not work, and the
operators were not trained to run the VxWorks command line. To solve the problem we did
tests and determined the most common failure modes of the IOCs. I then designed and wrote
the “IOC Reboot” program which uses the “expect” command to log into the VME crate,
interactively evaluate the IOC status, and pick one of three different methods to reboot the
crate. Any number of VME crates can be selected and rebooted sequentially. The boot
program also provides IOC status indicators, the EPICS channel list for each IOC in a
searchable window, and buttons that open windows containing IOC log files and boot logs
which were recorded for troubleshooting and diagnostics.
10

11. At MIT I created a control system to my specifications and one of the first high level
applications I wrote was a new version of HystArea that provided complete magnet control,
including fault reports, as well as the Save/Restore/Compare functions for all devices in the
save sets. It also allowed save/restores of BPM offsets so beam positions could be compared
between past beam tunes both live and offline. Not only could a save be made of any machine
area at any time, automatic save sets were created every 15 minutes to avoid losing the active
machine status. The save data was compressed into “tar” files at the end of each day and
11

12. HystArea allowed for the recovery and save/store/compare functions on these tar files as well.
With this program we could take all 500+ magnets, clear their faults, turn them on, ramp them
to their operating set points, cycle them and then check for faults with a few mouse clicks.
Likewise, all the menu trees, filters, and other list building elements were included in the
program. RF controls would have been included, but we never got more than read backs into
EPICS due to time and priorities. The Bates HystArea program demonstrates how an operator
can monitor and control virtually an entire accelerator with a single application.
12

13. During the commissioning period at JLAB, there was a frequent need to monitor various
systems that were not yet in the alarm handlers. The result was my invention of the program
CHIME. The original function of CHIME was to take an arbitrary EPICS channel and then
alarm with a series of 5 selectable tones when an EPICS channel read back exceeded a
specified limit. Typically, if an alarm sounded, an operator was required to take some action
such as pressing a reset button on a faulted device. So, I added the capability of executing
command line outputs into the program. CHIME continued to gain flow control functionality
such as the ability to monitor the beam status and turn itself off when the beam was not on.
For example, when used to maintain a beam position on a BPM, the program will not steer the
beam while the beam is off. Since CHIME can launch any command line process, it can be
used as an event logger and most importantly as a tool to prototype feedback systems and
automation of operator functions. This feature proved extremely useful at MIT where we had
few resources and many software needs.
The BLAST experiment required 850 MeV polarized electrons running on a plasma target in
the storage ring. The gas target rapidly degraded the beam intensity so the ring had to be
refilled with beam about every 15 minutes. This was a process requiring the manipulation of a
significant number of devices in a very specific sequence. Performing the task manually was
tedious, inefficient and error prone. The obvious solution was automation, and by running two
different CHIME programs we could perform the entire fill process by executing shell scripts
at different trigger points in the refill process. Once the sequence was established and
optimized, it was time to write the requirements document and then code and debug the EPICS
Automated Ring Fill (ARF) sequencing program. A given ring fill had about 20 steps in the
sequencer and there were various fault conditions that would stop the program to avoid
equipment damage from the beam.
13

14. One of the key design features of ARF was the enable/disable feature. The operators and
experiments each had a different enable/disable button. Before the software would activate,
both enable buttons had to be pressed. Either button could disable the software at any time.
This insured communication between operations and the experimenters before automated
14

15. filling could begin and provide both parties the ability to halt the fills at any time. The system
worked very well and ARF performed about 45,000 ring fills during three years of operation.
The picture above is the Operations ARF screen using the Bates MEDM screen formatting
standards. The experimenters screen was identical except for the Enable button. Below is the
ARF “expert” screen which has all the controls for testing, debugging, and configuration
changes. The green indicator box on the left shows the active sequence step.
Automating the ring fill process had another benefit as well. The optimal ring tune for
injection and the optimal tune for beam storage were significantly different. Therefore, the
tune would become a compromise between the two solutions. By using a CHIME program to
monitor the ARF sequence, we would trigger the CHIME program to execute a shell script to
then transition a couple dozen steering magnets between an injected tune and a stored tune.
15

16. This feature alone increased our integrated beam on target by about 50%. The BLAST
experiment required one MegaCoulomb of charge to be successful and the Bates Lab delivered
three MegaCoulombs of charge to target. ARF contributed significantly this success. As the
Bates Lab approached shutdown, layoffs were required. Due to the successful software
automation and EPICS integration, Operations productivity had doubled, so 5 of the 10
operators were laid off without significantly hindering beam delivery.
At the end of BLAST I was laid off as well and was recruited to Operations here at the NSCL.
As with Bates, I first focused on alarm handling. The ALERT program in use at the time only
alarmed on set/read changes, and during the cyclotron tuning process, the program had to be
disabled, so device failures were difficult to notice. The solution was to monitor the difference
between the sets and reads and alarm when that difference indicated a hardware problem. Greg
Humenik rewrote the LabView ALERT program to be what is now called SPRED. This
software has proved invaluable to NSCL operations.
I also created the Ops Tool Bar in Windows which provides links to most of the operations
software and related information. Easily navigating a file system is critical for efficient
16

17. operation of a facility. There is a panel mate called ColorTemplate.mt2 that I created with the
standardized color coding for all of operations panel mates. Most of the cells on the operations
panel mates have been modified to this standard although it has yet to become a lab wide
standardd. Randy Rencsok’s deflector conditioning program DefCon is another software
project I have worked on. It uses various feedback methods to condition deflectors and is a
good example of an “expert” system where the knowledge of the system expert Dave Poe, is
directly incorporated into the software. This software allows expert deflector control
capabilities for non-expert operators and has greatly improved the overall deflector
performance.
Since my arrival at the NSCL I have been advocating getting a Linux computer in the control
room so we can start using the standard EPICS alarm handler and other tools. This effort is
now underway, and as a preliminary step towards enhanced alarm handling we now have the
read only device monitor 2BAAD, which Greg also wrote at my behest. I did not design the
GUIs of the LabView applications, they follow the NSCL convention rather than the JLAB or
Bates standard.
17

18. While SPRED monitors beam line elements used in tuning and setup, 2BAAD monitors read
only channels such as temperatures, pressures, and intercepted beam currents. 2BAAD uses
the same two layer alarm limit logic as EPICS. For the past few months I have been fine
tuning these limits to provide a minimum of false alarms. Once the EPICS Alarm Handler is
ready, these limits will be entered into the EPICS database channels and provide immediate
functionality.
CHIME has been modified to run on the NSCL control system as well. By using it to control
the Coupling Line slits, it can typically regulate current to the experiment with <1% beam
intensity variation over a period of hours. Normally, delivered beam currents vary by a factor
of 2 or more as the stripper foil and beam tune degrade over time. One experiment got 20%
more data when CHIME was running to deliver beam just below the rate limit of the detectors.
Below is a screen shot of a HystArea type list building type program affect called CYCLOPS,
that I have prototyped for the NSCL. Due to inconsistencies in the Barney save sets it won’t
work reliably. However, when that issue is resolved completing the project is straight forward.
18

19. CYCLOPS builds the leftmost beam menu by reading the contents of that system directory.
No maintenance of the software is required when new beams are added. A simple click on a
Print/Save file name will put it in the “Selections 1 or 2”. These selections can be compared by
pressing the COMPARE button. Even beams of different elements can be compared and no
typing is required. Double clicking on a file name will open a double paned window with the
file contents displayed in the left side. Clicking and/or dragging on elements in the left side
will move them to the right side. Restoring device parameters can be made from either
window. The basic concept is that once a device list is created, the user can then choose a
range of actions to manipulate those devices: turn on, turn off, set to zero, scale settings, load
set points, save values, check faults, reset faults, calibrate, etc. With this methodology, one
operator with one software interface can control virtually an entire accelerator regardless of
size due to the scalability of the software design concept.
It is important to note that every program I have implemented has three things that are
immediately visible: consistent color standards, a version number, and a HELP button that
actually provides a searchable window with useful information. What you can’t see in the
screen shots are the formatting and comments in the code that make it reliable and easy to
maintain. This is important when modifying programs to work on various accelerators and
adding upgrades months or years after the software is written. Likewise, there is no disconnect
between the software users and myself, the designer. I have actually used all of these programs
extensively, and I worked closely with other users to providing the exact functionality desired.
19

20. The large version numbers on the programs are a testimony of the number of feature additions
added to the software during their lifetime. For effective controls software, the noted features
included in these high level Tcl/Tk programs are not optional.
The high level software applications presented in this document are all relevant to ReA3 and
FRIB if only conceptually. However, programs like CHIME can already be used during the
development phase. Given the opportunity, who knows what new and revolutionary solutions
to accelerator automation and control I will devise in the future.
20