This article describes the construction of a 650-cpu massively parallel supercomputer using MATLAB as its API and programming language. The system was designed for general use in the solution of commodity supply/demand/pricing algorithms. With this system, a model that would take 24 hours to run on a desktop could instead by run in minutes.

1. High Performance
Computing Using
MATLAB
Michael Patterson1
December 2003
Abstract
Williams has created a high performance MATLAB based computational platform to support its financial
analysis and enterprise risk management activities. Williams chooses to develop its analysis software with
MATLAB to benefit from that product's high productivity toolset. MATLAB's productivity features enable
Williams to quickly develop sophisticated computer models in step with rapidly evolving markets and
customer needs. These models, in turn, provide increased value to Williams' customers and shareholders.
This article will present an overview of Williams' computational platform, consisting of 650 low­cost
Linux­OS servers governed by a MATLAB based batch system. This platform has been running
continuously for two years. The platform is used for both impromptu calculations submitted by analysts
and scheduled nightly calculations. This article will cover the parallel platform's architectural design, the
benefits derived from that design, and the design's resulting capabilities.
Introduction
Williams' businesses produce, gather, process and transport clean­burning natural gas to heat homes and
power electric generation across the country. Williams is a publicly held corporation that trades on the
NYSE under ticker symbol WMB. In support of its business units Williams has created a suite of software
applications that provides financial analysis models and enterprise risk management capabilities. This
software suite models, for example, gas storage scenarios, electric power generation demands and
capacities, and commodity price behavior. Use of this software suite enables Williams to better serve its
customers, and thereby creates increased value for Williams' investors. Full details of the Williams
company are available online at http://www.williams.com.
Williams has chosen to create its modeling software using MATLAB®, a proprietary product developed by
the MathWorks. The MathWorks is the leading developer and supplier of technical computing software in
the world. A privately held corporation, The Mathworks employs more than 1000 people and serves more
than 500,000 customers. Full details of this company are available online at http://www.mathworks.com.
MATLAB is comprised of an integrated development environment that is recognized for its ease of use,
high­productivity development toolset, integrated math libraries and graphical interfaces. Williams
develops its modeling software in MATLAB's fourth­generation interpreted language to capitalize upon
these productivity features.
MATLAB is not, however, generally recognized as an optimal choice for use with applications that require
substantial computing time. Researchers and application developers seeking high computing performance
have typically relied upon multithreaded, third­generation languages such as Fortran, C, or C++, and the
compilers that accompany these languages. When parallel calculations are desired, researchers and
application developers rely upon distributed communication libraries such as MPI, PVM or OpenMP. Such
1
Manager, Portfolio Engineering, Williams.
© Williams Companies Inc. All rights reserved. Page 1 11/29/2008

2. third­generation languages, particularly when coupled with distributed communication libraries, can
produce models that exhibit much higher performance than MATLAB based models.
The time required to write and optimize models in third­generation languages, however, can be many
multiples of the time required to develop MATLAB based models. Further, the time required to learn a
third­generation language, its underlying compiler, and the communication library required for parallel
calculations can be substantial. High computing performance using third generation languages, in
summary, demands a significant up­front investment of personnel time.
A fourth­generation programming language such as MATLAB requires much less time to learn and use.
But MATLAB is a single­threaded, interpreted language and thus will not offer the performance of a
compiled, multithreaded third­generation language. If computation times are prohibitive using MATLAB,
one must find some manner in which to optimize the MATLAB solution. Because MATLAB is single­
threaded implicit parallelization is not possible. Explicit parallelization, however, is available. In
particular, coarse­grain parallelization of MATLAB code can harness the power of multiple computers on a
network.
Williams' financial analysis and enterprise risk management models are computationally demanding.
Execution of models on a desktop system would require significant wait times and would preclude iterative
modeling scenarios. To counter this performance obstacle, Williams has created a massively parallel
computational platform upon which to perform its MATLAB based computations. As with the models
themselves, Williams has chosen to write the software that controls its parallel platform in the MATLAB
language. The combination of MATLAB and a massively parallel platform provides a unique High
Performance Computing (HPC) environment that enables Williams to perform timely, valuable analyses in
support of its businesses and customers.
The software suite that controls Williams' HPC environment has been termed the HPC Toolbox. The HPC
Toolbox enables one to perform large­scale, massively parallel computations from the MATLAB user
interface using only MATLAB commands or graphical user interfaces. Using the HPC Toolbox, Williams
benefits from both the productivity of MATLAB and the computational performance of parallel processing.
The HPC Toolbox exhibits the following key features:
• The HPC Toolbox is written purely in the MATLAB language and requires no external C, C++ or
Fortran coding. MATLAB users can thus quickly learn the HPC Toolbox syntax and can continue
to leverage MATLAB’s many productivity tools in the HPC environment.
• The HPC Toolbox requires neither an underlying communication library such as MPI nor a system
dependent communication protocol. As such, the HPC Toolbox is platform agnostic and is
presently operational on the Linux, Solaris and Windows operating systems.
• The HPC Toolbox may be used on shared memory, distributed shared memory (NUMA) or
distributed memory (e.g., Beowulf) platforms. This Toolbox thus fits naturally alongside
MATLAB in any existing corporate or educational computing environment where multiple
computers are available for use.
• The HPC Toolbox provides a complete user API for parallel processing of MATLAB scripts,
methods or models. A developer is able to create a MATLAB model on a desktop computer and
then quickly adapt that model for execution on multiple CPUs using this flexible, intuitive API.
• The HPC Toolbox provides a complete API for scheduling, submittal and retrieval of batch jobs,
and runtime monitoring of jobs. As such, users of the HPC Toolbox are sheltered from the
© Williams Companies Inc. All rights reserved. Page 2 11/29/2008

3. intricacies of the actual parallel processing hardware, its operating system, and its network.
• The HPC Toolbox is capable of spawning arbitrary applications on its parallel nodes. As such, the
HPC Toolbox provides a robust API with which to perform parallel computations using any
application.
© Williams Companies Inc. All rights reserved. Page 3 11/29/2008

4. Current Capabilities
At Williams, the HPC Toolbox resides on a 650­CPU Linux system. This system has become a true HPC
environment for a diverse community of MATLAB developers and users. A usage chart showing user
CPU utilization in days of computing time is presented below for the year 2003. In this chart, the vertical
bars present CPU usage for a given day of a given month. A CPU day corresponds to a single CPU fully
utilized for 24 hours.
The above chart shows that Williams' HPC system routinely provides its users 300 CPU days per business
day. Restated, on any given business day this system provides an entire year's worth of CPU time.
The HPC Toolbox has achieved some notable accomplishments:
• The HPC Toolbox has accommodated dozens of users submitting hundreds of batch jobs daily.
• The HPC Toolbox is configured to execute multiple jobs simultaneously, with each job scaling to
hundreds of CPUs. The system has proven capable of handling in excess of 2200 concurrent
MATLAB processes.
• User­friendly Graphical User Interfaces (GUIs) provide an intuitive interface to the processing
system. These GUIs allow for job queue monitoring, job runtime monitoring, job results retrieval,
and job diagnostics examination.
• Since April 2002 the HPC Toolbox has processed 60,000 batch jobs, accumulating nearly five
billion user CPU seconds of processing time. The average batch job submitted to the system
required 15 minutes elapsed time, employed 100 CPUs, and accumulated 24 hours of CPU time.
Thus, the system allows users to obtain model results within minutes rather than tomorrow.
• Nearly one hundred individuals across two continents with varying MATLAB experience have
quickly learned to use the HPC Toolbox. These users have employed the system interactively, in
batch, and with automated, overnight scheduling.
© Williams Companies Inc. All rights reserved. Page 4 11/29/2008

5. It should be mentioned that installation of the HPC hardware and development of the HPC Toolbox were
completed in less than one year's time. Such a short development period to obtain the above outlined
results is testament to the productivity one can achieve with MATLAB.
© Williams Companies Inc. All rights reserved. Page 5 11/29/2008

6. Architecture
From its inception, the HPC Toolbox has targeted three design objectives:
1. For end­users: the system should be easy to employ. End­users should be able to access the
system using only MATLAB, with no additional desktop software required. User should not be
required to employ additional languages or data formats to access the system.
2. For developers: the system should allow MATLAB model development primarily or entirely upon
a single CPU desktop system. Distribution of computations to multiple CPUs should be achieved
with simple calls to a library of HPC Toolbox methods.
3. For designers of the HPC Toolbox: the system should readily adapt to changing business needs
and growing user sophistication.
The HPC Toolbox architecture helps achieve these goals by dividing the HPC Toolbox computational
environment into five parts, as per the following diagram.
In the above figure, dozens of users submit and retrieve batch jobs to a single Gateway system.
Approximately ten parent nodes pull jobs from the Gateway and distribute their workload to hundreds of
child nodes. A file server that is accessible to the parent and child nodes provides a simple yet highly
efficient communication mechanism with which the parent and child nodes can pass data and commands.
The five parts of the HPC Toolbox computational environment are next discussed in detail.
A user prepares an HPC batch job on his or her desktop workstation using the usual MATLAB interface.
Preparation consists of constructing a job object that contains information about the user, the job’s
commands and input and output data, and job meta­data such as job priorities and process priorities. A
final preprocessing step submits the job object to the HPC Gateway, which queues the job for parallel
processing.
The HPC Gateway acts as a buffer between users and the parallel processing system. Users never need to
log into the parallel processing system, and in fact are disallowed from accessing any HPC components
beyond the Gateway. Users instead submit batch jobs to the parallel system via the Gateway, which stores
the job input files and metadata. Using the Gateway as an intermediary, users can monitor their job’s
progression through the batch queues, monitor the runtime activity of their executing job, and retrieve their
completed job results from the Gateway. The Gateway thus plays a simple, yet prominent role in the HPC
architecture: it queues user jobs, stores job results, acts as a firewall between the parallel processing system
and the user network, and coordinates administrative activity for the entire system.
Once a job is submitted to the Gateway, one of the HPC parent nodes will detect the job. A daemon script
running on the parent performs the actual job detection, and moves the job to a directory that is private to
that parent node. The daemon script then initiates MATLAB upon the parent node itself, whereby
MATLAB assumes control of the job processing. The parent node's MATLAB process matches the job to
a set of child node resources, initiates MATLAB upon those child nodes, loads the user's job object and
© Williams Companies Inc. All rights reserved. Page 6 11/29/2008

7. distributes the object’s data and commands to the child nodes. Upon completion of the job’s commands on
the child nodes, the parent node's MATLAB process collates the resulting output and stores the output back
on the Gateway for user retrieval. MATLAB then terminates on the child nodes and on the parent node,
and the parent node daemon renews a search for jobs on the Gateway.
When executing a job, the parent node's MATLAB process performs numerous administrative tasks. Job
progression details are constantly forwarded to the Gateway so that the user may view runtime details of
the job. Warning and error messages, if any, are also forwarded to the user. If a child node happens to fail,
the parent will detect its failure and take that node offline. Lastly, the parent node will record job log files
and accounting files for administrative purposes.
The child nodes comprise the fourth part of the HPC environment. At Williams, child nodes number in the
hundreds, and evidence suggests that HPC architecture can scale to more than 2000 nodes. Child nodes can
be viewed as ‘obedient children’ for they do only as instructed by their parents. Child nodes listen for
commands from their parents and then act upon each command as it is received. Parent commands issued
to children fall into three categories: assign data into a child workspace; evaluate a command in a child
workspace; and save data out of a child workspace. Conceptually, the relationship between parent and
child is simple. However, this model enables children to execute arbitrary MATLAB commands of any
complexity.
Lastly, a network file server provides a common data storage area for the parent and child nodes. This fifth
and last part of the HPC environment provides a simple, efficient, and highly dependable communication
mechanism with which the parent and child nodes can pass data and commands. In implementing this
message­passing technique, the HPC Toolbox uses multithreaded file management commands of the
operating system. For example, the Toolbox employs shell wildcards and symbolic linking to quickly
replicate and destroy files. And importantly, this message­passing technique is robust and readily
adaptable to new code developments. The sequence of events by which a parent issues a command to its
children is illustrated in the figure below.
When data, rather than just commands,
is being transferred between nodes the
above sequence of messages changes
only slightly. A parent would send an
array to its children, for example, by
placing a MAT­file containing that
array onto the file server. The
accompanying command file would
instruct each child to load that MAT­
file into its workspace. Using this
technique, one can send data of any
type (doubles, strings, objects, etc.) to
the child nodes.
The architecture outlined above is easy
to deploy, easy to develop models upon,
and adaptable to evolving business
conditions. In fact, the architecture has
continued to evolve. To optimize use of
the network when transferring large
data files to its child nodes, a parent node will automatically bypass the file server by broadcasting the data
to its child nodes using a multicast (UDP protocol) technique. Using this technique, users routinely
broadcast 100 Mbyte data sets to the child nodes. The transmittal requires less than a minute using the
broadcast technique but would require thirty minutes if it were to proceed through the file server.
© Williams Companies Inc. All rights reserved. Page 7 11/29/2008
Data Passing Schematic

8. Depending upon the type of information being transferred between parent and child, the corresponding
message can be either text or binary formatted, small or large, private to a node or shared among many
nodes. A single job will typically create, move, and destroy thousands of messages. Large jobs may create
and move hundreds of gigabytes of data. The messaging and its optimization, however, are transparent to
the user.
The HPC Toolbox architecture targets the so­called embarrassingly parallel problem set. All child nodes
perform the same tasks, as instructed by their parent node. This architecture thus corresponds to a Single
Instruction Multiple Data (SIMD) model. Children nodes communicate with only their parent node; in this
architecture child nodes do not communicate with one another. It is possible to adapt the current
architecture to a MIMD model (Multiple Instruction, Multiple Data) but Williams presently has no need for
that model.
User Interface
HPC Toolbox Graphical User Interfaces
(GUIs) provide queue monitoring, runtime
job monitoring, job retrieval, job
management, and job diagnostics to aid
HPC Toolbox users. Many users of the
HPC Toolbox, in fact, interact with the
system using only the GUIs that
accompany the system. The convenience
and ease of use that GUIs provide
significantly lower the barrier to entry that
accompanies most HPC environments.
The primary GUI, as pictured to the right,
is the Job Control Panel for the HPC
Toolbox. This GUI is the entry­point to the
HPC system for all users. New users
appreciate the simplicity of the GUI's
design, while advanced users have
routinely added new capabilities to the
GUI. For all users, this GUI provides a
visual means of monitoring and managing
jobs that they have submitted to the HPC
system.
The Job Control Panel enables a user to
perform numerous job­related management
tasks:
1. View a job's metadata.
2. View a job's diagnostics.
3. Delete a job.
4. Retrieve a job's output data.
5. Export a job object to the MATLAB desktop workspace.
6. Refresh the job list.
7. Monitor and manage queued jobs.
8. View the runtime activity of any running job.
9. View the cluster status.
© Williams Companies Inc. All rights reserved. Page 8 11/29/2008

9. 10. Initiate a desktop job listener, which awaits completion of the requested job.
© Williams Companies Inc. All rights reserved. Page 9 11/29/2008

10. The HPC Control Panel launches additional GUIs to perform many of its tasks. For example, a queue
monitor GUI, pictured below, allows one to monitor a job’s progression through the queue system. All
queued jobs are noted, along with several key details about the jobs.
GUIs such as those illustrated here are yet another productivity feature that MATLAB provides to its user
base. New users are more comfortable with these GUIs than with the underlying MATLAB commands.
Advanced users, however, may choose to forego the GUIs in favor of the (typically) more powerful
underlying MATLAB commands. In either case, such GUIs are easy to construct and maintain. Further,
these GUIs may be configured to present different options to different people, so that advanced features are
hidden from new users.
The GUIs being presented here aid the user in managing submitted jobs. To actually submit a job, the user
can employ a simple command line interface that consists of just five commands:
• initialize(hpcjob) to create a new job object
• assignin(hpcjob,'variable',value) to add data to the job object
• evalin(hpcjob,'command') to add one or more commands to the job object
• store(hpcjob,'variable') to store output data into the job object
• submit(hpcjob,options) to send the job object to the Gateway
Because the variables and command strings of the above methods are job­specific, the HPC Toolbox does
not offer a GUI to create and submit jobs. Of course, users can hide these commands underneath job­
specific GUIs, and many users do just that.
A job diagnostics GUI, pictured above, provides a complete, point and click interface with which users can
examine warnings or errors that might have occurred during execution of a job. Child nodes are presented
in an interactive, color­coded matrix. By right­clicking on a child node button, one may request to see the
© Williams Companies Inc. All rights reserved. Page 10 11/29/2008

11. commands which the node executed, any warnings or errors that occurred during that processing, CPU and
elapsed timings of each command, input/output metrics, and a complete runtime log of activity on that
node. By clicking on the 'Parent Log' button, one will obtain the same information for the parent node.
And by clicking on the 'Reduced Diagnostics' button, one will obtain an aggregate summary of child node
diagnostic information.
A runtime job monitor, pictured next, provides the user with an up to date view of his running job. The
most recently issued parent command is displayed at the bottom right­hand corner of the figure. The status
of each child node is color coded in a display matrix: nodes are considered available for processing, busy
processing, or unavailable. The monitor is updated by the job’s parent node throughout the job’s lifetime,
and users can choose between a visual monitor and a simple text monitor to follow a job’s progression.
© Williams Companies Inc. All rights reserved. Page 11 11/29/2008

12. Other graphics and GUI capabilities enable a system administrator to interrogate the parallel system for
status and health information. The combination of MATLAB and Linux allows one to easily collect,
collate, and present a great deal of system performance information, accounting information, or diagnostic
information. For example, the following figure presents a 24­hour aggregate summary of system memory
usage for each of 600 system nodes.
Other Features
The HPC Toolbox incorporates several other features that enhance its usability, reliability, and
expandability.
• Object Oriented design: the HPC Toolbox is comprised of five classes and approximately 70
methods. This modular design facilitates ease of maintenance and code enhancements.
• Multiple layers of parallelism: the HPC Toolbox provides coarse­grain parallelism across system
nodes. Using the Linux operating system, the HPC Toolbox also allows a single CPU to
concurrently process multiple MATLAB processes. Alternatively, multiple CPUs on a single
node are able to time­share one or more MATLAB processes, thereby reducing the I/O and
memory access latency of multiple processes. The HPC Toolbox currently allows two instances
of MATLAB to simultaneously execute on a single node, which is the limit allowed by
Mathworks licensing.
• Optimized communication techniques: the HPC Toolbox automatically incorporates advanced
communication methodologies to optimize data transfer across a cluster. Users are able to
efficiently transfer data (arrays, objects, constants, etc.) across a cluster without regard for the size
of the data. The HPC Toolbox examines a user's data to determine if the data is small enough to
use standard file system transfer techniques, or if the data is large enough to warrant advanced
© Williams Companies Inc. All rights reserved. Page 12 11/29/2008

13. broadcasting techniques. The HPC Toolbox efficiently transfers files ranging from several bytes
to hundreds of Megabytes, with recipients ranging from a single node to hundreds of nodes.
• Security: the HPC Toolbox Gateway acts as a firewall across which users or intruders cannot pass.
For further security, the Gateway itself can be configured such that it cannot on its own initiative
pass data or commands to the parallel processing system; rather, the parent nodes must initiate all
transactions with the Gateway.
• User abstraction: MATLAB on the parallel processing system executes all of its jobs as a single,
predefined user. This abstract user can be issued restricted privileges within the parallel
processing system, thereby further increasing system security.
• Resiliency: parent nodes are effective at detecting non­responsive child nodes. Such child nodes
are placed into an offline nodes list that an administrator can monitor. Offline nodes are not used
by new jobs, thereby promoting system stability. The system administrator monitors the offline
nodes list, performs investigations and repairs, and then returns offline nodes to an online status.
• Multiple source code environments: the HPC Toolbox allows a user to specify that a job should
run in a predefined source code environment. For example, an installation may allow users to
choose from Alpha code, Beta code, or production code.
• Dynamic source code environment: the HPC Toolbox allows users to dynamically update source
code in a prescribed set of environments. For example, users may be permitted to dynamically
update source code in an Alpha environment. This feature facilitates model development.
• Multiple MATLAB versions: the HPC Toolbox is configured to allow the user to choose between
multiple MATLAB versions. This feature allows developers to test models on new releases of
MATLAB, and protects legacy models by allowing them to run on older releases.
• Job­level and process­level priorities: jobs and their child­node processes may be given user or
administrator specified priorities. Such specifications ensure that critical jobs can be initiated and
completed within an acceptable timeframe.
• Job scheduling: jobs may be scheduled to initiate at a prescribed time. This feature may be used to
ensure, for example, that new data arrives before a job begins.
• Single­CPU job scheduling: the HPC Toolbox allows users to request a single node for job
execution, rather than a full cluster. This feature allows users to access the memory, CPU and
disk of a high­capacity HPC server without tying up an entire cluster of nodes.
• Log file recording and retrieval: users can request a runtime log of their jobs and retrieve the
resulting log files for debugging purposes. Presently the HPC Toolbox logs parent and child node
commands, warning and error messages, input/output statistics, and performance metrics. Users
can easily retrieve this information and examine it either in aggregate or in node­level detail.
• Ease of system administration: numerous, simple scripts provide essential system administrative
tasks to the HPC Toolbox. These scripts assist one in updating source files, collating accounting
files, and tracking system usage.
Latest Features
© Williams Companies Inc. All rights reserved. Page 13 11/29/2008

14. The HPC Toolbox continues to evolve. The Toolbox's multi­layer architecture allows one to replace
components as system demands increase or business demands evolve. Late in 2003 we replaced the
Matlab­based job scheduling system with a sophisticated Java­based system. The new scheduling system
records node information into a database and then matches users' job resource requirements with available
hardware. System information recorded into the database includes hardware characteristics, monitoring
alerts, allowed job and process priorities, and number of currently executing processes. When a user job is
queued into the database, the scheduler examines the job's metadata to determine, among other things, the
number of requested CPUs, the requested job and process priorities, and the requested start time. When the
required resources become available the scheduler places the job onto the parallel system for execution.
Late in 2003 we also enhanced the communication technique by which the HPC Toolbox transmits small
messages (the cmd, begin, and done messages of the Data Passing Schematic, page 5) between parent and
child nodes. For these messages, the HPC Toolbox now uses TCP/IP sockets, thereby bypassing the file
server. Use of TCP/IP sockets offloads the file server and improves the performance of the cluster. Use of
sockets also opens the HPC Toolbox to a new level of functionality. In particular, this new technique will
allow the Toolbox to address MIMD problems with an API similar in functionality to MPI.
Summary
The HPC Toolbox builds upon the success and power of MATLAB, bringing High Performance
Computing to a new class of customers and problems. The HPC Toolbox is particularly well suited to
solving stochastic problems such as those found in the engineering and finance industries. Use of the HPC
Toolbox enables Williams to support its business units with financial analysis and enterprise risk
management capabilities, thereby creating customer efficiencies and shareholder wealth.
The HPC Toolbox meets its design targets. The system is easy to use, easy to develop models upon, and
easy to evolve with changing business needs. Scaling in size from tens of nodes to hundreds, with
thousands of nodes realizable, the HPC Toolbox can be sized to meet virtually any computing need.
Acknowledgements
Any success that the HPC Toolbox has enjoyed results from the concerted efforts of numerous contributors.
Edward V. Byrns2
conceived of the Toolbox and produced its proof of concept. Romel Chatterjee3
designed the Toolbox GUIs. The Java­based job scheduler was designed and implemented by Pablo
Galiasso and John Urban. The HPC Toolbox user community has provided extensive debugging,
suggestions and enhancements to the Toolbox. Lastly, numerous Williams' network administrators and
system administrators have provided invaluable contributions to the HPC Toolbox's feature set and
stability.
Additional Information
For further information about the HPC Toolbox, interested parties are encouraged to contact this paper's
author:
Michael Patterson
One Williams Center
Resource Center
PO Box 2848
Tulsa, OK 74172
Phone: (918) 583­8582
Email: Michael.Patterson@williams.com
2
Senior Manager, Quantitative Analytics. Now with Citadel Investment Group.
3
Quantitative Analyst. Now with ConocoPhillips.
© Williams Companies Inc. All rights reserved. Page 14 11/29/2008