This document describes a real-time workflow for analyzing streaming synchrotron data using high-performance computing resources. Synchrotron experiments produce large amounts of data that need to be reconstructed in real-time. The workflow includes data acquisition from multiple simulated beamlines, distribution to compute nodes, tomographic reconstruction using TraceX, denoising using TomoGAN, and visualization of results. A demonstration of this workflow is being run on Argonne Leadership Computing Facility's Theta supercomputer to process streaming data from 16,000 cores in real-time and provide reconstructed volumes and feedback to experiments.

1. TUESDAY, NOVEMBER 19, 2019
REAL-TIME ANALYSIS OF
STREAMING SYNCHROTRON
DATA
SC’19 TECHNOLOGY CHALLENGE
DEMOerhtjhtyhy
TEKIN BICER
Assistant Computer Scientist
Data Science and Learning Division, CELS
X-Ray Science Division, APS
Argonne National Laboratory
Denver, Colorado US
COLLABORATORS
StarLight & Northwestern University
Argonne Leadership Computing Facility, ANL
Advanced Photon Source, ANL
University of Chicago
Northern Illinois University

2. SYNCHROTRON EXPERIMENTS
 Synchrotron light sources help scientific
experiments of many fields
– Studying internal morphology of
materials/samples with very high spatial and
temporal resolutions
 Real-time analysis of synchrotron experiments
– Change data acquisition for dynamic systems
– Adjust experimental parameters on the fly
– Detect errors early in experiments
– Enables smart and efficient experimentation
 High performance network and compute resources
are necessary
2

3. TOMOGRAPHIC DATA ACQUISITION AND
ITERATIVE RECONSTRUCTION
3
Data AcquisitionIterative Tomographic Reconstruction
advanced object and data models and can provide better image
quality than does FBP on a limited number of projections.
Although many variations exist, the basic iterative recon-
struction method involves three major steps, as depicted in
Fig. 2. First, an initial guess of the volume object, which
might simply be an empty volume, is used to calculate the
simulated data through a forward model. Second, the simulated
data are compared with the measured data. Third, an update
of each model estimate is performed based on the employed
algorithm. Reconstruction of an object might require hundreds
of iterations, depending on the experiment and sample.
Theearliest and most basic form of iterativereconstruction is
the algebraic reconstruction technique (ART), which involves
solving a sparse linear system of equations in the form of
Af = p where p is the projection data, A is the forward
projection operator, and f is the unknown 3D object to be
determined. While ART provides satisfactory images and has
a fast convergence rate, the iterations must be stopped before a
deteriorating “salt and pepper” or “checkerboard” effect begins
to degrade the object estimate. A variation of the ART method
is the simultaneous iterative reconstruction technique (SIRT)
in which the updates to the solution are computed by taking
into account all rotation angles simultaneously in one iteration,
as follows:
f k+ 1
= f k
+ λAT
(p − Af k
). (3)
As in the ART method, a relaxation parameter λ can be
used to control convergence in certain cases. SIRT typically
produces better quality reconstructions than does ART and is
more robust to outliers in the measurement data. In our system,
we use more advanced iterative reconstruction algorithms.
However, the main computational steps remain the same.
Fig. 3: Reconstructed image of a shale sam
streamed projections: (a) fixed angle, offset=1
offset=5◦
; (c) optimized interleaved. The ra
[0,180)◦
.
processes; (2) the analysis system, which i
analysis and reconstruction of streaming d
controller, which analyzes reconstructed data
subsections, we explain each of these compo
A. Data Acquisition and Distribution
Data acquisition at current tomography b
cally performed with one of two methods: fix
or interleaved.
With fixed-angle rotation, acquisition sta
starting point and then increments by a spec
to a specified ending point. If, for example,
ending angles are 0◦
and 180◦
, respectively,
◦

4. HIGH-PERFORMANCE TOMOGRAPHIC IMAGE
RECONSTRUCTION
11,283
*M. Hidayetoglu, T. Bicer et al., Supercomputing 2019
*T. Bicer, D. Gursoy et al., Advanced Structural and Chemical Imaging 2017
Dataset: Dyer et al., Society for Neuroscience (eNeuro) 2017

5. SYSTEM OVERVIEWSYSTEM OVERVIEW

6. A REAL-TIME TOMOGRAPHIC RECONSTRUCTION
WORKFLOW (DATA ACQUISITION)
* V. De Andrade et al., Nanoscale 3D imaging at the Advanced Photon Source, SPIE’16
Continuous vs. Interleaved DAQ

7. A REAL-TIME TOMOGRAPHIC RECONSTRUCTION
WORKFLOW (DISTRIBUTOR)
.
..
Normalization
Distribution

8. A REAL-TIME TOMOGRAPHIC RECONSTRUCTION
WORKFLOW (TRACE-X)
* T. Bicer et al., Advanced Structural and Chemical Imaging, 2017
* T. Bicer et al., eScience, 2017
• * TraceX: A High-Throughput Tomographic
Reconstruction Engine for Large-Scale Datasets
• Sliding window with adjustable runtime params.
• Length (w), iteration (i), func. trigger freq (s).
• Reduction-based processing model
• Highly scalable and efficient
• Replicated reduction objects
• 32K cores on Mira, 64K cores on Theta

9. A REAL-TIME TOMOGRAPHIC RECONSTRUCTION
WORKFLOW (TOMOGAN: DENOISER)
* Z. Liu, T. Bicer et al., Deep Learning on Supercomputer, SC’19
* Z. Liu, et al., JOSA A (Under review)

10. A REAL-TIME TOMOGRAPHIC RECONSTRUCTION
WORKFLOW (VISUAL OUTPUTS)
Measurement
Normalized
Measurement
Reconstructed
Image (3D vol.)
Denoized
Image (3D Vol.)

11. DEMO SETUP
16K Cores100GigE Conn.
* 100GigE network enables simulation of 10 beamlines each with 10GigE detector
Gateway Node(s)
Argonne Leadership Computing Facility
Compute Nodes
(Theta)
...
Starlight / TC Booth @ SC Floor
Data Acquisition Simulator
and Data Transfer
100 GigE...
Visualization and
Feedback
Data Analysis Reconstructed
Volume
TomoGAN
Denoised
Volume
...
...
Sinogram
Distribution
...
Normalized
Measurement
Data
...
Volume
Gather
Measurement
Data
...
*Multiple
Beamline
Simulation
Volume
Rendering
(Cooley)
Reconstructed
Volumes
ØMQ

12. DEMO

13. THANKS
QUESTIONS?
Acknowledgement
• DSL & MCS: Zhengchun Liu, Raj Kettimuthu,
Joaquin Chung, Stefan Wild, Ian T. Foster
• ALCF: Venkatram Vishwanath, Mike Papka,
Suport & Allocation Teams
• APS: Doga Gursoy, Junjing Deng, Jeff Klug,
Vincent De Andrade, Pavel Shevchenko,
Francesco De Carlo, Stefan Vogt
• StarLight & Northwestern: Se-Young Yu, Jim
Chen, Fei Yeh, Joe Mambretti
• Many others…
Papers at SC’19
• Mert Hidayetoglu et al., “MemXCT: Memory-
Centric X-Ray CT Reconstruction with Massive
Parallelization”, Technical Paper
• Zhengchun Liu et al., “Deep Learning
Accelerated Light Source Experiments”, Deep
Learning on Supercomputers