UC San Diego's BioBurst cluster provides additional resources for bioinformatics workloads through an I/O accelerator, FPGA-based computational accelerator, and 672 additional compute cores. The I/O accelerator uses 40TB of flash memory to alleviate small block/file I/O issues in bioinformatics applications. The FPGA accelerator can perform genome analysis tasks much faster than standard hardware. The resources are integrated with UC San Diego's existing high performance computing cluster to improve research productivity and address bottlenecks in genomics and other bioinformatics applications and pipelines.

1. UC San Diego Research Computing
BioBurst Cluster for Bioinformatics
11/13/2017
RonHawkins
Director of Industry Relations
TSCC Program Manager

2. Acknowledgement
• This material is based upon work supported by the
National Science Foundation under Grant No. ACI-
1659104
• Any opinions, findings, and conclusions or
recommendations expressed in this material are those of
the author(s) and do not necessarily reflect the views of
the National Science Foundation

3. UC San Diego’s Triton Shared
Computing Cluster (TSCC)
• Provide research HPC, primarily
for UC San Diego campus users
• Hybrid business model:
“condo” (buy in) and “hotel”
(pay-as-you-go) options
• Officially launched in June 2013
• Currently at ~300 nodes (~6,000
cores, not incl. GPU cores)

4. Program Objectives
• Provide a robust research computing program at UC San
Diego that
1. Enhances research competitiveness
2. Provides a medium- to large-scale computing resource
• Access to a larger resource than most PI’s could afford just for their lab
3. Is readily accessible (without competitive proposals or long wait
times)
4. Follows best practices at other universities
5. Is cost-effective and energy-efficient
• Alternative to “closet clusters”
6. Provides for professional administration/maintenance (freeing up
postdocs & grad students to focus on research)

5. Condo Model Mechanics
Condo Cluster
Group 1’s
Purchased Nodes
Group 2’s
Purchased Nodes
TSCC Group Purchases
Common Infrastructure
Nodes are purchased directly and
are property of the lab/group or
funding agency
Common equipment is purchased
via assessment of a one-time, per-
node “infrastructure fee”
• Once purchased nodes are in place, group
may run on purchased nodes or entire
cluster according to usage rules
• Labs/groups are assessed an annual per-
node operations fee (~27% of total cost)
Copyright 2017, Regents of the University of California, All Rights Reserved

6. TSCC Operations
Condo Users Hotel Users• Purchase Nodes
• Pay initial
“infrastructure fee”
• Pay annual
operations fee
($495)
• Can run on
purchased nodes or
entire cluster
• Purchase Time (2.5
c per core-hour e.g.
$250 for 10,000
core-hours)
• Run only on hotel
nodes
Copyright 2017, Regents of the University of California, All Rights Reserved

7. Node Characteristics
• Nodes comprise dual-socket, Sandy Bridge,
Haswell and Broadwell processors, 16-28
cores/node, and 64-128GB main memory
• Mixed 10GbE and QDR InfiniBand interconnect
(BioBurst cluster is EDR)
• GPU nodes are mix of NVIDIA GTX
980/1080/1080Ti and Titan-X GPUs

8. CC* BioBurst for TSCC
• NSF Campus Cyberinfrastructure (CC*) Award
• Objective is to augment TSCC with capabilities
to address the growing bioinformatics workload
• Award value: $500K
• Award start date: Feb 1, 2017

9. Objective
• The overall objective of BioBurst
• Improve research productivity by providing a separately-
scheduled campus computing resource designed to
address performance bottlenecks found in a class of
applications important to campus researchers, including
genomics, transcriptomics, and other bioinformatics
pipelines.
• Specifically, the small block / small file I/O
problem with codes such as GATK – see
references 1-4

10. Key Features
More specifically, BioBurst will incorporate the following major
components and operational characteristics:
• A software-defined I/O accelerator appliance with 40 terabytes of
non-volatile (“flash”) memory and software designed to alleviate the
small-block/small-file random access I/O problem characteristic of
many bioinformatics codes;
• Derived from Exascale program “burst buffer” technology
• An FPGA-based computational accelerator node (Edico Genome
DRAGEN) that has been shown to conduct demultiplexing, mapping,
and variant calling of a single human genome in 22 minutes as
compared to ~10 hours on standard computing hardware [2];
• 672 commodity (x86) computing cores providing a separately
scheduled resource for running various bioinformatics
computations;
• Integration with a Lustre parallel file system, which supports
streaming I/O, and has the capacity to stage large amounts of data
characteristic of many bioinformatics studies; and,

11. Overall Architecture
Copyright 2017, Regents of the University of California, All Rights Reserved

12. More Detail
Copyright 2017, Regents of the University of California, All Rights Reserved
(DDN logo Copyright DDN, Edico Genome logo Copyright Edico Genome)
DDN IME System

13. IME® I/O Acceleration Architecture
13
OBJECT STORAGE &
TAPE LIBRARIES
ARCHIVE STORAGE
DISK/TAPE TIER
IME’s Active I/O Tier, is inserted
right between compute and the
parallel
file system
IME software intelligently
virtualizes disparate
NVMe SSDs into a
single pool of shared memory that
accelerates
I/O, PFS & Applications
ACTIVE I/O TIER
IME
I/O APPLIANCES
COMPUTE
CLUSTER
Slide used with permission of DDN

14. DRAGEN Bio-IT Platform
14
Ultra-Rapid Genomic Analysis
Platform
• The power of the platform makes it possible
to perform an extremely fast and accurate
secondary analysis, which results in
significant cost savings.
• Pipelines currently available include Whole
Genome, Exome, RNASeq, Methylome,
Microbiome, Joint Genotyping, Population
Calling, Cancer and more.
• DRAGEN accepts FASTQ/BCL, and
BAM/CRAM files as input and provides
output in standard BAM/VCF/gVCF file
formats.
• DRAGEN offers supreme flexibility of data
analysis with both the ability to stream BCL
data directly from sequencer storage.
• DRAGEN also offers the ability to convert
BCL to FASTQ or BAM/CRAM. DRAGEN can
read and output compressed or
uncompressed files.
DRAGEN is a fully reconfigurable FPGA-based platform that can be reconfigured in
seconds to host a number of different highly optimized analysis pipelines.
Slide used with permission of Edico Genome

15. Science Use Cases
• Investigating Genetic Causes and Treatments for
Pediatric Brain Disease – Dr. Joe Gleeson, UCSD
• Understanding the Role of Gene Expression in
Development and Aging – Dr. Gene Yeo, UCSD
• Revolutionizing the Development of Human
Vaccines – Dr. Richard Scheurmann (J. Craig
Venter Institute) and Dr. Robert Sinkovits (SDSC)
• Molecular Basis of Neuropsychiatric Disorders –
Dr. Jonathan Sebat, UCSD

16. Status
• All equipment has been received and installed in the
cluster
• New cluster nodes up and running
• Still working on software/scheduler integration for
Dragen node and IME system
• Expect full production by December

17. Questions?
Contact:
Ron Hawkins
rhawkins@sdsc.edu

18. References
1. Kovatch, P., Costa, A., Giles, Z., Fluder, E., Cho, H., Mazurkova, S., Big Omics Data
Experience. Proceedings of the International Conference for High Performance
Computing, Networking, Storage and Analysis, SC ’15, pages 39:1– 39:12, New York,
NY, USA, 2015. ACM.
2. P. Carns, S. Lang, R. Ross, M. Vilayannur, J. Kunkel, and T. Ludwig, “Small-file access
in parallel file systems,” in Proceedings of IEEE International Parallel and Distributed
Processing Symposium, 2009.
3. Lin, H., Ma, X., Feng, W., Samatova, N., “Coordinating Computation and I/O in
Massively Parallel Sequence Search,” in IEEE Transactions on Parallel and
Distributed Systems, Vol. 22, No. 4, April, 2011.
4. Lee, S., Min, H., Yoon, S., “Will solid-state drives accelerate your bioninformatics? In-
depth profiling, performance analysis, and beyond,” in Briefings in Bioinformatics,
Vol. 17, Issue 4, pp. 713-727, 1 Sep 2015.