Authors: Ivan Jimenez-Ruiz, Ricardo Gonzalez-Mendez, Alexander Ropelewski

1. Optimizing High Performance Big Data
Cancer Workflows
Iván L. Jiménez Ruiz
University of Puerto Rico, Río Piedras Campus
Ricardo González Méndez
University of Puerto Rico, School of Medicine
Alexander Ropelewski
Pittsburgh Supercomputing Center

2. Outline
Ø Aims
Ø Bridges architecture
Ø Workflow and software
Ø Timings and performance
Ø Recommendations for
similar workflows

3. Aims
Ø Implement workflows on Bridges supercomputer.
Ø Measure performance of file systems using NGS data.
Ø Determine where to run NGS programs to improve overall
workflow efficiency.
Ø Generate recommendations based on benchmarked data

4. 8 LM Nodes
Ø 3TB RAM per node
2 ESM Nodes
Ø 12TB RAM per node
752 RSM Nodes
Ø 128G RAM per node
Bridges – Phase 1
$RAMDISK
$LOCAL
3.7 TB
$RAMDISK
$LOCAL
15 TB
$RAMDISK
$LOCAL
64 TB
Pylon2
Ø 1 PB storage
Ø Slash2
Pylon1
Ø 1 PB storage
Ø Lustre

5. Phase 1 - File systems
$LOCAL
Ø Disk on the node
Ø Volatile
Ø Capacity B
/pylon1
Ø $SCRATCH storage
Ø Lustre
Ø Hi-speed
Ø Capacity C
$RAMDISK
Ø Memory storage
Ø Volatile
Ø Capacity A
/pylon2
Ø Archival storage
Ø Distributed, high-
storage capabilities
Ø Wide-area file system
Ø Capacity D

6. http://cole-trapnell-lab.github.io/cufflinks/manual/
Reads Reads
Align to genome
TopHat (Bowtie, Bowtie2), HISAT2
Merge transcript assemblies
Cuffmerge
Find differentially expressed genes & transcripts
Cuffdiff
Condition A Condition B
Mapped
Reads
Final Transcriptome Assembly
Assembled
Transcripts
Mapped
Reads
Mapped
Reads
Mapped
Reads
Differential Expression Results
Assemble transcripts
CufflinksAssembled
Transcripts
Quality Control

7. Software used in workflows
Bowtie 1&2
Ø Low memory
Ø Dozens of cores
Ø Demanding I/O
HISAT2
Ø Low memory
Ø Dozens of cores
Ø Demanding I/O
TopHat
Ø Low memory
Ø Dozens of cores
Ø Demanding I/O
FastQC
Ø Low memory
Ø Single core
Ø Demanding read I/O
Cufflinks
Ø Moderate to large
memory
Ø Dozens of cores
Ø Demanding I/O

8. Benchmark Data Sets
Two public transcriptomic datasets of glioblastomas (malignant
brain cancer) in human subjects were used:
Ø Primary Tumor (SRR3477485)
- Size: 2.2GB, 3,792 MBases
Ø Recurrent Tumor (SRR3477486)
- Size: 3.6GB, 6,474 MBases

9. Timings: Mapping – HISAT2
Time (minutes)
Partition – File system:
0 5 10 15 20 25 30 35
Recurrent Tumor (2nd Run)
Recurrent Tumor
Primary Tumor
RM-Pylon1
LM-Ram
RM-Local
RM-Pylon2
LM-Local
LM-Pylon2

10. Time (minutes)
Partition –
File system:
0 200 400 600 800 1000
Recurrent Tumor (2nd Run)
Recurrent Tumor
Primary Tumor
RM-Pylon1
LM-Ram
RM-Local
RM-Pylon2
LM-Local
LM-Pylon2
Timings: Mapping – TopHat (Bowtie1)

11. Time (minutes)
Partition – File system:
0 20 40 60 80 100 120
Recurrent Tumor
Primary Tumor
LM-Ram
RM-Local
RM-Pylon2
LM-Pylon2
Timings: Assembling – Cufflinks (TopHat, Bowtie1)

12. Summary: Quality Control, Aligning and Mapping
0 100 200 300 400 500 600
Tophat_Bowtie1
Tophat_Bowtie2
HiSat2
FastQC
RM-Pylon1
RM-Local
RM-Pylon2
LM-Local
LM-Pylon2
Time (minutes)
Partition –
File system:
Programs

13. Bridges changes since internship
Current:
Ø /pylon 5
Ø 10 PB Pylon storage
Ø LM Nodes: 8 4x16 cores
+ 34 4x20 cores
Ø ESM Nodes: 2 16x18
cores + 2 16x22 cores
Original:
Ø /pylon 1
Ø 2 PB Pylon storage
Ø LM Nodes: 8 4x16 cores
Ø ESM Nodes: 2 16x18
cores

14. Conclusions
Ø Bioinformatics workflows need to be reengineered regularly to
perform optimally on HPC systems.
Ø $LOCAL and $RAMDISK both performed comparably
- $RAMDISK had service usage charges associated
- Recommendation is to prefer $LOCAL over $RAMDISK
Ø /pylon1 performed similarly to both $LOCAL and $RAMDISK
- staging results and intermediate storage for output files.

15. Conclusions
Ø /pylon2 had:
- most variability
- worst performance
Our recommendation for using the file system on a similar
workflow would be to use /pylon2 for long-term storage
and archiving needs.

16. Acknowledgements
University of Puerto Rico, Rio Piedras Campus
Dr. Humberto Ortiz-Zuazaga
University of Pittsburgh
Department of Biomedical Informatics
Dr. David Boone
Dr. Uma Chandran
Funding:
• The NIH Big Data to Knowledge (BD2K) Enhancing Diversity in Biomedical Data Science Grant [9]
5R25MD010399-002 to the UPRRP
• The National Institutes of Health Minority Access to Research Careers (MARC) grant T36- GM-095335 and
National Institutes of Health Biomedical Technology Resource grant P41-GM-103712 to the Pittsburgh
Supercomputing Center (PSC)
• The computing resources used were provided through the Extreme Science and Engineering Discovery
Environment (XSEDE), which is supported by the National Science Foundation grant OCI-1053575.
• The Bridges supercomputer system at the PSC was acquired through NSF Award ACI-1445606.

17. Questions?
goo.gl/ifrr98