Reproducible pipelines using Nextflow

1. U-DAWS: Developing and
maintaining reproducible
workflows for bioinformatics data
1
Venkat Malladi
=: : katatonikkat
: vsmalladi
Nextflow Meeting
2019-09-20
CRG, Barcelona

2. Bioinformatics Department
2
A computer science dept in the heart of a medical center

3. Bioinformatics & BICF
3

4. Data and processing requirements are diverse:
–Functional Genomics (ChIP-seq, ATAC-seq, RNA-seq)
–Single cell Genomics
–Clinical Genomics and Precision Medicine
– Proteomics
–Machine Learning and Image Analysis
Research groups with different scientific questions and data types
We are not big enough to run all analysis for the university and people want
to learn to run their own analysis
Bioinformatics @ UTSW
4

5. Goals of the BICF
5
Initial data analysis for common assays
Standardized quality metrics and output as a starting point for further
analysis
 Provide easily reproducible workflows that can be cited by community
Easy to user interface for a diverse population (Post-docs, PhD-students,
Clinicians)
Deployable pipelines for community use

6. Today we’re going to cover:
Principles of Pipeline Development
Astrocyte: Web Interface for Nextflow Pipelines
Migrating to Cloud: Azure
Outline
6

7. Principles of Pipeline Development
7
1. Steps have defined input and output
2. Defined computational resource allocation
3. Parallelization across samples
4. Serial steps executed only if input criteria are met
5. Workflow reproducibility
6. Common steps shared between different analyses
7. Visualization of workflow output to aid researchers in understanding complex data

8. Steps have defined input and output
8

9. Defined computational resource allocation
9
• Take advantage of native config files
• Develop multiple profiles

10. 10
Defined computational resource allocation per process

11. Defined computational resource allocation Containers
11

12. Defined computational resource allocation Containers
12

13. Parallelization across samples
13
Define a design file to define parallelization and groupings

14. Parallelization across samples
14
Define complicated groupings

15. Parallelization across samples
15
Define process metadata

16. Parallelization across samples
16
By Sample

17. Parallelization across samples
17
By Experiment

18. Serial steps executed only if input criteria are met
18

19. Workflow reproducibility: CI
19
• Unit Tests
• Process specific logic
• Integration Tests
• End-to-end run with multiple real datasets
• Validate each process output

20. Workflow reproducibility: CI
20
Branch Specific Tests
Master/Merge Specific Tests

21. Workflow reproducibility: CI Test
21

22. Workflow reproducibility: CI Unit Testing
22

23. Workflow reproducibility: CI Test Process output
23

24. Workflow reproducibility: Versioning
24
2. 1. 3
MAJOR: Increase when you make
changes that break the parameter
structure of previous versions
MINOR: Increase when you add
functionality
PATCH: Increase when you make
changes that are small bug fixes.

25. Workflow reproducibility: Versioning
25
Zenodo:
• Overview of Pipeline

26. Workflow reproducibility: Versioning
26
Zenodo:
• Overview of Pipeline
• Unique Document Object Identifier

27. Workflow reproducibility: Versioning
27
Zenodo:
• Overview of Pipeline
• Unique Document Object Identifier
• Display all versions of the pipeline

28. Common steps shared between different analyses
28
• Each process is self contained script in : BASH, R, PERL
or Python
• Each process has its own Docker or Singularity image
• Plan to use Nextflow Modules to share steps (e.g.
ChIP-seq and ATAC-seq)

29. Visualization of workflow: MultiQC
29

30. Visualization of workflow: MultiQC
30

31. Visualization of workflow: Shiny Server
31

32. Principles of Pipeline Development: Overview
32

33. Today we’re going to cover:
Principles of Pipeline Development
Astrocyte: Web Interface for Nextflow Pipelines
Migrating to Cloud: Azure
Outline
33

34. Astrocyte: Web Interface for Nextflow Pipelines
34
Allows groups to give easy-access to their analysis pipelines via the web
Standardized Workflows
Simple Web Forms
Online documentation &
results visualization*
Django Webfamework
Workflows run on HPC cluster without developer or user needing cluster knowledge

35. Astrocyte: List Pipelines
35

36. Astrocyte: New Projects
36

37. Astrocyte: User Interface
37
SELECT YOUR FILES
Library
Type
Assembly
Skip
Options

38. Astrocyte: User Interface YAML
38

39. Astrocyte: Finished Pipeline
39
Export
ShinyApp

40. Today we’re going to cover:
Principles of Pipeline Development
Astrocyte: Web Interface for Nextflow Pipelines
Migrating to Cloud: Azure
Outline
40

41. Azure: Overview
41
Virtual Network
Avere vFXT
Cluster
On-Premises
HPC Cluster
/data/nas
Burst Compute
Virtual HPC Cluster
HPC Job Scheduling, Licensing, & Instrumentation:
- Azure Batch
- CycleCloud
- Virtual Machine Scale Sets
Bring your own Scheduler & Software:
- IBM LSF, Slurm, Grid Engine
- BioBuilds, Bioconda
- Homegrown/Custom
Azure
subscription
ExpressRoute
Gateway
Subnet Subnet
Azure Blob
/data/blob
<- NFS/SMB ->

42. Azure: CycleCloud
42
 Setup the programs that you want installed by default on nodes using the Azure CLI
–We chose Nextflow and Singularity
–Works remarkably similar to building Docker/Singularity images, but BASH
 All steps are run on the VM that you have setup, not on the master node of the HPC cluster
 Setup the installation instructions on a file
 Can only be tested once machine is started
 Best practice – One image per-program, like Docker/Singularity
 https://docs.microsoft.com/en-us/azure/cyclecloud/tutorials/modify-cluster-template
 https://docs.microsoft.com/en-us/azure/cyclecloud/tutorials/deploy-custom-application

43. Singularity script Example
43

44. 44
 From the Azure CLI:
 Enter the required information
 Setup SSH Keypair
 Deploy CycleCloud following the instructions:
–https://docs.microsoft.com/en-us/azure/cyclecloud/quickstart-install-cyclecloud

45. Azure: Setup Cluster
45
 We used SLURM
–Note: SLURM has unique setup requirements not present in some of the other options
–Counts CPUs as machines
–100 machines means
 Setup any alerts for cost
 Add the programs that you want to have available
–These will be installed on node creation each time

46. Azure: Future Work
46
 Setup images for compute nodes, rather than application installs
–Speed up the process of node launches
–Consistency in the node programs
 Further refinement of HTC/HPC nodes
–Customizations for each pipeline
 Streamline data transfer from current system to CycleCloud

47. Thank you
47
Microsoft:
• Shimail Gillani
• Jer-ming Chia
BICF
Not Pictured:
• David Trudgian
• Jeremy Mathews

48. Questions
48
Questions?