Common Workflow
Language Viewer
https://view.commonwl.org
Mark Robinson
Stian Soiland-Reyes
Michael Crusoe
Carole Goble
http://slides.com/markrobinson/cwlviewer
http://orcid.org/0000-0002-8184-7507
http://orcid.org/0000-0001-9842-9718
http://orcid.org/0000-0002-2961-9670
http://orcid.org/0000-0003-1219-2137
@soilandreyes
These slides are licensed under a
Creative Commons Attribution 4.0 International License
2017-07-22 BOSC 2017
ISMB/ECCB 2017, Prague
Difficulties in sharing CWL
Workflow spread over multiple text files
https://github.com/common-workflow-language/
workflows/tree/master/workflows/lobSTR/lobSTR-workflow.cwl
Difficulties in sharing CWL
Often missing:
Detailed description (what, how)
Provenance (who, where, when)
Versioning (which)
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow
inputs:
#SCWRL
scw_script: File
scw_input_pdb_path: File
scw_output_pdb_path: string
scw_mutation: string
scw_scwrl_path: string
scw_log_path: string
scw_error_path: string
#PDB2GMX
p2g_script: File
p2g_output_gro_path: string
p2g_output_top_path: string
p2g_output_itp_path: string
p2g_output_top_tar_path: string
p2g_water_type: string
p2g_force_field: string
p2g_ignh: string
p2g_gmx_path: string
p2g_log_path: string
p2g_error_path: string
# ...
We need better incentives!
#!/usr/bin/env cwl-runner
cwlVersion: "cwl:draft-3"
class: Workflow
inputs:
- id: p1
type:
- "null"
- type: array
items: File
description: list of files containing the first end of paired end reads in fasta or fastq format
- id: p2
type:
- "null"
- type: array
items: File
description: list of files containing the second end of paired end reads in fasta or fastq format
- id: output_prefix
type: string
description: prefix for output files. will output prefix.aligned.bam and prefix.aligned.stats
- id: reference
type: File
description: "lobSTR's bwa reference files"
- id: rg-sample
type: string
description: Use this in the read group SM tag
- id: rg-lib
type: string
description: Use this in the read group LB tag
- id: strinfo
type: File
description: File containing statistics for each STR.
- id: noise_model
type: File
description: File to read noise model parameters from (.stepmodel)
secondaryFiles:
- "^.stuttermodel"
outputs:
- id: bam
type: File
source: "#samindex/bam_with_bai"
- id: bam_stats
type: File
source: "#lobSTR/bam_stats"
- id: vcf
type: File
source: "#allelotype/vcf"
- id: vcf_stats
type: File
source: "#allelotype/vcf_stats"
hints:
- class: DockerRequirement
dockerLoad: https://workbench.qr1hi.arvadosapi.com/collections/download/qr1hi-4zz18-x2ae13tsx5jqg8d/1nduktd8dpvhdpgsva82lje0i710kgzb6rttks5jldx7s2y7k9/7e0c0ae3bf4e70442f9b8eee816ec23426d9e1169a2925316e5c932745e21613.tar
dockerImageId: 7e0c0ae3bf4e70442f9b8eee816ec23426d9e1169a2925316e5c932745e21613
steps:
- id: lobSTR
run: lobSTR-tool.cwl
inputs:
- { id: p1, source: "#p1" }
- { id: p2, source: "#p2" }
- { id: output_prefix, source: "#output_prefix" }
- { id: reference, source: "#reference" }
- { id: rg-sample, source: "#rg-sample" }
- { id: rg-lib, source: "#rg-lib" }
outputs:
- { id: bam }
- { id: bam_stats }
- id: samsort
run: samtools-sort.cwl
inputs:
- { id: input, source: "#lobSTR/bam" }
- { id: output_name, default: "aligned.sorted.bam" }
outputs:
- { id: output_file }
- id: samindex
run: samtools-index.cwl
inputs:
- { id: input, source: "#samsort/output_file" }
outputs:
- { id: bam_with_bai }
- id: allelotype
run: allelotype.cwl
inputs:
- { id: bam, source: "#samindex/bam_with_bai" }
- { id: reference, source: "#reference" }
- { id: output_prefix, source: "#output_prefix" }
- { id: noise_model, source: "#noise_model" }
- { id: strinfo, source: "#strinfo" }
outputs:
- { id: vcf }
- { id: vcf_stats }
Workflow Inputs
Workflow Outputs
reference
allelotype
lobSTR
rg-samplep1p2 output_prefixrg-lib strinfonoise_model
vcfvcf_statsbam_stats bam
samsort
samindex
"aligned.sorted.bam"
Visualisation = quick and easy understanding
CWL Viewer
https://view.commonwl.org
View a Workflow
https://view.commonwl.org/workflows/github.com/common-workflow-language/
workflows/tree/master/workflows/lobSTR/lobSTR-workflow.cwl
Interactivity
https://view.commonwl.org/workflows/github.com/ProteinsWebTeam/ebi-
metagenomics-cwl/tree/master/tools/rRNA_selection.cwl
Download and Share
Visualisation Images
(various formats)
GraphViz DOT source
Research Object Bundle/
Origin Information
COMMON
WORKFLOW
LANGUAGE
A Research Object bundles and relates digital
resources of a scientific experiment or
investigation:
Data used and results produced in experimental
study
Methods employed to produce and analyse that
data
Provenance and settings for the experiments
People involved in the investigation
Annotations about these resources, to improve
understanding and interpretation
bundle.zip
├── mimetype
├── .ro
│ └── manifest.json
└── workflow
├── completeWorkflow.cwl
├── concatFasta.cwl
├── fastqc.cwl
├── generateRReports.cwl
├── qiimeAddMetadata.cwl
├── readPair.yml
├── uparseChimeraCheck.cwl
├── uparseDerepWorkAround.cwl
├── uparseFastqMerge.cwl
├── uparseFilter.cwl
└── uparseTruncateLength.cwl
{
"@context" : [ "https://w3id.org/bundle/context" ],
"id" : "/",
"manifest" : [ "manifest.json" ],
"createdOn" : "2017-02-27T11:48:54.986Z",
"createdBy" : {
"uri" : "http://view.commonwl.org",
"name" : "Common Workflow Language Viewer"
},
"authoredBy" : [ {
"uri" : "https://github.com/grbot",
"name" : "Gerrit Botha"
}, {
"uri" : "https://github.com/mr-c",
"name" : "Michael R. Crusoe"
}, {
"name" : "Ubuntu"
}, {
"uri" : "https://github.com/mepstein-gh",
"name" : "Milt Epstein"
}, {
"uri" : "https://github.com/ShakunBaichoo",
"name" : "ShakunBaichoo"
} ],
"importedFrom" : "https://github.com/h3abionet/h3abionet16S/tree/70498847fd6bfc63f44c59c748233cac7f8dd85d/workfl
"importedOn" : "2017-02-27T11:48:54.986Z",
"importedBy" : {
"uri" : "http://view.commonwl.org",
"name" : "Common Workflow Language Viewer"
},
"aggregates" : [ {
"uri" : "/workflow/completeWorkflow.cwl",
"mediatype" : "text/x-yaml",
"createdOn" : "2017-02-27T11:48:55.21Z",
"authoredBy" : [ {
"uri" : "https://github.com/grbot",
"name" : "Gerrit Botha"
}, {
"uri" : "https://github.com/mr-c",
"name" : "Michael R. Crusoe"
}, {
"uri" : "https://github.com/mepstein-gh",
"name" : "Milt Epstein"
}, {
"uri" : "https://github.com/ShakunBaichoo",
"name" : "ShakunBaichoo"
} ],
"retrievedFrom" : "https://github.com/h3abionet/h3abionet16S/blob/70498847fd6bfc63f44c59c748233cac7f8dd85d/wor
"conformsTo" : "https://w3id.org/cwl/v1.0",
"bundledAs" : {
"uri" : "urn:uuid:fbc4f48a-b111-4e1b-b6b9-d6a97c7c7f90",
"folder" : "/workflow/"
}
}, {
"uri" : "/workflow/concatFasta.cwl",
"mediatype" : "text/x-yaml",
Provenance, attribution
Who made which part of the workflow
when
CWL Viewer Research Objects
BOSC 2017
Birds of a
Feather
Today 12:20-13:40
BoF #2: Common Workflow Language
community meetup
Farah Zaib Khan & Stian Soiland-Reyes:
ResearchObject.org & CWL integration
COMMON
WORKFLOW
LANGUAGE
Discovery of Workflows
https://view.commonwl.org/workflows
In the Works!
JSON API Use of the cwltool reference
implementation for parsing
Input labelling
More import options
And More!
Workflow Inputs
Workflow Outputs
p2
lobSTR
p2
rg-lib
rg-lib
p1
p1
reference
reference
allelotype
reference
output_prefix
output_prefix
output_prefix
noise_model
noise_model
rg-sample
rg-sample
strinfo
strinfo
vcf_stats vcfbambam_stats
samsort
input
samindex
bam
input
"aligned.sorted.bam"
output_name
cwl-svg
https://view.commonwl.org/
Join us!
https://github.com/
common-workflow-language/cwlviewer
Poster (A-124):
https://doi.org/10.7490/f1000research.1114375.1

2017-07-22 Common Workflow Language Viewer

  • 1.
    Common Workflow Language Viewer https://view.commonwl.org MarkRobinson Stian Soiland-Reyes Michael Crusoe Carole Goble http://slides.com/markrobinson/cwlviewer http://orcid.org/0000-0002-8184-7507 http://orcid.org/0000-0001-9842-9718 http://orcid.org/0000-0002-2961-9670 http://orcid.org/0000-0003-1219-2137 @soilandreyes These slides are licensed under a Creative Commons Attribution 4.0 International License 2017-07-22 BOSC 2017 ISMB/ECCB 2017, Prague
  • 2.
    Difficulties in sharingCWL Workflow spread over multiple text files https://github.com/common-workflow-language/ workflows/tree/master/workflows/lobSTR/lobSTR-workflow.cwl
  • 3.
    Difficulties in sharingCWL Often missing: Detailed description (what, how) Provenance (who, where, when) Versioning (which) #!/usr/bin/env cwl-runner cwlVersion: v1.0 class: Workflow inputs: #SCWRL scw_script: File scw_input_pdb_path: File scw_output_pdb_path: string scw_mutation: string scw_scwrl_path: string scw_log_path: string scw_error_path: string #PDB2GMX p2g_script: File p2g_output_gro_path: string p2g_output_top_path: string p2g_output_itp_path: string p2g_output_top_tar_path: string p2g_water_type: string p2g_force_field: string p2g_ignh: string p2g_gmx_path: string p2g_log_path: string p2g_error_path: string # ... We need better incentives!
  • 4.
    #!/usr/bin/env cwl-runner cwlVersion: "cwl:draft-3" class:Workflow inputs: - id: p1 type: - "null" - type: array items: File description: list of files containing the first end of paired end reads in fasta or fastq format - id: p2 type: - "null" - type: array items: File description: list of files containing the second end of paired end reads in fasta or fastq format - id: output_prefix type: string description: prefix for output files. will output prefix.aligned.bam and prefix.aligned.stats - id: reference type: File description: "lobSTR's bwa reference files" - id: rg-sample type: string description: Use this in the read group SM tag - id: rg-lib type: string description: Use this in the read group LB tag - id: strinfo type: File description: File containing statistics for each STR. - id: noise_model type: File description: File to read noise model parameters from (.stepmodel) secondaryFiles: - "^.stuttermodel" outputs: - id: bam type: File source: "#samindex/bam_with_bai" - id: bam_stats type: File source: "#lobSTR/bam_stats" - id: vcf type: File source: "#allelotype/vcf" - id: vcf_stats type: File source: "#allelotype/vcf_stats" hints: - class: DockerRequirement dockerLoad: https://workbench.qr1hi.arvadosapi.com/collections/download/qr1hi-4zz18-x2ae13tsx5jqg8d/1nduktd8dpvhdpgsva82lje0i710kgzb6rttks5jldx7s2y7k9/7e0c0ae3bf4e70442f9b8eee816ec23426d9e1169a2925316e5c932745e21613.tar dockerImageId: 7e0c0ae3bf4e70442f9b8eee816ec23426d9e1169a2925316e5c932745e21613 steps: - id: lobSTR run: lobSTR-tool.cwl inputs: - { id: p1, source: "#p1" } - { id: p2, source: "#p2" } - { id: output_prefix, source: "#output_prefix" } - { id: reference, source: "#reference" } - { id: rg-sample, source: "#rg-sample" } - { id: rg-lib, source: "#rg-lib" } outputs: - { id: bam } - { id: bam_stats } - id: samsort run: samtools-sort.cwl inputs: - { id: input, source: "#lobSTR/bam" } - { id: output_name, default: "aligned.sorted.bam" } outputs: - { id: output_file } - id: samindex run: samtools-index.cwl inputs: - { id: input, source: "#samsort/output_file" } outputs: - { id: bam_with_bai } - id: allelotype run: allelotype.cwl inputs: - { id: bam, source: "#samindex/bam_with_bai" } - { id: reference, source: "#reference" } - { id: output_prefix, source: "#output_prefix" } - { id: noise_model, source: "#noise_model" } - { id: strinfo, source: "#strinfo" } outputs: - { id: vcf } - { id: vcf_stats } Workflow Inputs Workflow Outputs reference allelotype lobSTR rg-samplep1p2 output_prefixrg-lib strinfonoise_model vcfvcf_statsbam_stats bam samsort samindex "aligned.sorted.bam" Visualisation = quick and easy understanding
  • 5.
  • 6.
  • 7.
  • 8.
    Download and Share VisualisationImages (various formats) GraphViz DOT source Research Object Bundle/ Origin Information
  • 9.
    COMMON WORKFLOW LANGUAGE A Research Objectbundles and relates digital resources of a scientific experiment or investigation: Data used and results produced in experimental study Methods employed to produce and analyse that data Provenance and settings for the experiments People involved in the investigation Annotations about these resources, to improve understanding and interpretation
  • 10.
    bundle.zip ├── mimetype ├── .ro │└── manifest.json └── workflow ├── completeWorkflow.cwl ├── concatFasta.cwl ├── fastqc.cwl ├── generateRReports.cwl ├── qiimeAddMetadata.cwl ├── readPair.yml ├── uparseChimeraCheck.cwl ├── uparseDerepWorkAround.cwl ├── uparseFastqMerge.cwl ├── uparseFilter.cwl └── uparseTruncateLength.cwl { "@context" : [ "https://w3id.org/bundle/context" ], "id" : "/", "manifest" : [ "manifest.json" ], "createdOn" : "2017-02-27T11:48:54.986Z", "createdBy" : { "uri" : "http://view.commonwl.org", "name" : "Common Workflow Language Viewer" }, "authoredBy" : [ { "uri" : "https://github.com/grbot", "name" : "Gerrit Botha" }, { "uri" : "https://github.com/mr-c", "name" : "Michael R. Crusoe" }, { "name" : "Ubuntu" }, { "uri" : "https://github.com/mepstein-gh", "name" : "Milt Epstein" }, { "uri" : "https://github.com/ShakunBaichoo", "name" : "ShakunBaichoo" } ], "importedFrom" : "https://github.com/h3abionet/h3abionet16S/tree/70498847fd6bfc63f44c59c748233cac7f8dd85d/workfl "importedOn" : "2017-02-27T11:48:54.986Z", "importedBy" : { "uri" : "http://view.commonwl.org", "name" : "Common Workflow Language Viewer" }, "aggregates" : [ { "uri" : "/workflow/completeWorkflow.cwl", "mediatype" : "text/x-yaml", "createdOn" : "2017-02-27T11:48:55.21Z", "authoredBy" : [ { "uri" : "https://github.com/grbot", "name" : "Gerrit Botha" }, { "uri" : "https://github.com/mr-c", "name" : "Michael R. Crusoe" }, { "uri" : "https://github.com/mepstein-gh", "name" : "Milt Epstein" }, { "uri" : "https://github.com/ShakunBaichoo", "name" : "ShakunBaichoo" } ], "retrievedFrom" : "https://github.com/h3abionet/h3abionet16S/blob/70498847fd6bfc63f44c59c748233cac7f8dd85d/wor "conformsTo" : "https://w3id.org/cwl/v1.0", "bundledAs" : { "uri" : "urn:uuid:fbc4f48a-b111-4e1b-b6b9-d6a97c7c7f90", "folder" : "/workflow/" } }, { "uri" : "/workflow/concatFasta.cwl", "mediatype" : "text/x-yaml", Provenance, attribution Who made which part of the workflow when
  • 11.
  • 12.
    BOSC 2017 Birds ofa Feather Today 12:20-13:40 BoF #2: Common Workflow Language community meetup Farah Zaib Khan & Stian Soiland-Reyes: ResearchObject.org & CWL integration COMMON WORKFLOW LANGUAGE
  • 13.
  • 14.
    In the Works! JSONAPI Use of the cwltool reference implementation for parsing Input labelling More import options And More! Workflow Inputs Workflow Outputs p2 lobSTR p2 rg-lib rg-lib p1 p1 reference reference allelotype reference output_prefix output_prefix output_prefix noise_model noise_model rg-sample rg-sample strinfo strinfo vcf_stats vcfbambam_stats samsort input samindex bam input "aligned.sorted.bam" output_name cwl-svg
  • 15.