2011 07-16 SCUFL2 - because a workflow is more than its definition (BOSC 2011)


Published on

Scufl2 – because a workflow is more than its definition

Stian Soiland-Reyes, Alan R Williams, Stuart Owen, David Withers and Carole Goble

School of Computer Science, University of Manchester, UK

Project site: http://www.taverna.org.uk/

Source code: https://github.com/mygrid/scufl2 http://taverna.googlecode.com/

License: GNU Lesser General Public License (LGPL) 2.1

Taverna is a scientific workflow management system that has gained popularity amongst scientists in bioinformatics, chemistry, astronomy and other domains. Since its inception in 2003, Taverna's workflow language has evolved from Scufl which was designed for the FreeFluo workflow engine. Scufl defines a directed acyclic graph of the data flow between processors (services) which receive and produce data at input/output ports. Scufl workflows are stored in a lightweight XML format.

t2flow adapted these concepts for the Taverna 2 workflow engine, including new capabilities such as extensible execution control. The t2flow XML format was created as a serialisation of internal Java beans, which unfortunately makes it difficult to consume or produce by non-Taverna clients compared to Scufl. There have also been growing demands for bundling a workflow with related resources such as example data and semantic annotations. Addressing these issues culminated in forming a new workflow language Scufl2, which is presented here.

By adapting linked data technology and preservation methodologies for research objects (http://bit.ly/dKiGz7), Scufl2 is not only a platform-independent workflow language which can be inspected, modified, created and executed by third-party tools and systems; it is extensible to allow the capture of workflow execution inputs and outputs, reference data sets, provenance, annotations, documentation, publications, alternative formats and representations, and even embedded binary service implementations.

Scufl2 comes with a Java API independent of Taverna that can be used for programmatic access to read and write Scufl2 workflow bundles. A workflow bundle is a structured ZIP-file based on Adobe UCF, ePub OCF and the Open Document Format (ODF), with the workflow definitions included as XML Schema-conformant documents which are also valid RDF/XML. The constraints from the schema allow clients to read and write Scufl2 workflow definitions as regular structured XML. Richer RDF-enabled clients may link workflow definitions with external resources and additional annotations stored in separate RDF files in the bundle, perhaps using vocabularies such as Dublin Core.

The workflow structure is defined using an OWL ontology, and annotated with URIs so that third parties can form semantic statements about any component of any Scufl2 workflow, for instance to say that a particular service produces outputs of a certain type, or that a given data link was added by a different researcher to the builder of the rest of the workflow. An unzipped workflow bundle can be made public using any standard web server and become part of the Linked Open Data cloud.

Semantic annotations and a manifest for the bundle declare the purpose of, and links between the different components forming a workflow. This allows third parties to extract and append annotations about data and services used by the workflow. The evolution of the workflow definition itself can also be included in these annotations, with references to previous versions and authors.

In recent years, Taverna has become an interesting target for researchers investigating published workflows, such as those found on myExperiment. Exploring these workflow definitions using the Scufl2 API can reveal compatibilities and implicit data types of web services, providing new annotations for service registries like BioCatalogue. Sites and tools can add additional descriptions of the workflow and the experiment directly to the Scufl2 workflow bundle without affecting execution behaviour or requiring any d

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

2011 07-16 SCUFL2 - because a workflow is more than its definition (BOSC 2011)

  1. 1. Stian Soiland-Reyes<br />myGrid, School of Computer Science<br />University of Manchester, UK<br />http://taverna.org.uk/<br />Scufl2: Because a workflow is more than its definition<br />BOSC 2011<br />Vienna, 2011-07-16<br />http://www.mygrid.org.uk/<br />http://www.wf4ever-project.org/<br />
  2. 2. Taverna workflows<br />A set of (local and remote) services to analyze or manage data<br />Nested workflows are also services<br />Data-links connects services<br />i.e. output from service A is input to service B and C<br />Describes the desired dataflow instead of process coordination<br />Automatic iterations<br />Can customize list handling and control links<br />Taverna 2.3 released 2011-07-14<br />
  3. 3. Taverna workflow features<br />Nested workflows<br />Reuse existing components<br />Implicit iterations<br />With customizable list handling<br />Pipelining<br />Process partial iteration results early<br />Parallelisation<br />Run as soon as data is available<br />Retries, failover, looping<br />For stability and conditional testing<br />Plugin-extensible execution control<br />Ideas: caching, error detection, dynamic service lookup<br />
  4. 4. Extensible workflow engine<br />New service types/protocols<br />Execution control like looping/branching/start/stop<br />Service discovery at runtime<br />
  5. 5. Taverna’s existing wf formats<br />Taverna 1: SCUFL 2004-2007<br />Lightweight XML format<br />Extensions in arbitrary formats (“Put your XML element here”)<br />Parsed and written by many 3rd parties<br />Taverna 2: t2flow 2007-2011<br />Supports control and service extensions of T2<br />Annotation system<br />‘Heavy’ XML serialisation of Java beans<br />Tricky to use by 3rd parties<br />
  6. 6. SCUFL2 motivations<br />Easy to use for third-parties<br />Semantic web/linked open data compatible identifiers and ontology<br />Extensible in a controlled manner<br />Plugins should provide a minimal ‘schema’<br />But parsers should not need to know about plugin<br />Bundle relevant resources<br />Input/output data, provenance, scripts, binaries<br />Annotations on anything<br />Reproducible workflow results<br />
  7. 7. Research objects<br />Reusable – used as part of new study;<br />Repurposeable – reuse the pieces in a new (and different) study. Substitute alternative data sets, methods;<br />Repeatable – repeat the study, possibly years later;<br />Reproducible – a special case of repeatability with a complete set of information/results to work towards;<br />Replayable – go back and see what happened;<br />Referenceable – cite in publications;<br />Revealable – provenance and audit;<br />Re-interpretable – crossing boundaries;<br />Respectful – appropriate credit and attribution;<br />Retrievable – discover and acquire.<br />
  8. 8. Research Object vision<br />ROs: Aggregationsto support sharing/publication<br />Incorporating methods, data, people<br />Research Objects will allow us to conduct research inways that are<br />Efficient: cheaper to borrow than recreate<br />Effective: larger scale through reuse<br />Ethical: Benefiting wider communities, not just individuals<br />Could I have a copy of your Research Object please?<br />
  9. 9. WfEver project<br />Architecture/implementation for workflow preservation, sharingand reuse<br />Research Object models<br />Workflow Decay, Integrity and Authenticity<br />Workflow Evolution and Recommendation<br />Provenance<br />Driven by Use Cases: Biology & Astronomy<br />…technological infrastructure for the preservation and<br />efficient retrieval and reuse of scientific workflows in a range<br />of disciplines.<br />
  10. 10. What is SCUFL2?<br />File format for (Taverna) workflows and data<br />Specification (wiki)<br />Ontology (OWL)<br />Schema (XSD)<br />API (Java +Ruby?)<br />Conversion Tool (Java)<br />
  11. 11. SCUFL2 bundles<br />ZIP archive<br />mimetype<br />META-INF/manifest.xml<br />http://myexperiment.org/workflow/1337<br />META-INF/container.xml<br />workflowBundle.rdf<br />annotations/myExperiment.rdf<br />http://example.com/largeData<br />workflow/HelloWorld.rdf<br />inputs.rdf<br />workflow/someNestedWf.rdf<br />inputs/data.txt<br />profile/tavernaServer.rdf<br />outputs.rdf<br />http://example.net/tavernaServer<br />outputs/result.txt<br />profile/tavernaCloud.rdf<br />resources/script.pl<br />run/provenance.rdf<br />http://example.org/binary<br />
  12. 12. Bundle manifest<br />“Structured” zip-file <br />(can be unpacked to be exposed on the web)<br />the-workflow-bundle.scufl2<br />mimetype<br />application/vnd.taverna.scufl2.workflow-bundle<br />Self-documenting media type (OpenOffice ODF, ePub OCF, Adobe UCF) – for tools like file(1) and mime magic<br />Root file: Primary document of the bundle (ePub OCF/Adobe UCF). <br />Alternative representations of same workflow bundle allowed (Turtle, JSON, HTML, etc)<br />META-INF/container.xml<br /><container version="1.0“xmlns="urn:oasis:names:tc:opendocument:xmlns:container"><br />    <rootfiles><br />        <rootfilefull-path="workflowBundle.rdf" media-type="application/rdf+xml" /><br />        <!-- <rootfile full-path="workflowBundle.ttl"<br />         media-type="text/turtle" /> Alternative repr. --> <br />    </rootfiles><br /></container><br />
  13. 13. Workflow bundle<br />Unique identifier which can be used as prefix for all relative references in bundle<br />workflowBundle.rdf<br />Suggests main workflow and main profile, but executor could (e.g. by parameter) run a different workflow or profile<br /><rdf:RDF xmlns="http://ns.taverna.org.uk/2010/scufl2#" xmlns:rdf=".."xmlns:rdfs=".."xmlns:xsi=".."xsi:schemaLocation="http://ns.taverna.org.uk/2010/scufl2# http://ns.taverna.org.uk/2010/scufl2/scufl2.xsd .."xsi:type="WorkflowBundleDocument" xml:base="./"><br />      <WorkflowBundle rdf:about=""><br />          <name>HelloWorld</name><br />          <sameBaseAsrdf:resource= "http://ns.taverna.org.uk/2010/workflowBundle/28f7c..a0ef731/" />        <br />          <mainWorkflow rdf:resource="workflow/HelloWorld/" /><br />          <workflow><br />            <Workflow rdf:about="workflow/HelloWorld/"><br />              <rdfs:seeAlso rdf:resource="workflow/HelloWorld.rdf" /><br />           </Workflow> <!-- plus each nested <Workflow> --><br />          </workflow><br />          <mainProfile rdf:resource="profile/tavernaWorkbench/" /><br />          <profile><br />           <Profile rdf:about="profile/tavernaWorkbench/"><br />             <rdfs:seeAlso rdf:resource="profile/tavernaWorkbench.rdf" /><br />            </Profile> <!– plus optional alternative <Profile>--><br />          </profile><br />          <rdfs:seeAlso rdf:resource="annotation/user_annotations.rdf" /><br />          <rdfs:seeAlso rdf:resource="annotation/myExperiment-wf-765.rdf" /><br />      </WorkflowBundle><br /></rdf:RDF><br />Additional (non-executable) annotations and metadata<br />
  14. 14. Technologies and standards<br />OpenOffice ODF manifest<br />ePub OCF/Adobe Universal Container Format (UCF) container rootfile, mimetype<br />RDF/XML and OWL structure and annotations<br />XML Schema for ‘structured’ RDF/XML<br />Open Annotation Collaboration (OAC)?<br />ORI-ORE? Aggregation<br />Open Provenance Model (OPM)?<br />
  15. 15. Status:<br />Want people to try it out<br />Open for suggestions and feedback<br />Beta to be relased in Autumn 2011<br />Native format for upcoming Taverna 3 in 2012<br />Load/Save plugin for Taverna 2.3 2011 Q4<br />Future plans<br />Explore further research object & provenance<br />Converting to/from wf systems like Galaxy?<br />Alpha<br />
  16. 16. http://www.mygrid.org.uk/<br />
  17. 17. http://www.wf4ever-project.org/<br />
  18. 18. Acknowledgements<br />Funded by: EPSRC EP/G026238/1; European Commission’s 7th FWP FP7-ICT-2007-6 270192;  FP7-ICT-2007-6 270137<br />
  19. 19. More information<br />Poster 21<br />http://www.taverna.org.uk/<br />http://www.wf4ever-project.org/<br />http://www.mygrid.org.uk/<br />http://github.com/mygrid/scufl2<br />