Developers don’t in general really ‘get’ classloading, not helped by confusing error messages when things go wrong. This is currently the main problem people hit when using Taverna’s code, they don’t understand the need for Raven and friends, and when they do appreciate it they have trouble using it.Applications using Taverna will already potentially have an environment in which they run, this could be an application server, from the classpath, within an Axis application or similar. We should allow this to remain, and cope with the potential classloading issues.If the developer just wants to load a workflow, run it and get the results they shouldn’t have to deal with all the configuration, editing etc. Conversely if the developer needs to drill down deeply into the models this should also be possible. A lot of this comes down to service selection within the platform and convenience wrappers for common tasksProvide a sensible API containing interfaces, do not let the developers get hold of the (changeable) implementation.The platform is used to provide Taverna to the developer, but it could be used to provide other functionality in a similar fashion such as services from the rest of OMII-UK. Make sure this is possible.
This code is pretty much ‘real’. While the reference service can throw a spectrum of exceptions they’re all runtime derived. This is partly so that code which can’t handle the exceptions anyway (and which would just re-throw them) doesn’t get cluttered. It’s the same approach used in the Spring project itself and while there are problems it’s not an entirely unreasonable way to go about things.
We’re ignoring monitoring, provenance capture and security here. The idea is that if the developer doesn’t want to bother with them they why force her? I’m assuming that there would be another method on the ‘enactor’ object that would allow us to specify a context explicitly, and that it’s in the context that things like the monitor would be specified.
We support optional dependencies by fetching jar files with raven but actually manually creating the class loader rather than relying on raven’s native mechanism. This means that we can reference artifacts in the plug-in description for ease of use, but that all jar files (artifact and non artifact) will be merged in a single URL class loader subclass carrying the plug-in metadata. Dependency analysis happens at the plug-in creation stage, adding an artifact to the dependencies for the plug-in will add all its transitive dependencies but from that point in the pom parsing is no longer used by the plug-in manager.
Shows dependency injection structure used when configuring a full plug-in manager instance.
The Taverna 2 Platform
Developing with the Taverna Code<br />An introduction to the T2 Platform<br />Tom Oinn, firstname.lastname@example.org, 7th October 2008<br />Updated 4th November 2008<br />
A warning!<br />In this presentation ‘Taverna’ is distinct from ‘The Taverna Workbench’.<br />‘Taverna’ should be interpreted as ‘the functionality provided by the core components’<br />Workflow creation and modification<br />Workflow enactment, monitoring, provenance<br />... and many more<br />‘Taverna’ in this context applies to the version 2 of the codebase in cases where a distinction is significant.<br />‘Service’ is used in the sense of a software component providing a service rather than any particular technology such as SOAP or REST. ‘Services’ here are plain Java objects.<br />
The Problem<br />Taverna (both 1.x and 2) is complex<br />Inherent complexity <br />Wide ranging functionality<br />Mechanistic complexity<br />Loaded through Raven to resolve dependency issues<br />Plug-in mechanism<br />Many different extension points<br />Requires unusual knowledge of classloading to resolve issues<br />Taverna is valuable<br />Developers want to use Taverna to:<br />Extend Taverna through plug-in implementations<br />Embed and use Taverna in other systems<br />The complexity obscures the value<br />
Application Support<br />Guiding principles:<br />Do not require knowledge of classloading<br />Do not require a specific application environment<br />Allow minimal code<br />Provide ‘exactly as much’ functionality as required<br />Explicitly support common usage patterns<br />Prevent dependency on implementation<br />Extensibility and openness<br />
What’s in an application?<br />An application using Taverna will contain:<br />The Taverna API jars<br />Support jars used by the application<br />The application code itself<br />Applies to all applications<br />Command Line<br />GUI<br />Web app<br />Axis service<br />...<br />Platform Libraries, T2 interfaces<br />Application Specific Libraries for your code<br />T2 Core API<br />Platform API, Spring etc.<br />Platform service APIs<br />...<br />...<br />...<br />Your Application Code<br />
Taverna Implementation Classes<br />The application must also contain...<br />Implementations of the enactor, workflow model and similar.<br />Loaded through Raven<br />Inherit and implement Taverna APIs from the parent class loader<br />Raven Class Loaders<br />Implementations of T2 Interfaces<br />Platform Libraries, T2 interfaces<br />T2 Core Impl<br />Implementation Dependencies<br />T2 Core API<br />Platform API, Spring etc.<br />Platform service APIs<br />
Plug-ins<br />Taverna is plug-in based<br />Each plug-in is isolated in its own class loader and contains:<br />Plug-in specific libraries<br />The extension point implementation code<br />The plug-in inherits the Taverna APIs from the parent class loader<br />These APIs include the extension point interfaces<br />A plug-in can contain many extension point implementations<br />Platform Libraries, T2 interfaces<br />Plug-in Class Loaders 1..n<br />T2 Core API<br />Platform API, Spring etc.<br />Platform service APIs<br />Plug-in specific libraries (not T2 APIs)<br />...<br />...<br />...<br />...<br />Plug-in Implementation<br />
Service Beans<br />Configured instances<br />Implement service APIs<br />Use T2 implementation classes from Raven<br />Act as a bridge between Raven driven implementations and application code<br />Think of as a ‘toolbox’ for working with Taverna facilities<br />Configured, linked and instantiated through Spring<br />XML configuration<br />Set shown here is indicative<br />We will provide more than this<br />3rd parties can also provide components here<br />Raven Class Loaders<br />Implementations of T2 Interfaces<br />Platform Libraries, T2 interfaces<br />Workflow Exporter<br />Reference Service<br />T2 Core Impl<br />Implementation Dependencies<br />T2 Core API<br />Platform API, Spring etc.<br />Platform service APIs<br />Workflow Parser<br />Enactor<br />Edit Kit<br />Monitor Factory<br />Activity Kit<br />...<br />Plug-in Service<br />XML Context Configuration<br />
Root Class Loader from Application, Web-App, Web Service ...<br />Raven Class Loaders<br />Platform Libraries, T2 interfaces<br />Application Specific Libraries for your code<br />Your Application Code<br />Implementations of T2 Interfaces<br />Workflow Exporter<br />Plug-in Class Loaders 1..n<br />T2 Core API<br />Platform API, Spring etc.<br />Platform service APIs<br />...<br />...<br />...<br />T2 Core Impl<br />Implementation Dependencies<br />Workflow Parser<br />Plug-in specific libraries (not T2 APIs)<br />...<br />...<br />...<br />...<br />Edit Kit<br />Activity Kit<br />Plug-in Implementation<br />Reference Service<br />Enactor<br />Classes only available through services<br />Classes available to all code<br />Service Beans<br />Monitor Factory<br />...<br />XML Context Configuration<br />Plug-in Service<br />Final Taverna Application Structure<br />
Using the platform 1-2-3<br />Initialize platform (once per application)<br />Downloads implementation code where required<br />Configures and instantiates service beans<br />Configures plug-in manager, loads plug-ins<br />Specified by Spring XML based configuration files<br />We will provide base configurations as part of the platform<br />Fetch appropriate service bean(s) by name<br />The standard Taverna platform will define a set of base services and their service names<br />Use service bean(s) to access Taverna functionality<br />Code purely against interface based APIs – service beans perform any object instantiation such as creation of new workflow model instances.<br />
An example use of the platform to load a workflow, acquire input data, run the workflow and handle the output.<br />I’m using pseudo-code here rather than Java but in general one line of pseudo-code will correspond to one line of Java, not taking into account standard Java constructs such as casting, type declarations etc. for brevity.<br />Also consider that the final ‘base’ set of service beans is not yet defined, so service names and interfaces are illustrative rather than definitive.<br />A short example<br />
initialization & workflow loading<br />plat = new T2Platform(“path/to/conf.xml”);<br />loader = plat.getBean(“t2.workflow.loader”);<br />workflow = loader.parseWorkflow(“http://foo.com/wf.xml”);<br />The ‘loader’ here is a platform service bean, a tool in the toolbox defined by the platform.<br />This particular tool can be used to create a new workflow model from an XML definition, in this case from a URL.<br />Important point – only one explicit object instantiation: the platform itself.<br />In a web application or web service this would already have been provided in the servlet context, but for conventional applications we need it explicitly. <br />Aside – the workflow model in Taverna 2 is read-only. If we wanted to modify the model we just loaded we would need an appropriate ‘workflow editor’ service. In our current code this is the Edits interface.<br />
Data registration<br />ref_service = plat.getBean(“t2.reference.service”);<br />input1 = ref_service.register(new File(“some.data”));<br />input2 = ref_service.register(“string value”);<br />‘ref_service’ is a Reference Service. This is the component of Taverna used to register data and obtain an internal identifier for it. This identifier is then used as input when running the workflow.<br />Two inputs are registered:<br />‘input1’ is data held in a file locally<br />‘input2’ is a literal string value<br />The real Reference Service interface is slightly more complex as it registers data asynchronously but the principle is the same. As before there is no direct object construction (other than the file), so no direct link to any implementation classes.<br />
Enactment<br />enactor = plat.getBean(“t2.enactor”);<br />wf_instance = enactor.createInstance(workflow);<br />wf_instance.push(“input1”,input1);<br />wf_instance.push(“input2”,input2);<br />Following the same pattern as before, we obtain an enactor service bean from the platform.<br />This tool creates workflow instances from workflow definitions. In this particular example we’re ignoring the workflow context, following the ‘minimal code’ principle the service bean API would include this simple version and use some sensible default context.<br />Having obtained the workflow instance we can push the previously registered data into its (in this case two) inputs. Workflows in T2 are pipeline based, so the act of pushing data in starts the enactment, there is no explicit ‘start’ operation (except for cases where there are no workflow inputs).<br />
Handling results<br />result = wf_instance.blocking_fetch(“output”);<br />System.out.println(ref_service.getAsString(result));<br />I’m taking some liberty with the ‘real’ API here but the principle is the same. We don’t have a blocking fetch method at the moment but maybe we should!<br />We call a method on the workflow instance to fetch the output which will block until it’s available.<br />The result is in the form of an internal identifier in the reference system, so we then use the reference service to render the data referenced by that identifier to a string and print it to the console.<br />
Conclusion<br />Going back to the ‘guiding principles’ does this code satisfy them?<br />There’s no reference to classloading, and the application would just run from the normal command line.<br />There’s no more code than required to do the job. This is a simple example and didn’t address various issues such as context configuration, but those issues weren’t in the summary of the application either so their corresponding absence in the code is another sign of success.<br />The code never refers to implementation types, it is therefore resilient in the face of implementation updates and changes.<br />The code is almost self documenting.<br />We can’t make any statements about the extensibility of the platform, but all the other requirements are satisfied.<br />
Plug-in Management In Taverna 1.x<br />Plug-in code is used to extend Taverna<br />Includes all ‘base’ functionality such as web service invocation, http references etc.<br />Plug-in development issues<br />No tooling support<br />Testing is problematic<br />Problems with optional maven dependencies<br />Can’t use libraries which are not maven artifacts<br />Plug-in usage issues<br />Insufficient metadata to property describe plug-in functionality<br />Plug-in management service API missing<br />Plug-ins have no presence in the code once loaded<br />
Plug-in Development<br />Provide tool support<br />Plug-in description generator<br />Plug-in verifier<br />Plug-in registry<br />Modified class loading strategy<br />Single class loader per plug-in<br />Allow mix of artifact and non-artifact jars<br />Use Raven to fetch artifact jars, but not to obtain class loaders<br />Testing is a special case of application development<br />See previous section<br />We can provide archetypes for integration tests using the platform to simplify testing<br />
Plug-in Description<br />Plain text description<br />Short name<br />Version, triple numeric with periods<br />Development status (enumeration, for example ‘alpha | beta | stable’ to be decided)<br />Author list<br />Author name<br />Author affiliation (optional)<br />Author email (optional)<br />Author URL (optional)<br />Author Icon (optional, 128x128 true colour PNG with transparency)<br />Tag list<br />Free text tag<br />Homepage URL (can be generated by tooling along with the page itself)<br />Related link list<br />Link description<br />Link URI (most likely HTTP but potentially other reference)<br />Icon, 128x128 true colour PNG with transparency<br />Plugin manager background image (arbitrary size, true colour PNG with transparency, used as a background for the plugin manager panel for this plugin)<br />Core API version targeted specified as an artifact (group, artifact, version)<br />SPI implementation list<br />Plain text description<br />Short name<br />
Plug-in Manager Implications<br />Most places in application code explicitly referencing artifacts and raven will now reference the plug-in manager<br />Code using SPI and instance registries is unchanged<br />Workflow serializer changes to reference plugin rather than artifact specifier<br />Plug-ins can be disabled on the fly<br />Plug-in manager appears as infrastructure bean in platform<br />Plug-in management functionality is generic<br />Part of the generic platform rather than the Taverna specific part<br />
Plug-in Manager Implementation<br />The plug-in manager is instantiated and configured through Spring<br />Plug-in manager and related components are available in the root class loader for an application<br />Exposed as named beans in the Spring context<br />Applications access infrastructure properties through the Spring context<br />Access through interfaces not implementation types<br />Next slide shows actual implementation components in the current code<br />
Timescale<br />1st November – Initial generic platform implementation<br />Early version released to public CVS 4th November<br />1st December – Specification of Taverna specific platform services<br />Early implementation of enactor, workflow loader and reference service components expected mid-November<br />1st January – Initial implementation of Taverna services<br />16th & 17th February – 2 day Developer Workshop in Manchester<br />Sign up at http://spreadsheets.google.com/viewform?key=pGRSW--IktWUFVQVLleNzIw&hl=en<br />