Successfully reported this slideshow.
2014-03-18 Konstantin Baierer: Omnom & Pubby 1/36
DM2E
Interoperability
platform
–
Omnom & Pubby
Konstantin Baierer
konsta...
2014-03-18 Konstantin Baierer: Omnom & Pubby 2/36
● Components of the DM2E Software Stack
● What is Omnom?
– Modelling wor...
2014-03-18 Konstantin Baierer: Omnom & Pubby 3/36
Omnom and Pubby in DM2E
2014-03-18 Konstantin Baierer: Omnom & Pubby 4/36
Omnom and Pubby in DM2E
Omnom
2014-03-18 Konstantin Baierer: Omnom & Pubby 5/36
Omnom and Pubby in DM2E
PubbyOmnom
2014-03-18 Konstantin Baierer: Omnom & Pubby 6/36
● dm2e-ws
– Workflow Engine
– Web Services
● dm2e-gui
– Graphical User I...
2014-03-18 Konstantin Baierer: Omnom & Pubby 7/36
● dm2e-ws
– Workflow Engine
– Web Services
● dm2e-gui
– Graphical User I...
2014-03-18 Konstantin Baierer: Omnom & Pubby 8/36
● dm2e-ws
– Workflow Engine
– Web Services
● dm2e-gui
– Graphical User I...
2014-03-18 Konstantin Baierer: Omnom & Pubby 9/36
Omnom: Workflows and Configurations
● Workflow
– Abstract description of...
2014-03-18 Konstantin Baierer: Omnom & Pubby 10/36
Omnom: What's in a Workflow?
● Typed Input and Output Parameters
● Posi...
2014-03-18 Konstantin Baierer: Omnom & Pubby 11/36
Simple XSLT Publish Workflow→
2014-03-18 Konstantin Baierer: Omnom & Pubby 12/36
Simple XSLT Publish Workflow→
Workflow Input Workflow Output
2014-03-18 Konstantin Baierer: Omnom & Pubby 13/36
Simple XSLT Publish Workflow→
Workflow Input Workflow Output
Input
Web ...
2014-03-18 Konstantin Baierer: Omnom & Pubby 14/36
Simple XSLT Publish Workflow→
Workflow Input Workflow Output
Input
Web ...
2014-03-18 Konstantin Baierer: Omnom & Pubby 15/36
Omnom: What's in a configuration?
● Parameter Values
– Resolvable URLs
...
2014-03-18 Konstantin Baierer: Omnom & Pubby 16/36
Web services: Protocol
● WS describe themselves in RDF:
– Input / Outpu...
2014-03-18 Konstantin Baierer: Omnom & Pubby 17/36
Webservice: Execution
2014-03-18 Konstantin Baierer: Omnom & Pubby 18/36
Convention for description/thing
Accept conflicts for RDF Inf. Res.
● R...
2014-03-18 Konstantin Baierer: Omnom & Pubby 19/36
Types of Services
● Backend Services
– Workflow Service
– Configuration...
2014-03-18 Konstantin Baierer: Omnom & Pubby 20/36
Web Services for Workflows
● Transformation
– Transform XML with XSLT t...
2014-03-18 Konstantin Baierer: Omnom & Pubby 21/36
More complex: ZIP XSLT-ZIP Publish→ →
2014-03-18 Konstantin Baierer: Omnom & Pubby 22/36
Even more complex: Nested Workflows
2014-03-18 Konstantin Baierer: Omnom & Pubby 23/36
Provenance in DM2E
● All Omnom resources are described in RDF
● All res...
2014-03-18 Konstantin Baierer: Omnom & Pubby 24/36
Provenance: PROV Omnom↔
2014-03-18 Konstantin Baierer: Omnom & Pubby 25/36
Omnom Web User Interface
● Single-Page Application
● BackboneJS
– MVC f...
2014-03-18 Konstantin Baierer: Omnom & Pubby 26/36
Pubby
● Frontend for Linked Data
● Make triplestores web-accesible
– De...
2014-03-18 Konstantin Baierer: Omnom & Pubby 27/36
DM2E Extensions to Pubby
● CONSTRUCT instead of DESCRIBE
– Support Name...
2014-03-18 Konstantin Baierer: Omnom & Pubby 28/36
Example Pubby Use Case
● Want to expose library data
– Book data from t...
2014-03-18 Konstantin Baierer: Omnom & Pubby 29/36
Omnom: Workflow Edit
2014-03-18 Konstantin Baierer: Omnom & Pubby 30/36
Configuration Edit
2014-03-18 Konstantin Baierer: Omnom & Pubby 31/36
Job Page
2014-03-18 Konstantin Baierer: Omnom & Pubby 32/36
Pubby View of the result
2014-03-18 Konstantin Baierer: Omnom & Pubby 33/36
Provenance: Follow your nose
● Resource → Versioned Dataset
● Versioned...
2014-03-18 Konstantin Baierer: Omnom & Pubby 34/36
Omnom: Next steps
● Performance
● UI/UX improvements
● Easier deployment
2014-03-18 Konstantin Baierer: Omnom & Pubby 35/36
Pubby: Next steps
● Handle weird URIs more gracefully
● Properly handle...
2014-03-18 Konstantin Baierer: Omnom & Pubby 36/36
Thank You!
● Demos:
– http://omnom.dm2e.eu
– http://data.dm2e.eu/data/
...
Upcoming SlideShare
Loading in …5
×

DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

483 views

Published on

Presentation of the DM2E Interoperability platform – Omnom & Pubby at the ExLibris Semantic Web Special Interest Group (18 March 2014)

Published in: Software, Technology
  • Be the first to comment

DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

  1. 1. 2014-03-18 Konstantin Baierer: Omnom & Pubby 1/36 DM2E Interoperability platform – Omnom & Pubby Konstantin Baierer konstantin.baierer@ibi.hu-berlin.de
  2. 2. 2014-03-18 Konstantin Baierer: Omnom & Pubby 2/36 ● Components of the DM2E Software Stack ● What is Omnom? – Modelling workflows – HTTP Protocol – Provenance ● What is Pubby ● Example Pubby setup ● Omnom/Pubby Demo Agenda
  3. 3. 2014-03-18 Konstantin Baierer: Omnom & Pubby 3/36 Omnom and Pubby in DM2E
  4. 4. 2014-03-18 Konstantin Baierer: Omnom & Pubby 4/36 Omnom and Pubby in DM2E Omnom
  5. 5. 2014-03-18 Konstantin Baierer: Omnom & Pubby 5/36 Omnom and Pubby in DM2E PubbyOmnom
  6. 6. 2014-03-18 Konstantin Baierer: Omnom & Pubby 6/36 ● dm2e-ws – Workflow Engine – Web Services ● dm2e-gui – Graphical User Interface (Omnom / dm2e-gui) ● dm2e-silk – Linking ● dm2e-pubby – RDF Browser Components of WP2 Interop Platform ● dm2e-ontologies – Data models/ Validation ● dm2e-direct – Command line ingestion ● dm2e-josso – Single Sign On
  7. 7. 2014-03-18 Konstantin Baierer: Omnom & Pubby 7/36 ● dm2e-ws – Workflow Engine – Web Services ● dm2e-gui – Graphical User Interface (Omnom / dm2e-gui) ● dm2e-silk – Linking ● dm2e-pubby – RDF Browser Components of WP2 Interop Platform Omn m ● dm2e-ontologies – Data models/ Validation ● dm2e-direct – Command line ingestion ● dm2e-josso – Single Sign On
  8. 8. 2014-03-18 Konstantin Baierer: Omnom & Pubby 8/36 ● dm2e-ws – Workflow Engine – Web Services ● dm2e-gui – Graphical User Interface (Omnom / dm2e-gui) ● dm2e-silk – Linking ● dm2e-pubby – RDF Browser Components of WP2 Interop Platform Omn m ● dm2e-ontologies – Data models/ Validation ● dm2e-direct – Command line ingestion ● dm2e-josso – Single Sign On
  9. 9. 2014-03-18 Konstantin Baierer: Omnom & Pubby 9/36 Omnom: Workflows and Configurations ● Workflow – Abstract description of algorithm – „Blueprint“ or „Data Wiring“ – Less than a dozen for DM2E ● Configuration – Instantiation of a workflow with concrete values – One for every ingestion
  10. 10. 2014-03-18 Konstantin Baierer: Omnom & Pubby 10/36 Omnom: What's in a Workflow? ● Typed Input and Output Parameters ● Positions wrapping Web Services Input and Output Parameters Iterative positions for repeatable output ● Connectors wiring the parameters => Define data flow
  11. 11. 2014-03-18 Konstantin Baierer: Omnom & Pubby 11/36 Simple XSLT Publish Workflow→
  12. 12. 2014-03-18 Konstantin Baierer: Omnom & Pubby 12/36 Simple XSLT Publish Workflow→ Workflow Input Workflow Output
  13. 13. 2014-03-18 Konstantin Baierer: Omnom & Pubby 13/36 Simple XSLT Publish Workflow→ Workflow Input Workflow Output Input Web Service Output Position 1 Input Web Service Output Position 2
  14. 14. 2014-03-18 Konstantin Baierer: Omnom & Pubby 14/36 Simple XSLT Publish Workflow→ Workflow Input Workflow Output Input Web Service Output Position 1 Input Web Service Output Position 2 Connectors
  15. 15. 2014-03-18 Konstantin Baierer: Omnom & Pubby 15/36 Omnom: What's in a configuration? ● Parameter Values – Resolvable URLs – Strings ● Examples – URL of result RDF/XML – URL of input XSLT – Name of a dataset for URI generation – Name-Value pair for XSLT parameters
  16. 16. 2014-03-18 Konstantin Baierer: Omnom & Pubby 16/36 Web services: Protocol ● WS describe themselves in RDF: – Input / Output parameters – Human readable description / label ● WS expect configuration in RDF: – Concrete values for input parameters – Either resolvable URLs or Literals ● WS immediately return a Job in RDF: – Asynchronously work in the background – Write (intermediate) results to the Job resource
  17. 17. 2014-03-18 Konstantin Baierer: Omnom & Pubby 17/36 Webservice: Execution
  18. 18. 2014-03-18 Konstantin Baierer: Omnom & Pubby 18/36 Convention for description/thing Accept conflicts for RDF Inf. Res. ● RDF/XML at http://mydomain/file/123 – Document: Output of a transformation – Metadata: File Size, Creator, MD5 ... ● HTTP Accept: application/rdf+xml – Return the document ?! – Return the metadata ?! ● Our solution: – No Accept / application/octet-stream => Data – Accept: RDF/JSON => Metadata
  19. 19. 2014-03-18 Konstantin Baierer: Omnom & Pubby 19/36 Types of Services ● Backend Services – Workflow Service – Configuration Service – Job Service – File Services ● DM2E File Service ● MINT ● Web Services to be used in workflows
  20. 20. 2014-03-18 Konstantin Baierer: Omnom & Pubby 20/36 Web Services for Workflows ● Transformation – Transform XML with XSLT to RDF/XML – Transform RDF with SPARQL to RDF ● Publishing – With provenance to Triplestore ● Iteration – Feed every contained resource to another web service – ZIP of XML files, OAI-PMH harvesting ● Execute a workflow → Nested Workflows
  21. 21. 2014-03-18 Konstantin Baierer: Omnom & Pubby 21/36 More complex: ZIP XSLT-ZIP Publish→ →
  22. 22. 2014-03-18 Konstantin Baierer: Omnom & Pubby 22/36 Even more complex: Nested Workflows
  23. 23. 2014-03-18 Konstantin Baierer: Omnom & Pubby 23/36 Provenance in DM2E ● All Omnom resources are described in RDF ● All results are accesible on the Web – Either Directly in RDF – Or as Web Documents, with RDF metadata ● All data is in versioned datasets ● [ Dm2e-pubby offers version-aware API ] ● Consumers should "Follow their nose"
  24. 24. 2014-03-18 Konstantin Baierer: Omnom & Pubby 24/36 Provenance: PROV Omnom↔
  25. 25. 2014-03-18 Konstantin Baierer: Omnom & Pubby 25/36 Omnom Web User Interface ● Single-Page Application ● BackboneJS – MVC framework – Client/Server: Same data representation: JSON-LD (ish, kind of) ● Bootstrap – Looks like Twitter – Sane Defaults
  26. 26. 2014-03-18 Konstantin Baierer: Omnom & Pubby 26/36 Pubby ● Frontend for Linked Data ● Make triplestores web-accesible – Dereferenceable URI – 303 Redirects – Content-Negotiation ● HTML interface
  27. 27. 2014-03-18 Konstantin Baierer: Omnom & Pubby 27/36 DM2E Extensions to Pubby ● CONSTRUCT instead of DESCRIBE – Support Named Graphs – Configuration defines the relevant subgraph that describes a URI – Multiple Resources per output graph ● More flexible URIs – 'data' and 'page' configurable – Dataset-specific URIs – Regex Matching in URIs ● Generate thumbnails
  28. 28. 2014-03-18 Konstantin Baierer: Omnom & Pubby 28/36 Example Pubby Use Case ● Want to expose library data – Book data from triplestore 1 – Subject data from triplestore 2 ● Using the same consistent interface ● Full example here http://data.dm2e.eu/pubby-example
  29. 29. 2014-03-18 Konstantin Baierer: Omnom & Pubby 29/36 Omnom: Workflow Edit
  30. 30. 2014-03-18 Konstantin Baierer: Omnom & Pubby 30/36 Configuration Edit
  31. 31. 2014-03-18 Konstantin Baierer: Omnom & Pubby 31/36 Job Page
  32. 32. 2014-03-18 Konstantin Baierer: Omnom & Pubby 32/36 Pubby View of the result
  33. 33. 2014-03-18 Konstantin Baierer: Omnom & Pubby 33/36 Provenance: Follow your nose ● Resource → Versioned Dataset ● Versioned Dataset → Dataset ● Versioned Dataset → Publish Job ● Publish Job → Publish Job Configuration ● Publish Job Configuration → Workflow Job ● Workflow Job → Workflow ● Workflow → User ● … Let's start here
  34. 34. 2014-03-18 Konstantin Baierer: Omnom & Pubby 34/36 Omnom: Next steps ● Performance ● UI/UX improvements ● Easier deployment
  35. 35. 2014-03-18 Konstantin Baierer: Omnom & Pubby 35/36 Pubby: Next steps ● Handle weird URIs more gracefully ● Properly handle blank nodes – Collection Ontology Lists ● Caching (hard) ● Re-Merge with upstream Pubby
  36. 36. 2014-03-18 Konstantin Baierer: Omnom & Pubby 36/36 Thank You! ● Demos: – http://omnom.dm2e.eu – http://data.dm2e.eu/data/ ● Source Code / Issue Tracker – http://github.com/DM2E/dm2e-ws – http://github.com/DM2E/dm2e-gui – http://github.com/DM2E/pubby ● Get in touch: – konstantin.baierer@ibi.hu-berlin.de – http://dm2e.eu, @DM2Europeana

×