Infrastructure and Workflow for the Formal Evaluation of Semantic Search Technologies

1,321 views
1,242 views

Published on

This paper describes an infrastructure for the automated evaluation of semantic technologies and, in particular, semantic search technologies. For this purpose, we present an evaluation framework which follows a service-oriented approach for evaluating semantic technologies and uses the Business Process Execution Language (BPEL) to define evaluation workflows that can be executed by process engines. This framework supports a variety of evaluations, from different semantic areas, including search, and is extendible to new evaluations. We show how BPEL addresses this diversity as well as how it is used to solve specific challenges such as heterogeneity, error handling and reuse.

Presented at Data infrastructurEs for Supporting Information Retrieval Evaluation (DESIRE 2011) Workshop, Co-located with CIKM 2011, the 20th ACM Conference on Information and Knowledge Management

Friday 28th October 2011, Glasgow, UK

http://www.promise-noe.eu/events/desire-2011/

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,321
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Infrastructure and Workflow for the Formal Evaluation of Semantic Search Technologies

  1. 1. Infrastructure and Workflow for the Formal Evaluation of Semantic Search Technologies Stuart N. Wrigley1, Raúl García-Castro2 and Cassia Trojahn3 1University of Sheffield, UK 2Universidad Politécnica de Madrid, Spain 3INRIA, France Data infrastructurEs for Supporting Information Retrieval Evaluation: DESIRE 2011 Workshop
  2. 2. SEALS Project• SEALS: Semantic Evaluation At Large Scale• EU FP7 funded Infrastructures project• June 2009 – June 2012.• Initial areas: ontology engineering, ontology storage and reasoning tools, ontology matching, semantic web service discovery, semantic search tiv ing Re tivi• Objectives: s Ac o rk Ac itie se ties tw ar – SEALS Platform. ch Ne • A lasting reference infrastructure. • Evaluations executed on-demand on SEALS Platform. Service – SEALS Evaluation Campaigns. Activities • Two public evaluation campaigns – SEALS Community.28.10.20112
  3. 3. Key (non-technical) features• Infrastructure characteristics: – Open (both in terms of use and development – Apache 2.0 license) – Scalable (to users and data size – cluster-based) – Extensible (new evals, new tool types, new metrics) – Sustainable (beyond funded period) – Independent (unbiased, trustworthy) – Repeatable (eval results can be reproduced)• Core criteria: – Interoperability – Scalability – Tool-specific measures (e.g., alignment precision, etc)28.10.20113
  4. 4. Evaluation dependencies Tools ER Execution Evaluation Results Request Test data Evaluation descriptions28.10.20114
  5. 5. Platform architecture28.10.20115
  6. 6. Evaluation overview Infrastructure Tools Tools Infrastructure request deployment undeployment release Execution Execution Evaluation Execution request environment description environment analysis preparation execution clean-up ER Test data Activity Results Test data stage-in Execution storage stage-out28.10.20116
  7. 7. REPOSITORIES28.10.20117
  8. 8. Repositories External Interface HTTP File Request Restlet Resources Internal Interface Java Test Data Tools Results objects Repository Repository Repository Implementation Implementation Implementation Storage RDF RDF File File Triples Repository Repository (Metadata) (Artifacts)28.10.20118
  9. 9. Test Data Repository Service (TDRS)• Storage of, and access to: – persistent test data sets (aka suites) – test data generators• Suites are stored as ZIP files and accompanied by metadata.• Suites can be versioned.• ZIP-internal metadata allows structuring and repository-based iteration. Data Metadata Artifact Data Entity Artifact Artifact Item Discovery Exploitation28.10.20119
  10. 10. Results Repository Service (RRS)• Storage of, and access to, suites of: – raw results – interpretations• Suites stored as metadata and optional ZIP files.• Metadata allows structuring and linking to ensure backlinks: – interpretation links to raw result dataItem – raw result links to tool and test suite dataItem28.10.201110
  11. 11. TOOLS28.10.201111
  12. 12. Tool wrapping and storage• Tools are evaluated on within the Platform (ie, locally)• Tools must have bi-directional communication with Platform.• Each campaign (e.g., search) defines its own Java API specific to their tool type.• Participating tools create a tool ‘wrapper’ to implement API.• Also includes ‘setup’ and ‘tear down’ scripts and any 3rd party libraries / packages required.• Bundles stored in Tool Repository Service (TRS).• Tutorials on SEALS portal (http://www.seals-project.eu/).28.10.201112
  13. 13. Semantic Search APIMethod Functionalityboolean loadOntology( Load an ontology URL ontology, String ontologyName, String ontologyNamespace)void showGUI(boolean show) Switch the GUI on or offboolean executeQuery(String query) Execute a queryboolean isResultSetReady() Are query results ready?URL getResults() Retrieve the URL of the results fileboolean isUserInputComplete() Has user hit ‘go’ (or equivalent)?String getUserQuery() Retrieve the query as entered by the user28.10.201113
  14. 14. WORKFLOWS28.10.201114
  15. 15. Evaluation overview (workflow) Infrastructure Tools Tools Infrastructure request deployment undeployment release Execution Execution Evaluation Execution request environment description environment analysis preparation execution clean-up ER Test data Activity Results Test data stage-in Execution storage stage-out28.10.201115
  16. 16. Workflow• Business Process Execution Language (BPEL) – orchestrates manipulation of information using (only) web services interfaces.• All entities involved must be exposed as web services (TDRS, RRS, the tool wrapper, custom services) defined using WSDL interfaces.• ‘Custom services’ allow out-of-band processing: – computation of analyses – data / metadata manipulation – timestamping – etc.• Workflow defined according to campaign requirements.• Stored in the Evaluation Repository Service (ERS).28.10.201116
  17. 17. Conceptual workflow for search Start Start Ontology No Get Ontology Test Data Loaded? URL Repository Load Test Suite Test Data Yes Repository Test Data Tool: Load Get Query Repository Ontology Loaded No Record error successfully? Tool: Execute Query Yes Tool: Switch off GUI Tool: Results No Pause Ready? Yes More test No Store Results End cases? Tool: Get Results Yes Store Results Results Results Repository Process Repository Test Case End28.10.201117
  18. 18. BPEL workflow snippets28.10.2011 Stored as XML S18
  19. 19. BPEL exception handling28.10.201119
  20. 20. Summary• SEALS Platform provides functionality to simplify and automate evaluations.• Powerful cluster-based compute.• Storage of test data, results and interpretations in perpetuity.• Workflows specified in industry-standard BPEL.• All for free!28.10.201120
  21. 21. Thank you for your attention! http://www.seals-project.eu/
  22. 22. Links to resources• Evaluation campaigns – http://www.seals-project.eu/seals-evaluation-campaigns/• Test suites – Mooney (NL questions) • http://seals.sti2.at/tdrs-web/testdata/persistent/Mooney+NL+Testsuite/2010/ – EvoOnt (SPARQL questions) • http://seals.sti2.at/tdrs-web/testdata/persistent/EvoOnt+1K+SPARQL+Testsuite/2010/ • http://seals.sti2.at/tdrs-web/testdata/persistent/EvoOnt+10K+SPARQL+Testsuite/2010/ • http://seals.sti2.at/tdrs-web/testdata/persistent/EvoOnt+100K+SPARQL+Testsuite/2010/ • http://seals.sti2.at/tdrs-web/testdata/persistent/EvoOnt+1000K+SPARQL+Testsuite/2010/ • http://seals.sti2.at/tdrs-web/testdata/persistent/EvoOnt+10000K+SPARQL+Testsuite/2010/ – QALD • http://seals.sti2.at/tdrs-web/testdata/persistent/QALD-MusicBrainz+SPARQL+Testsuite/2010/• Last year’s campaign outcomes – http://www.seals-project.eu/seals-evaluation-campaigns/semantic-search-tools/results-201028.10.201122

×