Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SEALS @ I-Semantics 2011


Published on

SEALS presented the results of its first SEALS Evaluation Campaigns at I-SEMANTICS 2011, held from September 7-9 in Graz, Austria.

Read more about SEALS at

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

SEALS @ I-Semantics 2011

  1. 1. The state of semantic technology today: overview of the first SEALS evaluation campaings Lyndon Nixon , STI International Raul Garcia-Castro , Universidad Politécnica de Madrid Stuart Wrigley , University of Sheffield Mikalai Yatskevich , University of Oxford Cassia Trojahn , INRIA Liliana Cabral , The Open University 07.09.11
  2. 2. Outline <ul><li>SEALS Project </li></ul><ul><li>SEALS Platform </li></ul><ul><li>Evaluation Process </li></ul><ul><li>Evaluation Campaign Results </li></ul><ul><li>Final remarks </li></ul>I-SEMANTICS 2011
  3. 3. SEALS (Semantic Evaluation At Large Scale) Universidad Politécnica de Madrid, Spain (Coordinator) University of Sheffield, UK Forschungszentrum Informatik, Germany University of Innsbruck, Austria Institut National de Recherche en Informatique et en Automatique, France University of Mannheim, Germany University of Zurich, Switzerland STI International, Austria Open University, UK Oxford University, UK Contact person: Asunción Gómez Pérez <> Duration: June 2009-May 2012 I-SEMANTICS 2011 1 3 1 2 1 2
  4. 4. SEALS Objectives <ul><li>The SEALS Platform </li></ul><ul><li>A lasting reference infrastructure for semantic technology evaluation </li></ul><ul><li>The evaluations to be executed on-demand at the SEALS Platform </li></ul><ul><li>The SEALS Evaluation Campaigns </li></ul><ul><li>Two public evaluation campaigns including the best-in-class semantic technologies: </li></ul><ul><li>Ontology engineering tools </li></ul><ul><li>Ontology storage and reasoning systems </li></ul><ul><li>Ontology matching tools </li></ul><ul><li>Semantic search tools </li></ul><ul><li>Semantic Web Service tools </li></ul><ul><li>The SEALS Community </li></ul><ul><li>Around the evaluation of semantic technologies </li></ul>Service Activities Research Activities Networking Activities I-SEMANTICS 2011
  5. 5. The SEALS Platform <ul><li>Provides the infrastructure for evaluating semantic technologies </li></ul><ul><li>Open (everybody can use it) </li></ul><ul><li>Scalable (to users, data size) </li></ul><ul><li>Extensible (to more tests, different technology, more measures) </li></ul><ul><li>Sustainable (beyond SEALS) </li></ul><ul><li>Independent (unbiased) </li></ul><ul><li>Repeatable (evaluations can be reproduced) </li></ul><ul><li>According to criteria: </li></ul><ul><li>Interoperability </li></ul><ul><li>Scalability </li></ul><ul><li>Specific measures (e.g., completeness of query answers, alignment precision) </li></ul>I-SEMANTICS 2011
  6. 6. SEALS Service Manager Runtime Evaluation Service SEALS Portal Test Data Repository Tools Repository Results Repository Evaluation Descriptions Repository Technology Adopters Software agents, i.e., technology evaluators SEALS Repositories The SEALS Platform Evaluation organizers I-SEMANTICS 2011 Technology Providers Entity management requests Evaluation requests
  7. 7. The Evaluation Process <ul><li>Initiation </li></ul><ul><ul><li>Identify organizers and scenarios </li></ul></ul><ul><li>Involvement </li></ul><ul><ul><li>Announce the campaigns and involve participants </li></ul></ul><ul><li>Preparation and execution </li></ul><ul><ul><li>Provide material, execute the scenarios </li></ul></ul><ul><li>Dissemination </li></ul><ul><ul><li>Publish and discuss results. </li></ul></ul>I-SEMANTICS 2011
  8. 8. The SEALS Evaluation Campaigns 2010 Next evaluation campaigns during 2011 - 2012 Ontology Engineering Tools Reasoning Tools Ontology Matching Tools Semantic Search Tools Semantic Web Service Tools OET Conformance 2010 Conformance OET Interoperability 2010 Interoperability OET Scalability 2010 Efficiency Scalability DLBS Classification 2010 DLBS Class satisfiability 2010 DLBS Ontology satisfiability 2010 DLBS Logical entailment 2010 MT Benchmark 2010 Compliance Efficiency MT Anatomy 2010 Compliance Efficiency MT Conference 2010 Compliance Efficiency Alignment coherence SST Automated Search Performance 2010 Search quality SST Automated Performance and Scalability 2010 Resource consumption SST Automated Query Expressiveness 2010 Query expressiveness SST Automated Quality of Documentation 2010 Usability SST User Usability 2010 Usability SST User Query Expressiveness 2010 Query expressiveness SWS Tool Discovery Evaluation 2010 : Performance
  9. 9. Ontology Engineering Tools <ul><li>Goal: To evaluate the ontology management capabilities of ontology engineering tools </li></ul><ul><li>Evaluation services for: </li></ul><ul><ul><li>Conformance </li></ul></ul><ul><ul><li>Interoperability </li></ul></ul><ul><ul><li>Scalability </li></ul></ul><ul><li>Test data: </li></ul><ul><ul><li>RDF(S) Import Test Suite </li></ul></ul><ul><ul><li>OWL Lite Import Test Suite </li></ul></ul><ul><ul><li>OWL DL Import Test Suite </li></ul></ul><ul><ul><li>OWL Full Import Test Suite </li></ul></ul><ul><ul><li>Scalability test data </li></ul></ul>I-SEMANTICS 2011
  10. 10. Ontology Engineering Results <ul><li>3 ontology management frameworks (Jena, the OWL API and Sesame) and 3 editors (the NeOn Toolkit, Protégé 4 and Protégé OWL) </li></ul>I-SEMANTICS 2011
  11. 11. Storage and Reasoning Tools <ul><li>Goal: to evaluate the interoperability and performance of DLBSs </li></ul><ul><ul><li>Standard reasoning services </li></ul></ul><ul><ul><ul><li>Classification </li></ul></ul></ul><ul><ul><ul><li>Class satisfiability </li></ul></ul></ul><ul><ul><ul><li>Ontology satisfiability </li></ul></ul></ul><ul><ul><ul><li>Logical entailment </li></ul></ul></ul><ul><li>Evaluation Criteria </li></ul><ul><ul><li>Conformance </li></ul></ul><ul><ul><li>Performance </li></ul></ul><ul><li>Metrics </li></ul><ul><ul><li>The number of tests passed without I/O errors </li></ul></ul><ul><ul><li>Time </li></ul></ul>I-SEMANTICS 2011
  12. 12. Storage and Reasoning Results <ul><li>Tools: HermiT, FaCT++ and jcel </li></ul><ul><li>The DLBSs systems have shown an acceptable level of performance </li></ul><ul><li>Trade-off between expressivity and performance </li></ul><ul><li>Most errors were related to features not supported in the current implementations </li></ul><ul><ul><li>e.g. jcel does not support entailment inference </li></ul></ul>I-SEMANTICS 2011
  13. 13. Ontology Matching Tools <ul><li>Goals: </li></ul><ul><ul><li>To evaluate the competence of matching systems with respect to different evaluation criteria. </li></ul></ul><ul><ul><li>To demonstrate the feasibility and benefits of automating matching evaluation. </li></ul></ul><ul><li>Criteria </li></ul><ul><ul><li>Compliance </li></ul></ul><ul><ul><ul><li>precision and recall </li></ul></ul></ul><ul><ul><ul><li>restricted semantic precision and recall </li></ul></ul></ul><ul><ul><ul><li>alignment coherence </li></ul></ul></ul><ul><ul><li>Efficiency </li></ul></ul><ul><ul><ul><li>runtime </li></ul></ul></ul><ul><ul><ul><li>memory consumption </li></ul></ul></ul>I-SEMANTICS 2011
  14. 14. Matching Tools <ul><ul><li>15 participants </li></ul></ul><ul><ul><li>11 for the benchmark </li></ul></ul><ul><ul><li>ASMOV and RiMOM ahead, with AgrMaker as close follower </li></ul></ul>I-SEMANTICS 2011
  15. 15. <ul><li>Anatomy </li></ul><ul><ul><li>AgreementMaker followed by Eff2Match, NBJLM and SOBOM </li></ul></ul><ul><ul><li>Reference alignment available </li></ul></ul><ul><li>Conference </li></ul><ul><ul><li>CODI, ASMOV and Eff2Match </li></ul></ul><ul><li>There is no better matcher for all tracks </li></ul><ul><li>New technology affected both participants and organizers </li></ul>Ontology Matching Tools I-SEMANTICS 2011
  16. 16. Semantic Search Tools <ul><li>Goals: </li></ul><ul><ul><li>Benchmark effectiveness of search tools </li></ul></ul><ul><ul><li>Emphasis on tool usability since search is a inherently user-centered activity. </li></ul></ul><ul><ul><li>Still interested in automated evaluation for other aspects </li></ul></ul><ul><ul><li>Two phase approach: </li></ul></ul><ul><ul><ul><li>Automated evaluation : runs on SEALS Platform </li></ul></ul></ul><ul><ul><ul><li>User-in-the-loop : human experiment </li></ul></ul></ul><ul><ul><li>Criteria for User-centred search: </li></ul></ul><ul><ul><ul><li>Query expressiveness </li></ul></ul></ul><ul><ul><ul><li>Usability (effectiveness, efficiency, satisfaction) </li></ul></ul></ul><ul><ul><ul><li>Scalability </li></ul></ul></ul><ul><ul><ul><li>Interoperability </li></ul></ul></ul>I-SEMANTICS 2011
  17. 17. <ul><li>Automated phase </li></ul><ul><li>Many system have problems to load ontologies (not able to cope with standards) </li></ul><ul><li>User in the loop </li></ul><ul><li>Systems in either best and worst categories </li></ul>Semantic Search Results I-SEMANTICS 2011 Tool  UITL  Auto  K-Search  x  x Ginseng x  x  NLP-Reduce x  x  Jena Arq   x  PowerAqua x  x 
  18. 18. Semantic Web Services <ul><li>Goal: to evaluate Semantic Web Service discovery </li></ul><ul><li>Test data: </li></ul><ul><ul><li>OWLS Test Collection (OWLS-TC) </li></ul></ul><ul><ul><li>SAWSDL Test Collection (SAWSDL-TC) </li></ul></ul><ul><ul><li>Seekda Services </li></ul></ul><ul><ul><li>OPOSSum Services </li></ul></ul>I-SEMANTICS 2011
  19. 19. Semantic Web Services Results <ul><li>A number of queries/datasets shows bias towards OWLS-MX (recall = 1.0) </li></ul><ul><li>Inconclusive between semantic and syntactic settings </li></ul><ul><li>Variants (different matching algorithms) are behaving the same for same settings! </li></ul>I-SEMANTICS 2011
  20. 20. Coming next: 2nd campaign <ul><li>The 2nd evaluation campaign has been launched in July 2011 </li></ul><ul><li>Details are available on the website </li></ul><ul><li>A world first – the next campaign will use the SEALS platform for remote evaluation </li></ul><ul><li>Campaigns are open for all – your semantic tool can and should participate </li></ul>I-SEMANTICS 2011
  21. 21. Thank you for your attention <ul><li>SEALS evaluation campaigns, see their webpages at </li></ul><ul><li> </li></ul>I-SEMANTICS 2011