SEALS @ I-Semantics 2011

The state of semantic technology today: overview of the first SEALS evaluation campaings Lyndon Nixon , STI International Raul Garcia-Castro , Universidad Politécnica de Madrid Stuart Wrigley , University of Sheffield Mikalai Yatskevich , University of Oxford Cassia Trojahn , INRIA Liliana Cabral , The Open University 07.09.11

Outline
SEALS Project
SEALS Platform
Evaluation Process
Evaluation Campaign Results
Final remarks
I-SEMANTICS 2011

SEALS (Semantic Evaluation At Large Scale) Universidad Politécnica de Madrid, Spain (Coordinator) University of Sheffield, UK Forschungszentrum Informatik, Germany University of Innsbruck, Austria Institut National de Recherche en Informatique et en Automatique, France University of Mannheim, Germany University of Zurich, Switzerland STI International, Austria Open University, UK Oxford University, UK Contact person: Asunción Gómez Pérez <asun@fi.upm.es> http://www.seals-project.eu/ Duration: June 2009-May 2012 I-SEMANTICS 2011 1 3 1 2 1 2

SEALS Objectives
The SEALS Platform
A lasting reference infrastructure for semantic technology evaluation
The evaluations to be executed on-demand at the SEALS Platform
The SEALS Evaluation Campaigns
Two public evaluation campaigns including the best-in-class semantic technologies:
Ontology engineering tools
Ontology storage and reasoning systems
Ontology matching tools
Semantic search tools
Semantic Web Service tools
The SEALS Community
Around the evaluation of semantic technologies
Service Activities Research Activities Networking Activities I-SEMANTICS 2011

The SEALS Platform
Provides the infrastructure for evaluating semantic technologies
Open (everybody can use it)
Scalable (to users, data size)
Extensible (to more tests, different technology, more measures)
Sustainable (beyond SEALS)
Independent (unbiased)
Repeatable (evaluations can be reproduced)
According to criteria:
Interoperability
Scalability
Specific measures (e.g., completeness of query answers, alignment precision)
I-SEMANTICS 2011

SEALS Service Manager Runtime Evaluation Service SEALS Portal Test Data Repository Tools Repository Results Repository Evaluation Descriptions Repository Technology Adopters Software agents, i.e., technology evaluators SEALS Repositories The SEALS Platform Evaluation organizers I-SEMANTICS 2011 Technology Providers Entity management requests Evaluation requests

The Evaluation Process
Initiation
Identify organizers and scenarios
Involvement
Announce the campaigns and involve participants
Preparation and execution
Provide material, execute the scenarios
Dissemination
Publish and discuss results.
I-SEMANTICS 2011

The SEALS Evaluation Campaigns 2010 Next evaluation campaigns during 2011 - 2012 Ontology Engineering Tools Reasoning Tools Ontology Matching Tools Semantic Search Tools Semantic Web Service Tools OET Conformance 2010 Conformance OET Interoperability 2010 Interoperability OET Scalability 2010 Efficiency Scalability DLBS Classification 2010 DLBS Class satisfiability 2010 DLBS Ontology satisfiability 2010 DLBS Logical entailment 2010 MT Benchmark 2010 Compliance Efficiency MT Anatomy 2010 Compliance Efficiency MT Conference 2010 Compliance Efficiency Alignment coherence SST Automated Search Performance 2010 Search quality SST Automated Performance and Scalability 2010 Resource consumption SST Automated Query Expressiveness 2010 Query expressiveness SST Automated Quality of Documentation 2010 Usability SST User Usability 2010 Usability SST User Query Expressiveness 2010 Query expressiveness SWS Tool Discovery Evaluation 2010 : Performance

Ontology Engineering Tools
Goal: To evaluate the ontology management capabilities of ontology engineering tools
Evaluation services for:
Conformance
Interoperability
Scalability
Test data:
RDF(S) Import Test Suite
OWL Lite Import Test Suite
OWL DL Import Test Suite
OWL Full Import Test Suite
Scalability test data
I-SEMANTICS 2011

Ontology Engineering Results
3 ontology management frameworks (Jena, the OWL API and Sesame) and 3 editors (the NeOn Toolkit, Protégé 4 and Protégé OWL)
I-SEMANTICS 2011

Storage and Reasoning Tools
Goal: to evaluate the interoperability and performance of DLBSs
Standard reasoning services
Classification
Class satisfiability
Ontology satisfiability
Logical entailment
Evaluation Criteria
Conformance
Performance
Metrics
The number of tests passed without I/O errors
Time
I-SEMANTICS 2011

Storage and Reasoning Results
Tools: HermiT, FaCT++ and jcel
The DLBSs systems have shown an acceptable level of performance
Trade-off between expressivity and performance
Most errors were related to features not supported in the current implementations
e.g. jcel does not support entailment inference
I-SEMANTICS 2011

Ontology Matching Tools
Goals:
To evaluate the competence of matching systems with respect to different evaluation criteria.
To demonstrate the feasibility and benefits of automating matching evaluation.
Criteria
Compliance
precision and recall
restricted semantic precision and recall
alignment coherence
Efficiency
runtime
memory consumption
I-SEMANTICS 2011

Matching Tools
15 participants
11 for the benchmark
ASMOV and RiMOM ahead, with AgrMaker as close follower
I-SEMANTICS 2011

Anatomy
AgreementMaker followed by Eff2Match, NBJLM and SOBOM
Reference alignment available
Conference
CODI, ASMOV and Eff2Match
There is no better matcher for all tracks
New technology affected both participants and organizers
Ontology Matching Tools I-SEMANTICS 2011

Semantic Search Tools
Goals:
Benchmark effectiveness of search tools
Emphasis on tool usability since search is a inherently user-centered activity.
Still interested in automated evaluation for other aspects
Two phase approach:
Automated evaluation : runs on SEALS Platform
User-in-the-loop : human experiment
Criteria for User-centred search:
Query expressiveness
Usability (effectiveness, efficiency, satisfaction)
Scalability
Interoperability
I-SEMANTICS 2011

Automated phase
Many system have problems to load ontologies (not able to cope with standards)
User in the loop
Systems in either best and worst categories
Semantic Search Results I-SEMANTICS 2011 Tool UITL Auto K-Search x x Ginseng x x NLP-Reduce x x Jena Arq x PowerAqua x x

Semantic Web Services
Goal: to evaluate Semantic Web Service discovery
Test data:
OWLS Test Collection (OWLS-TC)
SAWSDL Test Collection (SAWSDL-TC)
Seekda Services
OPOSSum Services
I-SEMANTICS 2011

Semantic Web Services Results
A number of queries/datasets shows bias towards OWLS-MX (recall = 1.0)
Inconclusive between semantic and syntactic settings
Variants (different matching algorithms) are behaving the same for same settings!
I-SEMANTICS 2011

Coming next: 2nd campaign
The 2nd evaluation campaign has been launched in July 2011
Details are available on the website
A world first – the next campaign will use the SEALS platform for remote evaluation
Campaigns are open for all – your semantic tool can and should participate
I-SEMANTICS 2011

Thank you for your attention
SEALS evaluation campaigns, see their webpages at
www.seals-project.eu
I-SEMANTICS 2011

SEALS @ I-Semantics 2011

by SEALS - Semantic Evaluation at Large Scale on Sep 22, 2011

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 936

Statistics

SEALS @ I-Semantics 2011 Presentation Transcript

http://www.seals-project.eu	933
http://seals.sti2.org	3