Methodology and Campaign Design for the Evaluation of Semantic Search Tools


Published on

The main problem with the state of the art in the semantic search domain is the lack of comprehensive evaluations. There exist only a few efforts to evaluate semantic search tools and to compare the results with other evaluations of their kind.
In this paper, we present a systematic approach for testing and benchmarking semantic search tools that was developed within the SEALS project. Unlike other semantic web evaluations our methodology tests search tools both automatically and interactively with a human user in the loop. This allows us to test not only functional performance measures, such as precision and recall, but also usability issues, such as ease of use and comprehensibility of the query language.
The paper describes the evaluation goals and assumptions; the criteria and metrics; the type of experiments we will conduct as well as the datasets required to conduct the evaluation in the context of the SEALS initiative. To our knowledge it is the first effort to present a comprehensive evaluation methodology for Semantic Web search tools.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Methodology and Campaign Design for the Evaluation of Semantic Search Tools

  1. 1. Methodology and Campaign Design for the Evaluation of Semantic Search Tools<br />Stuart N. Wrigley1, Dorothee Reinhard2, Khadija Elbedweihy1, Abraham Bernstein2, Fabio Ciravegna1<br />4/27/10<br />1<br />1University of Sheffield, UK<br />2University of Zurich, Switzerland<br />
  2. 2. Outline<br />SEALS initiative<br />Evaluation design<br />Criteria<br />Two phase approach<br />API<br />Workflow<br />Data<br />Results and Analyses<br />Conclusions<br />4/27/10<br />2<br />
  3. 3. Seals Initiative<br />4/27/10<br />3<br />
  4. 4. SEALS goals<br />Develop and diffuse best practices in evaluation of semantic technologies<br />Create a lasting reference infrastructure for semantic technology evaluation<br />This infrastructure will be the SEALS Platform<br />Organise two worldwide Evaluation Campaigns<br />One this summer<br />Next in late 2011 / early 2012<br />Facilitate the continuous evaluation of semantic technologies<br />Allow easy access to both: <br />evaluation results (for developers and researchers) <br />technology roadmaps (for non-technical adopters)<br />Transfer all infrastructure to the community<br />4/27/10<br />4<br />
  5. 5. Targeted technologies<br />Five different types of semantic technologies:<br />Ontology Engineering tools<br />Ontology Storage and Reasoning Systems<br />Ontology Matching tools<br />Semantic Web Service tools<br />Semantic Search tools<br />4/27/10<br />5<br />
  6. 6. What’s our general approach?<br />Low overhead to the participant<br />Automate as far as possible<br />We provide the compute<br />We initiate the actual evaluation run<br />We perform the analysis<br />Provide infrastructure for more than simply running high profile evaluation campaigns<br />reuse existing evaluations for your personal testing<br />create new ones evaluations <br />store / publish / download test data sets<br />Encourage participation in evaluation campaign definitions and design<br />Open Source (Apache 2.0)<br />4/27/10<br />6<br />
  7. 7. SEALS Platform<br />4/27/10<br />7<br />Technology<br />Developers<br />Evaluation Organisers<br />Technology<br />Users<br />SEALS Service Manager<br />Runtime Evaluation Service<br />Result<br />Repository<br />Service<br />Tool<br />Repository<br />Service<br />Test Data<br />Repository<br />Service<br />Evaluation<br />Repository<br />Service<br />
  8. 8. Search Evaluation Design<br />4/27/10<br />8<br />
  9. 9. What do we want to do?<br />Evaluate / benchmark semantic search tools<br />with respect to their semantic peers.<br />Allow as wide a range of interface styles as possible<br />Assess tools on basis of a number of criteria including usability<br />Automate (part) of it<br />4/27/10<br />9<br />
  10. 10. Evaluation criteria<br />User-centred search methodologies will be evaluated according to the following criteria:<br />Query expressiveness<br />Usability (effectiveness, efficiency, satisfaction)<br />Scalability<br />Quality of documentation<br />Performance<br />4/27/10<br />10<br /><ul><li>Is the style of interface suited to the type of query?
  11. 11. How complex can the queries be?
  12. 12. Is it easy to understand?
  13. 13. Is it well structured?
  14. 14. How easy is the tool to use?
  15. 15. How easy is it to formulate the queries?
  16. 16. How easy is it to work with the answers?
  17. 17. Ability to cope with a large ontology
  18. 18. Ability to query a large repository in a reasonable time
  19. 19. Ability to cope with a large amount of results returned</li></ul>Resource consumption:<br /><ul><li>execution time (speed)
  20. 20. CPU load
  21. 21. memory required</li></li></ul><li>Two phase approach<br />Semantic search tools evaluation demands a user-in-the-loop phase<br />usability criterion<br />Two phases:<br />User-in-the-loop<br />Automated<br />4/27/10<br />11<br />
  22. 22. Evaluation criteria<br />Each phase will address a different subset of criteria.<br />Automated evaluation: query expressiveness, scalability, performance, quality of documentation<br />User-in-the-loop: usability, query expressiveness<br />4/27/10<br />12<br />
  23. 23. Running the Evaluation<br />4/27/10<br />13<br />
  24. 24. Automated evaluation<br />4/27/10<br />14<br />Tools uploaded to platform. Includes:<br />wrapper implementing API<br />supporting libraries<br />Test data and questions stored on platform<br />Workflow specifies details of evaluation sequence<br />Evaluation executed offline in batch mode<br />Results stored on platform<br />Analyses performed and stored on platform<br />SEALS Platform<br />API<br />Runtime Evaluation Service<br />Search tool<br />
  25. 25. User in the loop evaluation<br />4/27/10<br />15<br />Performed at tool provider site<br />All materials provided<br />Controller software<br />Instructions (leader and subjects)<br />Questionnaires<br />Data downloaded from platform<br />Results uploaded to platform<br />Over the web<br />API<br />Tool provider machine<br />SEALSPlatform<br />Search tool<br />Controller<br />
  26. 26. API<br />A range of information needs to be acquired from the tool in both phases<br />In automated phase, the tool has to be executed and interrogated with no human assistance.<br />Interface between the SEALS platform and the tool must be formalised<br />4/27/10<br />16<br />
  27. 27. API – common<br />Load ontology<br />success / failure informs the interoperability<br />Determine result type<br />ranked list or set?<br />Results ready?<br />used to determine execution time<br />Get results<br />list of URIs<br />number of results to be determined by developer<br />4/27/10<br />17<br />
  28. 28. API – user in the loop<br />User query input complete?<br />used to determine input time<br />Get user query<br />String representation of user’s query<br />if NL interface, same as text inputted<br />Get internal query<br />String representation of the internal query<br />for use with…<br />4/27/10<br />18<br />
  29. 29. API – automated<br />Execute query<br />mustn’t constrain tool type to particular format<br />tool provider given questions shortly before evaluation is executed<br />tool provider converts those questions into some form of ‘internal representation’ which can be serialised as a String<br />serialised internal representation passed to this method<br />4/27/10<br />19<br />
  30. 30. Data<br />4/27/10<br />20<br />
  31. 31. Data set – user in the loop<br />Mooney Natural Language Learning Data<br />used by previous semantic search evaluation<br />simple and well-known domain<br />using geography subset<br />9 classes<br />11 datatype properties<br />17 object properties and <br />697 instances<br />877 questions already available<br />4/27/10<br />21<br />
  32. 32. Data set – automated<br />EvoOnt<br />set of object-oriented software source code ontologies<br />easy to create different ABox sizes given a TBox<br />5 data set sizes: 1k, 10k, 100k, 1M, 10M triples<br />questions generated by software engineers<br />4/27/10<br />22<br />
  33. 33. Results and Analyses<br />4/27/10<br />23<br />
  34. 34. Questionnaires<br />3 questionnaires:<br />SUS questionnaire<br />Extended questionnaire<br />similar to SUS in terms of type of question but more detailed<br />Demographics questionnaire<br />4/27/10<br />24<br />
  35. 35. System Usability Scale (SUS) score<br />SUS is a Likert scale<br />10-item questionnaire<br />Each question has 5 levels (strongly disagree to strongly agree)<br />SUS scores have a range of 0 to 100.<br />A score of around 60 and above is generally considered as an indicator of good usability.<br />4/27/10<br />25<br />
  36. 36. Demographics<br />Age<br />Gender<br />Profession<br />Number of years in education<br />Highest qualification<br />Number of years in employment<br />knowledge of informatics<br />knowledge of linguistics<br />knowledge of formal query languages<br />knowledge of English<br />…<br />4/27/10<br />26<br />
  37. 37. Automated<br />Results<br />Execution success (OK / FAIL / PLATFORM ERROR)<br />Triples returned<br />Time to execute each query<br />CPU load, memory usage<br />Analyses<br />Ability to load ontology and query (interoperability)<br />Precision and Recall (search accuracy and query expressiveness)<br />Tool robustness: ratio of all benchmarks executed to number of failed executions<br />4/27/10<br />27<br />
  38. 38. User in the loop<br />Results (other than core results similar to automated phase)<br />Query captured by the tool<br />Underlying query (e.g., SPARQL)<br />Is answer in result set? (user may try a number of queries before being successful)<br />time required to obtain answer<br />number of queries required to answer question<br />Analyses<br />Precision and Recall<br />Correlations between results and SUS scores, demographics, etc<br />4/27/10<br />28<br />
  39. 39. Dissemination<br />Results browsable on the SEALS portal<br />Split into three areas:<br />performance<br />usability<br />comparison between tools<br />4/27/10<br />29<br />
  40. 40. Conclusions<br />4/27/10<br />30<br />
  41. 41. Conclusions<br />Methodology and design of a semantic search tool evaluation campaign<br />Exists within the wider context of the SEALS initiative<br /><ul><li>First version</li></ul>feedback from participants and community will drive the design of the second campaign<br />Emphasis on the user experience (for search)<br />Two phase approach<br />4/27/10<br />31<br />
  42. 42. Thank You<br />4/27/10<br />32<br />
  43. 43. Get involved!<br />First Evaluation Campaign in all SEALS technology areas this Summer<br />Get involved – your input and participation is crucial<br />Workshop planned for ISWC 2010 after campaign<br />Find out more (and take part!) at:<br /><br />or talk to me, or email me (<br />4/27/10<br />33<br />
  44. 44. Timeline<br />May 2010: Registration opens<br />May-June 2010: Evaluation materials and documentation are provided to participants<br />July 2010: Participants upload their tools<br />August 2010: Evaluation scenarios are executed<br />September 2010: Evaluation results are analysed<br />November 2010: Evaluation results are discussed at ISWC 2010 workshop (tbc)<br />4/27/10<br />34<br />
  45. 45. Best paper award<br />SEALS is proud to be sponsoring the best paper award here at SemSearch2010<br />Congratulations to the winning authors!<br />4/27/10<br />35<br />