• Save
Evaluating Semantic Search Systems to Identify Future Directions of Research
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


Evaluating Semantic Search Systems to Identify Future Directions of Research

Uploaded on

Recent work on searching the Semantic Web has yielded a wide range of approaches with respect to the style of input, the underlying search mechanisms and the manner in which results are presented.......

Recent work on searching the Semantic Web has yielded a wide range of approaches with respect to the style of input, the underlying search mechanisms and the manner in which results are presented. Each approach has an impact upon the quality of the information retrieved and the user's experience of the search process. This highlights the need for formalised and consistent evaluation to benchmark the coverage, applicability and usability of existing tools and provide indications of future directions for advancement of the state-of-the-art. In this paper, we describe a comprehensive evaluation methodology which addresses both the underlying performance and the subjective usability of a tool. We present the key outcomes of a recently completed international evaluation campaign which adopted this approach and thus identify a number of new requirements for semantic search tools from both the perspective of the underlying technology as well as the user experience.

More in: Technology , Design
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. IWEST 2012 workshop located at ESWC 2012 Evaluating Semantic Search Systems to Identify Future Directions of Research Khadija Elbedweihy1, Stuart N. Wrigley1, Fabio Ciravegna1, Dorothee Reinhard2, Abraham Bernstein2 1University of Sheffield, UK 18.06.2012 2University of Zurich, Switzerland 1
  • 2. Outline• Introduction• Evaluation Design• Evaluation Execution• Usability Feedback and Analysis• Future Directions for Research• Conclusions18.06.20122
  • 3. INTRODUCTION18.06.20123
  • 4. Semantic Search• Semantic Search tools have different • querying approaches (e.g., forms, graphs, keywords). • search strategies during processing and query execution. • format and content of the results presented to the user.• These factors influence the users perceived performance usability of the tool.• Searching is a user-centric process; usability evaluation is as important as – if not more than – assessing the performance.18.06.20124
  • 5. Previous evaluation efforts• Kaufmann (2007): compared 4 SW query interfaces (NL and Graph- based)• SemSearch Challenge: ad-hoc object retrieval using keywords• Question Answering Over Linked Data (QALD): two NL interfaces• TREC Entity List Completion (ELC) Task: similar to SemSearch• All previous evaluations based upon the Cranfield methodology – test collection; set of tasks; set of relevance judgments.• Little or no focus on usability18.06.20125
  • 6. EVALUATION DESIGN18.06.20126
  • 7. Evaluation DesignAspect DetailsTools • Any query input style • Answers extracted from data (e.g., list of URIs or literals but not documents)Data Mooney Natural Language Learning Data • known within the search community • simple and well-known domain for subjects (geography) • questions already available • Give me all the state capitals of the USA? • Which rivers in Arkansas are longer than Alleghany river?Subjects 38 subjects (26 males, 12 females); aged between 20 and 35 years oldCriteria • Usability: • query input (expressiveness, etc.) • usefulness and suitability of returned answers (data) and presentation • Performance: speed of execution (also affects user satisfaction) 18.06.2012 7
  • 8. Data Captured• Results for each question: – time required to formulate query – number of attempts required to answer question – success rate (user found satisfying answer or not) – query execution time• Questionnaires capturing user experience – System Usability Scale (SUS) questionnaire – Extended questionnaire – Demographics questionnaire04.08.20108
  • 9. EVALUATION EXECUTION18.06.20129
  • 10. Participating toolsTool DescriptionK-Search Form-basedGinseng Natural language with constrained vocabulary and grammarNLP-Reduce Natural language for full English questions, sentence fragments, and keywords.PowerAqua Natural language interface18.06.201210
  • 11. Running the experiment18.06.201211
  • 12. ANALYSIS AND FEEDBACK18.06.201212
  • 13. ResultsCriterion K-Search Ginseng Nlp- PowerAqua ‘Bad’ Bad Form- Controlled Reduce NL-based based NL-based NL-basedMean experiment time (s) 4313.84 3612.12 4798.58 2003.9 ‘Awful’Mean SUS (0 – 100) 44.38 40 25.94 72.25 ‘Good’Mean Ext.Questionnaire (0-100) 47.29 45 44.63 80.67Mean number of attempts 2.37 2.03 5.54 2.01 Twice # of attemptsMean answer found rate 0.41 0.19 0.21 0.55Mean execution time (s) 0.44 0.51 0.51 11 slowestMean input time (s) 69.11 81.63 29 16.03 slowest 18.06.2012 13
  • 14. Feedback: input styleInput Positive NegativeFree NL • fast (16 and 29 sec on average) mismatch (habitability) problem: “I need to • most natural (query in plain natural know and use the terms expected by the language) system and not my own terms to get results”Contr. NL • guidance: suggestions and auto- very restricted language model: completion • frustration (low SUS) • avoids habitability problem (only valid • limit flexibility and expressiveness queries) • slow query formulation (highest input time: 81.63 sec)Form • allow users to build more complex • more difficult to use than NL queries than NL • time consuming (input time: 69.11 sec on • helpful to know the search space average) (concepts & relations) 18.06.2012 14
  • 15. Feedback: resultsAspect CommentsPresentation Results not user-friendly • provided full URIs of the concepts (e.g. `http://www.mooney.net/geo#tennesse2’) • used ontology labels for providing a NL representation of the answer (e.g. `montgomeryAI’)Management Users have high expectations; requested advanced means of managing the results such as: • storing and reusing results of previous queries • filtering results according to some suitable criteria • checking the provenance of the results • basic manipulations such as sorting results18.06.201215
  • 17. Input Style• Visualising the search space shows: • what type of information is available (exploration) • what queries are supported (query formulation guidance).• Typing queries in natural language is fast and easy• Provide ‘dual query formulation’ approach • users unfamiliar with domain can correctly formulate their intended queries using view-based • users familiar with domain can use faster NL queries18.06.201217
  • 18. Input Style• Comparatives and Superlatives still a challenge e.g., FREyA uses an ‘intervention approach’ • if a numerical datatype property is found in user query: 1. generates maximum, minimum and sum functions 2. user chooses the required function18.06.201218
  • 19. Query ExecutionDelays in response time negatively affect user experience andsatisfaction.• Provide feedback • reduces the effect of delays (more willing to wait if they know the status of their search process).• Provide intermediate (partial) results • gradually incremented to provide the complete result set. • similar to (arguably better than) basic feedback18.06.201219
  • 20. Results • Presentation • Attractive, accessible, understandable and user-friendly. • Augment with associated information: `richer’ user experience. • Management • Filter, sort • Some complex questions require multiple sub-queries • Ability to store and reuse the result set could be helpful. • Queries can then be constructed by combining saved queries with logical operators such as `AND and `OR’.18.06.201220
  • 21. CONCLUSIONS18.06.201221
  • 22. Conclusions & Recommendations• Query input approaches serve different purposes: – View-based: explore and understand – NL-based: efficiency and simplicity• Dual query approach to input – natural language and view-based input styles – improve search effectiveness and user satisfaction• More sophisticated results presentation and management – customise: sort, filter, provenance and (temporary save) – enrich: supplementary information18.06.201222
  • 23. THANK YOU18.06.201223