Evaluating Semantic Search QueryApproaches with Expert and Casual Users               Khadija Elbedweihy, Stuart N. Wrigle...
Outline•   Motivation•   Research Question•   Evaluation Design•   Evaluation Setup•   Findings•   Conclusions
Motivation – Semantic Search• Wikipedia states that Semantic Search “seeks to improve  search accuracy by understanding se...
Motivation - Evaluations• Evaluation of software is critical.• Large-scale evaluations foster research and development.• S...
Research QuestionHow do different types of users perceive the usabilityof different query approaches?• Method  - Assess us...
Query Approaches                                           Controlled-NLFree-NL                                           ...
Evaluation Design: Dataset• Mooney Natural Language Learning Data   - simple and well-known domain (geography)   - used by...
Evaluation Design: Data Collected• Objective data:  1) Input time  2) Number of attempts  3) Success rate• Subjective data...
Evaluation Setup• 20 subjects   – 10 casual users, 10 expert users   – 12 females, 8 males• Within-subjects: allows direct...
Results• Evaluated tools:   – Free-NL: NLP-Reduce   – Controlled-NL: Ginseng   – Form-based: K-Search   – Graph- based:   ...
QUERY APPROACH
Results for expert users     • Expert users prefer graph- and form- based approach.     • View-based allow more complex qu...
Results for casual users         • Casual users prefer form-based query approach.         • Required less input time than ...
ONTOLOGY VISUALIZATION
Results for expert users• Visualizing the entire ontology supports query formulation   – Semantic Crystal: shows the entir...
Results for casual users• Not showing ontology more complex for casual users:  – Semantic Crystal receiving higher scores....
CONTROLLED-NL APPROACH
Results for expert users    • Controlled-NL very restrictive for expert users (least-liked)    • Highest query input time ...
Results for casual users• Controlled-NL provided most support for casual users.• Users’ positive feedback for controlled-N...
RESULTS INDEPENDENT OF USER TYPE
Free-NL approach  + simplest and most natural  - suffer from habitability problem.• Feedback:     “I have to guess the rig...
Negation• Tell me which rivers do not traverse the state with the  capital Nashville?                           1         ...
Negation  Tell me which states does the river Mississippi does not  traverse.• “Closed world assumption (CWA): presumption...
Formal Query• Formal Query (e.g., SPARQL)
Formal Query• Benefit of showing formal query depends on user type.• Formal query perceived by:   – Casual users: not unde...
Results presentation• Results presentation and format affected usability and user  satisfaction.   – Unless users are very...
Results Content• Results should be augmented with associated information  to provide a `richer’ user experience.• Users fe...
CONCLUSIONS
Research Question & ApproachHow do different types of users perceive the usabilityof different query approaches?  - Assess...
ConclusionsExpert Users                            Casual Users• Graph-based most preferred            • Form-based mid-po...
RecommendationsCater to both expert and casual users:• Hybridized query approach: Combine a view-based  approach (visualiz...
Limitations & Future work• Limitation: Small size of the dataset.• Assess learnability of different query approaches.• Ass...
QuestionsQuestions?
Upcoming SlideShare
Loading in …5
×

Evaluating Semantic Search Query Approaches with Expert and Casual Users

587 views

Published on

Usability and user satisfaction are of paramount importance
when designing interactive software solutions. Furthermore, the optimal
design can be dependent not only on the task but also on the type of
user. Evaluations can shed light on these issues; however, very few studies
have focused on assessing the usability of semantic search systems.
As semantic search becomes mainstream, there is growing need for standardised,
comprehensive evaluation frameworks. In this study, we assess
the usability and user satisfaction of di erent semantic search query input
approaches (natural language and view-based) from the perspective
of di erent user types (experts and casuals). Contrary to previous studies,
we found that casual users preferred the form-based query approach
whereas expert users found the graph-based to be the most intuitive.
Additionally, the controlled-language model o ered the most support for
casual users but was perceived as restrictive by experts, thus limiting
their ability to express their information needs.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
587
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
11
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Evaluating Semantic Search Query Approaches with Expert and Casual Users

  1. 1. Evaluating Semantic Search QueryApproaches with Expert and Casual Users Khadija Elbedweihy, Stuart N. Wrigley and Fabio Ciravegna OAK Research Group, Department of Computer Science, University of Sheffield, UK
  2. 2. Outline• Motivation• Research Question• Evaluation Design• Evaluation Setup• Findings• Conclusions
  3. 3. Motivation – Semantic Search• Wikipedia states that Semantic Search “seeks to improve search accuracy by understanding searcher intent and the contextual meaning of terms as they appear in the searchable dataspace, whether on the Web or within a closed system, to generate more relevant results”.• Covers broad category of applications in Semantic Web: – Search engines (e.g., Swoogle, FalconS, Sindice) – Closed-domain query interfaces (e.g., AquaLog, Querix) – Open-domain query interfaces (e.g., PowerAqua)
  4. 4. Motivation - Evaluations• Evaluation of software is critical.• Large-scale evaluations foster research and development.• Semantic search evaluations (SemSearch, TREC ELC, QALD) focused on assessing retrieval performance. Assessing usability of tools and user satisfaction is important in Semantic Search.
  5. 5. Research QuestionHow do different types of users perceive the usabilityof different query approaches?• Method - Assess usability and user satisfaction of: * Free-NL, Controlled-NL, Form-based, Graph-based - from the perspective of * expert users and casual users
  6. 6. Query Approaches Controlled-NLFree-NL Specific vocabularyNatural language queries Which state has river Submit capitalWhat is the capital of Alabama? Submit lake mountai capital Alabama Submit n a anyForm-based Graph-basedVisualize the Visualize thesearch space search space
  7. 7. Evaluation Design: Dataset• Mooney Natural Language Learning Data - simple and well-known domain (geography) - used by other studies within the search community - questions already available (877 NL questions)• Geography Dataset: – Concepts: State, City, Lake, Mountain, Capital, River, etc – Properties: population of state, length of river, etc – Relations linking concepts: State ‘hasCity’ City
  8. 8. Evaluation Design: Data Collected• Objective data: 1) Input time 2) Number of attempts 3) Success rate• Subjective data, collected using: 1) Questionnaires (e.g., System Usability Scale ‘SUS’) 2) Ranking of the tools (w.r.t: system, query approach, results content, results presentation) 3) Observations
  9. 9. Evaluation Setup• 20 subjects – 10 casual users, 10 expert users – 12 females, 8 males• Within-subjects: allows direct comparison.• Randomising tool order: normalize learning or tiredness effects.• Randomising question order: normalize learning effects.
  10. 10. Results• Evaluated tools: – Free-NL: NLP-Reduce – Controlled-NL: Ginseng – Form-based: K-Search – Graph- based: • Semantic-Crystal (Graph-based 1) • Affective Graphs (Graph-based 2)
  11. 11. QUERY APPROACH
  12. 12. Results for expert users • Expert users prefer graph- and form- based approach. • View-based allow more complex queries than NL-based.Best 1 0.9 0.8Query Language Rank 0.7 Graph-based1 0.6 Graph-based2 0.5 Form-based 0.4 Controlled-NL 0.3 Free-NL 0.2 0.1 0 Worst
  13. 13. Results for casual users • Casual users prefer form-based query approach. • Required less input time than graph-based approach. Best 100 1 90 0.9 Query Language Rank 80 0.8 70 0.7 Graph-based1Input Time (Sec) 60 0.6 Graph-based2 50 0.5 Form-based 40 0.4 Controlled-NL 30 0.3 Free-NL 20 0.2 10 0.1 0 0 Worst
  14. 14. ONTOLOGY VISUALIZATION
  15. 15. Results for expert users• Visualizing the entire ontology supports query formulation – Semantic Crystal: shows the entire ontology. – Affective Graphs: shows selected concepts & relations.
  16. 16. Results for casual users• Not showing ontology more complex for casual users: – Semantic Crystal receiving higher scores. – Affective Graphs perceived as complex and difficult to use • 50% of the users found it to increase complexity and difficulty
  17. 17. CONTROLLED-NL APPROACH
  18. 18. Results for expert users • Controlled-NL very restrictive for expert users (least-liked) • Highest query input time 120 Best 1 0.9 100 0.8 Query Language Rank 0.7 Graph-based1 80Input Time (Sec) 0.6 Graph-based2 60 0.5 Form-based 0.4 Controlled-NL 40 0.3 Free-NL 20 0.2 0.1 0 0 Worst
  19. 19. Results for casual users• Controlled-NL provided most support for casual users.• Users’ positive feedback for controlled-NL: – allow only correct queries (50%) – suggestions and guidance to formulate queries (40%) Example: Although Ginseng is limited to specific vocabulary, I knew that I will get answers once I can do the query because it only allows the correct ones and thus I didnt keep trying a lot of queries that I wasnt sure about.
  20. 20. RESULTS INDEPENDENT OF USER TYPE
  21. 21. Free-NL approach + simplest and most natural - suffer from habitability problem.• Feedback: “I have to guess the right words” – Example: `run through’ with `river’ but not `traverse’.• NLP-Reduce: – lowest success rate: 20% – highest number of attempts: 4.2
  22. 22. Negation• Tell me which rivers do not traverse the state with the capital Nashville? 1 0.9 0.8 0.7 Answer Found Rate Graph-based1 0.6 Graph-based2 0.5 Form-based 0.4 Controlled-NL 0.3 Free-NL 0.2 0.1 0 Expert Users Casual Users
  23. 23. Negation Tell me which states does the river Mississippi does not traverse.• “Closed world assumption (CWA): presumption that what is not currently known to be true is false”. <Mississippi, traverse, Louisiana>• “Open world assumption (OWA): assumption that the truth-value of a statement is independent of whether or not it is known by any single observer or agent to be true”. <Mississippi, not_traverse, Alabama>
  24. 24. Formal Query• Formal Query (e.g., SPARQL)
  25. 25. Formal Query• Benefit of showing formal query depends on user type.• Formal query perceived by: – Casual users: not understandable and confusing – Expert users: increased confidence Also, performing direct changes to the formal query increased the expressiveness of the query language.
  26. 26. Results presentation• Results presentation and format affected usability and user satisfaction. – Unless users are very familiar with the data, presenting URIs alone is not very helpful. – Example: A query for rivers returns one of the answers: http://www.mooney.net/geo#tennesse2
  27. 27. Results Content• Results should be augmented with associated information to provide a `richer’ user experience.• Users feedback: – Maybe a `mouse over function to show more information. – Perhaps related information with the results. – Results very limited, would be good to have more context.
  28. 28. CONCLUSIONS
  29. 29. Research Question & ApproachHow do different types of users perceive the usabilityof different query approaches? - Assess usability and user satisfaction of: * Free-NL, Controlled-NL, Form-based, Graph-based - from the perspective of * expert users and casual users
  30. 30. ConclusionsExpert Users Casual Users• Graph-based most preferred • Form-based mid-point - Intuitive - Allow more complex queries than - Support complex queries NL. - Easier than graph-based• Controlled-NL least preferred - Faster than graph-based - Very restrictive. - Limited expressiveness • Controlled-NL most supportive• Prefer flexibility of free-NL - Only valid queries: Confidence• Formal query provides confidence - Vocabulary suggestions: guidance - Ability to change query increases • Formal Query not understandable expressiveness. and confusing. • Users want search results to be augmented with more information to have a better understanding of the answers.
  31. 31. RecommendationsCater to both expert and casual users:• Hybridized query approach: Combine a view-based approach (visualize search space) with a NL-input feature (balance difficulty and speed) while including optional suggestions for the NL input (provide guidance).• Results Content: Augment results with ‘extra’ and ‘related’ information. – extra information: for ‘State’: capital, area, population. – related information: for ‘State’: rivers, lakes, mountains.
  32. 32. Limitations & Future work• Limitation: Small size of the dataset.• Assess learnability of different query approaches.• Assess how interaction with the search tools affect the information seeking process: usefulness. – Use questions with an overall goal and compare users knowledge before and after the search task.
  33. 33. QuestionsQuestions?

×