A variety of query approaches have been proposed by the semantic web community to explore and query semantic data. Each was developed for a specific task and employed its own interaction mechanism; each query mechanism has its own set of advantages and drawbacks. Most semantic web search systems employ only one approach, thus being unable to exploit the benefits of alternative approaches. Motivated by a usability and interactivity perspective, we propose to combine two query approaches (graph-based and natural language) as a hybrid query approach. In this paper, we present NL-Graphs which aims to exploit the strengths of both approaches, while ameliorating their weaknesses. NL-Graphs was conceptualised and developed from observations, and lessons learned, in several evaluations with expert and casual users. The results of evaluating our approach with expert and casual users on a large semantic dataset are very encouraging; both types of users were highly satisfied and could effortlessly use the hybrid approach to formulate and answer queries. Indeed, success rates showed they were able to successfully answer all the evaluation questions.

1. NL-Graphs: A Hybrid Approach toward
Interactively Querying Semantic Data
Khadija Elbedweihy
Suvodeep Mazumdar
Stuart N. Wrigley
Fabio Ciravegna
OAK Group
University of Sheffield
Sheffield, UK
k.Elbedweihy | s.mazumdar | s.wrigley | f.ciravegna | @ shef.ac.uk
ovus00

2. Users and User Experience
• Semantic Search System
• Assist users satisfy their information needs
• User Satisfaction
– Query approach
– Performance of Search system
– Presentation of results
– ….
• Usability and User Experience largely ignored
– Need to focus on users

3. Query Approaches
• Types
• Interaction mechanisms
• Formalisms (Natural language – Forms – Formal Query)
• Often Task or application Specific
– Keyword-based approach to quickly identify possible relevant
instances
– Formal query for more complex tasks (negation, group-by etc.)
• Most SW applications focus on one approach

4. Esther Kaufmann and Abraham Bernstein. Evaluating the usability of natural language query languages and
interfaces to semantic web knowledge bases. Journal of Web Semantics, 8(4):377–393, 2010.
Spectrum of Query Approaches
4

5. Issues + Motivations
• Comparison of Query Approaches (Expert and Casual) [2]
 NL Approach
+ Prior experience with Google-like approaches
+ Easy and quick to write questions/keywords
- Less support for Complex queries
 View Based Approach
+ Good support in complex queries
+ Visualising the search space
- Time and effort required formulating queries
• Learnability study with Graph based approach (Experts) [3]
+ Improvement in user satisfaction and performance over time
- Still needed time and effort to formulate queries
- Extensive use of keyword-search for concepts
[2] K. Elbedweihy, S. N. Wrigley, and F. Ciravegna. Evaluating Semantic Search Query Approaches
with Expert and Casual Users. In Proceedings of the Evaluations and Experiments Track, ISWC,
2012.
[3] S.Mazumdar, D.Petrelli, K. Elbedweihy, V. Lanfranchi, and F. Ciravegna. Affective Graphs: The
visual appeal of linked data. Semantic web : interoperability, usability, applicability, 2013. In Press.

6. Affective Graphs

7. Affective Graphs
• Demo

8. HybridGraphs
• NL [4] + Graph [3] = Hybrid Query Approach
• Builds on Advantages of either approach
• Quick access to relevant data
• Easy and visual means to formulate complex queries
• Visually inspect search space
• Alleviates the disadvantages
• Lesser time taken to build complex queries
[3] S.Mazumdar, D.Petrelli, K. Elbedweihy, V. Lanfranchi, and F. Ciravegna. Affective Graphs: The visual appeal of linked
data. Semantic web : interoperability, usability, applicability, 2013. In Press.
[4] K. Elbedweihy, S.N. Wrigley, F. Ciravegna, and Z. Zhang (2013). Using BabelNet in bridging the gap between natural
language queries and linked data concepts. NLP & DBpedia Workshop, 12th International Semantic Web Conference
(ISWC2013), 21-25 October, Sydney, Australia.

9. Hybrid Approach

10. Evaluation
• Previous evaluations on NL component and Affective Graphs [3,4]
(performance, usability, effectiveness etc.)
• Focus on hybrid approach as a new query mechanism and the
support provided
• Methodology
– Expert (SW experts in Computer Science + Software development
firm) and Casual users (staff and students at the University of
Sheffield)
– 24 subjects (12 + 12)
– Aged between 18 and 53, mean age 31 years
– Short demonstration of a scenario based example
– Familiarisation with the system
[3] S.Mazumdar, D.Petrelli, K. Elbedweihy, V. Lanfranchi, and F. Ciravegna. Affective Graphs: The visual appeal of linked
data. Semantic web : interoperability, usability, applicability, 2013. In Press.
[4] K. Elbedweihy, S.N. Wrigley, F. Ciravegna, and Z. Zhang (2013). Using BabelNet in bridging the gap between natural
language queries and linked data concepts. NLP & DBpedia Workshop, 12th International Semantic Web Conference
(ISWC2013), 21-25 October, Sydney, Australia.

11. Evaluation Tasks
• 5 questions derived from QALD dataset [5]
• Different levels of difficulty
– 1. When was Capcom founded?
– 2. What did Bruce Carver die from?
– 3. Who was the wife of U.S. president Lincoln?
– 4. Give me all cities in Alaska with more than 10000 inhabitants.
– 5. Show me all songs from Bruce Springsteen released between 1980
and 1990.
[5] V. Lopez, C. Unger, P. Cimiano, and E. Motta. Evaluating question answering over linked
data. Web Semantics: Science, Services and Agents on the World Wide Web, 2013.

12. Evaluation Analysis
• Objective and Subjective data collected
• Input Time, number of attempts, Success rate
• Post evaluation questionnaires – System Usability Scale (SUS),
Extended questionnaire
– The query construction process was X. (5-point Likert scale, ranging
from Laborious to Effortless)
– What did you like about the hybrid approach as a mechanism for
expressing your query? and why?
– What things you didn't like about the hybrid approach as a mechanism
for expressing your query? and why?
• Aesthetics questionnaire

13. Results
• Overall, experts (73.75) liked NL Graphs more than casuals (61.25)
– Graph based approach more complicated
– Experts more familiar with Semantic Web and graph data
• Success rate 100% (all users were able to successfully complete the
tasks) mostly on one attempt
• Query Construction process scored effortless
• Most users commented positively on the usability and support
provided by the hybrid approach
• Four users (1e + 3c)
– Complicated and non-intuitive
– Why are there more steps required to build queries than with Google?
– “It seemed an extra step to get to your answer rather than just typing
in a search and it appearing in results.”

14. Results Contd.
• More assistance needed/desired
• Need for more Training
• Experts took longer (94.48s) to formulate queries than casual
(76.88s) users
– Follow logic and prior understanding of SW concepts
– Increased interest in the interface, trying out new features, exploring
technical aspects (e.g. SPARQL query generated)

15. Results Contd.

16. Results Contd.
• Q4 – Give me all cities in Alaska with
more than 10000 inhabitants
– Use of terms like “Alaskan”
– DbPedia properties such as isPartOf
– Numerical constraints >10000
• Q5 - Show me all songs from Bruce Springsteen released between
1980 and 1990
– Setting dates as a constraint is a manual process

17. Summary
• Hybrid approach to querying Linked Data
• It works ! 
• High SUS scores indicate usability and usefulness
• Future work
– Several re-designs identified/suggested in the evaluation
– A longer term study directly comparing Hybrid approach to other
approaches in identical conditions
– Study with larger set of user groups
– Improving NL system
[3] S.Mazumdar, D.Petrelli, K. Elbedweihy, V. Lanfranchi, and F. Ciravegna. Affective Graphs: The visual appeal of linked
data. Semantic web : interoperability, usability, applicability, 2013
Aesthetic design in Semantic Web Applications [3]

18. Thank You
• Questions ?