User Satisfaction of a Hybrid Ontology-Engineering Tool C. Debruyne and I. Ciuciu LNCS 8186, p. 414 ff.

1. User Satisfaction of a Hybrid
Ontology-engineering Tool
Christophe Debruyne and Ioana Ciuciu
13-09-2013 @ META4eS, OTM 2013

2. Overview
‣ Motivation
‣ Background
‣ Usability Test Design
‣ User Satisfaction
‣ Results and Recommendations
‣ Conclusions and Future Work
2

3. Motivation
‣ Semantic interoperability and vocabularies for Linked Data initiatives
‣ Ontology engineering is far from trivial
‣ Requires appropriate methods and tools
‣ Hybrid Ontology Engineering platform in the context of the Open
Semantic Cloud for Brussels (OSCB) project
‣ Evaluate usability and user satisfaction of the hybrid ontology platform
3

4. Background
‣ IBM Post-study System Usability Questionnaire (PSSUQ)
4
Information Quality
System Use
Interface Quality
Overall

5. Background
‣ Hybrid Ontology Engineering
‣ Community promoted to first-class citizen
‣ Social interactions are
‣ formalized
‣ lead to ontology evolution
‣ supported by a glossary
‣ Concepts are described
‣ formally --> with binary fact-types called lexons
e.g., <Research Domain, Call, with, of, Deadline>
‣ informally --> with glosses
5

6. GOSPL Method and Tool
6

7. GOSPL Method and Tool
7

8. GOSPL Method and Tool
‣ With respect to the previous study
‣ System usefulness scored best
‣ Information quality needed most improvement
‣ Changes made to the system to tackle problems
‣ More (online) documentation
‣ Changes to stylesheets
‣ RSS to keep track of discussions
‣ Additional functionalities
‣ Tool support for managing annotations
‣ A reputation framework
8

9. Usability Test Design
‣ 23 MSc student volunteers part of an ontology engineering course
‣ Objective:
‣ prototype ontology related to research domain
‣ use ontology to annotate
‣ “reverse engineered” existing application
‣ existing research information system
‣ Evaluate formative and summative user satisfaction of the ontology
engineering platform
‣ Compare with results from a previous study with similar experiment
9

10. Usability Test Design
‣ Tasks on the platform
‣ Propose changes to the hybrid ontology
‣ Discuss and vote
‣ Conclude discussions --> ontology evolution
‣ Create and manage communities
‣ Annotate information systems
‣ Participants
‣ 36 students, 9 groups for the ontology engineering experiment
‣ Background in computer science or similar
‣ 23 students participated in the survey (of which 4 complete groups)
10

11. Summative User Satisfaction
‣ Summative evaluation
‣ System usefulness and interface quality remained the same
‣ Information quality and overall satisfaction improved
‣ Formative evaluation
‣ Tracking “hottest” discussions
‣ Correcting mistakes
‣ Voting mechanism (!) and verbalization of constraints (!)
‣ Positive outcome given the problems encountered with application
management and the reputation framework
11
[2] Ioana Ciuciu, Christophe Debruyne:Assessing the User Satisfaction with an Ontology Engineering Tool Based
on Social Processes. OTM Workshops 2012: 242-251

12. Recommendations
‣ Overview of discussions need improvement
‣ (Error-) management of application commitments
‣ Verbalization
‣ Removing voting mechanism
12

13. Conclusions and Future Work
‣ Conclusions
‣ System usefulness performs best
‣ Information quality improved a lot
‣ Overall satisfaction remained stable
‣ Given the fact that new functionalities posed problems, this can be
interpreted as a considerable improvement of the tool
‣ Future work
‣ Third iteration planned in May 2014
‣ Discussion overview + lightweight user participation
13

14. Questions?