Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Protégé4US: Harvesting Ontology Authoring Data with Protégé

1,123 views

Published on

The inherent complexity of ontologies poses a number of cognitive and perceptual challenges for ontology authors. We investigate how users deal with the complexity of the authoring process by analysing how one of the most widespread ontology development tools (i.e. Protégé) is used. To do so, we build Protégé4US (Protégé for User Studies) by extending Protégé in order to generate log files that contain ontology authoring events. These log files not only contain data about the interaction with the environment, but also about OWL entities and axioms. We illustrate the usefulness of Protégé4US with a case study with 15 participants. The data generated from the study allows us to know more about how Protégé is used (e.g. most frequently used tabs), how well users perform (e.g. task completion times) and identify emergent authoring strategies, including moving down the class hierarchy or saving the cur- rent workspace before running the reasoner. We argue that Protégé4US is an valuable instrument to identify ontology authoring patterns.

Published in: Science
  • Be the first to comment

  • Be the first to like this

Protégé4US: Harvesting Ontology Authoring Data with Protégé

  1. 1. Protégé4US: Harvesting Ontology Authoring Data with Protégé Markel Vigo, Caroline Jay, Robert Stevens firstname.lastname@manchester.ac.uk @markelvigo, @CarolineEJay, @stevensrd65 Workshop on Human-Semantic Web Interaction, HSWI 2014 May 26. Crete (Greece)
  2. 2. Little is known about the human factors of ontology authoring Protégé4US: Harvesting Ontology Authoring Data with Protégé HSWI @ ESWC 2014 • What we know is based on anecdotal evidence • We asked about problems and strategies – Making sense – Search and retrieval – Efficient population – On-the-fly reasoning – Overloaded explanations – Lack of evaluation methods Design insights for the next wave ontology authoring tools. CHI 2014 http://dx.doi.org/10.1145/2556288.2557284 Introduction > Protégé4US > Results > Conclusion
  3. 3. Lack of sound HCI research in the Semantic Web Protégé4US: Harvesting Ontology Authoring Data with Protégé HSWI @ ESWC 2014 • HCI does not pervade all computing disciplines • Instruments to run user studies are scarce • Consequences for the OWL realm – No understanding about the authoring process – Authoring tools are not human-centered • What if we want to go further? – Automatic detection of authoring patterns – Intelligent support for authoring Introduction > Protégé4US > Results > Conclusion
  4. 4. Protégé4US: a step towards having observational instruments Protégé4US: Harvesting Ontology Authoring Data with Protégé HSWI @ ESWC 2014 • Protégé4US: Protégé for User Studies • Logging capabilities of: – Interaction events: click, hover, expand hierarchy... – Authoring events: add siblings, add restrictions... – Environment commands: reason, search, undo... 76585,2,Classes,Element edited,Juliette subclass of: Potato and hasCroppingTime some ’Main cropping’ 77786,3,Classes,Save ontology,http://owl.cs.manchester.ac.uk/ontology/start-here.owl 80204,3,Classes,Reasoner invoked,HermiT 1.3.8 80647,1,Classes,Mouse entered, Class hierarchy (inferred) 82910,1,Classes,Element hovered,Early_cropping_potato 83049,1,Classes,Element selected,Early_cropping_potato 83661,1,Classes,Hierarchy expanded,Early_cropping_potato Introduction > Protégé4US > Results > Conclusion
  5. 5. User study to show the strengths of Protégé4US Protégé4US: Harvesting Ontology Authoring Data with Protégé HSWI @ ESWC 2014 • Experimental design: – Participants: 15 expert authors – Stimuli: a potato ontology and Protégé4US – 3 authoring tasks with an increased complexity • Collected data (apart from Protégé4US logs): – Completion times – Self reported expertise – Perceived task difficulty Introduction > Protégé4US > Results > Conclusion
  6. 6. Protégé4US in action Protégé4US: Harvesting Ontology Authoring Data with Protégé HSWI @ ESWC 2014 Introduction > Protégé4US > Results > Conclusion
  7. 7. Data analysis to check the intention of the experimental design Protégé4US: Harvesting Ontology Authoring Data with Protégé HSWI @ ESWC 2014 • Tasks given had an increased complexity – Completion times significantly different: T1<T2<<T3 – Task difficulty significantly different: T1<T2<T3 – Expertise with Protégé negatively correlated with completion times for T1 – No correlation between completion times and perceived difficulty • Completion time indicator of editing action than of cognitive difficulty Introduction > Protégé4US > Results > Conclusion
  8. 8. Some events may be indicators of problematic situations Protégé4US: Harvesting Ontology Authoring Data with Protégé HSWI @ ESWC 2014 • Positive and significant correlations between task completion time and: – Expansions of the class hierarchy – Running the reasoner – Renaming entities • Log analysis allows to profile users based on their tab use Introduction > Protégé4US > Results > Conclusion
  9. 9. Reconstructing the interaction allows to identify patterns through visualisation Protégé4US: Harvesting Ontology Authoring Data with Protégé HSWI @ ESWC 2014 • Web diagrams show the most frequent transitions between statesP15 log Back Check property Class addition Convert into defined class:finished Convert into defined class:start Entity renamed Explanation invoked Entity deleted Entity dragged Entity edited:finishEntity edited:start Entity selected Link clicked Property addition Reasoner finished Reasoner invoked Save Set active ontology Hierarchy collapsed Hierarchy expanded Undo P9 log Back Check property Class addition Convert into defined class:finished Convert into defined class:start Entity renamed Explanation invoked Entity deleted Entity dragged Entity edited:finishEntity edited:start Entity selected Link clicked Property addition Reasoner finished Reasoner invoked Save Set active ontology Hierarchy collapsed Hierarchy expanded Undo Introduction > Protégé4US > Results > Conclusion
  10. 10. Reconstructing the interaction allows to identify patterns through visualisation Protégé4US: Harvesting Ontology Authoring Data with Protégé HSWI @ ESWC 2014 • Time diagrams show the authoring rhythm Introduction > Protégé4US > Results > Conclusion
  11. 11. Reconstructing the interaction allows to identify patterns through visualisation Protégé4US: Harvesting Ontology Authoring Data with Protégé HSWI @ ESWC 2014 • The analysis of diagrams across users allows to sketch decision trees – Reasoner invoked after 1. A class is converted into a defined class 2. Ontology is saved 3. An entity is selected – Hierarchy is expanded after 1. Reasoner finishes 2. Hierarchy is expanded 3. Entity addition is invoked Introduction > Protégé4US > Results > Conclusion
  12. 12. Future work and Conclusions Protégé4US: Harvesting Ontology Authoring Data with Protégé HSWI @ ESWC 2014 • Future work – Statistical analysis of patterns and identification of strategies – Incorporation of eye-tracking data – Ontology authoring into the wild • Conclusions – Protégé4US may give urgency to a more human- centred Semantic Web – Better tools for ontology and linked data authoring Introduction > Protégé4US > Results > Conclusion
  13. 13. Protégé4US: Harvesting Ontology Authoring Data with Protégé Markel Vigo, Caroline Jay, Robert Stevens firstname.lastname@manchester.ac.uk @markelvigo, @CarolineEJay, @stevensrd65 Workshop on Human-Semantic Web Interaction, HSWI 2014 May 26. Crete (Greece) WhatIf: Answering “What if...” questions for Ontology Authoring. EPSRC reference EP/J014176/1

×