SlideShare a Scribd company logo
Constructing Conceptual Knowledge Artefacts: 

Activity Patterns in the Ontology Authoring Process
Markel Vigo, Caroline Jay, Robert Stevens
University of Manchester (UK)

CHI 2015, Seoul (Korea)
@markelvigo
markel.vigo@manchester.ac.uk
What are ontologies?
Mad cow
Lion
Giraffe
Cow
Animal
Classes
eats
SubClassOf
PropertiesAxioms
Cow, Giraffe, Lion à SubClassOf Animal
Mad cow à SubClassOf Cow
Mad cow, lion à Animal eats some Animal
What are ontologies?
Mad cow
SubClassOf
Lion
Giraffe
 Cow
Animal
Classes
PropertiesAxioms
Cow, Giraffe, Lion à SubClassOf Animal
Mad cow à SubClassOf Cow
Mad cow, lion à Animal eats some Animal
eats
What are ontologies?
Mad cow
SubClassOf
Lion
Giraffe
 Cow
Animal
Classes
PropertiesAxioms
Cow, Giraffe, Lion à SubClassOf Animal
Mad cow à SubClassOf Cow
Mad cow, lion à Animal eats some Animal
Vegetarian à Animal and (eats only (not (Animal)))
eats
What are ontologies?
Mad cow
SubClassOf
Lion
Giraffe
 Cow
Animal
Classes
PropertiesAxioms
Cow, Giraffe, Lion à SubClassOf Animal
Mad cow à SubClassOf Cow
Mad cow, lion à Animal eats some Animal
Vegetarian à Animal and (eats only (not (Animal)))
eats
Vegetarian
What are ontologies?
Mad cow
SubClassOf
Lion
Giraffe
 Cow
Animal
Classes
PropertiesAxioms
Cow, Giraffe, Lion à SubClassOf Animal
Mad cow à SubClassOf Cow
Mad cow, lion à Animal eats some Animal
Vegetarian à Animal and (eats only (not (Animal)))
eats
Vegetarian
Complex artefacts
•  Large size
•  Domain expertise required
•  Critical
•  Highly expressive
Problem
We don’t know...
§  if existing ontology authoring tools are
effective
§  how ontologist go about authoring
§  the authoring workflows
Authoring environment: Protégé
Authoring environment: Protégé
File menu
Class hierarchy
Property hierarchy
Annotations-Usage
Description
Mad cow
SubClassOf
Lion
Giraffe
 Cow
Animal
Classes
Properties
Axioms
Cow, Giraffe, Lion à SubClassOf Animal
Mad cow à SubClassOf Cow
Mad cow, lion à Animal eats some Animal
Vegetarian à Animal and (eats only (not (Animal)))
eats
File menu
Class hierarchy
Property hierarchy
Annotations-Usage
Description
Pop up
Explanation
Edit entity
Authoring environment: Protégé
Mad cow
SubClassOf
Lion
Giraffe
 Cow
Animal
Classes
Properties
Axioms
Cow, Giraffe, Lion à SubClassOf Animal
Mad cow à SubClassOf Cow
Mad cow, lion à Animal eats some Animal
Vegetarian à Animal and (eats only (not (Animal)))
eats
Study
•  16 ontology authors, 3 typical tasks
•  Protégé à Protégé4US 
•  Eye-tracking
•  Sync and merge interaction + fixation data
•  N-gram analysis
1389973572771,eye,Class hierarchy!
1389973577038,eye,Description!
1389973584775,log,EntitySelected!
1389973586349,log,DescriptionSelected!
1389973598978,log,EntityModified!
1389973603166,log,EntitySelected!
1389973605053,log,DescriptionSelected!
1389973607847,log,EntityModified!
1389973616404,eye,Description!
1389973616754,eye,Class hierarchy!
1389973617221,eye,Description!
Findings: interaction log data
•  Interaction events account for 65% of events
while authoring events are 30%
•  The top 3 events (entity selection, description
selection and invocation of editing menu) account
for 56% of events
6
12
19
23
28
39
47
61
82
113
139
142
182
199
267
314
332
426
617
960
1004
1405
2793
Back
Undo
Get explanation
Entity renamed
Set property
Entity dragged
Property addition
Entity deleted
Load ontology
Hierarchy collapsed(i)
Save
Description selected(i)
Run reasoner
Hierarchy collapsed
Convert into defined
Hierarchy expanded(i)
Class addition
Entity selected(i)
Hierarchy expanded
Entity edited:finish
Entity edited:start
Description selected
Entity selected
0 1000 2000
Findings: eye-tracking data
The class hierarchy is the pivotal area
•  Index of the ontology
•  External memory
Transitions between AOIs
from
to
Ann−Usage
Class hierarchy
Description
Explanation
File menu
Pop up
Edit Entity
Prop. hierarchy
Ann−U
sageC
lass
hierarchy
D
escription
Explanation
File
m
enu
Pop
up
EditEntityProp.hierarchy
0
1000
2000
3000
4000
5000
6000
Findings: eye-tracking data
The class hierarchy receives users’ attention
45% of the time
0
100
200
300
400
File
m
enu
Ann.−U
sage
C
lass
hierarchy
D
escription
Popup
Editentity
Prop.hierarchy
Explanation
AOI
time(sec)
Findings: workflows
Select
description
Select
entity
0.29 Modify
entity
0.37
0.63
0.59
Editing activity
Run
reasoner
Convert into
defined class
Save
Select
description
0.16
0.15
0.17
0.40
Expand
inferred
hierarchy
0.30
Select
entity
0.41
0.37
0.43
Select
inferred
entity
0.54
0.25 0.12
Reasoning activity
Select
entity
Expand
hierarchy
0.48
0.31
Select
inferred
entity
Expand
inferred
hierarchy
0.25
0.43
0.12
0.54
Load
ontology
0.52
0.31
Expand
hierarchy
Select
description
0.29
0.37
Exploration activity
Exploration workflow Editing workflow
Reasoning workflow
Implications: from raw data to workflows
Workflows can be automatically identified
raw data
cleaning
merging
filtering
workflow
detection
~7K rows
~200 rows
•  Different authoring styles
•  Time distribution per workflow
•  Identification of confounding variables
Implications for design
•  Support for bulk editing
•  Anticipation of reasoner invocation
•  Automatic detection of authoring problems
•  Make changes to the inferred hierarchy
explicit
tl;dr
•  Identification of activity patterns when dealing
with complex interactive artefacts
•  Interaction log data + eye gaze data
•  Data-driven 
•  Application on knowledge artefacts
Markel Vigo, Caroline Jay, Robert Stevens
University of Manchester (UK)

@markelvigo
markel.vigo@manchester.ac.uk

More Related Content

More from Markel Vigo

Protégé4US: Harvesting Ontology Authoring Data with Protégé
Protégé4US: Harvesting Ontology Authoring Data with ProtégéProtégé4US: Harvesting Ontology Authoring Data with Protégé
Protégé4US: Harvesting Ontology Authoring Data with Protégé
Markel Vigo
 
Design Insights for the Next Wave Ontology Authoring Tools
Design Insights for the Next Wave Ontology Authoring ToolsDesign Insights for the Next Wave Ontology Authoring Tools
Design Insights for the Next Wave Ontology Authoring Tools
Markel Vigo
 
Identifying ontology authoring strategies and patterns
Identifying ontology authoring strategies and patternsIdentifying ontology authoring strategies and patterns
Identifying ontology authoring strategies and patterns
Markel Vigo
 
Challenging Information Foraging Theory: Screen Reader Users are not Always D...
Challenging Information Foraging Theory: Screen Reader Users are not Always D...Challenging Information Foraging Theory: Screen Reader Users are not Always D...
Challenging Information Foraging Theory: Screen Reader Users are not Always D...
Markel Vigo
 
Considering People with Disabilities as Überusers for Eliciting Generalisable...
Considering People with Disabilities as Überusers for Eliciting Generalisable...Considering People with Disabilities as Überusers for Eliciting Generalisable...
Considering People with Disabilities as Überusers for Eliciting Generalisable...
Markel Vigo
 
Evaluating Accessibility-in-Use
Evaluating Accessibility-in-UseEvaluating Accessibility-in-Use
Evaluating Accessibility-in-Use
Markel Vigo
 
Benchmarking Web Accessibility Evaluation Tools: Measuring the Harm of Sole R...
Benchmarking Web Accessibility Evaluation Tools: Measuring the Harm of Sole R...Benchmarking Web Accessibility Evaluation Tools: Measuring the Harm of Sole R...
Benchmarking Web Accessibility Evaluation Tools: Measuring the Harm of Sole R...
Markel Vigo
 
Adaptive web accessibility metrics
Adaptive web accessibility metricsAdaptive web accessibility metrics
Adaptive web accessibility metrics
Markel Vigo
 
Acceptance of Mobile Technology in Hedonic Scenarios
Acceptance of Mobile Technology in Hedonic ScenariosAcceptance of Mobile Technology in Hedonic Scenarios
Acceptance of Mobile Technology in Hedonic Scenarios
Markel Vigo
 
Enriching Web Information Scent for Blind Users
Enriching Web Information Scent for Blind UsersEnriching Web Information Scent for Blind Users
Enriching Web Information Scent for Blind Users
Markel Vigo
 
Transition of Accessibility Evaluation Tools to New Standards
Transition of Accessibility Evaluation Tools to New StandardsTransition of Accessibility Evaluation Tools to New Standards
Transition of Accessibility Evaluation Tools to New Standards
Markel Vigo
 
Automatic Creation of User Profiles for Achieving Personal Web Accessibility
Automatic Creation of User Profiles for Achieving Personal Web AccessibilityAutomatic Creation of User Profiles for Achieving Personal Web Accessibility
Automatic Creation of User Profiles for Achieving Personal Web Accessibility
Markel Vigo
 
Evaluating Web Accessibility For Specific Mobile Devices
Evaluating Web Accessibility For Specific Mobile DevicesEvaluating Web Accessibility For Specific Mobile Devices
Evaluating Web Accessibility For Specific Mobile Devices
Markel Vigo
 

More from Markel Vigo (13)

Protégé4US: Harvesting Ontology Authoring Data with Protégé
Protégé4US: Harvesting Ontology Authoring Data with ProtégéProtégé4US: Harvesting Ontology Authoring Data with Protégé
Protégé4US: Harvesting Ontology Authoring Data with Protégé
 
Design Insights for the Next Wave Ontology Authoring Tools
Design Insights for the Next Wave Ontology Authoring ToolsDesign Insights for the Next Wave Ontology Authoring Tools
Design Insights for the Next Wave Ontology Authoring Tools
 
Identifying ontology authoring strategies and patterns
Identifying ontology authoring strategies and patternsIdentifying ontology authoring strategies and patterns
Identifying ontology authoring strategies and patterns
 
Challenging Information Foraging Theory: Screen Reader Users are not Always D...
Challenging Information Foraging Theory: Screen Reader Users are not Always D...Challenging Information Foraging Theory: Screen Reader Users are not Always D...
Challenging Information Foraging Theory: Screen Reader Users are not Always D...
 
Considering People with Disabilities as Überusers for Eliciting Generalisable...
Considering People with Disabilities as Überusers for Eliciting Generalisable...Considering People with Disabilities as Überusers for Eliciting Generalisable...
Considering People with Disabilities as Überusers for Eliciting Generalisable...
 
Evaluating Accessibility-in-Use
Evaluating Accessibility-in-UseEvaluating Accessibility-in-Use
Evaluating Accessibility-in-Use
 
Benchmarking Web Accessibility Evaluation Tools: Measuring the Harm of Sole R...
Benchmarking Web Accessibility Evaluation Tools: Measuring the Harm of Sole R...Benchmarking Web Accessibility Evaluation Tools: Measuring the Harm of Sole R...
Benchmarking Web Accessibility Evaluation Tools: Measuring the Harm of Sole R...
 
Adaptive web accessibility metrics
Adaptive web accessibility metricsAdaptive web accessibility metrics
Adaptive web accessibility metrics
 
Acceptance of Mobile Technology in Hedonic Scenarios
Acceptance of Mobile Technology in Hedonic ScenariosAcceptance of Mobile Technology in Hedonic Scenarios
Acceptance of Mobile Technology in Hedonic Scenarios
 
Enriching Web Information Scent for Blind Users
Enriching Web Information Scent for Blind UsersEnriching Web Information Scent for Blind Users
Enriching Web Information Scent for Blind Users
 
Transition of Accessibility Evaluation Tools to New Standards
Transition of Accessibility Evaluation Tools to New StandardsTransition of Accessibility Evaluation Tools to New Standards
Transition of Accessibility Evaluation Tools to New Standards
 
Automatic Creation of User Profiles for Achieving Personal Web Accessibility
Automatic Creation of User Profiles for Achieving Personal Web AccessibilityAutomatic Creation of User Profiles for Achieving Personal Web Accessibility
Automatic Creation of User Profiles for Achieving Personal Web Accessibility
 
Evaluating Web Accessibility For Specific Mobile Devices
Evaluating Web Accessibility For Specific Mobile DevicesEvaluating Web Accessibility For Specific Mobile Devices
Evaluating Web Accessibility For Specific Mobile Devices
 

Recently uploaded

NuGOweek 2024 Ghent programme overview flyer
NuGOweek 2024 Ghent programme overview flyerNuGOweek 2024 Ghent programme overview flyer
NuGOweek 2024 Ghent programme overview flyer
pablovgd
 
The binding of cosmological structures by massless topological defects
The binding of cosmological structures by massless topological defectsThe binding of cosmological structures by massless topological defects
The binding of cosmological structures by massless topological defects
Sérgio Sacani
 
Micronuclei test.M.sc.zoology.fisheries.
Micronuclei test.M.sc.zoology.fisheries.Micronuclei test.M.sc.zoology.fisheries.
Micronuclei test.M.sc.zoology.fisheries.
Aditi Bajpai
 
SAR of Medicinal Chemistry 1st by dk.pdf
SAR of Medicinal Chemistry 1st by dk.pdfSAR of Medicinal Chemistry 1st by dk.pdf
SAR of Medicinal Chemistry 1st by dk.pdf
KrushnaDarade1
 
Applied Science: Thermodynamics, Laws & Methodology.pdf
Applied Science: Thermodynamics, Laws & Methodology.pdfApplied Science: Thermodynamics, Laws & Methodology.pdf
Applied Science: Thermodynamics, Laws & Methodology.pdf
University of Hertfordshire
 
Immersive Learning That Works: Research Grounding and Paths Forward
Immersive Learning That Works: Research Grounding and Paths ForwardImmersive Learning That Works: Research Grounding and Paths Forward
Immersive Learning That Works: Research Grounding and Paths Forward
Leonel Morgado
 
Direct Seeded Rice - Climate Smart Agriculture
Direct Seeded Rice - Climate Smart AgricultureDirect Seeded Rice - Climate Smart Agriculture
Direct Seeded Rice - Climate Smart Agriculture
International Food Policy Research Institute- South Asia Office
 
Randomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNERandomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNE
University of Maribor
 
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdfMending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Selcen Ozturkcan
 
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
hozt8xgk
 
8.Isolation of pure cultures and preservation of cultures.pdf
8.Isolation of pure cultures and preservation of cultures.pdf8.Isolation of pure cultures and preservation of cultures.pdf
8.Isolation of pure cultures and preservation of cultures.pdf
by6843629
 
20240520 Planning a Circuit Simulator in JavaScript.pptx
20240520 Planning a Circuit Simulator in JavaScript.pptx20240520 Planning a Circuit Simulator in JavaScript.pptx
20240520 Planning a Circuit Simulator in JavaScript.pptx
Sharon Liu
 
Pests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdfPests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdf
PirithiRaju
 
Equivariant neural networks and representation theory
Equivariant neural networks and representation theoryEquivariant neural networks and representation theory
Equivariant neural networks and representation theory
Daniel Tubbenhauer
 
Authoring a personal GPT for your research and practice: How we created the Q...
Authoring a personal GPT for your research and practice: How we created the Q...Authoring a personal GPT for your research and practice: How we created the Q...
Authoring a personal GPT for your research and practice: How we created the Q...
Leonel Morgado
 
在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样
在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样
在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样
vluwdy49
 
Shallowest Oil Discovery of Turkiye.pptx
Shallowest Oil Discovery of Turkiye.pptxShallowest Oil Discovery of Turkiye.pptx
Shallowest Oil Discovery of Turkiye.pptx
Gokturk Mehmet Dilci
 
Eukaryotic Transcription Presentation.pptx
Eukaryotic Transcription Presentation.pptxEukaryotic Transcription Presentation.pptx
Eukaryotic Transcription Presentation.pptx
RitabrataSarkar3
 
11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf
PirithiRaju
 
The cost of acquiring information by natural selection
The cost of acquiring information by natural selectionThe cost of acquiring information by natural selection
The cost of acquiring information by natural selection
Carl Bergstrom
 

Recently uploaded (20)

NuGOweek 2024 Ghent programme overview flyer
NuGOweek 2024 Ghent programme overview flyerNuGOweek 2024 Ghent programme overview flyer
NuGOweek 2024 Ghent programme overview flyer
 
The binding of cosmological structures by massless topological defects
The binding of cosmological structures by massless topological defectsThe binding of cosmological structures by massless topological defects
The binding of cosmological structures by massless topological defects
 
Micronuclei test.M.sc.zoology.fisheries.
Micronuclei test.M.sc.zoology.fisheries.Micronuclei test.M.sc.zoology.fisheries.
Micronuclei test.M.sc.zoology.fisheries.
 
SAR of Medicinal Chemistry 1st by dk.pdf
SAR of Medicinal Chemistry 1st by dk.pdfSAR of Medicinal Chemistry 1st by dk.pdf
SAR of Medicinal Chemistry 1st by dk.pdf
 
Applied Science: Thermodynamics, Laws & Methodology.pdf
Applied Science: Thermodynamics, Laws & Methodology.pdfApplied Science: Thermodynamics, Laws & Methodology.pdf
Applied Science: Thermodynamics, Laws & Methodology.pdf
 
Immersive Learning That Works: Research Grounding and Paths Forward
Immersive Learning That Works: Research Grounding and Paths ForwardImmersive Learning That Works: Research Grounding and Paths Forward
Immersive Learning That Works: Research Grounding and Paths Forward
 
Direct Seeded Rice - Climate Smart Agriculture
Direct Seeded Rice - Climate Smart AgricultureDirect Seeded Rice - Climate Smart Agriculture
Direct Seeded Rice - Climate Smart Agriculture
 
Randomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNERandomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNE
 
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdfMending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
 
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
 
8.Isolation of pure cultures and preservation of cultures.pdf
8.Isolation of pure cultures and preservation of cultures.pdf8.Isolation of pure cultures and preservation of cultures.pdf
8.Isolation of pure cultures and preservation of cultures.pdf
 
20240520 Planning a Circuit Simulator in JavaScript.pptx
20240520 Planning a Circuit Simulator in JavaScript.pptx20240520 Planning a Circuit Simulator in JavaScript.pptx
20240520 Planning a Circuit Simulator in JavaScript.pptx
 
Pests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdfPests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdf
 
Equivariant neural networks and representation theory
Equivariant neural networks and representation theoryEquivariant neural networks and representation theory
Equivariant neural networks and representation theory
 
Authoring a personal GPT for your research and practice: How we created the Q...
Authoring a personal GPT for your research and practice: How we created the Q...Authoring a personal GPT for your research and practice: How we created the Q...
Authoring a personal GPT for your research and practice: How we created the Q...
 
在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样
在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样
在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样
 
Shallowest Oil Discovery of Turkiye.pptx
Shallowest Oil Discovery of Turkiye.pptxShallowest Oil Discovery of Turkiye.pptx
Shallowest Oil Discovery of Turkiye.pptx
 
Eukaryotic Transcription Presentation.pptx
Eukaryotic Transcription Presentation.pptxEukaryotic Transcription Presentation.pptx
Eukaryotic Transcription Presentation.pptx
 
11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf
 
The cost of acquiring information by natural selection
The cost of acquiring information by natural selectionThe cost of acquiring information by natural selection
The cost of acquiring information by natural selection
 

Constructing Conceptual Knowledge Artefacts: 
 Activity Patterns in the Ontology Authoring Process

  • 1. Constructing Conceptual Knowledge Artefacts: 
 Activity Patterns in the Ontology Authoring Process Markel Vigo, Caroline Jay, Robert Stevens University of Manchester (UK) CHI 2015, Seoul (Korea) @markelvigo markel.vigo@manchester.ac.uk
  • 2. What are ontologies? Mad cow Lion Giraffe Cow Animal Classes eats SubClassOf PropertiesAxioms Cow, Giraffe, Lion à SubClassOf Animal Mad cow à SubClassOf Cow Mad cow, lion à Animal eats some Animal
  • 3. What are ontologies? Mad cow SubClassOf Lion Giraffe Cow Animal Classes PropertiesAxioms Cow, Giraffe, Lion à SubClassOf Animal Mad cow à SubClassOf Cow Mad cow, lion à Animal eats some Animal eats
  • 4. What are ontologies? Mad cow SubClassOf Lion Giraffe Cow Animal Classes PropertiesAxioms Cow, Giraffe, Lion à SubClassOf Animal Mad cow à SubClassOf Cow Mad cow, lion à Animal eats some Animal Vegetarian à Animal and (eats only (not (Animal))) eats
  • 5. What are ontologies? Mad cow SubClassOf Lion Giraffe Cow Animal Classes PropertiesAxioms Cow, Giraffe, Lion à SubClassOf Animal Mad cow à SubClassOf Cow Mad cow, lion à Animal eats some Animal Vegetarian à Animal and (eats only (not (Animal))) eats Vegetarian
  • 6. What are ontologies? Mad cow SubClassOf Lion Giraffe Cow Animal Classes PropertiesAxioms Cow, Giraffe, Lion à SubClassOf Animal Mad cow à SubClassOf Cow Mad cow, lion à Animal eats some Animal Vegetarian à Animal and (eats only (not (Animal))) eats Vegetarian
  • 7. Complex artefacts •  Large size •  Domain expertise required •  Critical •  Highly expressive
  • 8. Problem We don’t know... §  if existing ontology authoring tools are effective §  how ontologist go about authoring §  the authoring workflows
  • 10. Authoring environment: Protégé File menu Class hierarchy Property hierarchy Annotations-Usage Description Mad cow SubClassOf Lion Giraffe Cow Animal Classes Properties Axioms Cow, Giraffe, Lion à SubClassOf Animal Mad cow à SubClassOf Cow Mad cow, lion à Animal eats some Animal Vegetarian à Animal and (eats only (not (Animal))) eats
  • 11. File menu Class hierarchy Property hierarchy Annotations-Usage Description Pop up Explanation Edit entity Authoring environment: Protégé Mad cow SubClassOf Lion Giraffe Cow Animal Classes Properties Axioms Cow, Giraffe, Lion à SubClassOf Animal Mad cow à SubClassOf Cow Mad cow, lion à Animal eats some Animal Vegetarian à Animal and (eats only (not (Animal))) eats
  • 12. Study •  16 ontology authors, 3 typical tasks •  Protégé à Protégé4US •  Eye-tracking •  Sync and merge interaction + fixation data •  N-gram analysis 1389973572771,eye,Class hierarchy! 1389973577038,eye,Description! 1389973584775,log,EntitySelected! 1389973586349,log,DescriptionSelected! 1389973598978,log,EntityModified! 1389973603166,log,EntitySelected! 1389973605053,log,DescriptionSelected! 1389973607847,log,EntityModified! 1389973616404,eye,Description! 1389973616754,eye,Class hierarchy! 1389973617221,eye,Description!
  • 13. Findings: interaction log data •  Interaction events account for 65% of events while authoring events are 30% •  The top 3 events (entity selection, description selection and invocation of editing menu) account for 56% of events 6 12 19 23 28 39 47 61 82 113 139 142 182 199 267 314 332 426 617 960 1004 1405 2793 Back Undo Get explanation Entity renamed Set property Entity dragged Property addition Entity deleted Load ontology Hierarchy collapsed(i) Save Description selected(i) Run reasoner Hierarchy collapsed Convert into defined Hierarchy expanded(i) Class addition Entity selected(i) Hierarchy expanded Entity edited:finish Entity edited:start Description selected Entity selected 0 1000 2000
  • 14. Findings: eye-tracking data The class hierarchy is the pivotal area •  Index of the ontology •  External memory Transitions between AOIs from to Ann−Usage Class hierarchy Description Explanation File menu Pop up Edit Entity Prop. hierarchy Ann−U sageC lass hierarchy D escription Explanation File m enu Pop up EditEntityProp.hierarchy 0 1000 2000 3000 4000 5000 6000
  • 15. Findings: eye-tracking data The class hierarchy receives users’ attention 45% of the time 0 100 200 300 400 File m enu Ann.−U sage C lass hierarchy D escription Popup Editentity Prop.hierarchy Explanation AOI time(sec)
  • 16. Findings: workflows Select description Select entity 0.29 Modify entity 0.37 0.63 0.59 Editing activity Run reasoner Convert into defined class Save Select description 0.16 0.15 0.17 0.40 Expand inferred hierarchy 0.30 Select entity 0.41 0.37 0.43 Select inferred entity 0.54 0.25 0.12 Reasoning activity Select entity Expand hierarchy 0.48 0.31 Select inferred entity Expand inferred hierarchy 0.25 0.43 0.12 0.54 Load ontology 0.52 0.31 Expand hierarchy Select description 0.29 0.37 Exploration activity Exploration workflow Editing workflow Reasoning workflow
  • 17. Implications: from raw data to workflows Workflows can be automatically identified raw data cleaning merging filtering workflow detection ~7K rows ~200 rows •  Different authoring styles •  Time distribution per workflow •  Identification of confounding variables
  • 18. Implications for design •  Support for bulk editing •  Anticipation of reasoner invocation •  Automatic detection of authoring problems •  Make changes to the inferred hierarchy explicit
  • 19. tl;dr •  Identification of activity patterns when dealing with complex interactive artefacts •  Interaction log data + eye gaze data •  Data-driven •  Application on knowledge artefacts
  • 20. Markel Vigo, Caroline Jay, Robert Stevens University of Manchester (UK) @markelvigo markel.vigo@manchester.ac.uk