This document summarizes a literature review of 87 papers from major Semantic Web conferences between 2015-2017 that were analyzed for their inclusion of empirical user studies. 46 papers included user studies, which were coded based on purpose, type of users, evaluation methods, and other attributes. The review found that nearly half of papers did not include user studies, and of those that did, few involved both Semantic Web experts and domain experts or comparisons of experts and novices. It concludes by calling for more and higher quality reporting of user studies in Semantic Web research to improve design and evaluation.

1. Empirical user studies in
semantic web contexts
Catia Pesquita 1, Valentina Ivanova 2, Steffen Lohmann 3,
Patrick Lambrix 4
1 LASIGE, Universidade de Lisboa, Lisbon, Portugal
2 RISE, Sweden
3 Fraunhofer IAIS, Sankt Augustin, Germany
4 Linkoping University, Linkoping, Sweden
EKAW 2018

2. Special Theme 2018:
Empirical Research
International Workshop on Visualization and
Interaction for Ontologies and Linked Data

3. What is the state of empirical evaluation in SW
contexts?

4. What is the state of empirical evaluation in SW
contexts?
Can we help improve it?

5. Why are user studies different in
Semantic Web?

6. Knowledge Engineer vs. Domain Expert

7. Knowledge Engineering expert vs. novice

8. Different datasets with varying semantic complexity

9. Different users and datasets

10. external validity
ecological validity
population validity

11. 2015-2017
• ISWC (International Semantic Web Conference)
• ESWC (Extended Semantic Web Conference)
• VOILA (International Workshop on Visualization and Interaction
for Ontologies and Linked Data)
• IESD (International Workshop on Intelligent Exploration of
Semantic Data)
• “user study”
• “user evaluation”
• “empirical evaluation”
• “interaction”
• “visualization”

12. Literature review
user
study
46
no user
study
35
NA
6
inductive
analysis
coding
purpose
users
evaluation
methods
codes shared
by all
researchers
list of codes is
revised
assigned codes
checked by
second
researcher
87 reviewed papers
46 user study papers were
coded

13. Purpose
exploration 17
querying 10
search 8
explanation 2
question answering 1
navigation 1
learning/understanding
modelling 10
editing 9
annotation 5
mapping 2
validation 1creating/managing

14. Number of users in reported study
not reported
2
[1-9]
9
[10-19]
16
[20-29]
18
[30+]
8

15. Users expertise
0 2 4 6 8 10 12 14 16
not reported
SW
IT
domain
non-expert
diverse

16. Users recruitment method
not reported 25
researchers 4
students 11
clients/users 4 crowdsourcing 1

17. Evaluation Methods
• Qualitative 12
• Quantitative
• Questionnaire
• Standard 8
• Custom 21
• Task performance
• Success 10
• Time 2
• Success & Time 10
• Non-tracked 3
• Design
• Comparative within subjects 5
• Comparative between subjects 3

18. User expertise by type of purpose and operation

19. How to design SW user studies?
How to report on SW user studies?

20. • support the interpretation of the conducted user study
• enable the comparison to similar evaluations
• permit the replication of the user study

21. Minimum information about a user study in SW
Purpose
Users
Tasks
Setup
Procedure
Analysis

22. Purpose
goal is general discovery and insight
generationexploration
focused examination of SW content with a
clear information needsearch
includes operations such as modeling
ontologies or RDF content, and creating
mappings between SW resources
creation
includes assessment, validation, annotation
and editing of SW resources;management

23. Users
expected
demographics
expertise
levels in
SW and
domain
target users vs. study
participants
participant
recruitment

24. Tasks
Include the exact task
descriptions
For multi-purpose systems,
categorize tasks according to
purpose
Describe data and make it
available

25. Final thoughts
• Nearly half of the papers did not present a user study
• Of the 46 papers with user study reports we reviewed, only
four conducted studies for SW experts and domain experts,
and three for SW experts vs. novices
• Are we struggling to find the right participants?
• Are we lacking space in our publications to devote to user
testing results? Are these considered a lesser contribution?
• Are SW researchers publishing their user studies elsewhere?
• Or is there something else holding user testing and reporting in
SW back?

26. Acknowledgements
Valentina Ivanova, RISE Research Institutes of Sweden
Patrick Lambrix, Linköping University
Steffen Lohmann, Fraunhofer IAIS
CP is funded by FCT (UID/CEC/00408/2013, PTDC/EEI-ESS/4633/2014).
P is funded by the Swedish e-Science Society (SeRC).
SL is partly funded by the Fraunhofer Cluster of Excellence Cognitive Internet Technologies (CIT).
VOILA community