Networks of citations are a key tool for referencing, disseminating and evaluating research results. The task of characterising the functional role of citations in scientific literature is very difficult, not only for software agents but for humans, too. The main problem is that the mental models of different annotators hardly ever converge to a single shared opinion. The goal of this paper is to investigate how an existing reference model for classifying citations, namely CiTO (Citation Typing Ontology), is interpreted and used by annotators of scientific literature. We present an experiment capturing the cognitive processes behind subjects’ decisions in annotating papers with CiTO, and we provide initial ideas to refine future releases of CiTO.

1. STLab University of Bologna#eswc2014Ciancarini
!
Evaluating citation functions in CiTO:
cognitive issues
!
!
!
!
!
28 May 2014 - Heraklion, Crete
Paolo Ciancarini1,2,Angelo Di Iorio1, Andrea Giovanni Nuzzolese1,2,
Silvio Peroni1 and FabioVitali1
1Department of Computer Science and Engineering, University of Bologna, Italy
2STLab, Institute of Cognitive Science and Technology, National Research Council, Rome, Italy

2. STLab University of Bologna#eswc2014Ciancarini
Outline
2
• Motivations
• CiTO
• Experiment
• Evaluation
• Lessons learnt and conclusions

3. STLab University of Bologna#eswc2014Ciancarini
• Bibliographic citations can be seen as tools for:
• linking, disseminating, exploring, evaluating research
• The task of characterising the functional role of citations in
scientific literature is very difficult for agents and humans
• Investigating how an existing reference model for classifying
citations, i.e., CiTO, is interpreted and used by human annotators
• We want to study human’s behaviour in order to simulate it
within CiTalO, a tool that automatically classifies citations with
CiTO
Motivations and goals
3

4. STLab University of Bologna#eswc2014Ciancarini
• An OWL ontology for describing factual as well as rhetorical
functions of citations in scholarly articles
• Defines
• a top-level property cites (and its inverse isCitedBy)
• 41 sub-properties of cites that allow users to characterise precisely
the semantics of a citation act
• Has been successfully used in large projects, like CiteULike,
data.open.ac.uk and the Open Citation Corpus
• Several tools have been developed to annotate citations with
CiTO
• e.g., Chrome and WordPress plug-ins
CiTO
4

5. STLab University of Bologna#eswc2014Ciancarini
• The richness of properties in CiTO (CiTO-Ps) is
• a key feature of CiTO: this aspect has contributed to the
adoption of the ontology by the Semantic Publishing
Users’ adoption of CiTO
5

6. STLab University of Bologna#eswc2014Ciancarini
• The richness of properties in CiTO (CiTO-Ps) is
• a key feature of CiTO: this aspect has contributed to the
adoption of the ontology by the Semantic Publishing
• an hindrance: most tools actually employ a sub-set of the CiTO
properties,
• e.g., 6 CiTO-Ps enabled for user annotation by Pensoft Publishers
and 9 in the Chrome plug-in
Users’ adoption of CiTO
5

7. STLab University of Bologna#eswc2014Ciancarini
6
user
author
“It extends the research
outlined in earlier work [3]”
CiTO annotations and mental
models
CiTO

8. STLab University of Bologna#eswc2014Ciancarini
6
Interpretation
of author’s
text
Understanding
of CiTO
user
author
“It extends the research
outlined in earlier work [3]”
CiTO annotations and mental
models
CiTO

9. STLab University of Bologna#eswc2014Ciancarini
6
Interpretation
of author’s
text
Understanding
of CiTO
user
mental
model
author
“It extends the research
outlined in earlier work [3]”
CiTO annotations and mental
models
CiTO

10. STLab University of Bologna#eswc2014Ciancarini
6
Interpretation
of author’s
text
Understanding
of CiTO
user
mental
model
Annotation
cito:extends
author
“It extends the research
outlined in earlier work [3]”
CiTO annotations and mental
models
CiTO

11. STLab University of Bologna#eswc2014Ciancarini
mental
model
mental
model
6
Interpretation
of author’s
text
Understanding
of CiTO
user
mental
model
Annotation
cito:extends
author
“It extends the research
outlined in earlier work [3]”
mental
model
mental
model
CiTO annotations and mental
models
CiTO

12. STLab University of Bologna#eswc2014Ciancarini
mental
model
mental
model
6
Interpretation
of author’s
text
Understanding
of CiTO
user
mental
model
Annotation
cito:extends
author
“It extends the research
outlined in earlier work [3]”
mental
model
mental
model
CiTO annotations and mental
models
CiTO!
cito:citesForInformation
cito:givesSupportTo
…
The mental models of
different annotators
hardly ever converge to
a single shared opinion

13. STLab University of Bologna#eswc2014Ciancarini
7
What we did
We performed an experiment to
investigate how humans use CiTO
to annotate citations with a type
• 20
• 105 citations chosen among the
seventh volume of the
proceedings of the Balisage
Conference Series
subjects

14. STLab University of Bologna#eswc2014Ciancarini
7
What we did
We performed an experiment to
investigate how humans use CiTO
to annotate citations with a type
• 20
• 105 citations chosen among the
seventh volume of the
proceedings of the Balisage
Conference Series
(lucky) subjects

15. STLab University of Bologna#eswc2014Ciancarini
• The experiment had one independent variable, i.e., the number of
CiTO-Ps available to subjects for the annotation
• Condition T41: 10 subjects used 41 CiTO-Ps
• Condition T10: 10 subjects used a subset of 10 CiTO-Ps
• i.e., citesAsDataSource, citesAsPotentialSolution,
citesAsRecommendedReading, citesAsRelated, citesForInformation,
credits, critiques, includesQuotationFrom, obtainsBackgroundFrom,
usesMethodIn
• T10 CiTO-Ps were chosen among those that had shown a moderate
inter-rater agreement (Fleiss’k>0.33) in a preliminary experiment on
the same data sample
The experiment
8

16. STLab University of Bologna#eswc2014Ciancarini
!
Evaluation framework
9
Citation context CITO-Ps
Explanations and examples
of the usage of the CiTO-Ps
Available on line at http://www.cs.unibo.it/~nuzzoles/cito_1/?user=r

17. STLab University of Bologna#eswc2014Ciancarini
10
1. Which properties have been used by subjects during the experiment?
2. Which were the most used properties?
3. What was the global inter-rater agreement of the subjects?
4. Did the number of available choices bias the global inter-rater agreement?
5. Which properties showed an acceptable positive agreement among
subjects?
6. Could properties be organized according to their similarity in subjects’
annotations?
7. What was the perceived usability of the CiTO-Ps?
8. Which were the features of CiTO-Ps that subjects perceived as most
useful or problematic?
Target questions

18. STLab University of Bologna#eswc2014Ciancarini
11
!
!
• Condition T41
• used 37 different CiTO-Ps over 41(avg: 21.7 CiTO-Ps per subject)
• 4 properties not selected by any subject
• i.e., parodies, plagiarizes, repliesTo and ridicules
!
• Condition T10
• used all the 10 CiTO-Ps
Results
“Which properties have been used by subjects during the experiment?”

19. STLab University of Bologna#eswc2014Ciancarini
12
Results
“Which were the most used properties?”

20. STLab University of Bologna#eswc2014Ciancarini
“What was the global inter-rater agreement of the subjects?”
“Did the number of available choices bias the global inter-rater agreement?”
“Which properties showed an acceptable positive agreement among subjects?”
13
• Condition T41
• Global Fleiss’k = 0.13
• 5 CiTO-Ps with moderate local positive agreement (k > 0.5)
• i.e., citesAsPotentialSolution (0.66), citesAsRecommendedReading (0.6), agreesWith (0.54),
citesAsDataSource (0.52), usesMethodIn (0.54)
• Condition T10
• Global Fleiss’k = 0.15
• 4 CiTO-Ps with moderate local positive agreement
• i.e., citesAsPotentialSolution (0.71), citesAsDataSource (0.63),
citesAsRecommendedReading (0.52), includesQuotationFrom (0.69)
Data evaluation

21. STLab University of Bologna#eswc2014Ciancarini
“What was the global inter-rater agreement of the subjects?”
“Did the number of available choices bias the global inter-rater agreement?”
“Which properties showed an acceptable positive agreement among subjects?”
13
• Condition T41
• Global Fleiss’k = 0.13
• 5 CiTO-Ps with moderate local positive agreement (k > 0.5)
• i.e., citesAsPotentialSolution (0.66), citesAsRecommendedReading (0.6), agreesWith (0.54),
citesAsDataSource (0.52), usesMethodIn (0.54)
• Condition T10
• Global Fleiss’k = 0.15
• 4 CiTO-Ps with moderate local positive agreement
• i.e., citesAsPotentialSolution (0.71), citesAsDataSource (0.63),
citesAsRecommendedReading (0.52), includesQuotationFrom (0.69)
Data evaluation
The global
agreement is
very low

22. STLab University of Bologna#eswc2014Ciancarini
“What was the global inter-rater agreement of the subjects?”
“Did the number of available choices bias the global inter-rater agreement?”
“Which properties showed an acceptable positive agreement among subjects?”
13
• Condition T41
• Global Fleiss’k = 0.13
• 5 CiTO-Ps with moderate local positive agreement (k > 0.5)
• i.e., citesAsPotentialSolution (0.66), citesAsRecommendedReading (0.6), agreesWith (0.54),
citesAsDataSource (0.52), usesMethodIn (0.54)
• Condition T10
• Global Fleiss’k = 0.15
• 4 CiTO-Ps with moderate local positive agreement
• i.e., citesAsPotentialSolution (0.71), citesAsDataSource (0.63),
citesAsRecommendedReading (0.52), includesQuotationFrom (0.69)
Data evaluation
The global
agreement is
very low
The # of CiTO-Ps
does not affect the
agreement

23. STLab University of Bologna#eswc2014Ciancarini
“What was the global inter-rater agreement of the subjects?”
“Did the number of available choices bias the global inter-rater agreement?”
“Which properties showed an acceptable positive agreement among subjects?”
13
• Condition T41
• Global Fleiss’k = 0.13
• 5 CiTO-Ps with moderate local positive agreement (k > 0.5)
• i.e., citesAsPotentialSolution (0.66), citesAsRecommendedReading (0.6), agreesWith (0.54),
citesAsDataSource (0.52), usesMethodIn (0.54)
• Condition T10
• Global Fleiss’k = 0.15
• 4 CiTO-Ps with moderate local positive agreement
• i.e., citesAsPotentialSolution (0.71), citesAsDataSource (0.63),
citesAsRecommendedReading (0.52), includesQuotationFrom (0.69)
Data evaluation
The global
agreement is
very low
The # of CiTO-Ps
does not affect the
agreement
The set of CiTO-PS with
moderate local positive
agreement is little
affected by the # of
CiTO-Ps

24. STLab University of Bologna#eswc2014Ciancarini
• We applied the Chinese Whispers clustering algorithm
• Input: 2 graphs built by combining all the pairs of different CiTO-Ps as
annotated by subjects for each citation
• Gr - it takes into account repetitions in annotations for a each CiTO
property
•
e.g.,“extends”,“extends”, and “updates” on a citation generate
(extends,updates) and (extends,updates)
• Gn - it does not take into account repetitions
•
e.g.,“extends”,“extends”, and “updates” generate (extends,updates)
14
Clustering CiTO-Ps
“Could properties be organized
according to their similarity in subjects’ annotations?”

25. STLab University of Bologna#eswc2014Ciancarini
14
Clustering CiTO-Ps
“Could properties be organized
according to their similarity in subjects’ annotations?”
Gr Gn
disputes
critiques
derides
refutes confirms
credits
obtainsSupportFrom

26. STLab University of Bologna#eswc2014Ciancarini
14
Clustering CiTO-Ps
“Could properties be organized
according to their similarity in subjects’ annotations?”
Gr Gn
disputes
critiques
derides
refutes confirms
credits
obtainsSupportFrom
There exist some sort of
relations (e.g., taxonomical,
equivalence) among the
CiTO-Ps of each cluster

27. STLab University of Bologna#eswc2014Ciancarini
15
• We computed the System Usability Scale (SUS)
Measuring the usability of CiTO-Ps
0.00!
10.00!
20.00!
30.00!
40.00!
50.00!
60.00!
70.00!
80.00!
90.00!
100.00!
SUS mean! Usability mean! Learnability mean!
T41!
T10!
“What was the perceived usability of the CiTO-Ps? ”

28. STLab University of Bologna#eswc2014Ciancarini
15
• We computed the System Usability Scale (SUS)
Measuring the usability of CiTO-Ps
0.00!
10.00!
20.00!
30.00!
40.00!
50.00!
60.00!
70.00!
80.00!
90.00!
100.00!
SUS mean! Usability mean! Learnability mean!
T41!
T10!
“What was the perceived usability of the CiTO-Ps? ”
Only the usability
score approaches the
statistical significance

29. STLab University of Bologna#eswc2014Ciancarini
16
Grounded theory analysis
• The subjects filled a final text questionnaire aimed at capturing
positive and negative aspects of CiTO-Ps
• We used the text answers for performing a grounded theory analysis,
used in Social Science to extract relevant concepts from
unstructured text
“Which were the features of CiTO-Ps that subjects perceived as most useful or
problematic?”

30. STLab University of Bologna#eswc2014Ciancarini
16
Grounded theory analysis
“Which were the features of CiTO-Ps that subjects perceived as most useful or
problematic?”

31. STLab University of Bologna#eswc2014Ciancarini
17
Conclusions
• Lessons learnt and suggestions to improve CiTO
• Reduce the number of less-used properties
• Identify the most-used neutral properties
• Investigate motivations for low inter-rater agreement
• Define explicit relations between CiTO properties
• Add support for customised properties
• Extend examples, labels and explanations
• Future work
• Improve CiTalO, a tool for identifying automatically the nature of
citations
• e.g., by investigating cognitive architectures in order to simulate
humans’ behaviour

32. STLab University of Bologna#eswc2014Ciancarini
18
Thank you!