Characterising citations in scholarly articles: an experiment

782 views

Published on

This work presents some experiments in letting humans annotate citations according to CiTO, an OWL ontology for describing the function of citations. We introduce a comparison of the performance of different users, and show strengths and difficulties that emerged when using that particular model to characterise citations of scholarly articles.

Published in: Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
782
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
6
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Characterising citations in scholarly articles: an experiment

  1. 1. Characterising Citations in Scholarly Articles: an Experiment Paolo Ciancarini – ciancarini@cs.unibo.it Angelo Di Iorio – diiorio@cs.unibo.it Andrea Giovanni Nuzzolese – nuzzoles@cs.unibo.it Silvio Peroni – essepuntato@cs.unibo.it Fabio Vitali – fabio@cs.unibo.it International Workshop on Artificial Intelligence and Cognition co-located with XIII Conference of the Italian Association for Artificial Intelligence Turin (Italy), December 3, 2013 http://creativecommons.org/licenses/by-sa/3.0
  2. 2. What and Why
  3. 3. What and Why • What: to analyse how humans use a particular citation model for the annotation of citations in scientific article ✦ Citation model under investigation: the Citation Typing Ontology (CiTO), an OWL-2 DL ontology describing various functions of citation (e.g. uses method in, extends, critiques)
  4. 4. What and Why • What: to analyse how humans use a particular citation model for the annotation of citations in scientific article ✦ • Citation model under investigation: the Citation Typing Ontology (CiTO), an OWL-2 DL ontology describing various functions of citation (e.g. uses method in, extends, critiques) Why: final aims (of future works): ✦ ✦ To improve CiTO by adding new properties, and by creating cluster of semantically-similar properties (e.g. disagrees with and critiques) by analysing how humans use them when dealing with concrete tasks of annotation To improve CiTalO, a web application that tries to infer citation functions from sentences containing a citation, so as to reflect human behaviour when annotating citations
  5. 5. Who users
  6. 6. Who users
  7. 7. Who citation model (CiTO) e mod ls were d users eriv ed f rom mental model etal mental model m The mental model mental model mental model
  8. 8. Who author ls were d users eriv ed f rom mental model e mod mental model etal Interpretation of author’s text mental model mental model mental model m The “It extends the research outlined in earlier work [3]” citation model (CiTO)
  9. 9. Who author users eriv ed f rom mental model ls were d mental model e mod Mapping personal interpretation with citation functions etal Interpretation of author’s text mental model mental model mental model m The “It extends the research outlined in earlier work [3]” citation model (CiTO)
  10. 10. How, When and Where
  11. 11. How, When and Where • How: preliminary user testing session with users to whom we asked to assign CiTO properties to the citations in the Proceedings of Balisage 2011: ✦ ✦ ✦ 18 papers 104 citations selected (out of 377) 5 users
  12. 12. How, When and Where • How: preliminary user testing session with users to whom we asked to assign CiTO properties to the citations in the Proceedings of Balisage 2011: ✦ ✦ ✦ • 18 papers 104 citations selected (out of 377) 5 users When: well, few months ago, but we are still experimenting involving a larger set of users and using different experimental settings
  13. 13. How, When and Where • How: preliminary user testing session with users to whom we asked to assign CiTO properties to the citations in the Proceedings of Balisage 2011: ✦ ✦ ✦ 18 papers 104 citations selected (out of 377) 5 users • When: well, few months ago, but we are still experimenting involving a larger set of users and using different experimental settings • Where: the whole test was/is performed online, without any supervision
  14. 14. How, When and Where • How: preliminary user testing session with users to whom we asked to assign CiTO properties to the citations in the Proceedings of Balisage 2011: ✦ ✦ ✦ 18 papers 104 citations selected (out of 377) 5 users By the way, would you like to help us? • When: well, few months ago, but we are still experimenting involving a larger set of users and using different experimental settings • Where: the whole test was/is performed online, without any supervision
  15. 15. How, When and Where • How: preliminary user testing session with users to whom we asked to assign CiTO properties to the citations in the Proceedings of Balisage 2011: ✦ ✦ ✦ 18 papers 104 citations selected (out of 377) 5 users By the way, would you like to help us? • When: well, few months ago, but we are still experimenting involving a larger set of users and using different experimental settings • Where: the whole test was/is performed online, without any Please ... ;-( ... supervision
  16. 16. Results • Users have selected 34 different CiTO properties over 40 ✦ • average: 22.4 properties per user Properties used many times ✦ ✦ ✦ • cites for information (110) cites as related (39) cites as data source (38) Low positive agreement for the 5 raters over all 104 subjects ✦ • k = 0.16 Moderate positive local agreement on ✦ ✦ ✦ ✦ cites as data source: k = 0.5 cites as potential solution: k = 0.45 cites as recommended reading: k = 0.34 includes quotation from: k = 0.49
  17. 17. Results • Users have selected 34 different CiTO properties over 40 ✦ • average: 22.4 properties per user Properties used many times Summarising: it ✦ ✦ ✦ • cites for information (110) is a difficult task cites as related (39) humans cites as data source (38) for Low positive agreement for the 5 raters over all 104 subjects ✦ • k = 0.16 Moderate positive local agreement on ✦ ✦ ✦ ✦ cites as data source: k = 0.5 cites as potential solution: k = 0.45 cites as recommended reading: k = 0.34 includes quotation from: k = 0.49
  18. 18. Results • Users have selected 34 different CiTO properties over 40 ✦ • average: 22.4 properties per user Properties used many times Summarising: it ✦ ✦ ✦ • cites for information (110) is a difficult task cites as related (39) humans cites as data source (38) for Which reminds me... Low positive agreement for the 5 raters over all 104 subjects ✦ • k = 0.16 ... positive are Moderate that welocal agreement on ✦ ✦ ✦ ✦ cites as data source:users for looking for k = 0.5 cites asadditional tests 0.45 potential solution: k = cites as recommended reading: k = 0.34 includes quotation from: k = 0.49
  19. 19. Results • Users have selected 34 different CiTO properties over 40 ✦ • average: 22.4 properties per user Properties used many times Summarising: it ✦ ✦ ✦ • cites for information (110) is a difficult task cites as related (39) humans cites as data source (38) for Which reminds me... Low positive agreement for the 5 raters over all 104 subjects ✦ • k = 0.16 ... positive are Moderate that welocal agreement on ✦ ✦ ✦ ✦ cites as data source:users for looking for k = 0.5 cites asadditional tests 0.45 potential solution: k = cites as recommended reading: k = 0.34 includes quotation from: k = 0.49 Would you like to help us? Please ... ;-( ...
  20. 20. Thanks for your attention

×