Journal Evaluation and Science 2.0 Using Social Bookmarks to Analyze Reader Perception | Stefanie Haustein, Evgeni Golov, ...
Outline <ul><ul><li>Introduction </li></ul></ul><ul><ul><ul><li>Multidimensional Journal Evaluation </li></ul></ul></ul><u...
Introduction: Multidimensional Journal Evaluation <ul><li>one indicator cannot measure true influence of periodicals  </li...
Introduction: Journal Perception <ul><li>measuring problematic despite standards </li></ul><ul><li>only available for usag...
Introduction: Social Bookmarking in STM <ul><li>modeled on  delicious </li></ul><ul><li>allow academic users to store refe...
Introduction: Social Bookmarking in STM
Introduction: Social Bookmarking in STM
Introduction: Social Bookmarking in STM
Introduction: Social Bookmarking in STM 1  Approximate number of estimated cookies on a site for April 2010 (Google, Inc.,...
Introduction: Social Bookmarking in STM 1  Approximate number of estimated cookies on a site for April 2010 (Google, Inc.,...
Retrieval of Bookmarking Data <ul><li>bookmarks from  BibSonomy ,  CiteULike  and  Connotea  to articles published in 45 P...
Retrieval of Bookmarking Data <ul><li>completeness / correctness of bookmarking data </li></ul>
Retrieval of Bookmarking Data <ul><li>completeness / correctness of bookmarking data </li></ul>
Retrieval of Bookmarking Data <ul><li>download was checked and limited to bookmarks to documents published in 45 journals ...
Retrieval of Bookmarking Data <ul><li>missing DOIs were replaced by  CrossRef  and manually </li></ul><ul><li>bookmarks we...
Retrieval of Bookmarking Data <ul><li>13,608 bookmarks were matched to 10,280 articles </li></ul><ul><li>2,441 unique user...
Journal Evaluation: Usage Indicators <ul><li>bookmarking an article indicates journal usage </li></ul><ul><li>Usage ratio ...
Journal Evaluation: Usage Indicators <ul><li>Usage diffusion </li></ul><ul><li>number of unique users per journal </li></u...
Journal Evaluation: Usage Indicators <ul><li>Article Usage Intensity </li></ul><ul><li>average  number of bookmarks per ar...
Journal Evaluation: Usage Indicators <ul><li>Journal Usage Intensity </li></ul><ul><li>mean  number of articles per user <...
Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage...
Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage...
Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage...
Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage...
Journal Evaluation: Tags <ul><li>88.4% of all bookmarks contained tags </li></ul><ul><li>8,208 unique tags assigned 38,241...
Journal Evaluation: Tags number of unique tags Number of tags per publication year for all bookmarked articles
Journal Evaluation: Tags <ul><li>tags assigned to journal articles depict the users’ perspective on content </li></ul><ul>...
Conclusions <ul><li>data from social bookmarking services can be used in journal evaluation </li></ul><ul><li>number of bo...
Literature <ul><li>BERNIUS, S., HANAUSKE, M. AND DUGALL, B. (2009). Von traditioneller wissenschaftlicher Kommunikation zu...
Upcoming SlideShare
Loading in …5
×

STM social bookmarking (by S. Haustein)

2,191 views

Published on

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,191
On SlideShare
0
From Embeds
0
Number of Embeds
298
Actions
Shares
0
Downloads
8
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • presentation is part of an ongoing research project about multidimensional journal evaltuation five dimensions test set Social bookmarking data is used to analyze two dimensions: journal content journal perception which is the readership or usage of scientific journals
  • analysis of readership is still problematic despite standards like Project Counter that try to measure usage by downloads of electronic articles above all, the usage statistics are not made available on a global scale but only on the level of one‘s own institute as an alternative, the analysis of data from social bookmarking services is introduced to measure worldwide journal usage bookmarking an article can be counted similar to counting downloads
  • social bookmarking platforms in STM are modeled on delicious, the social bookmarking service that allows you to store web bookmarks platforms specialized on science cater the specific needs of academic users and allow for bookmarking scientific literature since a resource is always connected with its user and the assigned tag, it is possible to find relevant content through other users and tags
  • this is how a scientific social bookmarking service works: if you have installed the so called bookmarklet, you can bookmark an article right from the publisher’s site
  • the metadata should be extracted and added to your library
  • if the bookmark was set successfully, you can also see who else already stored this paper and what other literature this user bookmarked and which tags he assigned
  • there are four social bookmarking services that provide the same service specialized on academics: CiteULike being the oldest, followed by Connotea which is a Nature product and BibSonomy and 2collab by Elsevier. as the numbers show the first three are comparable in terms of usage but 2collab lacks far behind this is because 2collab is suffering from severe spam attacks and not admitting new users for almost a year now
  • 2collab can be disregarded
  • Pearson correlation coefficients of the bookmarking indicators with the number of publications and common citation indicators
  • Usage ratio, Article usage intensity and 5-year Impact Factor correlate highly meaning that a journal with a high IF has many users per article and also a great share of publications being bookmarked
  • besides that, the other bookmarking indicators show different results
  • Usage diffusion, i.e. the number of different users correlate highly with the number of documents published in the periodical No correlation between Cited Half-Life and any of the other indicators
  • Tagcloud: all tags assigned at least 50 times
  • more tags to recent publications usage increases
  • tags, that were assigned to articles by users reflect the readers perspective of journal content if they are analyzed over time, shifts of focuses can be revealed Condensed matter = macroscopic and microscopic physical properties of matter static electricity diffusion colloid = high energy particle with large surfaces DMS = dilute magnetic semiconductors graphene = carbon crystal
  • STM social bookmarking (by S. Haustein)

    1. 1. Journal Evaluation and Science 2.0 Using Social Bookmarks to Analyze Reader Perception | Stefanie Haustein, Evgeni Golov, Kathleen Luckanus, Sabrina Reher & Jens Terliesner
    2. 2. Outline <ul><ul><li>Introduction </li></ul></ul><ul><ul><ul><li>Multidimensional Journal Evaluation </li></ul></ul></ul><ul><ul><ul><li>Journal Usage </li></ul></ul></ul><ul><ul><ul><li>Social Bookmarking in STM </li></ul></ul></ul><ul><ul><li>Retrieval of Bookmarking Data </li></ul></ul><ul><ul><li>Journal Evaluation </li></ul></ul><ul><ul><ul><li>Usage Indicators </li></ul></ul></ul><ul><ul><ul><li>Tags </li></ul></ul></ul><ul><ul><li>Conclusions </li></ul></ul><ul><ul><li>Literature </li></ul></ul>
    3. 3. Introduction: Multidimensional Journal Evaluation <ul><li>one indicator cannot measure true influence of periodicals </li></ul><ul><li>five dimensions of journal evaluation are identified </li></ul><ul><li>test set: </li></ul><ul><ul><li>45 solid state physics journals </li></ul></ul><ul><ul><li>all publications from 2004 to 2008 </li></ul></ul><ul><ul><li>bibliographic data for 168,109 documents from Web of Science </li></ul></ul>
    4. 4. Introduction: Journal Perception <ul><li>measuring problematic despite standards </li></ul><ul><li>only available for usage at own institutions </li></ul><ul><li>publishers shut global statistics away </li></ul><ul><li>alternative needed to measure worldwide journal usage </li></ul><ul><li>bookmarking articles on social bookmarking platforms indicates journal usage </li></ul>
    5. 5. Introduction: Social Bookmarking in STM <ul><li>modeled on delicious </li></ul><ul><li>allow academic users to store references online </li></ul><ul><li>indexing resources with tags </li></ul><ul><li>sharing literature with other users </li></ul><ul><li>finding new content through other users </li></ul>
    6. 6. Introduction: Social Bookmarking in STM
    7. 7. Introduction: Social Bookmarking in STM
    8. 8. Introduction: Social Bookmarking in STM
    9. 9. Introduction: Social Bookmarking in STM 1 Approximate number of estimated cookies on a site for April 2010 (Google, Inc., 2010) 2 Estimated number of times a site is accessed by unique visitors in April 2010 (Google, Inc., 2010). 3 Total estimated number of times pages on a site have been accessed by users (Google, Inc., 2010) CiteULike Connotea BibSonomy 2collab Launch 2004 2004 2006 2007 Responsibility Oversity Ltd. sponsored by Springer Nature Publishing Group University of Kassel Elsevier B.V. Unique visitors 1 360,000 380,000 350,000 40,000 Total visits 2 480,000 690,000 620,000 44,000 Page views 3 2,100,000 3,100,000 2,800,000 180,000
    10. 10. Introduction: Social Bookmarking in STM 1 Approximate number of estimated cookies on a site for April 2010 (Google, Inc., 2010) 2 Estimated number of times a site is accessed by unique visitors in April 2010 (Google, Inc., 2010). 3 Total estimated number of times pages on a site have been accessed by users (Google, Inc., 2010) CiteULike Connotea BibSonomy 2collab Launch 2004 2004 2006 2007 Responsibility Oversity Ltd. sponsored by Springer Nature Publishing Group University of Kassel Elsevier B.V. Unique visitors 1 360,000 380,000 350,000 40,000 Total visits 2 480,000 690,000 620,000 44,000 Page views 3 2,100,000 3,100,000 2,800,000 180,000
    11. 11. Retrieval of Bookmarking Data <ul><li>bookmarks from BibSonomy , CiteULike and Connotea to articles published in 45 Physics journals from 2004 to 2008 </li></ul><ul><li>search strategies applied to compensate incomplete and erroneous bookmarking data </li></ul><ul><ul><li>journal title </li></ul></ul><ul><ul><li>abbreviations </li></ul></ul><ul><ul><li>ISSNs </li></ul></ul><ul><ul><li>DOIs of articles </li></ul></ul><ul><li>download of bookmarks via API or by parsing </li></ul>
    12. 12. Retrieval of Bookmarking Data <ul><li>completeness / correctness of bookmarking data </li></ul>
    13. 13. Retrieval of Bookmarking Data <ul><li>completeness / correctness of bookmarking data </li></ul>
    14. 14. Retrieval of Bookmarking Data <ul><li>download was checked and limited to bookmarks to documents published in 45 journals from 2004 to 2008 </li></ul><ul><li>13,760 correct bookmarks retrieved </li></ul>
    15. 15. Retrieval of Bookmarking Data <ul><li>missing DOIs were replaced by CrossRef and manually </li></ul><ul><li>bookmarks were matched to the bibliographic entries from Web of Science via DOI </li></ul><ul><ul><li>98.9% of bookmarks matched </li></ul></ul><ul><li>2.8% doubles in user names between services </li></ul><ul><li>results were combined </li></ul>
    16. 16. Retrieval of Bookmarking Data <ul><li>13,608 bookmarks were matched to 10,280 articles </li></ul><ul><li>2,441 unique users </li></ul><ul><li>1,179 users posted one article </li></ul><ul><li>75% of content is created by 21% of users </li></ul><ul><li>8,511 articles were only bookmarked once </li></ul>
    17. 17. Journal Evaluation: Usage Indicators <ul><li>bookmarking an article indicates journal usage </li></ul><ul><li>Usage ratio </li></ul><ul><li>journal articles with bookmarks compared to all publications </li></ul>
    18. 18. Journal Evaluation: Usage Indicators <ul><li>Usage diffusion </li></ul><ul><li>number of unique users per journal </li></ul><ul><li>depicts the diffusion of journal content into the community </li></ul>
    19. 19. Journal Evaluation: Usage Indicators <ul><li>Article Usage Intensity </li></ul><ul><li>average number of bookmarks per article </li></ul><ul><li>shows how often articles are read </li></ul>
    20. 20. Journal Evaluation: Usage Indicators <ul><li>Journal Usage Intensity </li></ul><ul><li>mean number of articles per user </li></ul><ul><li>measures how intensive the average user reads the journal </li></ul>
    21. 21. Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage intensity P 5-year IF h-index Cited Half-Life Usage ratio 0.33 0.90 -0.22 -0.09 0.86 0.50 0.02 Usage diffusion 0.33 0.27 0.45 0.72 0.14 0.78 -0.24 Article usage intensity 0.90 0.27 -0.36 -0.16 0.88 0.46 0.09 Journal usage intensity -0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10 P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17 5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21 h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22 Cited Half-Life 0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22
    22. 22. Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage intensity P 5-year IF h-index Cited Half-Life Usage ratio 0.33 0.90 -0.22 -0.09 0.86 0.50 0.02 Usage diffusion 0.33 0.27 0.45 0.72 0.14 0.78 -0.24 Article usage intensity 0.90 0.27 -0.36 -0.16 0.88 0.46 0.09 Journal usage intensity -0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10 P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17 5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21 h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22 Cited Half-Life 0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22
    23. 23. Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage intensity P 5-year IF h-index Cited Half-Life Usage ratio 0.33 0.90 -0.22 -0.09 0.86 0.50 0.02 Usage diffusion 0.33 0.27 0.45 0.72 0.14 0.78 -0.24 Article usage intensity 0.90 0.27 -0.36 -0.16 0.88 0.46 0.09 Journal usage intensity -0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10 P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17 5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21 h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22 Cited Half-Life 0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22
    24. 24. Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage intensity P 5-year IF h-index Cited Half-Life Usage ratio 0.33 0.90 -0.22 -0.09 0.86 0.50 0.02 Usage diffusion 0.33 0.27 0.45 0.72 0.14 0.78 -0.24 Article usage intensity 0.90 0.27 -0.36 -0.16 0.88 0.46 0.09 Journal usage intensity -0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10 P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17 5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21 h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22 Cited Half-Life 0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22
    25. 25. Journal Evaluation: Tags <ul><li>88.4% of all bookmarks contained tags </li></ul><ul><li>8,208 unique tags assigned 38,241 times </li></ul>tagcloud: all tags assigned at least 50 times
    26. 26. Journal Evaluation: Tags number of unique tags Number of tags per publication year for all bookmarked articles
    27. 27. Journal Evaluation: Tags <ul><li>tags assigned to journal articles depict the users’ perspective on content </li></ul><ul><li>analysis over time can reveal shifts in thematic focus areas </li></ul>tags assigned to articles published in J Phys Condens Matter in 2004 tags assigned to articles published in J Phys Condens Matter in 2008
    28. 28. Conclusions <ul><li>data from social bookmarking services can be used in journal evaluation </li></ul><ul><li>number of bookmarks can be used alternatively to measure worldwide usage of journals </li></ul><ul><li>tags reveal the users’ perspective on journal content </li></ul><ul><li>poor quality of bookmarking data makes retrieval and analysis complex </li></ul><ul><li>social bookmarking in STM is still in its infancy and services are competing for a critical mass </li></ul>
    29. 29. Literature <ul><li>BERNIUS, S., HANAUSKE, M. AND DUGALL, B. (2009). Von traditioneller wissenschaftlicher Kommunikation zu &quot;Science 2.0&quot;. ABI-Technik 29, 214-226. </li></ul><ul><li>HAMMOND, T., HANNAY, T., LUND, B. AND SCOTT, J. (2005). Social Bookmarking Tools (I). D-Lib Magazine 11. </li></ul><ul><li>HAUSTEIN, S. (2010). Multidimensional Journal Evaluation. Proceedings of STI 2010, Leiden. </li></ul><ul><li>JUCHEM, K., SCHLOEGL, C. AND STOCK, W.G. (2006). Dimensionen der Zeitschriftenszientometrie am Beispiel von &quot;Buch und Bibliothek&quot;. Information Wissenschaft & Praxis 57, 31-37. </li></ul><ul><li>PROJECT COUNTER (2008). The COUNTER Code of Practice. Journals and Databases Release 3, in: http://www.projectcounter.org/r3/Release3D9.pdf. </li></ul><ul><li>REHER, S. AND HAUSTEIN, S. ( to be published ). Social Bookmarking in STM: Putting Services to the Acid Test. ONLINE. </li></ul><ul><li>ROUSSEAU, R. (2002). Journal Evaluation: Technical and Practical Issues. Library Trends 50, 418-439. </li></ul><ul><li>SHNEIDERMAN, B. (2008). Computer Science - Science 2.0. Science 319, 1349-1350. </li></ul>Thank you for your kind attention!

    ×