Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Journal Evaluation and Science 2.0 Using Social Bookmarks to Analyze Reader Perception | Stefanie Haustein, Evgeni Golov, ...
Outline <ul><ul><li>Introduction </li></ul></ul><ul><ul><ul><li>Multidimensional Journal Evaluation </li></ul></ul></ul><u...
Introduction: Multidimensional Journal Evaluation <ul><li>one indicator cannot measure true influence of periodicals  </li...
Introduction: Journal Perception <ul><li>measuring problematic despite standards </li></ul><ul><li>only available for usag...
Introduction: Social Bookmarking in STM <ul><li>modeled on  delicious </li></ul><ul><li>allow academic users to store refe...
Introduction: Social Bookmarking in STM
Introduction: Social Bookmarking in STM
Introduction: Social Bookmarking in STM
Introduction: Social Bookmarking in STM 1  Approximate number of estimated cookies on a site for April 2010 (Google, Inc.,...
Introduction: Social Bookmarking in STM 1  Approximate number of estimated cookies on a site for April 2010 (Google, Inc.,...
Retrieval of Bookmarking Data <ul><li>bookmarks from  BibSonomy ,  CiteULike  and  Connotea  to articles published in 45 P...
Retrieval of Bookmarking Data <ul><li>completeness / correctness of bookmarking data </li></ul>
Retrieval of Bookmarking Data <ul><li>completeness / correctness of bookmarking data </li></ul>
Retrieval of Bookmarking Data <ul><li>download was checked and limited to bookmarks to documents published in 45 journals ...
Retrieval of Bookmarking Data <ul><li>missing DOIs were replaced by  CrossRef  and manually </li></ul><ul><li>bookmarks we...
Retrieval of Bookmarking Data <ul><li>13,608 bookmarks were matched to 10,280 articles </li></ul><ul><li>2,441 unique user...
Journal Evaluation: Usage Indicators <ul><li>bookmarking an article indicates journal usage </li></ul><ul><li>Usage ratio ...
Journal Evaluation: Usage Indicators <ul><li>Usage diffusion </li></ul><ul><li>number of unique users per journal </li></u...
Journal Evaluation: Usage Indicators <ul><li>Article Usage Intensity </li></ul><ul><li>average  number of bookmarks per ar...
Journal Evaluation: Usage Indicators <ul><li>Journal Usage Intensity </li></ul><ul><li>mean  number of articles per user <...
Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage...
Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage...
Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage...
Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage...
Journal Evaluation: Tags <ul><li>88.4% of all bookmarks contained tags </li></ul><ul><li>8,208 unique tags assigned 38,241...
Journal Evaluation: Tags number of unique tags Number of tags per publication year for all bookmarked articles
Journal Evaluation: Tags <ul><li>tags assigned to journal articles depict the users’ perspective on content </li></ul><ul>...
Conclusions <ul><li>data from social bookmarking services can be used in journal evaluation </li></ul><ul><li>number of bo...
Literature <ul><li>BERNIUS, S., HANAUSKE, M. AND DUGALL, B. (2009). Von traditioneller wissenschaftlicher Kommunikation zu...
Upcoming SlideShare
Loading in …5
×

STM social bookmarking (by S. Haustein)

2,523 views

Published on

  • Be the first to comment

STM social bookmarking (by S. Haustein)

  1. 1. Journal Evaluation and Science 2.0 Using Social Bookmarks to Analyze Reader Perception | Stefanie Haustein, Evgeni Golov, Kathleen Luckanus, Sabrina Reher & Jens Terliesner
  2. 2. Outline <ul><ul><li>Introduction </li></ul></ul><ul><ul><ul><li>Multidimensional Journal Evaluation </li></ul></ul></ul><ul><ul><ul><li>Journal Usage </li></ul></ul></ul><ul><ul><ul><li>Social Bookmarking in STM </li></ul></ul></ul><ul><ul><li>Retrieval of Bookmarking Data </li></ul></ul><ul><ul><li>Journal Evaluation </li></ul></ul><ul><ul><ul><li>Usage Indicators </li></ul></ul></ul><ul><ul><ul><li>Tags </li></ul></ul></ul><ul><ul><li>Conclusions </li></ul></ul><ul><ul><li>Literature </li></ul></ul>
  3. 3. Introduction: Multidimensional Journal Evaluation <ul><li>one indicator cannot measure true influence of periodicals </li></ul><ul><li>five dimensions of journal evaluation are identified </li></ul><ul><li>test set: </li></ul><ul><ul><li>45 solid state physics journals </li></ul></ul><ul><ul><li>all publications from 2004 to 2008 </li></ul></ul><ul><ul><li>bibliographic data for 168,109 documents from Web of Science </li></ul></ul>
  4. 4. Introduction: Journal Perception <ul><li>measuring problematic despite standards </li></ul><ul><li>only available for usage at own institutions </li></ul><ul><li>publishers shut global statistics away </li></ul><ul><li>alternative needed to measure worldwide journal usage </li></ul><ul><li>bookmarking articles on social bookmarking platforms indicates journal usage </li></ul>
  5. 5. Introduction: Social Bookmarking in STM <ul><li>modeled on delicious </li></ul><ul><li>allow academic users to store references online </li></ul><ul><li>indexing resources with tags </li></ul><ul><li>sharing literature with other users </li></ul><ul><li>finding new content through other users </li></ul>
  6. 6. Introduction: Social Bookmarking in STM
  7. 7. Introduction: Social Bookmarking in STM
  8. 8. Introduction: Social Bookmarking in STM
  9. 9. Introduction: Social Bookmarking in STM 1 Approximate number of estimated cookies on a site for April 2010 (Google, Inc., 2010) 2 Estimated number of times a site is accessed by unique visitors in April 2010 (Google, Inc., 2010). 3 Total estimated number of times pages on a site have been accessed by users (Google, Inc., 2010) CiteULike Connotea BibSonomy 2collab Launch 2004 2004 2006 2007 Responsibility Oversity Ltd. sponsored by Springer Nature Publishing Group University of Kassel Elsevier B.V. Unique visitors 1 360,000 380,000 350,000 40,000 Total visits 2 480,000 690,000 620,000 44,000 Page views 3 2,100,000 3,100,000 2,800,000 180,000
  10. 10. Introduction: Social Bookmarking in STM 1 Approximate number of estimated cookies on a site for April 2010 (Google, Inc., 2010) 2 Estimated number of times a site is accessed by unique visitors in April 2010 (Google, Inc., 2010). 3 Total estimated number of times pages on a site have been accessed by users (Google, Inc., 2010) CiteULike Connotea BibSonomy 2collab Launch 2004 2004 2006 2007 Responsibility Oversity Ltd. sponsored by Springer Nature Publishing Group University of Kassel Elsevier B.V. Unique visitors 1 360,000 380,000 350,000 40,000 Total visits 2 480,000 690,000 620,000 44,000 Page views 3 2,100,000 3,100,000 2,800,000 180,000
  11. 11. Retrieval of Bookmarking Data <ul><li>bookmarks from BibSonomy , CiteULike and Connotea to articles published in 45 Physics journals from 2004 to 2008 </li></ul><ul><li>search strategies applied to compensate incomplete and erroneous bookmarking data </li></ul><ul><ul><li>journal title </li></ul></ul><ul><ul><li>abbreviations </li></ul></ul><ul><ul><li>ISSNs </li></ul></ul><ul><ul><li>DOIs of articles </li></ul></ul><ul><li>download of bookmarks via API or by parsing </li></ul>
  12. 12. Retrieval of Bookmarking Data <ul><li>completeness / correctness of bookmarking data </li></ul>
  13. 13. Retrieval of Bookmarking Data <ul><li>completeness / correctness of bookmarking data </li></ul>
  14. 14. Retrieval of Bookmarking Data <ul><li>download was checked and limited to bookmarks to documents published in 45 journals from 2004 to 2008 </li></ul><ul><li>13,760 correct bookmarks retrieved </li></ul>
  15. 15. Retrieval of Bookmarking Data <ul><li>missing DOIs were replaced by CrossRef and manually </li></ul><ul><li>bookmarks were matched to the bibliographic entries from Web of Science via DOI </li></ul><ul><ul><li>98.9% of bookmarks matched </li></ul></ul><ul><li>2.8% doubles in user names between services </li></ul><ul><li>results were combined </li></ul>
  16. 16. Retrieval of Bookmarking Data <ul><li>13,608 bookmarks were matched to 10,280 articles </li></ul><ul><li>2,441 unique users </li></ul><ul><li>1,179 users posted one article </li></ul><ul><li>75% of content is created by 21% of users </li></ul><ul><li>8,511 articles were only bookmarked once </li></ul>
  17. 17. Journal Evaluation: Usage Indicators <ul><li>bookmarking an article indicates journal usage </li></ul><ul><li>Usage ratio </li></ul><ul><li>journal articles with bookmarks compared to all publications </li></ul>
  18. 18. Journal Evaluation: Usage Indicators <ul><li>Usage diffusion </li></ul><ul><li>number of unique users per journal </li></ul><ul><li>depicts the diffusion of journal content into the community </li></ul>
  19. 19. Journal Evaluation: Usage Indicators <ul><li>Article Usage Intensity </li></ul><ul><li>average number of bookmarks per article </li></ul><ul><li>shows how often articles are read </li></ul>
  20. 20. Journal Evaluation: Usage Indicators <ul><li>Journal Usage Intensity </li></ul><ul><li>mean number of articles per user </li></ul><ul><li>measures how intensive the average user reads the journal </li></ul>
  21. 21. Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage intensity P 5-year IF h-index Cited Half-Life Usage ratio 0.33 0.90 -0.22 -0.09 0.86 0.50 0.02 Usage diffusion 0.33 0.27 0.45 0.72 0.14 0.78 -0.24 Article usage intensity 0.90 0.27 -0.36 -0.16 0.88 0.46 0.09 Journal usage intensity -0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10 P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17 5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21 h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22 Cited Half-Life 0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22
  22. 22. Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage intensity P 5-year IF h-index Cited Half-Life Usage ratio 0.33 0.90 -0.22 -0.09 0.86 0.50 0.02 Usage diffusion 0.33 0.27 0.45 0.72 0.14 0.78 -0.24 Article usage intensity 0.90 0.27 -0.36 -0.16 0.88 0.46 0.09 Journal usage intensity -0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10 P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17 5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21 h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22 Cited Half-Life 0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22
  23. 23. Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage intensity P 5-year IF h-index Cited Half-Life Usage ratio 0.33 0.90 -0.22 -0.09 0.86 0.50 0.02 Usage diffusion 0.33 0.27 0.45 0.72 0.14 0.78 -0.24 Article usage intensity 0.90 0.27 -0.36 -0.16 0.88 0.46 0.09 Journal usage intensity -0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10 P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17 5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21 h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22 Cited Half-Life 0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22
  24. 24. Journal Evaluation: Usage Indicators Pearson correlation Usage ratio Usage diffusion Article usage intensity Journal usage intensity P 5-year IF h-index Cited Half-Life Usage ratio 0.33 0.90 -0.22 -0.09 0.86 0.50 0.02 Usage diffusion 0.33 0.27 0.45 0.72 0.14 0.78 -0.24 Article usage intensity 0.90 0.27 -0.36 -0.16 0.88 0.46 0.09 Journal usage intensity -0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10 P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17 5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21 h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22 Cited Half-Life 0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22
  25. 25. Journal Evaluation: Tags <ul><li>88.4% of all bookmarks contained tags </li></ul><ul><li>8,208 unique tags assigned 38,241 times </li></ul>tagcloud: all tags assigned at least 50 times
  26. 26. Journal Evaluation: Tags number of unique tags Number of tags per publication year for all bookmarked articles
  27. 27. Journal Evaluation: Tags <ul><li>tags assigned to journal articles depict the users’ perspective on content </li></ul><ul><li>analysis over time can reveal shifts in thematic focus areas </li></ul>tags assigned to articles published in J Phys Condens Matter in 2004 tags assigned to articles published in J Phys Condens Matter in 2008
  28. 28. Conclusions <ul><li>data from social bookmarking services can be used in journal evaluation </li></ul><ul><li>number of bookmarks can be used alternatively to measure worldwide usage of journals </li></ul><ul><li>tags reveal the users’ perspective on journal content </li></ul><ul><li>poor quality of bookmarking data makes retrieval and analysis complex </li></ul><ul><li>social bookmarking in STM is still in its infancy and services are competing for a critical mass </li></ul>
  29. 29. Literature <ul><li>BERNIUS, S., HANAUSKE, M. AND DUGALL, B. (2009). Von traditioneller wissenschaftlicher Kommunikation zu &quot;Science 2.0&quot;. ABI-Technik 29, 214-226. </li></ul><ul><li>HAMMOND, T., HANNAY, T., LUND, B. AND SCOTT, J. (2005). Social Bookmarking Tools (I). D-Lib Magazine 11. </li></ul><ul><li>HAUSTEIN, S. (2010). Multidimensional Journal Evaluation. Proceedings of STI 2010, Leiden. </li></ul><ul><li>JUCHEM, K., SCHLOEGL, C. AND STOCK, W.G. (2006). Dimensionen der Zeitschriftenszientometrie am Beispiel von &quot;Buch und Bibliothek&quot;. Information Wissenschaft & Praxis 57, 31-37. </li></ul><ul><li>PROJECT COUNTER (2008). The COUNTER Code of Practice. Journals and Databases Release 3, in: http://www.projectcounter.org/r3/Release3D9.pdf. </li></ul><ul><li>REHER, S. AND HAUSTEIN, S. ( to be published ). Social Bookmarking in STM: Putting Services to the Acid Test. ONLINE. </li></ul><ul><li>ROUSSEAU, R. (2002). Journal Evaluation: Technical and Practical Issues. Library Trends 50, 418-439. </li></ul><ul><li>SHNEIDERMAN, B. (2008). Computer Science - Science 2.0. Science 319, 1349-1350. </li></ul>Thank you for your kind attention!

×