Usage Factor vs. Impact Factor: A case study with

1,934 views
1,797 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,934
On SlideShare
0
From Embeds
0
Number of Embeds
15
Actions
Shares
0
Downloads
14
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • First, we reviewed some published literature that surveyed authors and asked the question about whether article downloads are a good measure of usefullness (blue bars), and if author citations are a good measure (green bars). You’ll notice that not only do they overwhelming agree that citations are good measures of of the usefulness of a journal, but also that article downloads are also a good measure.
  • When compared directly, Rowlands and Nicholas found that article downloads offer a better measure of the usefulness of research than citation.
  • Since 2002, downloads per researcher has increased 3 to 10 fold, while articles published per researcher has remained flat (except in canada?!?)
  • Out of the research, an idea of what metrics could contribute to a Usage Factor measure began to emerge. Similar to Impact Factor, it was Total Usage over a Specified Time Period of the Articles Published during a Time Period, divided by the Total articles published during the Time Period
  • Sorted by Impact factor rank Nonrandom sample of Bioone journals (those attending PPM w/ IF rank) 7 journals have higher impact ranks than usage ranks Many high impact journals have medium to low impact
  • Sorted by Usage factor rank 5 journals have higher usage factor ranks than IF ranks Am bio teacher as prime example of practitioner journal NOT well eval by impact
  • Adding in journals attending PPM that are not ISI ranked (Green bars = no IF rank)
  • # represented here
  • Box plots dot = average, line = median, box includes central 50% of values
  • As would be expected from earlier correlations– local or consortial downloads are not related to usage factor, although there is a strong relationship among total downloads at the local, consortial, and global levels (data not shown)
  • Usage Favtor survey Phase 2 provided more information relating to how librarians would use Usage Factor. In ranking new journals, they would use it as the 2 nd most important factor, behind direct feedback, but ahead of price and Impact Factor.
  • On evaluating existing journals, it would be third, behind raw usage figures, but ahead of price again. A librarian was quoted during this time period as saying, “I would view Usage Factor as an aid for collection rather than for cancellation decisions. Usage per se is a more suitable tool for us when considering cancellation”
  • The plan is that the Project Steering Group was established with six publishers, one aggregator, one hosting service and broader geographical representation by libraries. Usage logs will need to be converted to a uniform standard report format suitable for analysis by an outside vendor. And RFP for this portion of the study is currently in draft form and is expected to be published early in 2009.
  • This report will outline the various metrics, recommend the best ones that are practical and scalable. In addition, the report will outline various ways that data providers can capture, structure, lable, and maintain their data to ensure the rapid and reliable dissemination of their Usage Factors. Also, the report will propose audit methods for all providers to ensure their accuracy.
  • Usage Factor vs. Impact Factor: A case study with

    1. 1. Usage Factor vs. Impact Factor: A case study with BioOne Journals Jason Price, PhD Head of Collections and Acquisitions, Claremont University Consortium eJournal Package Analyst, Statewide Calif. Electronic Library Consortium
    2. 2. Researcher opinion on usefulness metric: Downloads as good (or better) than citations CIBER: New journal publishing models : an international survey of senior researchers; Ian Rowlands and Dave Nicholas, 2005
    3. 3. Researcher opinion on usefulness metric: Downloads better than citations CIBER: New journal publishing models : an international survey of senior researchers; Ian Rowlands and Dave Nicholas, 2005
    4. 4. Downloads per researcher have increased more rapidly than articles per researcher Source: Elsevier Customer Research, Scopus
    5. 5. Usage Factor - Background <ul><li>Idea: Leverage COUNTER statistics to create a universal usage-based metric of relative journal quality </li></ul><ul><li>Information gathering process: </li></ul><ul><ul><li>Oct 2006: In-depth interviews </li></ul></ul><ul><ul><ul><li>~10 @ authors, librarians & publishers </li></ul></ul></ul><ul><ul><li>March 2007: Web survey </li></ul></ul><ul><ul><ul><li>150 librarians; 1500 Authors </li></ul></ul></ul><ul><ul><li>June 2007: Final Report: Stage 1 (Shepherd) </li></ul></ul><ul><ul><ul><li>http://www.uksg.org/usagefactors/final </li></ul></ul></ul>
    6. 6. Usage Factor Formula <ul><li>Usage Factor = </li></ul><ul><li>Total usage over period ‘x’ of articles published during period ‘y’ </li></ul><ul><li>÷ </li></ul><ul><li>Total articles published during period ‘y’ </li></ul>
    7. 7. Usage Factor - Advantages <ul><li>Can be applied to all online journals </li></ul><ul><li>Measures practitioner & student use </li></ul><ul><li>Easily calculated by publishers </li></ul><ul><li>Additional pre-purchase journal eval. metric </li></ul><ul><li>Not discipline dependent? </li></ul><ul><li>Data availability: more rapid? less proprietary? </li></ul>
    8. 8. Usage factor - Concerns <ul><li>More potential for abuse </li></ul><ul><li>Difficult to agree on length of publication period to assess </li></ul><ul><li>Many publishers unwilling to provide usage data to 3 rd parties – will require audit process </li></ul><ul><li>Many journals published on multiple platforms, making download # harder to count </li></ul>
    9. 9. Usage Factor Pilot Study – Bioone Journals <ul><li>Title-level Data (n = 134) </li></ul><ul><li>Impact Factor (n = 94) </li></ul><ul><li>Total Times Cited in ISI (2007) </li></ul><ul><li>2008 Global Downloads* </li></ul><ul><ul><li>* of abstracts & articles from the BioOne Platform </li></ul></ul><ul><ul><li>- adjusted for automated indexing hits </li></ul></ul><ul><li># of articles in Bioone (≈ Publication period) </li></ul><ul><li>2008 Claremont and Oberlin Group downloads </li></ul>
    10. 10. Questions <ul><li>How do impact and usage factor relate to total citations & downloads? </li></ul><ul><li>(How) Do impact and usage factor rankings differ? </li></ul><ul><li>How does usage factor compare: </li></ul><ul><ul><li>between ISI-ranked & unranked? </li></ul></ul><ul><ul><li>Between .1 and .2 ? </li></ul></ul><ul><li>Does usage factor differ from local usage? </li></ul><ul><li>[How could usage factor be used to promote or select Bioone journals?] </li></ul>
    11. 11. Strong? Impact Factor Usage Factor 2007 ISI Citations Strong? Weak? 2008 Global Downloads Weak? Strong? 2007
    12. 12. Impact and usage factor ranks are not related
    13. 13. Higher impact journals with lower usage factor
    14. 14. Higher usage journals with lower impact factor
    15. 16. Bioone Profile
    16. 17. Largely overlapping usage factor distributions
    17. 18. Local usage unrelated to usage factor
    18. 19. Back to Survey Data: Librarian results: new journals Ranking without Usage Factor Ranking with Usage Factor 1. Feedback from library users 1. Feedback from library users 2. Price 2. Usage Factor 3. Reputation/status of publisher 3.Price 4. Impact Factor 4. Impact Factor 5. Reputation/status of publisher
    19. 20. Survey Data: Librarian results: existing journals “ I would view Usage Factor as an aid for collection rather than cancellation decisions. Usage per se is a more suitable tool for us when considering cancellation.” Ranking without Usage Factor Ranking with Usage Factor 1. Feedback from library users 1. Feedback from library users 2. Usage 2. Usage 3. Price 3. Usage Factor 4. Cost per Download 4. Price 5. Impact Factor 5. Cost per Download 6. Reputation/status of publisher 6. Impact Factor 7. Reputation/status of Publisher
    20. 21. Usage Factor - Stage 2 <ul><li>Project Steering Group established </li></ul><ul><ul><li>6 publishers </li></ul></ul><ul><ul><li>1 aggregator </li></ul></ul><ul><ul><li>1 hosting service </li></ul></ul><ul><li>Usage logs converted to uniform standard report format for analysis by expert third party </li></ul><ul><li>RFP for third party selection process </li></ul>
    21. 22. Stage 2 – Deliverables <ul><li>A report (early 2009) which will: </li></ul><ul><ul><li>Outline the various metrics assessed </li></ul></ul><ul><ul><li>Recommend which of them prove consistent and robust enough to be adopted for scaled up onward monitoring </li></ul></ul><ul><ul><li>Suggest any ways in which data providers might amend the way they capture, structure, label, and maintain their data which would make the measurement of Usage Factors easier and more reliable. </li></ul></ul><ul><ul><li>Propose ways to audit Usage Factor for accuracy </li></ul></ul>
    22. 23. Summary: Usage Factor Implications <ul><li>Can break down the ISI ‘divide’ </li></ul><ul><li>More favorable evaluation of practitioner/student used journals </li></ul><ul><li>Can be used to show value of less well known journals (particularly Bioone.2) </li></ul><ul><li>Could be used to identify candidates for expansion </li></ul><ul><li>What others? </li></ul>
    23. 24. Comments <ul><li>Doesn’t address journals w/ narrow coverage / small audience </li></ul><ul><li>Basic vs. Applied Science journals might be differentially affected </li></ul><ul><li>Usage factor might correlate better with Impact factor if a dif’t publishing period was examined </li></ul><ul><li>Need a better estimate of downloads from all platforms to verify results </li></ul>
    24. 25. Sources/FMI <ul><ul><li>Usage Factor Website </li></ul></ul><ul><ul><ul><li>http://www.uksg.org/usagefactors </li></ul></ul></ul><ul><ul><li>Rowland & Nicholas. 2005 CIBER: New journal publishing models: an international survey of senior researchers </li></ul></ul><ul><ul><ul><li>http://www.ucl.ac.uk/ciber/pa_stm_final_report.pdf </li></ul></ul></ul><ul><ul><li>McDonald. 2008. Usage Factor: Final Rpt. & Next Steps </li></ul></ul><ul><ul><ul><li>http://www.nfais.org/Usage08McDonald.ppt </li></ul></ul></ul><ul><ul><li>---------------------- </li></ul></ul><ul><ul><li>Peter Shepherd AKA Tom Angus Novel Let Them Eat Cake </li></ul></ul><ul><ul><ul><li>A hilarious comedy caricature of the excesses of the publishing industry </li></ul></ul></ul><ul><ul><ul><li>‘ whose characters are 2 nd to none in their wild & offbeat excentricity’ </li></ul></ul></ul><ul><ul><ul><li>Read more & Order http://tinyurl.com/tomangus </li></ul></ul></ul>

    ×