1. The document discusses developing a Usage Factor (UF) metric to measure the usage and value of journals, similar to the Impact Factor. 2. An initial study found that both authors and librarians felt a UF would be helpful. However, there are challenges to address like data standards and accounting for print usage. 3. A second stage of testing real usage data from publishers against potential UF calculations is proposed, with a report on the findings. Developing a standardized UF metric could help demonstrate journal value.

1. Usage Factor: Final Report & Next Steps John McDonald, Director, Libraries of the Claremont Colleges Usage Statistics: New Developments and Practical Applications NFAIS Committee on Usage Statistics New York City, October 21, 2008

2. Usage Factor Genesis COUNTER statistics provide a reliable comparison of amount of use between journals But they don’t provide a meaningful usage-based measure of relative quality or value ISI's Impact Factor compensates for the fact that larger journals will tend to be cited more than smaller ones

3. Who will be interested? Is there a demand for it among:- Publishers Authors Librarians What are the practical issues that would need to be addressed? Who will do it & how much will it cost to develop and maintain?

4. Research review From New journal publishing models: an international survey of senior researchers; Ian Rowlands and Dave Nicholas, A CIBER report for the Publishers Association and the International Association of STM Publishers, 22 September 2005

5. Some initial evidence…… From New journal publishing models: an international survey of senior researchers; Ian Rowlands and Dave Nicholas, A CIBER report for the Publishers Association and the International Association of STM Publishers, 22 September 2005

6. Addressing the challenge……. ISI's Impact Factor provides an industry standard for journal quality. Can we do something similar for usage? In other words, should we seek to develop a usage-based measure of journal quality or value? What metrics should be used?

7. The Research……. In 2007 UK Serials Group and COUNTER published research which examined:- the various ways in which journal quality is currently assessed the degree to which any additional usage-based metrics might prove valuable to each stakeholder community practical ways in which such metrics might be derived and constructed to provide the maximum utility for all, within defined resource constraints

8. Stage 1 Phase 1 October 2006 – January 2007 COUNTER Director Peter Shepherd carried out a series of in-depth interviews with:- 7 authors 9 librarians 13 publishers Phase 2 March 2007 Broader web based survey of:- 155 librarians 1400 academic authors Results June 2007 Final report published by UKSG

9. Stage 1, Phase 1 Results - Users Would Journal Usage Factors be helpful to you in assessing the value, status and relevance of a journal? Librarians – YES 100% Authors - YES 100%

10. Phase 1 Usage factor advantages A useful counterweight to Impact Factors Especially helpful for journals and fields not covered by ISI Especially helpful for journals with high undergraduate or practitioner use Especially helpful for journals publishing relatively few articles Data available potentially sooner than with Impact Factors

11. Phase 1 Usage factor advantages Authors select journals that will give their articles prestige and reach. Impact Factor is a widely used surrogate for the former, while perceived circulation and readership reflect the latter. But usage is becoming more important as a measure of reach. Carol Tenopir A simple, usage based metric would make usage more understandable to editors and authors as a measure of value. There is currently much talk of usage and a lot of data, which the non-librarians find confusing. Publisher

12. Phase 1 Issues to address Impact Factors are well known and relied upon in the industry Not all publishers are COUNTER compliant. COUNTER data may not be robust enough for this. How would print usage, still significant for many journals, be taken into account? Would another global measure, such as usage half-life per journal or per discipline, be of greater value? Usage based metrics could stimulate publishers to inflate their usage or be influenced by sales forces and technology investments. Many journals publish in multiple hosting sites, making calculation of a single usage metric problematic.

13. Phase 1 Who will construct the measure? Publishers are, on the whole, unwilling to provide their usage data to a third party for consolidation calculation of Usage Factors. The majority appear to be willing to calculate UFs for their own journals and to have this process audited. This is generally perceived as a natural extension of the work already being done for COUNTER. While it may have implications for systems, these are not seen as being problematic. Some publishers already consolidate their usage data from several sources and also maintain figures on the total number of articles published in each of their journals.

14. Basis for Usage Factor Usage Factor = Total usage over period ‘x’ of articles published during period ‘y’ ÷ Total articles published during period ‘y’

15. Phase 1 Definitions Many views on how UF components should be defined. In particular, how to define total usage specified usage period total number of articles published online specified publication period Tests with real usage data will be required to refine the definitions for these terms.

16. Phase 1 What to measure TOTAL USAGE There was agreement that usage should be total downloads, as specified in COUNTER JR1. SPECIFIED USAGE PERIOD There was overwhelming agreement that the specified usage period be one calendar year. TOTAL NUMBER OF ARTICLES PUBLISHED ONLINE Need to account for journals not covered by ISI Is the ISI definition of “source items” too narrow when measuring usage?

17. Phase 1 What to measure SPECIFIED PUBLICATION PERIOD There was a diversity of responses to this question, with no clear consensus on any time period among authors, librarians or publishers. Insufficient data to support the selection of any one specific option Tests using real data will be required before any firm conclusions can be drawn.

18. Phase 2 Librarian results: new journals Ranking without Usage Factor Ranking with Usage Factor 1. Feedback from library users 1. Feedback from library users 2. Price 2. Usage Factor 3. Reputation/status of publisher 3.Price 4. Impact Factor 4. Impact Factor 5. Reputation/status of publisher

19. Phase 2 Librarian results: existing journals “ I would view Usage Factor as an aid for collection rather than cancellation decisions. Usage per se is a more suitable tool for us when considering cancellation.” Ranking without Usage Factor Ranking with Usage Factor 1. Feedback from library users 1. Feedback from library users 2. Usage 2. Usage 3. Price 3. Usage Factor 4. Cost per Download 4. Price 5. Impact Factor 5. Cost per Download 6. Reputation/status of publisher 6. Impact Factor 7. Reputation/status of Publisher

20. Stage 1 recommendations UF be developed to the point of testing as a practical, implementable measure of journal quality, value and status. On May 18, 2007 the UKSG Committee accepted these recommendations.

21. Next Steps: Stage 2 Test each of the individual elements in the UF equation using real publisher usage data. Compare UF ranks with IF ranks. Refine and further define the workflow/organization scenarios for the definition, calculation and dissemination of the Usage Factor.

22. Stage 2 – The Plan Project Steering Group established (6 publishers, 1 aggregator, 1 hosting service, wider geographical library representation Usage logs to be converted to uniform standard report format for analysis by expert third party RFP currently in DRAFT form for third party selection process

23. Stage 2 – The Deliverables A report (early 2009) which will: Outline the various metrics assessed Recommend which of them prove consistent and robust enough to be adopted for scaled up onward monitoring Suggest any ways in which data providers might amend the way they capture, structure, label, and maintain their data which would make the measurement of Usage Factors easier and more reliable. Propose ways to audit Usage Factors for accuracy

24. Importance….. “ Currently journal publishers are under a lot of pressure to demonstrate the value they provide. By participating in this process, publishers will influence it, helping to develop useful measures in which they can have confidence.” “ This is going to happen in any event, so it is best that UF is developed and implemented by a trusted organization in which publishers are represented.”

25. Usage Factor Project More information at: http://www.uksg.org/usagefactors Contact: Richard Gedye, Project Director [email_address]