This document discusses journal rankings and quality measures used in the 2008 UK Research Assessment Exercise. It makes three key points: 1. Journal rankings are based on subjective criteria and can be biased, but they are commonly used to assess researcher and paper quality. Quality is better assessed through peer review of individual papers. 2. Reconstructing the RAE grades shows the grades aligned with expected journal quality rankings. Journals submitted that were also in the ABS list received higher grades. 3. Citations can measure paper and researcher quality over time, but are influenced by various factors like journal prestige and author characteristics. The h-index may be a better metric than citations per paper.

1. Estimating Business and Management Journal Quality from the 2008 Research Assessment Exercise in the UKProfessor John MingersDirector of ResearchKent Business SchoolCentre for Evaluating Research Performance (CERP)July 2010

2. 1. Journal rankings Journal qualityJournal rankingsMany different journal rankings each with its own biases and prejudices

3. They are based on often arbitrary criteria. They can be by peer review or behavioural (e.g, impact factors)

4. The original Kent ranking was simply a statistical combination of other rankings

5. “Objectivity results from a combination of subjectivities” (Ackoff)Why are they so contentious?

6. Paper qualityJournal qualityJournal rankingsResearcher qualityPaper quality is unknown unless we peer review it – hence the RAE; so is researcher quality – no little Lion markSo we impute them from the journal ranking

7. THEORY 1: The quality of a journal purely reflects the quality of its papers (Editors/publishers/common sense)

8. THEORY 2: Low quality papers may be published in high quality journals and vice versa (RAE)

9. It matters in terms of publication strategy and decision-making Is journal a good proxy for quality?

10. 2. Reconstructing the 2008 RAE Grades Table 1 Submission statistics for the last three RAEsAdapted from Geary et al (2004), Bence and Oppenheim (2004), RAE (2009a)a Totals differ slightly between different sources. Figures for 2008 are after data cleaning as described later

11. Table 2 Number of publications by output typeAdapted from Geary et al (2004), Bence and Oppenheim (2004), RAE (2009a). Categories with zero entries have been suppressed

12. Figure 1 Pareto curve for the number of entries per journal in the 2008 RAE

13. Figure 2Numbers of journals in the RAE and the ABS list

15. 2.1 The LP Model Initial model (QP1)Let: j index the journals (j = 1 .. no. of journals)g index the grades 0* - 4* (g = 0 .. 4) i index the universities (i = 1 .. no. of institutions)eig be the estimated proportion of research at grade g for university ipjg be the estimated proportion of the outputs of journal j graded at grade guig be the actual proportion of research at grade g for university inij be the number of entries of journal j submitted by university i s.t. for each institution (i) and grade (g) for each journal (j)

16. 2.2 Validity of the Results

18. Table 9 Proportions of journals in particular ranks comparing ABS with RAE gradesNote: we show the proportions in terms of % for ease of comparison but all Chi-Square tests were performed on the underlying frequencies

19. Conclusions from Table 9 Overall RAE grades were higher than overall ABS grades (cols 1, 4) but this was because of selectivity of submissions

20. This can be seen by comparing the ABS submitted with the ABS not submitted (cols 2, 3)

21. Comparing those journals that are in common the level of grading is very similar (cols 3,6)

22. In the RAE , ABS journals were graded more highly than non-ABS journals (cols 5,6)Figure 3 Scattergram showing association between GPA and proportion of an institution’s submitted journals that are in ABS

23. There are at least 3 possible explanations of this:Better RAE gradesHigher % ABS journals“RAE Bias”Higher % ABS journalsHigher quality of department“Better depts. more mainstream”Better RAE gradesHigher quality of department“Greater selectivity”Higher % ABS journals

24. 2. Citations Paper qualityJournal qualityJournal rankingsResearcher qualityCitationsThe REF and the Leiden methodology

25. Mean citations per paper field normalised (cpp)

26. Paper quality generates citations which can then measure journal, paper and researcher quality but …Is there a journal effect or a researcher effect as well?