Estimating Business Journal Quality from 2008 UK Research Assessment
•
1 like•410 views
![1. Journal rankings Journal quality Journal rankings ,[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
What's hot
What's hot (19)
Viewers also liked
Similar to Estimating Business Journal Quality from 2008 UK Research Assessment
Similar to Estimating Business Journal Quality from 2008 UK Research Assessment (20)
More from Kent Business School
More from Kent Business School (20)
Recently uploaded
Recently uploaded (20)
Estimating Business Journal Quality from 2008 UK Research Assessment
- 1. Estimating Business and Management Journal Quality from the 2008 Research Assessment Exercise in the UK Professor John Mingers Director of Research Kent Business School Centre for Evaluating Research Performance (CERP) July 2010
- 3. They are based on often arbitrary criteria. They can be by peer review or behavioural (e.g, impact factors)
- 4. The original Kent ranking was simply a statistical combination of other rankings
- 5. “Objectivity results from a combination of subjectivities” (Ackoff)Why are they so contentious?
- 7. THEORY 1: The quality of a journal purely reflects the quality of its papers (Editors/publishers/common sense)
- 8. THEORY 2: Low quality papers may be published in high quality journals and vice versa (RAE)
- 9. It matters in terms of publication strategy and decision-making Is journal a good proxy for quality?
- 10. 2. Reconstructing the 2008 RAE Grades Table 1 Submission statistics for the last three RAEs Adapted from Geary et al (2004), Bence and Oppenheim (2004), RAE (2009a) a Totals differ slightly between different sources. Figures for 2008 are after data cleaning as described later
- 11. Table 2 Number of publications by output type Adapted from Geary et al (2004), Bence and Oppenheim (2004), RAE (2009a). Categories with zero entries have been suppressed
- 12. Figure 1 Pareto curve for the number of entries per journal in the 2008 RAE
- 13. Figure 2Numbers of journals in the RAE and the ABS list
- 15. 2.1 The LP Model Initial model (QP1) Let: j index the journals (j = 1 .. no. of journals) g index the grades 0* - 4* (g = 0 .. 4) i index the universities (i = 1 .. no. of institutions) eig be the estimated proportion of research at grade g for university i pjg be the estimated proportion of the outputs of journal j graded at grade g uig be the actual proportion of research at grade g for university i nij be the number of entries of journal j submitted by university i s.t. for each institution (i) and grade (g) for each journal (j)
- 16. 2.2 Validity of the Results
- 18. Table 9 Proportions of journals in particular ranks comparing ABS with RAE grades Note: we show the proportions in terms of % for ease of comparison but all Chi-Square tests were performed on the underlying frequencies
- 20. This can be seen by comparing the ABS submitted with the ABS not submitted (cols 2, 3)
- 21. Comparing those journals that are in common the level of grading is very similar (cols 3,6)
- 23. There are at least 3 possible explanations of this: Better RAE grades Higher % ABS journals “RAE Bias” Higher % ABS journals Higher quality of department “Better depts. more mainstream” Better RAE grades Higher quality of department “Greater selectivity” Higher % ABS journals
- 25. Mean citations per paper field normalised (cpp)
- 26. Paper quality generates citations which can then measure journal, paper and researcher quality but …Is there a journal effect or a researcher effect as well?
- 27. 2.1 Journal and citations? A study of 6 OR journals from Management Science to Omega Looked at all 600 papers published in 1990
- 28. 2.2 How long do citations take?
- 31. ManSci, EJOR
- 32. Rank of author’s institution
- 33. Moderate
- 34. Theoretical papers, review papers
- 35. No. of pages, no. of references
- 36. OpsRes
- 37. Not significant
- 38. Empirical, methodological, case study papers
- 39. JORS, DecSci
- 40. Author’s country, no. of publications
- 42. 4. Is cpp the best measure: h-index? Cites per paper = no. of cites/no. of papers Clearly this can be increased by getting more citations or by producing less papers. There is a built-in behavioural effect to lessen research productivity The h-index: “a scientist has index h if h of his/her N papers have at least h citations each and the other (N-h) papers have no more than h citations each” (Hirsch, 2005,p. 16569 It measures both impact and productivity
- 47. We can use the RAE data to reconstruct the judgements they made
- 48. Citations are a reasonable measure of impact - citations are in fact peer review by the world
- 49. There are significant problems in measuring citations
- 50. WoS (and Scopus) are extremely limited of their coverage for non-science
- 51. GS has better coverage but is unreliable
- 53. Narrow focus on types of outputs – ie “4*” English language journal articles
- 54. Narrow focus on types of measurements
- 55. Narrow focus on types of impact
- 56. Should we stop now and develop a system that aims to evaluate quality in a variety of forms, a variety of media, through a variety of measures with the ultimate goal of answering significant questions?Adler, N. and Harzing, A-W. (2009) “When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings”, Academy of Management Learning and Education 8, 1, 72-95
Editor's Notes
- Until about 5 years ago I had no interest in rankings at all. I knew the journals in my own disciplines and roughly where they stood. However, I then became Director of Research in the run-up to the RAE and developed a rather unhealthy , indeed salacious interest in anything to do with rankings, citations and so on. Suddenly colleagues keep wanting to know should they send their paper to journal X rather than Y, or having had it rejected by Z where should they send it next. Never having heard of most of the journals it was difficult to advise and so I turned to journal rankings. At that time, before the ABS one, the problem was that there were many different rankings covering different sets of journals using different scoring systems and each with its own particular biases or prejudices. So, being a bit of a statistician I decided to combine these together into one amalgamated list based on the RAE 1* - 4*. I did this and eventually came up with a list of about 650 journals based on 14 other rankings. The main claim I would make for this list if that it has no element of my individual subjectivity in it at all – it is simply a statistical combination of other peoples subjectivities, and as ackoff saidI had thought that it might take an afternoon but in fact it took several weeks because of the nature of the data and when I finished I thought that I would write it up as a paper – it would be of interest to other DoRs and it was quite a rigorous piece of work. There my troubles began – I have never had a paper rejected so many time before! Five rejections before it was eventually published. And the problem was not so much with the paper itself, but the referees, and editors, hated the whole idea of journal rankings. A few quotesIt was eventually published (in a 3* journal) because the editor said that he hated journal rankings and it would give him a chance to do a reply criticising them!
- Some people are interested in journal quality per se (editors/publishers/authors) but often the journal quality is just an indicator of the paper quality. Intrinsic paper quality is not obvious – it needs a a judgement – hence the RAE which judged the quality of over 12,000 papers. But we cannot generally do that Sometimes we are interested in the researcher quality for promotions or jobs and paper quality may be an indicator of this.But basically we cannot easily measure the intrinsic quality of either researchers or papers – we don’t go round with a british lion mark on our heads - so we need other ways of doing it and hence we use the quality of journal as measured by the rankings.There are 2 theories about this
- Lets now move on to citations.Clearly the REF has stimulated a great deal of interest and concern about the use of bibliometric data, especially citations, in evaluating research quality. Although problems with this approach, at least in SS, suggests that they are moving back to peer review but with some bibliometric data.Why are we interested in citations?Whilst citations are not a direct measure of quality they are generally thought to be a measure of impact which is one dimension of quality
- GS averages 71% but is very high for the journals – 92%WoS is only 50% - which is actually quite high in comparison with some studies eg Leiden analysis of RAE2001 had 46% of journal papers in WoS, Norris & Oppenheim had 41% of journalsCPP is 5 for WoS again about averageTwice as big for GSRAE submissions 118Papers in GS 114Papers in WoS 76Citations in GS 1278Citations in WoS 310GS CPP 11.2WoS CPP 4.1
- Could include books, conference papers, reports, websites, artefacts, practice (cf English and drama)Invisible research – new specialised innovativeNot just number or citations but more innovative measures of quality dependent on the areaAlso new measures, eg downloads, growth and decay rates, Google PageRank, co-citation analysis etcImpacts especially external impacts on society policy economy and community