Professor Michael Rowlinson, Queen Mary, University of London, UK presented this seminar "Where Next for the ABS Guide" as part of the Whitaker Institute Seminar Series at the Whitaker Institute on 1st February 2012.

1. 1
Galway 1st Feb 2012
Where next for
the ABS Guide?
Michael Rowlinson
(Queen Mary University of London)
Based on Charles Harvey, Aidan Kelly, Huw
Morris, & Michael Rowlinson eds.
The Association of Business Schools Academic
Journal Quality Guide Version 4. March 2010.
A brief history of the ABS Guide
• Originated as the Harvey Morris guide
• Adopted by the ABS
• Produced annually up to 2010, now in
version 4
• Revamped version to appear biannually
from 2013
• 90,000 downloads from 100 countries in
2010

2. 2
Purposes of academic journals
• A stamp of quality. Publication in a peer
reviewed journal indicates an article is worth
reading.
• To disseminate research findings and stimulate
debate.
• An official record of knowledge and information.
A definitive version of an article that can be
catalogued, indexed, accessed and cited.
• Managing intellectual property rights and
permissions. Ensuring rights are properly
assigned between authors and publishers so
that permission for reproduction can be granted.
Functions of Journal Quality Lists
• Help authors decide what to read and
where to publish, especially for early
career researchers, and inter-disciplinary
research.
• Inform staffing decisions for appointment,
promotion, and reward.
• Inform library purchasing decisions.
• To aid internal and external reviews of
research activity and evaluation of outputs

3. 3
Types of Journal Quality List
• Institutional lists
• Derived lists extrapolated from ratings awarded
in assessment and audit e.g. the UK RAE
(Geary, Marriott & Rowlinson, 2004; Mingers,
Watson & Scaparra 2009).
• Peer surveys made on the basis of assessment
of peers in a field or sub-field.
• Citation studies based on the number of times
an average article in a journal is cited.
• Hybrid lists (Harvey, Kelly, Morris & Rowlinson,
2010; Harzing, 2011).
Commonly perceived problems of
Journal Quality Lists
• Wheat and chaff. Sometimes highly rated
journals publish poor quality work and low rated
journals publish high quality work.
– But this is not commonplace and journals can move
up and down ratings
• Special issues dilute editorial standards
– No evidence for 3 and 4 ABS rated journals
• New journals get lower ratings
– Need for a track record
• Citation impact factors are sufficient to measure
journal quality
– Impact factors are problematic as a standalone proxy

4. 4
Limitations of Citation Impact Factors
as a measure of journal quality
• Incomplete coverage. 459/821 journals in ABS List 2010
have an ISI citation impact factor for 2008.
•• Differences in epistemological traditions need to be
recognized. High citation impact results from repeated
inclusion of articles in referential chains
• Herding means that academics associate their work with
highly cited authors and highly cited journals, reinforcing
perceived superiority of highly cited journals
• Content bias means that journals featuring survey
articles, lit reviews, methodology and conceptual
development have higher citation impact that journals
publishing original research
• Game playing and differences in maturity between fields
– Established fields such as economics have higher citation rates
than newer fields such as tourism and hospitality
– The rules of the game in mature fields require citation of journals
in referential networks
Main features of the ABS List
• Provides an expert assessment of the quality of
empirical and theoretical research articles typically
published in an academic journal
• A consensus list that rests on general acceptance of the
fair and balanced view taken by the editors and panel
members
• Is not comprehensive but includes all core journals with
2 or more submissions for bus & mgt in UK RAE2008
plus limited others
• Ratings not fixed and will be reviewed biennially from
2010 (the next version is due 2013)
• The rating scale is not the same as the UK RAE and
makes no reference to ‘national’ and ‘international’
standards

5. 5
ABS List specification of Journal
Quality Standards
Quality
Rating
Meaning of Quality Rating No. and
(%)
4*
World Elite Journals - a small number of grade four journals recognized
worldwide as exemplars of excellence. Their high status acknowledged by
inclusion well regarded international journal quality lists.
22
(2.7%)
4 publish the most original and best executed research. Typically have high
submission and low acceptance rates. Papers heavily refereed. Generally have
the highest citation impact factors within their field.
72
(8.7%)
3 publish original and well executed research papers and are highly regarded.
Typically have good submission rates and are very selective. Papers are heavily
refereed. Have fair to good citation impact factors relative to others in their field,
230
(27.9%)
but not all journals in this category carry a citation Impact factor.
2 publish original research of an acceptable standard. Well regarded journal in its
field, papers are fully refereed according to accepted standards and conventions.
Have modest citation impact factors or do not have one at all.
295
(35.8%)
1 publish research of a recognized standard and are modest standard journals
Within their field. Papers refereed relatively lightly according to accepted
conventions. Few journals in this category carry a citation impact factor.
204
(24.8%)
Proportion of journals at each ABS grade
Grade ABS 2008 ABS 2009 ABS 2010
4 10.3 10.5 11.4
3 24.9 24.4 27.9
2 37.8 37.4 35.9
1 27.0 27.8 24.8
Total 1017 1033 825

6. 6
The ABS List has a highly consistent pattern of
correlation with other ranking schemes, RAE 2008
outcomes and citation impact factors
Table 5: Mean scores on RAE 2008 data by ABS 2010
Grade
Mean cites in
BMS
Mean outputs
GPA
Mean Overall
GPA
4 24.5 3.5 3.8
3 19.9 2.9 3.2
2 8.7 2.5 2.7
1 5.2 2.2 2.3
Total 12.7 2.6 2.9
Should the REF use The ABS List?
Taylor’s (fc) results from his analysis of the RAE 2008:
‘support the use of journal quality indicators in the
research assessment process. Requiring the panels to
take bibliometric indicators such as journal quality scores
into account should help not only to reduce their
workload but also to mitigate the implicit bias indicated
by the statistical analyses reported in this paper. This
process would also be more consistent with the
widespread use of journal quality indices such as the
ABS Journal Quality Guide by business schools in
assessing the quality of their own research, as they did
in preparation for the 2008 RAE.’

7. 7
Should the REF use The ABS List?
According to Taylor’s (fc) analysis, the estimated
bias in favour of the Russell Group was much
greater for the esteem score than for research
outputs:
‘This may be a consequence of the fact that the
existence of the ABS Guide could have helped
to moderate the panel’s judgements of the
quality of a department’s research output,
whereas it was less constrained in the case of
judgements about a department’s esteem.’
Should the REF use The ABS List?
(we was robbed in the RAE 2008!)

8. 8
ABSv4 RAE08 outputs
21
33 2
160
140
120
37
30
66
13 12
37
27
10
2 8
10
4
6
18
7
100
80
60
no outputs
nonABS
ABS1
ABS2
ABS3
ABS4
32
39
24
36
23
45
39
40
20
0
QM08 King's08 Leicester08 Royal Holl08 St Andrews08
RAE08 output profiles
0 0 0 0 0
35
36
31 33 33
9
4
8 8 8
100
75
42
47
33
42 41
50
U/C
1
2
3
4
14 13
28
17 18
25
0
Leicester 08 StAndrews 08 King's 08 QM 08 Royal Holl08

9. 9
ABS scores
450
400
350
300
250
200
150
cum ABS scores
QM08
Kings08
Leicester08
RoyalHoll08
StAndrews08
100
50
0
0 5 10 15 20 25 30 35 40 45 50
staff nos
Issues to consider if the Guide is to
become more European/International
• Separation from the UK Research Excellence
Framework (the REF)
– The REF provides a valuable publicly available data set
• Weighting of Web of Science citation impact factors
– Difficulty of using Thomson Reuters proprietary data
• Distinctiveness from other rankings and ratings
– e.g. Publish or Perish
• Identify or generate alternative sources of accessible
non proprietary data
•• Avoid being bound by an algorithm, while not
succumbing to lobbying from special interests
• Facilitate validation of a wide variety of European and
international business school missions
• Retain the principle of sub-field normalization, in
recognition that measurements such as impact factors
vary between fields

10. 10
References
Geary, J., Marriott, L., & Rowlinson, M. 2004. Journal rankings in business and
management and the 2001 Research Assessment Exercise in the UK.
British Journal of Management, 15(2): 95-141.
Harvey, C., Kelly, A., Morris, H., & Rowlinson, M. 2010. The Association of
Business Schools Academic Journal Quality Guide Version 4.
Harzing, A-W. 2011. Journal Quality List Thirty-eight Edition.
Mingers, J., Watson, K., & Scaparra, P. 2009. Estimating Business and
Management Journal Quality from the 2008 Research Assessment Exercise
in the UK, Kent Business School Working Paper Series.
Taylor, J. forthcoming. The Assessment of Research Quality in UK Universities:
Peer Review or Metrics? British Journal of Management