Many contemporary research funding instruments and research policies aim for excellence at the level of individual scientists, teams or research programmes. Good bibliometric approximations of related specialties could be useful for instance to help assign reviewers to applications. This paper reports findings on the usability of reviewer suggestions derived from a recently developed specialty approximation method combining key sources, title words, authors and references (Rons, 2018). Reviewer suggestions for applications for Senior Research Fellowships were made available to the evaluation coordinators. Those who were invited to review an application showed a normal acceptance rate, and responses from experts and coordinators contained no indications of mismatched scientific focus. The results confirm earlier indications that this specialty approximation method can successfully support tasks in research management. Poster presented at the 23rd International Conference on Science and Technology Indicators (STI 2018) "Science, Technology and Innovation indicators in transition", Leiden, The Netherlands, 12-14 September 2018 (http://sti2018.cwts.nl/). Paper: Rons, N. (2018). Testing Reviewer Suggestions Derived from Bibliometric Specialty Approximations in Real Research Evaluations. In: Proceedings of the 23rd International Conference on Science and Technology Indicators (STI 2018) "Science, Technology and Innovation indicators in transition", 12-14 September 2018, Leiden, the Netherlands, Rodrigo Costas, Thomas Franssen, Alfredo Yegros-Yegros (Eds.), 170-173 (http://hdl.handle.net/1887/65261).

1. REVIEWER
SUGGESTIONS
derived from
bibliometric
SPECIALTY
APPROXIMATIONS
TESTING REVIEWER SUGGESTIONS DERIVED FROM
BIBLIOMETRIC SPECIALTY APPROXIMATIONS IN
REAL RESEARCH EVALUATIONS
Nadine Rons
Research and Data Management Department, Vrije Universiteit Brussel (VUB), Pleinlaan 2, BE-1050 Brussels, Belgium
Nadine.Rons@vub.ac.be, http://vub.academia.edu/NadineRons
STI 2018 Leiden
Bibliometric approximation of a research specialty
by partial combination of key values for 4 publication data fields: [7]
8 INVITATIONS
TO REVIEW
6 Applicants
(Engineering & Bioengineering)
30 Potential reviewers,
4-6 per application
Research
administration
34 Potential reviewers,
5-7 per application
APPLICATIONS
11 Potential reviewers,
0-4 per application
Academic evaluation
coordinator (per application)
18 INVITATIONS
TO REVIEW
10 INVITATIONS
TO REVIEW
10 Reviews
4 Reviews
5 Reviews
INVITATIONS aiming to obtain reviews from 3 experts:
≥1 suggested by the applicant,
≥1 not suggested by the applicant
OWN
RESOURCES
Board of the VUB
Research Council
⌷
Senior
Research
Fellowships
Concepts defining
DISCIPLINES [1]
and
SCHOLARLY
COMMUNICATION [2]
Observed regularities
- SOURCES [3a] [3b]
- TITLE WORDS [4]
- AUTHORS [5]
- REFERENCES [6]
[1] Sugimoto, C.R., & Weingart, S. (2015). The kaleidoscope of disciplinarity. Journal of Documentation, 71(4), 775-794.
[2] Ni, C., Sugimoto, C.R., & Cronin, B. (2013). Visualizing and comparing four facets of scholarly communication: Producers, artifacts, concepts, and gatekeepers. Scientometrics, 94(3), 1161-1173.
[3a] Bradford, S.C. (1934). Sources of Information on Specific Subjects. Engineering: An Illustrated Weekly Journal, 137(3550), 85-86.
[3b] Garfield, E. (1971). The Mystery of the Transposed Journal Lists -- Wherein Bradford’s law of Scattering is Generalized According to Garfield’s Law of Concentration. Current Contents, 17, 5-6.
[4] Zipf, G.K. (1935). The psycho-biology of language: An introduction to dynamic philology (Boston: Houghton Mifflin).
[5] Lotka, A.J. (1926). The frequency distribution of scientific productivity. Journal of the Washington Academy of Sciences, 16(12), 317-323.
[6] Price, D.J. (1965). Networks of Scientific Papers. The pattern of bibliographic references indicates the nature of the scientific research front. Science, 149(3683), 510-515.
[7] Rons, N. (2018). Bibliometric approximation of a scientific specialty by combining key sources, title words, authors and references. Journal of Informetrics, 12(1), 113–132.
[8] Rons, N. (2016). 4D Specialty Approximation: Ability to Distinguish between Related Specialties. In: Proceedings of the 21st International Conference on Science and Technology Indicators,
14-16 September 2016, València, Spain.
Applied earlier to subdomains of Biology [7] and Physics [8].
Many contemporary research funding instruments
and research policies aim for excellence
at the level of individual scientists, teams or
research programmes.
Bibliometric approximations of related specialties
could be useful for instance to help find reviewers.
This poster reports findings on the usability of
reviewer suggestions derived from a
new specialty approximation method combining
key sources, title words, authors and references.
Reviewer suggestions were made available to
academic evaluation coordinators during a real
research evaluation.
The coordinators were free to use or not to use
these bibliometric suggestions.
1. USAGE
of the bibliometric reviewer suggestions
by almost all
academic evaluation coordinators
↓ for 5 of the 6 applications
↑
for 8 of the 18
invitations to review
sent to experts
‘not suggested by the applicant’
2. ACCEPTATION RATE
after invitation is normal,
similar to results
for customary approaches
↓ 4✔ / 8☐ for bibliometric
reviewer
suggestions
↑ ↑
15✔ / 28☐
for the other experts,
◻ suggested by applicants
◼ found by coordinators
Responses from invited experts and academic evaluation coordinators
contained no indications of mismatched scientific focus.
The results show how the new
specialty approximation method
was successfully applied for the first
time during an evaluation, to support
academic evaluation coordinators
in their task to find reviewers
for applications by
individual scientists.
The method
builds on
conceptual and
empirical foundations,
and consists of 3 phases:
1. specification of the
seed record as starting point,
2. determination of the sets of most
frequently occurring key sources,
title words, authors and references
characterizing the seed record
3. identification of all publications associated to
key values for ≥ 3 of the 4 data fields
= the specialty approximation.