This document discusses three issues related to developing new scientific models and assessment tools: 1. Developing a more socially relevant bibliometric model of scientific development that tracks trends in publication output and international collaboration for countries. The model outlines four phases of development. 2. Reducing reliance on journal impact factors and developing indicators of the peer review process to provide more transparency and help improve review quality. A two-phase approach is proposed involving exploratory analysis and large-scale data mining. 3. Creating a multi-dimensional information system on universities that educates users and provides discovery tools beyond singular rankings. It argues for comparing different ranking systems and allowing users to observe patterns in complex, multi-faceted institutional data.

1. Towards new
scientific development models and
research assessment support tools
Henk F. Moed
Independent researcher and scientific advisor, Amsterdam, The
Netherlands. Email: hf.moed@gmail.com
OECD Blue Sky Conference, Ghent, 19-21 Sept. 2016

2. Short CV Henk F. Moed
1981-2009
Senior Staff member at Centre for Science and
Technology Studies (CWTS), Leiden Univ.
2009
Professor of Research Assessment Methodologies
at Leiden University
2010 – Sept
2012
Elsevier, SciVal Dept. Senior Scientific Advisor
Sept 2012 –
1 Nov 2014
Elsevier, AGRM Dept. Senior Scientific Advisor
and Head of Informetric Research Group
Sept 2014 –
July 2016
Visiting professor and advisor at Sapienza Univ of
Rome

3. Issues
1. More socially
relevant models
A bibliometric model of scientific development
2. Reduce role of
journal impact factor
Indicators of the manuscript peer review process
3. Educate users,
provide discovery tools
A multi-dimensional info-system on universities

4. 1. More socially relevant models
A bibliometric model of scientific development

5. Annual number of publications during 1980-2014
1
10
100
1000
10000
100000
#ARTICLES(INCITES)
YEAR
TURKEY
IRAN
ISRAEL
EGYPT
SAUDI ARABIA
UAE
QATAR
JORDAN
LEBANON
IRAQ
KUWAIT
OMAN
SYRIA
BAHRAIN
YEMEN

6. A bibliometric model for capturing the state of a country’s scientific development
Phase Description Trend in #
published
articles
Trend in %
internat. co-
auth. publ.
1. Pre-develop-
ment
Low research activity without clear policy
of structural funding of research ~ ~
2. Building-up Collaborations with developed countries
are established; national researchers enter
international scientific networks
+ ++
3. Consolidation
and expansion
The country develops its own
infrastructure; the amount of funds
available for research increases
++ -
4. Internationali-
zation
Research institutions in the country start
as fully-fledged partners, increasingly take
the lead in international collaboration
+ +

7. Percentage of internationally co-authored publications (relative to total publ. output)
0
20
40
60
80
100
03 04 05 06 07 08 09 10 11 12 13 14 15
%INTCOLLABPUB(SCOPUS)
YEAR
QATAR
SAUDI
ARABIA
UAE
LEBANON
JORDAN
IRAN

8. Scientific development phase of 6 major Persian Gulf/Neighbouring countries
Phase Country analysed (2003-2015) Trend in #
published
articles
Trend in %
internat. co-
auth. publ.
1. Pre-develop-
ment ~ ~
2. Building-up Saudi-Arabia, Qatar, UAE,
Lebanon, Jordan
+ ++
3. Consolidation
and expansion ++ -
4. Internationali-
zation
Iran
+ +

9. 2. Reduce role of the
journal impact factor
Towards indicators of the manuscript peer review
process

10. Why are indicators of the manuscript peer review process useful?
• They provide more direct indicators of journal quality
• They further enhance the transparency of the review process
• They can help educating and training reviewers
• They can help assessing the effect of peer review upon manuscript
quality.
• They can be used to monitor and further improve the review process

11. Two phases
Phase Brief description
1 Exploration phase
“Classical-
humanities
approach”
 Development of a conceptual model
 Construction of referee report profiles
 Communication modes between actors
 Based on well-selected, small data samples
2 Data mining
“Digital
humanities
approach”
 Use of computational linguistics tools
 Natural language processing
 Statistical analysis
 Data mining of large data samples

12. 3. Educate users, provide
discovery tools
A truly multi-dimensional info-system on universities

13. Principal indicators in Teaching, Research and Reputation
Aspect ARWU LEIDEN THE QS U-MULTIRANK
Teaching # Award & Prize
winners among
alumni
Teaching
Performance
Faculty per
Student
Quality of
Teaching
(per field)
Research -
Publications
# Publications
(WoS)
# Core Publ.
(WoS)
# Publications
(WoS)
Research -
Citations
# Highly Cited
Researchers
(WoS)
Mean Norm. Cit.
Score; % Top
cited publ.(WoS)
Citations
Performance
(Scopus)
Citations per
Faculty (Scopus)
% Top cited
Publications
(WoS)
Research –
Other
# Publicat. in
Nature, Science
Research
performance
External Income
Interdisc. Pub
Reputation,
recognition
# Award & Prize
winners among
current staff
Acad Reputation
(main factor in
Research Perf.,
Teaching Perf.)
Academic
Reputation;
Employers Rep.

14. Propositions
• One should raise among users the level of background knowledge
and hands-on experience with creating and interpreting indicators
• Ranking systems are still one-dimensional: It is like looking into the
outside world through a few vertical splits in a fence, one at the time
• A ranking system should not make too many ‘decisions’ for users but
rather a dataset and tools to observe patterns in multi-facetted data

15. THE 2016 Research/Reputation vs. Citations Performance: GERMANY
Bayreuth
Bielefeld
Charite Berlin
Free U Berlin
Frankfurt
Heidelberg
Humboldt Berlin
Mainz
Giessen
Karlsruhe IT
LMU Munich
Leibniz Hanover
Magdeburg
RWTH Aachen
Bochum
TU Dresden
TU Darmstadt
TU Dortmund
TU Munich
Ulm
Bonn
Bremen
Cologne
Duisburg-Essen
Erlangen-Nuremberg
Freiburg
Greifswald
Gottingen
Hohenheim
Kaiserslautern Kiel
Konstanz Mannheim
MunsterStuttgart
Tubingen
Wurzburg
0
20
40
60
80
0 20 40 60 80 100
THEResearchPerformance
THE Citations Performance
REPUTATION
CIT IMPACT

16. Concluding remark
• Each ranking system has a unique orientation in terms of
institutional coverage, rating methods, the selection of
indicators and normalizations
• By systematically comparing different systems, more insight
is provided into how these orientations influence the ranking
positions of given institutions

17. THE Research Performance (Reputation) vs. QS Acad Reputation
CHL Cathol Chile
THA Chulalongkorn
CHL Chile
JPN Waseda
COL los Andes
CHN Fudan
JPN Keio
ITA Bologna
BRA Fed Rio Janeiro
FRA Strasbourg
IDN Indonesia
CHL Santiago Chile
ESP Complutense
IND Delhi
CHN City Hong Kong
FRA Paris 7
ITA Pisa
IRL Trinity Coll Dublin
ITA Padua
POL Warsaw
NLD Erasmus
USA Vanderbilt
CHE Bern
USA Georgia
DEU Stuttgart
USA Virginia Poly Inst St
NLD Wageningen
NLD Nijmegen
DEU Erlangen
DEU Konstanz
SWE Karolinska
USA Notre Dame
USA Florida State
TWN Nat Sun Yat sen
TWN Nat Chiao Tung
NLD Twente
TWN Nat Taiwan S-T
NLD Maastricht
DEU Tech Darmstadt
DEU Bielefeld
0
20
40
60
80
100
0 20 40 60 80 100
THERESEARCH(REPUTATION)(PCTLS)
QS ACAD REPUTATION (PCTLS)
THE>>
QS:
NLD
DEU
USA
TWN
QS>>
THE:
CHL
ITA
FRA
JPN
QS REPUT
THE
REPUT
Traces of regional weightings
of reputation indic in QS?

18. Thank you for
your attention