The document discusses the importance of contextualized scientometric analysis, emphasizing the need for effective communication between domain experts and scientometricians. It highlights the limitations of both simple and advanced indicators in research evaluation and argues for a shift towards contextualized analysis to improve understanding and interpretability. The conclusions suggest focusing on enhancing contextual information access and guiding domain experts in utilizing this data for informed peer review.

1. The need for contextualized
scientometric analysis
Ludo Waltman and Nees Jan van Eck
Centre for Science and Technology Studies
Leiden University
STI Conference 2016, Valencia, Spain
September 16, 2016

2. Evaluative scientometrics:
Basic research at CWTS
1

3. Evaluative scientometrics:
Applied work at CWTS
2

4. To what degree does research evaluation
truly benefit from our work?
3
Do we fully exploit the possibilities of
scientometric analyses?
What should be the main focus of
research in evaluative scientometrics?

5. Trends in evaluative scientometrics
• Increasingly sophisticated indicators
• Do-it-yourself bibliometrics
• Informed peer review
• Altmetrics
4

6. Simple vs. advanced indicators
Simple indicators
• Low accuracy
• Easy to understand
• Limitations are evident
Advanced indicators
• High accuracy
• Hard to understand
• Limitations are hidden in
technical details
5

7. Dilemma of informed peer review
Domain experts
• Deep domain knowledge
• No scientometric
knowledge
Scientometricians
• Deep scientometric
knowledge
• No domain knowledge
6
How should these two groups communicate?
A common language is lacking

8. Dilemma of informed peer review
• Triangulation of expert knowledge and scientometric
indicators is difficult without a common language
• Scientometric indicators are either rejected or blindly
trusted
• Overemphasis on what can be quantified easily
7

9. Contextualized scientometric analysis
• Context
– Allow data underlying an indicator to be explored
• Simplicity
– Increased sophistication comes at a price; value simplicity
– Stop being overly negative about impact factors and h-indices
• Diversity
– To avoid overemphasizing what can be quantified easily, diversity of
indicators is essential
– Better to have indicators of low accuracy than no indicators at all
8

10. 9

11. 10

12. 11

13. Advanced bibliometric indicators vs.
contextualized scientometric analysis
Advanced bibliometric
indicators
• Offer reasonably accurate
but incomplete
information
• Little room for deeper
understanding
• To be used for macro-
level evaluations
Contextualized
scientometric analysis
• Offers detailed
information requiring
careful interpretation
• Allows triangulating
expert knowledge and
indicators
• To be used for micro-level
and meso-level
evaluations
12

14. Conclusions
• Scientometricians should shift their attention from
advanced bibliometric indicators to contextualized
scientometric analysis
• New research questions:
– How to make contextual information available (e.g., interactive user
interfaces, visualizations)?
– How to guide domain experts in the use of this information?
– How to fulfill the promise of informed peer review?
13

15. Thank you for your attention!
14