Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Responsible metrics: One size doesn't fit all

245 views

Published on

Presentation at the 23rd International Conference on Science and Technology Indicators (STI 2018). Leiden, the Netherlands, September 13, 2018.

Published in: Science

Responsible metrics: One size doesn't fit all

  1. 1. Responsible metrics: One size doesn’t fit all Ludo Waltman Centre for Science and Technology Studies (CWTS), Leiden University 23rd International Conference on Science and Technology Indicators (STI 2018) Leiden, the Netherlands September 13, 2018
  2. 2. Responsible metrics 1
  3. 3. One size doesn’t fit all • Research evaluation is an umbrella term • Responsible use of scientometrics depends on evaluative setting • Micro-level vs. macro-level evaluation 2
  4. 4. Macro-level research evaluation • Comprehensive in-depth evaluation is impossible at the macro level • Experts view the world through indicators • Indicators do not just support evaluation, they enable it 3Source: National Science Board Science & Engineering Indicators 2018
  5. 5. Micro-level research evaluation 4 Whom to hire?
  6. 6. Micro-level vs. macro-level research evaluation 5
  7. 7. Implications of the micro-macro distinction • Sophistication vs. simplicity • Professional vs. citizen scientometrics • Indicators vs. statistics 6
  8. 8. Sophistication vs. simplicity • Macro level: – Experts view the world through indicators – Indicators need to have an unambiguous conceptual foundation and a high validity – This typically requires a relatively high level of sophistication • Micro level: – Indicators do not so much provide information themselves; they point experts to potentially relevant information – Keeping indicators simple ensures that experts can truly reflect on what indicators tell them 7
  9. 9. Sophistication vs. simplicity 8
  10. 10. Professional vs. citizen scientometrics • Citizen scientometrics not meant to be understood in an offensive manner! • Macro level: – Need for professional scientometrics • Micro level: – To ensure indicators are helpful in supporting expert assessment, involvement of experts in choice and design of indicators is essential – Experts need to take on role as citizen scientometrician 9
  11. 11. Indicators vs. statistics • Terminology: – Metric: Direct measurement – Indicator: Proxy of an underlying concept – Statistic: Quantitative summary of a body of information • Macro level: – Concepts of interest need to be clearly specified a priori – Preferred term: Indicator • Micro level: – No need to impose strong a priori interpretations – Interpretation is given by experts during evaluation – Preferred term: Statistic 10
  12. 12. Conclusion • Responsible use of scientometrics depends on evaluative setting • Macro-level vs. micro-level distinction offers example – Macro level requires ‘classical scientometrics’ – Micro level requires ‘contextualized scientometrics’ • Meso-level evaluation • Interdependencies between levels • Need for more refined taxonomies of different types of evaluations 11
  13. 13. Thank you for your attention! 12

×