Blue Sky 19/09/16 Plenary 3

1. Towards more inclusive STI indicators
Ismael Ràfols1,2 and Jordi Molas-Gallart1,
Richard Woolley1 and Diego Chavarro2
1Ingenio (CSIC-UPV), Universitat Politècnica de València
2 SPRU (Science Policy Research Unit), University of Sussex, Brighton
Gent, OECD Blue Sky 2016

2. Re-shaping design and use for inclusion
• Indicators may be harming – via exclusion
 Current indicators are only (partially) appropriate for some types of
science and innovation.
 Biases against and potential suppression of creative and valuable
types of research (agro-, health,…). Threat to diversity.
 May count as positive harmful forms of innovation (Soete)
• Not only more, but other types of indicators needed
 Making visible “other” contributions and other types of research and
innovation (e.g. action research, co-creation)
 Enhancing visualisation for “opening up” perspectives rather than
facilitating “closing down”
• Understanding uses of indicators
 Better embedding in policy, evaluation, context
 Indicators used to pluralise debate, as tools for interpretation
and deliberation, not a substitute for judgement (Barré)

3. Uses of indicators: Pressing demands of research policy,
management and evaluation --- Can indicators help?
Yes, indicators can help make decisions…
 Reduce time and costs
 Increase transparency and sense of objectivity
 Reduce complexity, accessible to managers
but do they lead to the “right” decisions?
Evaluation gap (Wouters):
“discrepancy between [the] criteria [implicit in indicators] and the social and
economic functions of science”
*Academia – “excellence” *Innovation – economic “growth”
Missions not well covered: agriculture, public health, defence,
development, social inclusion,…
Often related to marginalised / “neglected” populations

4. Space of problems
Space of research
Research
well illuminated
by indicators
Problems, research, indicators and marginalisation

5. Problems, research, indicators and marginalisation
Multiple types of space:
STI Peripheries:
research spaces not
well captured
by indicators
Research
well illuminated
by indicators
Cognitive: SSH,
engineering
Linguistic:
non-English
Sectoral: low-tech,
agriculture, creative ind.
Social: gender,
minorities
Geographical: regional, “South”

6. Coverage: Knowledge production on rice by country
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
India China Japan USA
Publications
2000-2009
CABI
Scopus
WoS
Rafols, Ciarli and Chavarro (in preparation)

7. Coverage: Knowledge production on rice by topic
Rafols, Ciarli and Chavarro (in preparation)
2002-2012

8. Streetlight effect in indicators: mistaking light with “problems”
Space of problems
Space of research
Research
well illuminated
by indicators

9. Streetlight effect in indicators: mistaking light with “problems”
Space of problems
Space of research
Space of problems
Space of research

10. Space of problems
Hypothesis: reduced indicator coverage may contract research space
Space of research
Space of STI
indicators
The societal needs dealt by research that is under the streetlight effect, will be
better rewarded.
Reduced diversity of
research efforts...
…reduced coverage
of societal needs

11. Space of problems
Demands for expanding role of science in society…
Space of research
Space of STI
indicators

12. Space of problems
Demands for expanding role of science in society…
Space of research
Space of STI
indicators

13. Space of problems
…may require expanded sets of data and indicators
Space of research
Space of STI
indicators

14. Space of problems
…may require expanded sets of data and indicators
Space of research
Space of STI
indicators
• Comprehensive
databases
• Better normalisation
by context (e.g. citation bias)
• Indicators of process
or interactions
• New data sources

15. narrow
broad
closing-down opening-up
range of
appraisal's
inputs
(issues, perspectives,
scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Leach et al. 2010
Appraisal: Broadening out vs. Opening up

16. narrow
broad
closing-down opening-up
range of
appraisal’s
inputs
(issues, perspectives,
scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Most conventional
S&T indicators??
Appraisal: Broadening out vs. Opening up

17. narrow
broad
closing-down opening-up
range of
appraisal’s
inputs
(issues, perspectives,
scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Conventional
S&T indicators??
Broadening out
Incorporation multiple
analytical dimensions:
New analytical inputs:
media, blogsphere.
Appraisal: Broadening out vs. Opening up

18. narrow
broad
closing-down opening-up
range of
appraisals
inputs
(issues, perspectives,
scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Journal rankings
University rankings
Unitary measures
that may be translated into
prescriptionComposite
European Innovation
Scoreboard
Appraisal: Broadening out vs. Opening up

19. narrow
broad
closing-down opening-up
range of
appraisals
inputs
(issues, perspectives,
scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Appraisal: Broadening out vs. Opening up
Conventional
S&T Indicators??
opening-up
Making explicit underlying
conceptualisations and context
creating tools to facilitate exploration
NOT about the uniquely best method
Or about the unitary best explanation
Or the single best prediction

20. From S&T indicators for justification and disciplining…
… towards S&T indicators as tools for deliberation
Model 2: Plural and conditional
Exploring diverse choices
Facilitating options/choices in landscapes
Model 1: Unique and prescriptive
Proposing “best choices”
Rankings -- ranking list of preferences

21. An agenda for more inclusive metrics
• Inclusiveness in the inputs
 Broadening out: Create more diverse indicators
– Indicators of open science, RRI, hidden, social innovation
– Improve representation of SSH scholarship, languages other than
English, the “South”,…
• Inclusiveness in the outputs
 Opening up: develop toolkits that allow exploration of choices.
New ways of presenting indicators
– Multi-ranking tools
– Interactive visualisations
• Inclusiveness in the policy process (??)
 Develop new social processes on use of indicators
– STI indicators as tools for interpretation and deliberation (R. Barré)

23. Backup slides

24. 24
Rice Varieties
Classic Genetics
Transgenics
Mol. Biology
Genomics
Pests
Plant protection
Weeds
Plant protection
Plant nutrition
Production &
socioeconomic issues
Consumption
H. nutrition,
food techs
What are the “options” in rice research?
Not a fined-grain perspective

25. 25
US, 2000-12
How research priorities differ by country

26. 26
India 2000-12
How research priorities differ by country

27. 27
Thailand 2000-12
How research priorities differ by country

28. S&T Indicators Conference – First week
21st Science and Technology Indicators Conference
València, 14-16 September 2016, www.sti2016.org
The problem:
• Some STI activities and their outcome are badly captured by indicators
• Disincentive to carry out certain types of activity at certain spaces
• Empirically: problem more acute at peripheral spaces in science
• Potential effects: Reduction of diversity –particularly locally relevant?

29. Bias in citation within a subdiscipline
Good
Average
Bad
Van Eck, Waltman et al. (2013)
More basic
More applied
Clinical neurology
Is basic always
better than applied?
Citations: not stable to changes in classification and granularity (Zitt et al., 2005;
Adams et al., 2008).

30. Measures of “scientific excellence”
0
0.5
1
1.5
2
2.5
3
3.5
4
ISSTI SPRU MIoIR Imperial WBS LBS
ABSRank
0
1
2
3
4
5
ISSTI SPRU MIoIR Imperial WBS LBS
Citations/pub
Journal-fieldNormalised
0
0.05
0.1
0.15
0.2
ISSTI SPRU MIoIR Imperial WBS LBS
Citations/pub
Citing-paperNormalised
Which one is more meaningful??
0
1
2
3
4
ISSTI SPRU MIoIR Imperial WBS LBS
JournalImpactFactor
Rafols et al. (2012, Research Policy)