Countries throughout the world are looking to innovation (particularly derived from advances in science and engineering) not only for wealth creation and job growth but also to help solve the global grand challenges – in energy, food security, water supplies, climate change, environmental sustainability, social unrest and aging populations. And the debt crises in the US and the Euro zone have given an added urgency to the importance of understanding how to build a supply chain of innovation and entrepreneurship that goes all the way from quality, curiosity-driven research to the development of innovative products and services. Publicly supported research and research funding agencies are seen to play a key role in this supply chain. And much attention is being directed in many countries to designing ways to better harness the contributions made by this type of research to innovation. This comes at a time when Western governments in particular are questioning the sustainability of their investments in the research enterprise and looking for ways to maximise the impact of these investments. For some this has led to a preoccupation with measurement and an enhanced role of bibliometrics in the funding allocations process – sometimes with brutal consequences for researchers who have become disenfranchised as a result of the ‘concentration and selectivity’ drive. In my presentation I will overview some of the pluses and minuses of bibliometrics as used for judging research performance for funding purposes. Attention will also be directed to recent initiatives such as STAR-METRICS in the US and the Lattes Platform in Brazil and the SIAMPI project coordinated by KNAW. However, my main message is that governments first need to be clear about what it is they want to achieve from investing in research and to differentially support and measure research activity and outcomes based on these investment objectives. The European Research Area Board’s recent contribution to the Common Strategic Framework for Research and Innovation consultation provides important leadership in this quest.

1. It’s not how you measure – it’s
what you measure
Fiona Wood
Presentation to the ESF Member Organisation Forum
on Peer Review
6 December 2011
Copyright © 2011, Fiona Wood

2. Why invest in research?
To change the world
 Economically
 Socially
 Environmentally
Copyright © 2011, Fiona Wood 2

3. Where to invest?
 High risk-high return curiosity driven
research (internet to lasers to the sex
life of the screw worm)
 Research geared to the grand societal
challenges
 Creation of new innovation and
entrepreneurship skills
Copyright © 2011, Fiona Wood 3

4. How to invest?
 Support basic/frontier and applied
research separately
 Develop performance measures that
match the different expectations
Copyright © 2011, Fiona Wood 4

5. How well do bibliometrics measure the
value of investment in these areas?
Copyright © 2011, Fiona Wood 5

6. Not well…
http://nuclear-news.net/2010/10/16/the-emperors-new-nuclear-clothes/ 6

7. Use and abuse of Bibliometrics
 Count publications, patents, citations to develop S&T
performance indicators
 Treat them as proxies for performance
 Use them to guide policy development & the
allocation of funds, grants, promotion etc
OECD Frascati Manual; RN Kostoff Use and Misuse of Metrics in Research Evaluation, Science and
Engineering Ethics (1997) 3, 109-120
Copyright © 2011, Fiona Wood 7

8. BUT….
‘…bibliometrics does not properly take into
account the originality of the academic’s
work, conceptual innovation, research
applications, scientific and industrial utility’
Institut de France Académie des sciences 2011
Copyright © 2011, Fiona Wood 8

9. …and furthermore…
 Citation counts are a measure of quantity, not
quality
 Bias towards English language, established
researchers and low risk research
 Induce conformity to the ‘exigencies’ of the
measure (salami publications, risk-averse
research, fraud)
 Demoralises and disenfranchises researchers
Copyright © 2011, Fiona Wood 9

10. ….leading to….
 an enormous fragmented data sets of
dubious meaning
 a massive disaggregated literature
 an aggressive band of metrics zealots
Copyright © 2011, Fiona Wood 10

11. 11

12. HOW NOT TO FIX IT…
Copyright © 2011, Fiona Wood 12

13. Excellence in Research for
Australia ‘ERA’ - Objectives
 establish an evaluation framework that gives government, industry,
business and the wider community assurance of the excellence of
research conducted in Australia’s institutions;
 provide a national stocktake of discipline-level areas of research
strength and areas where there is opportunity for development in
Australia’s higher education institutions;
 identify excellence across the full spectrum of research performance;
 identify emerging research areas and opportunities for further
development; and
 allow for comparisons of Australia’s research nationally and
internationally for all discipline areas.
13

14. ‘ERA’ indicators
1. research quality (publications,
citations, research income, peer
review)
2. research volume and activity
3. research application
4. esteem measures
Unit of analysis – the research discipline at
each institution
14

15. Rating Scale (magically) derived
from indicators:
5 well above world standard evaluation
4 above world standard
3 at world standard
2 below world standard
1 well below world standard
NA research outputs did not meet volume
threshold
 No operational defn for ‘world standard’
Copyright © 2011, Fiona Wood 15

16. Problems
 Assessed basic and applied research
together
 Difficulties with apportioning MICT to
classification codes
 Used a ranked journals list
 Did not measure impact
 Process not transparent
Copyright © 2011, Fiona Wood 16

17. Undesirable outcomes
 Attributing meaning to a meaningless
classification system
 Discontinuation of fields of research and
and researcher positions
 Poaching of ‘star research teams’
Copyright © 2011, Fiona Wood 17

18. Undesirable outcomes
 Committed just over $35 million,
occupied over 600 people’s time, and
assessed 330,000 research outputs
from 41 higher education institutions
without addressing any of the real
questions
Fiona Wood 2011 ERA: an ailing emperor’s new
clothes. R&D Review Feb-Mar: 12-13. Invited Op. Ed.
Copyright © 2011, Fiona Wood 18

19. HOW TO FIX IT
 Develop measures that reflect the
different purposes for investments in
curiosity and mission-driven research
 Capture data at sufficiently granulated
levels
 Increase global collaboration between
funding agencies in developing &
trialling new measures
Copyright © 2011, Fiona Wood 19

20. Good Models
 Lattes Platform in Brazil
 SIAMPI pilot KNAW
 STAR-METRICS USA
Copyright © 2011, Fiona Wood 20

21. Lattes Platform Brazil
 Used in making funding funding decision by
government and in universities for tenure and
promotion
 High quality data on history of scientific,
academic and professional activities of
researchers and institutions
 Incentives for compliance
 Unique researcher identifiers
http://lattes.cnpq.br/english/index.htm
Copyright © 2011, Fiona Wood 21

22. SIAMPI
 Social Impact Assessment Methods for research and
funding instruments through the study of Productive
Interactions between science and society
 Offers a framework to assist in the identification and
assessment of social impact of research activities,
programmes and organizations.
 Case studies in Health, ICT, Nanotechnology and
SS&H
NB ESRC in the UK initiatives in capturing impact
http://www.siampi.eu/Pages/SIA/12/643.bGFuZz1FTkc.html
Copyright © 2011, Fiona Wood 22

23. STAR-METRICS
 Science and Technology in America’s
Reinvestment – Measuring the Effects
of Research on Innovation,
Competitiveness and Science
 NSF, NIH and OSTP
https://www.starmetrics.nih.gov/Star/Participate#about
Copyright © 2011, Fiona Wood 23

24. Phase 1 – focus on job
creation from stimulus
1. Build a clean data base
2. Use university administrative records
to calculate the employment impact of
federal science spending through the
American Recovery and Reinvestment
Act and agencies’ existing budgets
Copyright © 2011, Fiona Wood 24

25. Phase 2 - focus on impact
1. Economic growth – through indicators such as
patents and business start-ups
2. Workforce outcomes – measured by student
mobility into the workforce and employment markers
3. Scientific knowledge - measured through
publications and citations
4. Social outcomes -measured by long-term health and
environmental impact of funding.
Copyright © 2011, Fiona Wood 25

26. STAR-METRIC use of Visual
analytics
 Visualising techniques – applied to
complex data sets in trying to
understand and describe the scientific
and innovation enterprise as well as
explaining outcomes to policy makers.
 Utilising visual analytics developed by
Homeland Security to describe terrorist
networks to describe scientific ones.
Copyright © 2011, Fiona Wood 26

27. Where next for European
funding councils?
 Important to develop protocols about what data to
collect, what they mean and how to use them
 Need a better balance between accountability and
risk, requiring major cultural changes in the people
and organizations involved in the sponsorship and
evaluation of research
 E-Collaborate in the development of a universal
template for reporting scientific achievements as
proposed by Julia Lane, NSF
Copyright © 2011, Fiona Wood 27

28. ERAB has made a great start
 Recommendations and advice provided
by the European Research Area Board
in its 2011 contribution to the ‘Common
Strategic Framework for Research and
Innovation’ consultation
Copyright © 2011, Fiona Wood 28

29. Including…
 Be ambitious and be prepared to take managed risks
for the sake of the European economy.
 Divide support between curiosity and mission-driven.
The latter to include both high risk enabling
technologies and further support for Europeans
competitiveness.
 Create a number of independent arms length funding
agencies to support and govern different types of
excellent research and innovation.
Copyright © 2011, Fiona Wood 29

30. THANK YOU FOR YOUR ATTENTION!
fwood@une.edu.au
Copyright © 2011, Fiona Wood 30