How accurately does output reflect the nature and design of transdisciplinary research programmes? Presentation at STI2014 by Elizabeth Koier and Edwin Horlings

1. How accurately does output reflect the nature
and design of transdisciplinary research
programmes?
STI 2014, Leiden
Elizabeth Koier & Edwin Horlings

2. 2 | How accurately does output reflect the nature and design of transdisciplinary research programmes?
Introduction
• Governments want to stimulate research on grand
societal challenges
• Solutions to societal challenges affect the interests of
non-academic stakeholders
• A new approach arises: transdisciplinary research
• Researchers work together with stakeholders
• Not only scientific goals, also societal output

3. 3 | How accurately does output reflect the nature and design of transdisciplinary research programmes?
Evaluation of transdisciplinary research
• A new, multi-method approach is required
• The goal is to assess the extent to which integration of
knowledge of stakeholders and researchers was
achieved
• Scientometric methods are frequently used in evaluation
practises
Do scientometric methods accuratly reflect the nature
and design of transdisciplinary programmes?

4. 4 | How accurately does output reflect the nature and design of transdisciplinary research programmes?
Two large climate programmes with
transdisciplinary goals
• Climate changes Spatial Planning (2004-2011; €80m)
• Knowledge for Climate (2008-2014; €92m)

5. 5 | How accurately does output reflect the nature and design of transdisciplinary research programmes?
Goals of this study
Two large climate programmes with transdisciplinary
goals:
1. Does scientific output accurately reflect the activities,
nature and design of the programmes?
2. Do the programme and publication data accurately
reflect the contribution of programme members to the
performed research?
3. Are there alternative reproducible metrics that provide
a more accurate picture of the output of
transdisciplinary science?

6. 6 | How accurately does output reflect the nature and design of transdisciplinary research programmes?
Data
• Data on the participants and projects
• Data on output of projects
• articles
• other (often non-scientific) output
• Downloadstatistics of scientific and non-scientific output
from the programme website
• Authors of scientific output and project participants were
linked
• Scientific publications were linked with publication data
from the Web of Science

7. 7 | How accurately does output reflect the nature and design of transdisciplinary research programmes?
WoS coverage of scientific articles
0 10 20 30 40 50 60 70 80 90
Not found in WoS
Found in WoS
For other reasons not present
International
International locally oriented
published in a Dutch language journal

8. 8 | How accurately does output reflect the nature and design of transdisciplinary research programmes?
WoS coverage of scientific articles
0 10 20 30 40 50 60 70 80 90
Not found in WoS
Found in WoS
For other reasons not present
International
International locally oriented
published in a Dutch language journal
Adds up to 34% of this
group
Adds up to 5% of this
group

9. 9 | How accurately does output reflect the nature and design of transdisciplinary research programmes?
Retrieving output by means of WoS funder
information
Number of publications Percentage of the total number
of publications
Total publications 221 100%
Acknowledgement was correctly retrieved 142 64%
–the programmes’ funders were correctly identified 57 26%
–the programmes’ funders were not correctly identified 26 12%
–funders do not include one of the programmes 59 27%
Acknowledgement was mistakenly not retrieved 54 24%
–funders include one of the programmes 34 15%
–funders do not include one of the programmes 20 9%
There is an acknowledgement but it does not contain funder information 7 3%
There is no acknowledgement 18 8%
51% of the articles containing funding information on the programmes
cannot be found using the list of funders in the WoS

10. 10 | How accurately does output reflect the nature and design of transdisciplinary research programmes?
Co-authorship analyses as an indicator for
the involvement of non-academics
Organization type Total number of
projects
Projects not
recognizable from
author affiliations
Percentage not
recognizable from
author affiliations
Universities 23 4 17%
Public research
26 8 31%
organizations
Governments 3 2 66%
Firms 7 6 86%
NGOs 3 3 100%

11. 11 | How accurately does output reflect the nature and design of transdisciplinary research programmes?
Alternative forms of output
• Measuring impact of non-academic output or the impact
of academic output outside of academia is problematic
• Items are incomparable
• The database is based on self-reporting
• Possible solution:
• Citations may be substituted by download statistics

12. 12 | How accurately does output reflect the nature and design of transdisciplinary research programmes?
Measuring the impact of alternative output
via the programme’s download statistics
Methodological issues:
• Non-scientific output is not protected from alternative
publication (multiple download sources)
• Scientific output is dependent on the restrictions of the
publisher
• IP-addresses were not recognized so one person may have
been responsible for multiple downloads
• Not all output was registered (links to a website made by a
project were not counted)
There was no correlation whatsoever between scientific downloads
and citation patterns
Download statistics are in this setup not a reliable
indicator for the impact of alternative output

13. 13 | How accurately does output reflect the nature and design of transdisciplinary research programmes?
Conclusions
A scientometric study of the output does not (yet) give an
accurate insight in the nature and design of
transdisciplinary research programmes
1. Locally and practically oriented research is underrepresented in the WoS
2. By means of the WoS funder information only 51% of the programme
output could be found
3. Co-author analyses show only a small proportion of the stakeholder
involvement in research
4. Measuring impact of non-scientific output or the impact of scientific output
outside of academia is not (yet) possible via download statistics
If we want to take into account the non-scientific output in
evaluation, we need a common understanding of the value
or impact of the various sorts of output.