Panel talk on November 3, 2023 at the National Open Research Festival 2023 which took place at the Royal Irish Academy in Dublin, Ireland. Panel moderator: Yensi Flores Bueso Slides from early career researchers: Noémie Aubert Bonn, Postdoctoral Researcher at Hasselt University, Belgium, and the University of Manchester, UK; Melissa Sharp, Senior Postdoctoral Fellow and Honorary Lecturer, Royal College of Surgeons Ireland; Erzsébet Tóth Czifra Head of Programme at the Coalition for Advancing Research Assessment (CoARA); Stefan Müller, Assistant Professor and Ad Astra Fellow in the School of Politics and International Relations at University College Dublin, Member of the Young Academy of Ireland; Irene Castellano, Horizon Europe Health Cluster National Contact Point (NCP) for Ireland and Chair of the Ireland Chapter of the Marie Curie Alumni Association (MCAA).

1. ECR PERSPECTIVES OF
RESEARCH ASSESSMENT
NORFEST 2023

2. Panel Moderator
Yensi Flores Bueso
MSCA postdoctoral fellow at University College Cork
& the Institute for Protein Design, UW
Focus: Molecular Biologist and microbiome
researchers developing cancer therapies
Steering board member of CoARA, member of the
Executive Committee of the GYA, member of the
Young Academy of Ireland.
ECR PERSPECTIVES OF RESEARCH ASSESSMENT

3. https://www.unesco.org/en/recommendat
ion-science
https://council.science/publications/the-
future-of-research-evaluation-a-synthesis-of-
current-debates-and-developments/
https://www.researchsquare.com/article/rs-
3011208/latest

4. What is research assessment?
How does it shape our careers?
What are the current trends?
How can we adapt and engage?
what outcomes can we expect?

5. What is research assessment and how does it define our research
culture?
Noémie Aubert Bonn
Post-doctoral researcher at Hasselt University,
Belgium & University of Manchester, UK.
Focus: Research assessments' impact on
practices, wellbeing, and integrity.
Former policy advisor at UKRI: Future Research
Assessment Programme and Global Research
Council WG on Responsible Research Assessment.

6. Research assessment and
research culture
What is research assessment and how do they
define the research culture?
Panel discussion on research assessment and early career researchers at the National Open Research
Festival
2 November 2023
Noémie Aubert Bonn
CC-BY

7. ⦿ Focus groups and interviews with diverse stakeholders
⦿ Small-scale survey with researchers
⦿ Large-scale survey with researchers
Research2
Research assessment and
research culture
What is research assessment and how do they
define the research culture?
I want to be a
Researcher!
Publish or
perish….?
Wait… is this what is
means to be a
researcher?

8. PhD candidates
Post Doc
Faculty researchers
⦿ Focus groups and
interviews with diverse
stakeholders
56 stakeholders in the Flemish
biomedical science
What makes researchers
successful?
Aubert Bonn, N., Pinxten, W. (2021) https://doi.org/10.1186/s41073-020-00104-0 and https://doi.org/10.1186/s41073-020-00105-z
Past-Researchers
Integrity offices
Institution leaders
Policy makers
Editors/Publishers
Research funders
Lab technicians
Integrity network

9. Planification
Peer Reviewing
Data sharing
Collaboration
Analysis
Outreach
Real life application
Committee membership
Data collection
Methodology Publication Implementation
Collegiality
Teaching
Skills building
Keeping up-to-date with recent science
Community involvement
Dissemination in conferences
Mentoring and supervision
(Patent)
Funding application
Aubert Bonn, N., Pinxten, W. (2021) Research Integrity and Peer Review 6, 1 et 3. DOI: 10.1186/s41073-020-00104-0 et 10.1186/s41073-020-00105-z
Industry collaboration
Mobilisation
Support
Academic services and improvements
Expertise building
Networking
Networking
Funding acquisition
Scientific editing
Aubert Bonn, N., Pinxten, W. (2021) https://doi.org/10.1186/s41073-020-00104-0 and https://doi.org/10.1186/s41073-020-00105-z

10. Planification
Peer Reviewing
Data sharing
Collaboration
Analysis
Outreach
Real life application
Committee membership
Data collection
Methodology Publication Implementation
Collegiality
Teaching
Skills building
Keeping up-to-date with recent science
Community involvement
Dissemination in conferences
Mentoring and supervision
(Patent)
Funding application
Industry collaboration
Mobilisation
Support
Academic services and improvements
Expertise building
Networking
Networking
Funding acquisition
Scientific editing
Aubert Bonn, N., Pinxten, W. (2021) https://doi.org/10.1186/s41073-020-00104-0 and https://doi.org/10.1186/s41073-020-00105-z
Success is almost exclusively measured from research outputs
(publication!)

11. Success is almost exclusively measured from research outputs
(publication!), despite what researchers find important.

12. QUESTION:
…When a researcher's performance is being evaluated by an employer or potential
employer, how important do you think it is to include each of the following activities
in making an assessment of their performance?
• Societal impact of their research
• Teaching
• Leadership
• Editorship of journals and other publications
• Supervisory responsibilities
• Outreach and communication of research to public audiences
• Peer review
• Publication metrics (JIF, H index)
• Collegiality
• Participation in, or delivery of, research integrity training
⦿ Large scale survey with researchers
European Commission H2020 project
IRIS survey coordinated by Nick Allum and Abi Reid, University of Essex
n = ± 50 000

13. 0
5
10
15
20
25
30
35
40
45
50
Not important at all Somewhat important Fairly important Very important Extremely important
Societal impact of research Editorship of journals and other publica
Publication metrics (eg Journal Impact F Participation in, or delivery of, resear
…how important do you think it is to include each of
the following activities in making an assessment of
researchers’ performance
• Societal impact of their research
• Editorship of journals and other publications
• Publication metrics (e.g., Journal Impact Factor, H index)
• Participation in, or delivery of, research integrity training
PRELIMINARY DATA

14. 0
5
10
15
20
25
30
35
40
45
50
Not important at all Somewhat important Fairly important Very important Extremely important
Societal impact of research
Editorship of journals and other publica
Outreach and communication of research t
Publication metrics (eg Journal Impact F
Participation in, or delivery of, resear
…how important do you think it is to include each of
the following activities in making an assessment of
researchers’ performance
• Societal impact of their research
• Editorship of journals and other publications
• Outreach and communication of research to public audiences
• Publication metrics (e.g., Journal Impact Factor, H index)
• Participation in, or delivery of, research integrity training
PRELIMINARY DATA

15. 0
5
10
15
20
25
30
35
40
45
50
Not important at all Somewhat important Fairly important Very important Extremely important
Societal impact of research
Teaching
Peer review
Editorship of journals and other publica
Supervisory responsibilities
Outreach and communication of research t
…how important do you think it is to include each of
the following activities in making an assessment of
researchers’ performance
• Societal impact of their research
• Teaching
• Leadership
• Editorship of journals and other publications
• Supervisory responsibilities
• Outreach and communication of research to public audiences
• Peer review
• Publication metrics (e.g., Journal Impact Factor, H index)
• Collegiality
• Participation in, or delivery of, research integrity training
PRELIMINARY DATA

16. 0
5
10
15
20
25
30
35
40
45
50
Not important at all Somewhat important Fairly important Very important Extremely important
Societal impact of research
Teaching
Peer review
Editorship of journals and other publica
Supervisory responsibilities
Outreach and communication of research t
…how important do you think it is to include each of
the following activities in making an assessment of
researchers’ performance
• Societal impact of their research
• Teaching
• Leadership
• Editorship of journals and other publications
• Supervisory responsibilities
• Outreach and communication of research to public audiences
• Peer review
• Publication metrics (e.g., Journal Impact Factor, H index)
• Collegiality
• Participation in, or delivery of, research integrity training
PRELIMINARY DATA
Not research outputs!
Largely absent in
current assessments

17. Aubert Bonn, N., Pinxten, W. (2021) https://doi.org/10.1186/s41073-020-00104-0 and https://doi.org/10.1186/s41073-020-00105-z
…focus on (publication) outputs
ignore research processes, impacts,
and diverse outputs
…have exceptional results discourage transparency/integrity
…focus on individual achievements discourage collaboration
…be productive (quantity) discourage quality and rigour
…be competitive discourage openness
…follow the typical career path discourage diversity
Problems in current
research environments
1. What makes researchers
SUCCESSFUL?
⦿ Focus groups and interviews with diverse stakeholders
ignore research processes, impacts,
and diverse outputs
discourage transparency/integrity
discourage collaboration
discourage quality and rigour
discourage openness
discourage diversity

18. Inadequate assessments can have a damaging impact on how research is
performed.
Success is almost exclusively measured from research outputs
(publication!), despite what researchers find important.

19. Aubert Bonn, N., & Pinxten, W. (2021). https://doi.org/10.1371/journal.pone.0243664
Collaborating across borders, disciplines, and sectors is…
Having public outreach (e.g., social media, news, etc.) is…
Publishing papers is…
Publishing commentaries or editorials is…
Publishing more papers than others is…
Publishing open access is…
Peer reviewing is…
Replicating past research is…
Publishing findings that did not work (i.e., negative findings) is…
Sharing your full data and detailed methods is…
Reviewing raw data from students and collaborators is…
Conducting research with a high risk of failure is…
Connecting with renowned researchers is…
Getting cited in scientific literature is…
Having your papers read and downloaded is…
Having your results used or implemented in practice is…
Having luck is…
Publishing in high impact journals is…
18
statements
of
potential
indicators
for
success
⦿ Small scale survey
with researchers
126 respondents
Mostly from Flemish
research institutions

20. Aubert Bonn, N., & Pinxten, W. (2021). https://doi.org/10.1371/journal.pone.0243664
…in advancing my career …in advancing science …to my
personal satisfaction

21. Publishing findings that did not work
Conducting innovative
research with high risk of failure
Replicating past research
Sharing your full data
and detailed methods
Peer reviewing
Publishing open access
Reviewing raw data from students
and collaborators
Collaborating across borders, disciplines, and
sectors
Having your results used or implemented in
practice
Publishing more papers than others
Getting cited in scientific literature
Publishing in high impact journals
Connecting with renowned researchers
Having luck
Publishing papers
Having public outreach (e.g.,
social media, news)
Having your papers read and downloaded
Publishing commentaries and editorials
Aubert Bonn, N., & Pinxten, W. (2021). https://doi.org/10.1371/journal.pone.0243664

22. Publishing findings that did not work
Conducting innovative
research with high risk of failure
Replicating past research
Sharing your full data
and detailed methods
Peer reviewing
Publishing open access
Reviewing raw data from students
and collaborators
Collaborating across borders, disciplines,
and sectors
Having your results used or implemented in
practice
Getting cited in scientific literature
Publishing in high impact journals
Connecting with renowned researchers
Having luck
Publishing papers
Having public outreach (e.g.,
social media, news)
Having your papers read and downloaded
Publishing commentaries and editorials
advancing
SCIENCE
Publishing more papers than others
Aubert Bonn, N., & Pinxten, W. (2021). https://doi.org/10.1371/journal.pone.0243664

23. Publishing findings that did not work
Conducting innovative
research with high risk of failure
Replicating past research
Sharing your full data
and detailed methods
Peer reviewing
Publishing open access
Reviewing raw data from students
and collaborators
Collaborating across borders, disciplines,
and sectors
Having your results used or implemented in
practice
Getting cited in scientific literature
Publishing in high impact journals
Connecting with renowned researchers
Having luck
Publishing papers
Having public outreach (e.g.,
social media, news)
Having your papers read and downloaded
Publishing commentaries and editorials
advancing
SCIENCE
Openness
Transparency
Quality
Innovation
Publishing more papers than others
Aubert Bonn, N., & Pinxten, W. (2021). https://doi.org/10.1371/journal.pone.0243664

24. Publishing findings that did not work
Conducting innovative
research with high risk of failure
Replicating past research
Sharing your full data
and detailed methods
Peer reviewing
Publishing open access
Reviewing raw data from students
and collaborators
Collaborating across borders, disciplines,
and sectors
Having your results used or implemented in
practice
Getting cited in scientific literature
Publishing in high impact journals
Connecting with renowned researchers
Having luck
Publishing papers
Having public outreach (e.g.,
social media, news)
Having your papers read and downloaded
Publishing commentaries and editorials
advancing
CAREER
advancing
SCIENCE
Openness
Transparency
Quality
Innovation
Publishing more papers than others
Aubert Bonn, N., & Pinxten, W. (2021). https://doi.org/10.1371/journal.pone.0243664

25. Publishing findings that did not work
Conducting innovative
research with high risk of failure
Replicating past research
Sharing your full data
and detailed methods
Peer reviewing
Publishing open access
Reviewing raw data from students
and collaborators
Collaborating across borders, disciplines,
and sectors
Having your results used or implemented in
practice
Getting cited in scientific literature
Publishing in high impact journals
Connecting with renowned researchers
Having luck
Publishing papers
Having public outreach (e.g.,
social media, news)
Having your papers read and downloaded
Publishing commentaries and editorials
Prestige
Competition
Status
advancing
CAREER
advancing
SCIENCE
Openness
Transparency
Quality
Innovation
Publishing more papers than others
Luck
Aubert Bonn, N., & Pinxten, W. (2021). https://doi.org/10.1371/journal.pone.0243664

26. Indicators used to assess researchers don’t always capture elements that
are thought to advance science
Inadequate assessments can have a damaging impact on how research is
performed.
Success is almost exclusively measured from research outputs
(publication!), despite what researchers find important.

27. Publishing findings that did not work
Conducting innovative
research with high risk of failure
Replicating past research
Sharing your full data
and detailed methods
Peer reviewing
Publishing open access
Reviewing raw data from students
and collaborators
Collaborating across borders, disciplines, and
sectors
Having your results used or implemented in
practice
Publishing more papers than others
Getting cited in scientific literature
Publishing in high impact journals
Connecting with renowned researchers
Having luck
Publishing papers
Having public outreach (e.g.,
social media, news)
Having your papers read and downloaded
Publishing commentaries and editorials
Aubert Bonn, N., & Pinxten, W. (2021). https://doi.org/10.1371/journal.pone.0243664
27
⦿ Small scale survey with researchers
“Even though I have been publishing and speaking about my
failures, I also find it difficult to do so, as I am in a precarious
situation as a woman and early career”
“[T]here is little to no guidance on how to review for early career scholars.
So in that sense, peer reviewing is not that good to my personal
satisfaction. But in an ideal world, I like peer reviewing, I enjoy doing the
work, and it helps me broaden my focus, and I think we should be able to
spend time on it during working hours (and not just on top of everything
else).”
…in advancing my
career
…in advancing science
…to my personal
satisfaction

28. Indicators used to assess researchers don’t always capture elements that
are thought to advance science.
Inadequate assessments can have a damaging impact on how research is
performed.
Assessments shape what matters and what is satisfying to researchers,
thereby shaping practices and cultures.
Success is almost exclusively measured from research outputs
(publication!), despite what researchers find important.

29. Editors/Publishers
Research Funders
Institution Leaders
Lab Technicians
PhD Students
Policy Makers
Post Doc
Ex-Researchers
Faculty researchers
Research Integrity
Offices
⦿ Findings from focus groups and interviews
Aubert Bonn, N., Pinxten, W. (2021) https://doi.org/10.1186/s41073-020-00104-0 and https://doi.org/10.1186/s41073-020-00105-z
Who is responsible?
These problems are systematic and difficult to address.

30. Indicators used to assess researchers don’t always capture elements that
are thought to advance science.
Inadequate assessments can have a damaging impact on how research is
performed.
These problems are systematic and difficult to address.
Assessments shape what matters and what is satisfying to researchers,
thereby shaping practices and cultures.
Are we doomed?
Success is almost exclusively measured from research outputs
(publication!), despite what researchers find important.

31. Indicators used to assess researchers don’t always capture elements that
are thought to advance science.
Inadequate assessments can have a damaging impact on how research is
performed.
These problems are systematic and difficult to address.
Assessments shape what matters and what is satisfying to researchers,
thereby shaping practices and cultures.
Are we doomed? No! There is momentum for change!
Success is almost exclusively measured from research outputs
(publication!), despite what researchers find important.

32. Aubert Bonn, N., & Bouter, L. (2021, July 19). MetaArxiv preprint. https://doi.org/10.31222/osf.io/82rmj
Are we doomed? No! There is momentum for change!

33. How do current Research assessment practices impact ECRs
in Ireland?
Melissa Sharp
Senior Postdoctoral Fellow & Honorary
Lecturer at RCSI University of Medicine and
Health Sciences.
Research: Evidence synthesis & science
communication.
Postdoctoral Representative at RCSI, RCSI
Representative to Irish Research Staff
Association.

34. CoARA and early career researchers
Erzsébet Tóth Czifra
Program Manager of CoARA.
Research: Evidence synthesis & science
communication.
Cultural Linguistics and scholarly
communication.

35. WHAT IS COARA?

36. THE COALITION IN A NUTSHELL
• The Coalition offers a space for its members to learn from others'
experiences, to advance the process of research assessment reform in
Europe and beyond.
• Signatory organisations, having subscribed to the Guiding Principles and
Code of Conduct, are invited to become members of the coalition.
• Coalition members are invited to be involved in Working Groups
and National Communities, as well as other Coalition activities.

37. AGREEMENT ON
REFORMING RESEARCH
ASSESSMENT
https://coara.eu/agreement/

38. 4 CORE COMMITMENTS
1. Recognise the diversity of contributions to, and careers in, research,
in accordance with the needs and the nature of the research.
2. Base research assessment primarily on qualitative evaluation for
which
peer-review is central, supported by responsible use of quantitative
indicators.
3. Abandon the inappropriate uses in research assessment of journal-
and publication-based metrics, in particular the inappropriate uses of
journal impact factor (JIF) and h-index.
4. Avoid the use of rankings of research organisations
in research assessment.

39. WHY YOUR PARTICIPATION IS IMPORTANT
TO COARA?
✔ ECRs are most affected by the reform should be directly involved in the
reform
✔ECRs input is crucial to achieve CoARA’s mission of realising its
commitments towards a more diverse and more equitable assessment and
rewards criteria
✔ CoARA applies to all forms of research assessment (individuals, projects,
organisations, national frameworks...)
✔ It is essential that the control over the reform remains in the hands of those
who are primarily affected by the changes and who are best positioned to
innovate it: research communities themselves.

40. WAYS IN WHICH ECRS CAN ENGAGE WITH CoARA
Raise awareness:
If your organisation is
not yet a CoARA
member, you can
initiate internal
discussions about it.
More information: FAQ -
CoARA
Participate in knowledge
sharing
Join or propose Working
Groups
Or contribute to the Irish
National Chapter
Collect and document
success stories
Participate in the
governance
Experts on
research
assessment can
join the CoARA
Steering Board –
ECR
representation is
important! Ask
Yensi!
Reproduce this
panel
In your own work
environment. You
can find materials
in follow-up.

41. EARLY-AND-MID-CAREER RESEARCHERS (EMCRS) – ASSESSMENT
AND RESEARCH CULTURE WORKING GROUP
“This researcher-driven WG addresses specific needs for and vulnerability of
EMCRs, while recognizing the shift in research culture the reform will bring about.
Both combined, we aim at providing the basis for ensuring a future-ready
research system and culture for the next generation. “
Contact: Sebastian Dahle sebastian.dahle@eurodoc.net

42. The intended and unintended consequences of research
assessment for ECRs
Stefan Müller
Assistant Professor and Ad Astra Fellow in the
School of Politics and International Relations at
University College Dublin
Research: political representation, party
competition, political communication, and public
opinion.
Member of the Young Academy Ireland

43. How can ECRs be involved in the reform (globally and locally),
equally, how can institutions engage with ECRs?
Irene Castellanos
Project Officer in EU programmes and policy at the Health
Research Board (HRB). National Contact Point (NCP) position for
Horizon Europe Health in Ireland and works in open access
Biomedical researcher for over 8 years, working in different
countries and settings including industry and academia.
Chair of the Ireland Chapter of the Marie Curie Alumni
Association (MCAA) and active member of the MCAA policy
working group.

44. Email: noemie.aubertbonn@uhasselt.be
noemie.aubertbonn@manchester.ac.uk
Twitter: @naubertbonn
Drawing from James Graham, used with permission