Francesca Soldati, University of Aberdeen - Research metrics are widely used in recruitment and promotion decisions but reliance on these metrics has led to unintended consequences, including gender disparities within the academic landscape. It is well documented that women in research, particularly those in early to mid-career stages, often face challenges such as lower publication rates and fewer citations than their male counterparts, while remaining underrepresented in senior positions. However, the underlying causes of these discrepancies remain unclear; this project proposes an investigation into the relationship between research culture, gender, and career outcomes, aiming to identify scalable metrics that can better support gender equity in research.

1. GO BEYOND BOUNDARIES
UKSG2025
2nd of April 2025
Francesca Soldati
Open Research Officer
University of Aberdeen
Metrics that Matter:
Identifying, Testing and Validating
Scalable Metrics for a Gender
Equitable Research Culture

2. GO BEYOND BOUNDARIES
How did it start?
Institutional Research Leave Scheme
• Open to Academics, Technician and Professional Services
• Focus on Research Culture

3. GO BEYOND BOUNDARIES
‘We believe the future role of the university is as
an Open Knowledge Institution, a platform for
supporting groups to come together and create
and apply knowledge. Knowledge-making for, and
with society, requires us to rethink the boundaries
of our institutions and to create systems that
support effective communication, diversity and
coordination amongst groups’
www.open.coki.ac/about/

4. GO BEYOND BOUNDARIES
COKI Academic Observatory
• Fetches data about research publications
from multiple sources
Crossref Metadata
Crossref Funder Registry
Crossref Events
OpenAlex
Unpaywall
Research Organization Registry (ROR)
Open Citations
https://open.coki.ac

5. GO BEYOND BOUNDARIES
Why Gender?
• The gender gap in academia is a pressing issue
57%
43%
Postgraduate Students
F M
49%
51%
Total Academic Staff
F M
51%
49%
Other Contract Level
F M
44%
56%
Senior Academic
F M
32%
68%
Professors
F M
Data source: UK Higher Education Statistic Agency – HESA,
Year: 2022/23

6. GO BEYOND BOUNDARIES
Why should we worry?
• The issue goes beyond fairness - it's about creating equal opportunities.
• Gender equality fosters diverse perspectives in research and teaching.
• Greater diversity, in both gender and background, encourages innovation and
creativity by incorporating:
• Different backgrounds
• Different experiences
• Different issues
Research becomes more impactful and relevant to a broader audience

7. GO BEYOND BOUNDARIES
Topic of interest
• Increasing the number of female participants in
academia is not enough to achieve gender equality
(Huang et al., 2020)
• Several explanations/ theories:
• Metrics related
• Systems and Structure related
Number of papers published per year on topics related to
gender equality in academia. Source of data: Web of
Science
0
50
100
150
200
250
300
350
400
450
500
2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024
Publications
Year

8. GO BEYOND BOUNDARIES
Are research metrics part of the issue?
Researcher evaluation
relying on traditional
metrics H-index, counts
of papers and citations,
journal impact factors
(e.g. Cameron et al., 2013)
Bias in research
assessment, hiring
and promotions
(e.g. Fisher et al., 2012)
Limits women’s
success and
reinforces bias
(Cameron et al., 2016)
Increase gender
disparity
(Huang et al., 2020)
Career Interruptions – Traditional metrics do not
account for career breaks (e.g., parental leave),
disproportionately affecting women (Cameron et al.,
2016).
Citation Bias – Studies show that women’s research
tends to receive fewer citations than men’s (Wu, 2024).
Authorship and Collaboration Barriers – Women are
less likely to be in first- or last-author positions, which
are highly valued in academic evaluation (Bruck, 2023).
Peer review barriers– Women may face bias in peer
review and have fewer opportunities to publish in high-
impact journals and be awarded grants (Witteman et al.,
2019).

9. GO BEYOND BOUNDARIES
Project Aim
• To identify metrics that capture positive research outcomes for women and
support strategic intervention.
How?
1. Exploring metrics at a large scale using the COKI Observatory
2. Testing these metrics locally – Athena Swan Data

10. GO BEYOND BOUNDARIES
Challenges
Using sensitive data → DPIA assessment, ethical approval, DMP
Main issue: “The intended use of the data is different to the original purpose
for which it was collected”
Anonymised dataset No possibility of connecting personal
characteristics to publication metrics
Alternative approach: identifying academics currently employed at the university
and assigned gender manually by looking at profile pictures, or pronouns used in
biographies

11. GO BEYOND BOUNDARIES
Exploring Collaboration Diversity
Studies have shown a positive correlation between collaboration and
research outputs
• productivity
• citation impact Crucial for career development
• research quality
(e.g. Abramo et al., 2009)

12. GO BEYOND BOUNDARIES
Exploring Collaboration Diversity
Traditional approach:
• measure the percentages of collaborative
outputs of an author.
• Overlook the difference between diverse
vs. repetitive partnerships.
A researcher with 100% collaborative
outputs may still have low diversity if all
collaborations are with the same institution
Our Approach:
• Measure collaboration diversity—the variety
of institutions and countries researchers
engage with.
• Metric Used: The Gini index, which captures
both the number and diversity of an author’s
co-authors' affiliations and countries (Huang et al.,
2024).
Low diversity High diversity
0 1

13. GO BEYOND BOUNDARIES
Questions and expectations
• Do male and female authors differ in their ability to engage in diverse
collaborations?
• Do female authors tend to have less diverse collaborators than male authors?
• How does the relationship between gender and collaboration diversity
shape academic impact?
• If female authors have lower collaboration diversity, does this lead to fewer citations?
According to the homophily theory (Shen et al., 2022) we anticipated that women might
face greater collaboration barriers in fields which are male-dominated.

14. GO BEYOND BOUNDARIES
Data of 170 UK Universities
Data source: ETER project - The European Higher Education Sector Observatory

15. GO BEYOND BOUNDARIES
Local dataset
Tot academics included in the analysis=881
Tot publications analysed = 24,299
Year: 2015-2024
Data Source: OpenAlex
Limitations:
• Some individuals were excluded for lack
of profile on OpenAlex or because their
profile was not accurate (e.g. merged
with a homonymous author)
• Publications with no DOIs were excluded
from the analysis

16. GO BEYOND BOUNDARIES
Is there a difference between male and female authors in their ability to engage in
diverse collaborations?

17. GO BEYOND BOUNDARIES
Engaging with a broader
range of institutions is more
beneficial for women than for
men in terms of citation
impact.
Collaborating with
different countries enhance
citation impact, and the
benefit is similar for both
genders.
Does the relationship between gender and collaboration diversity
shape academic impact?

18. GO BEYOND BOUNDARIES
1.Our results show similar collaboration performance for female and male
researchers.
→Inclusivity
→Overcoming barriers
2.Diversity of collaboration is beneficial for both genders, but particularly for
women
→ Fostering an inclusive environment can support women in academia
Discussion

19. GO BEYOND BOUNDARIES
Next steps
Digging Deeper into Collaboration Diversity
•Which types of institutions do men and women collaborate with
most?
•What roles do men and women play in collaborations?
•Do collaboration gaps or discontinuities have any impact on
performance?

20. GO BEYOND BOUNDARIES
THANK YOU!
Dr. Francesca Soldati
Open Research Officer
University of Aberdeen
Susan Halfpenny
Head of Research & Learning Information Services
University of Aberdeen
Emma Francis
Open Research Manager
University of Aberdeen
Prof. Ben Tatler, Dean of Research Culture, University of Aberdeen
Dr. Maria Cascio, Athena Swan Coordinator, University of Aberdeen
Prof. Lucy Montgomery
COKI Co-Lead
Curtin University
Prof. Cameron Neylon
COKI co-lead
Curtin University
Dr. Karl Huang
Data scientist and COKI director
Curtin University

21. GO BEYOND BOUNDARIES
References
• Abramo, G., D’Angelo C.A., Di Costa, F. (2009). Research collaboration and productivity: is there a correlation? Higher Education, 57(2): 155–171.
• Brück, O. A (2023) Bibliometric analysis of the gender gap in the authorship of leading medical journals. Commun Med 3, 179. https://doi.org/10.1038/s43856-
023-00417-3
• Cameron, E.Z., Gray, M.E., White, A.M. (2013). Is publication rate an equal opportunity metric? Trends in Ecology and Evolution, 28: 7–8.
https//doi.org/10.1016/j.tree.2012.10.014
• Cameron, E.Z., White, A.M., Gray, M.E. (2016). Solving the Productivity and Impact Puzzle: Do Men Outperform Women, or are Metrics Biased?, BioScience,
66(3): 245–252. https://doi.org/10.1093/biosci/biv173
• Fischer, J., Ritchie, E.G., Hanspach, J. (2012). Academia’s obsession with quantity. Trends in Ecology and Evolution, 27: 473–474.
https//doi.org/10.1016/j.tree.2012.05.010
• Huang, C.K., Neylon, C., Montgomery, L. et al. Open-access research outputs receive more diverse citations. Scientometrics 129: 825–845 (2024).
https://doi.org/10.1007/s11192-023-04894-0
• Huang, J., Gates, A.J., Sinatra, R., Barabási, A. (2020). Historical comparison of gender inequality in scientific careers across countries and disciplines. Proc. Natl. Acad. Sci.
U.S.A. 117 (9): 4609–4616. https://doi.org/10.1073/pnas.1914221117
• Shen, H., Xie, J., Ao, W., Cheng, Y. (2022). The continuity and citation impact of scientific collaboration with different gender composition. Journal of
Informetrics,16(1): 101248. https://doi.org/10.1016/j.joi.2021.101248
• Witteman, H, O., Hendricks, M., Straus, S., Tannenbaum, C. (2019). Are gender gaps due to evaluations of the applicant or the science? A natural experiment at a
national funding agency. Lancet, 393:531-540.
• Wu, C. (2024). The Gender Citation Gap: approaches, explanations and implications. Sociology Compass, e13189. https://doi.org/10.1111/soc4.13189