Recommended
More Related Content
What's hot
What's hot (20)
Similar to CONUL 5 Nov 2020 (Katie Evans) Research Evaluation and Information Literacy
Similar to CONUL 5 Nov 2020 (Katie Evans) Research Evaluation and Information Literacy (20)
More from ldore1
More from ldore1 (20)
Recently uploaded
Recently uploaded (20)
CONUL 5 Nov 2020 (Katie Evans) Research Evaluation and Information Literacy
- 1. Research Evaluation and Information Literacy Katie Evans, Research Analytics Librarian University of Bath, K.T.Evans@bath.ac.uk
- 2. Library Research Services Research Data Open Access Archives & Research Collections Research Analytics Opening up, promoting & preserving the University’s research https://library.bath.ac.uk/research-services
- 3. Research Analytics Service Enhance research profile Monitor & benchmark Publishing strategy External assessment Build partnerships Present strengths
- 4. Research Evaluation & Information Literacy 1. Applying information literacy to research evaluation 2. Changing scholarly communications 3. Applying research evaluation to information literacy 4. Scope for collaboration
- 5. Applying information literacy to research evaluation: research evaluation literacy
- 6. Irises by Van Gough, Digital image courtesy of the Getty’s open content program Research quality and metrics
- 7. Bias What gets counted? • English language • Journal articles What biases in academia show up in the metrics? • Sexism • Racism • Ableism • … What biases are introduced by approaches to evaluation? • Cognitive & systems biases Image from DORA briefing https://sfdora.org/resources/
- 8. Research evaluation literacy Information literacy “The ability to think critically and make balanced judgements about any information we find and use. It empowers us as citizens to reach and express informed views and to engage fully with society.” CILIP ILG 2018 Research evaluation literacy “The ability to think critically and make balanced judgements about the use and interpretation of research indicators and assessment approaches.” Dr Elizabeth Gadd, 2019
- 9. Responsible use of metrics “The problem is that evaluation is now led by the data rather than by judgement” “The research community should develop a more sophisticated and nuanced approach to the contribution and limitations of quantitative indicators”
- 10. University of Bath Principles of Research Assessment & Management 1. Centred on expert judgement 2. Set in the broader environment 3. Supported by reliable data 4. Tailored: one size does not fit all 5. Transparent https://www.bath.ac.uk/corporate-information/principles-of- research-assessment-and-management/
- 11. Changing Scholarly Communications: “research evaluation is the keystone”
- 12. Why change research evaluation? • Bias • Excessive pressure, unhealthy culture • Hinders transition to open access • ‘Serials crisis’ – subscription/publishing costs • Lack of reward for peer review, data curation & sharing, societal impact • Wastes time and duplicates peer review efforts • Reproducibility crisis • Selective publishing, not publishing negative results • Incentivizes sensationalism For researchers For research For scholarly comms
- 13. Applying research evaluation to information literacy: boost your searches
- 14. Visualise key terms Example: VoS Viewer map of key terms from publications found by searching Scopus for ‘Quaker’
- 16. Map the literature using citation networks
- 17. Scope for collaboration • Team up on training • Literature searching • Publishing strategy • Bridge the skills gap • e.g. University of Reading used Bibliometric Competencies as basis for in-house training programme for Liaison Librarians • Decolonising scholarly communications • and more ….
- 18. Research Evaluation and Information Literacy Katie Evans, Research Analytics Librarian University of Bath, K.T.Evans@bath.ac.uk
Editor's Notes
- Scholarly Communications team ‘Inside out’ library service
- Enhance the profile of the university’s research by doing useful & interesting things with data about publishing and citations. These stands all involve evaluating research (or at least trying to!) Both backward looking and forward looking. But what has this got to do with information literacy?
- Research quality is multi-faceted and nuanced Metrics/indicators can never give the full picture Distilling key indicators can be helpful, but there are risks
- It’s not just that we lose nuance by representing research quality with metrics, the way the metrics are calculated and the way they are used can both introduce bias into evaluations. Existing biases in academic are also likely to show up in, & be reinforced by metrics. e.g. Gendered impact of Covid pandemic on publishing – an amplification of already existing biases So, research evaluation is fraught with difficulties. This is were information literacy comes in … https://blogs.lse.ac.uk/impactofsocialsciences/2018/08/07/a-vicious-circle-of-gender-bias-has-meant-differences-between-mens-and-womens-scholarly-productivity-have-not-changed-since-the-1960s/ https://blogs.lse.ac.uk/impactofsocialsciences/2020/10/12/what-we-know-about-the-academic-journal-landscape-reflects-global-inequalities/ https://thebibliomagician.wordpress.com/2020/06/09/the-truth-behind-the-numbers/
- We need to apply information literacy skills of thinking critically and making balanced judgements to research evaluation. At the Liverpool event which inspired today’s event, Lizzie Gadd suggested thinking of this as research evaluation literacy. Lizzie Gadd is the Research Policy Manager (Publications) at Loughborough University. She chairs the Lis-Bibliometrics forum and founded The Bibliomagician blog. She is also the co-Champion of the ARMA Research Evaluation Special Interest Group and chairs the International Network of Research Management Societies (INORMS) Research Evaluation Working Group
- The conversation in this area has gathered under the heading of ‘responsible metrics’. Quotes from: Hicks, D., Wouters, P., de Rijcke, S., & Rafols, I., 2015. The Leiden Manifesto for research metrics. Nature. 520 (7548) 429-431 doi:10.1038/520429a Wilsdon, J., et al. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. DOI: 10.13140/RG.2.1.4929.1363 DORA & University statements also advocate for responsible use of metrics Seeing growing attention & commitment to responsible research assessment, but putting this into practice is challenging
- So far I’ve been looking at this from the point of view of having a need to evaluate research for a particular task, e.g. to benchmark universities or to make a compete for research funding But how we do research evaluation has much wider ramifications for scholarly communications and for research. This brings us to 2nd topic: changing scholarly communications For our generation the major change in scholarly communications is the transition to open access publishing and open scholarship more generally. We’ve been working for this for 10/20 years, there’s broad consensus that it’s desirable, so why is it so hard? The European Commission report on the slide ( https://op.europa.eu/en/publication-detail/-/publication/464477b3-2559-11e9-8d04-01aa75ed71a1 ) is just one example of people asking this question, and the conclusion that they, and others, come to is that research evaluation is the key. We need to change research evaluation, to improve research evaluation literacy, if we’re to achieve the changes we want to see in scholarly comms.
- So, why do we need to change research evaluation? We’ve already seen how bias can be present in evaluations. We need to eliminate this – especially in assessments of individual researchers. I hadn’t really appreciated until I worked in research analytics just how competitive academia is. E.g. Nature editorial compared statistics on changes of getting permanent academic job vs chances of getting a job in professional football - three-quarters of doctoral researchers expect to pursue an academic career … but only 3%-4% of PhD students in UK become permanent members of university staff, Nature, Oct 2017 doi.org/10.1038/550429a In practice this means huge pressure to publish in prestigious, high impact journals ... … which hinders transition to OA. When I worked in OA my conversations with researchers were advocating for OA publishing. When I moved to research analytics, those conversations changed to be about publishing strategically – which often means targeting prestigious journals. This makes academic publishing conservation and puts a lot of power in the publishers hands. (Hence serials crisis) Narrow focus on publishing in high impact/prestigious journals because this is what’s rewarded in research evaluations means that peer review, data sharing, public engagement – which are all desirable for scholarly comms and for research – get relegated down the priority list. Replication studies and negative results are unlikely to be published in high impact journals, contributing to reproducibility crisis and gaps in research record. Current system can incentivize sensationalism over rigour. Conclusion: Research evaluation literacy is fundamental not just to the fair assessment of research, but also to creating the research environment and scholarly communications practices that we want.
- Example: using research analytics tool VoS Viewer to extract and map the key terms in a group of publications in order to hone your search strategy. Start with group of publication, extract key terms/phrase. Each bubble is a term. The larger the bubble the more publications the term appears in. Bubbles positioned to that terms that often co-occur in publications are closer together. Clustering (colours) so we can see different groups within the publication set.
- Quakers are a religious group Blue cluster is around theology & worship Red cluster is around Quaker history But relevance of the green cluster isn’t as clear, looking at the edge of it we find ‘Quaker oats’ and ‘Myiopsitta Mochasu’ the Quaker parrot – neither are related to the religious group. Can then refine search terms appropriated (e.g. exclude oats!)
- Suppose it is actually the Quaker parrots we’re interested in. Key term searching is hit and miss. Another approach is to use citation relationships between publications to identify publications with a common theme. Elsevier’s ‘Topics’ analysis does this. We can then see, e.g. authors contributing to a Topic and journals frequently publishing items in the Topic – both useful for literature searching and keeping up to date
- Scope for collaboration between information literacy specialists & research evaluation/scholarly communication specialists. Example – joint workshop for researchers, looking forward to Michelle’s presentation on this Research analytics is a relatively new area for academic libraries, there’s a skills gap. Particularly hard for colleagues whose role overlaps with this area occasionally, but it’s not their main focus so they can’t devote a lot of time to this. The Lis-Bibliometrics group developed Bibliometrics Competencies to set out competencies needed. Dr Karen Rowlett at University of Reading used this as basis for survey and then custom training programme. Decolonising – I’ve talked about how bias gets in to research evaluation, particularly with regards to what gets counted (mostly English language journal publishing). Information literacy specialists involved in decolonising the curriculum, the library & the reading list. We also need to decolonise the data sources used for research evaluation.