Successfully reported this slideshow.
Research Assessment<br />David Clay<br />Playing the game of Bibliometrics, DIT<br />20th November 2009<br />
Outline<br />Research Funding in England<br />Research Assessment Exercise (RAE)<br />Research Evaluation Framework<br />I...
RIN (2008) Making sense of research funding in UK higher education. London: RIN. http://www.rin.ac.uk/resources/factsheets...
Research Assessment Exercise 2008	<br />The purpose of the RAE is:<br />“to provide authoritative and comprehensive qualit...
RAE 2008<br />HEIs made submissions to the UOAs in which they specialised<br />All submissions were reviewed by expert pan...
Elements of assessment<br />Research outputs: 4 per person (70%)<br />Research<br />environment (20%) <br />Esteem & impac...
Selection of staff<br />HEFCE (2009) Selection of staff  for inclusion in RAE2008. London: HEFCE. http://www.hefce.ac.uk/p...
RAE 2008 quality criteria for outputs<br />* To knowledge, theory, policy or practice<br />Source: Thorpe, R (2009) Reflec...
RAE 2008 Quality Profile<br />http://www.rae.ac.uk/Submissions/submission.aspx?id=23&type=uoa&subid=1587<br />
From RAE to Research Evaluation Framework (REF)<br />In 2006 DfES proposed moving to a metrics-based system of allocating ...
Outputs (60%)<br />REF Framework__________________________________________________________________________________________...
Impact <br />Rosenberg, G (2009) The REF – taking account of the impact of research. AURIL Conference, 10 October. http://...
Outputs	<br />HEIs will select staff and outputs (3) to submit for assessment<br />The criteria for assessing outputs will...
Use of citation information<br />UOAs for which ‘robust’ data is available will make use of citation information<br />Pane...
REF Outcomes<br />For each UoA submitted the outcomes will be presented as:<br />a sub-profile for each of outputs, impact...
The influence of research assessment<br />“... for the clear majority of researchers, the RAE is the dominant concern when...
RIN (2009) Communicating knowledge: How & why UK researchers publish & disseminate their findings. London: JISC. p.33 http...
RIN (2009) Communicating knowledge: How & why UK researchers publish & disseminate their findings. London: JISC. p.38 http...
Beating the REF<br />Increase author self citation<br />Publish with US authors because they overcite their own papers<br ...
RAE Rules and Institutional Policies<br />Understanding of institutional policies has an important part in determining beh...
Open Access<br />The use of bibliometrics  in research assessment will lead to more researchers making there work availabl...
Conclusions<br />“Researchers are not fools. Whenever targets have changed in the past, academic’s behaviour has adjusted ...
This work is licensed under a Creative Commons Licence.<br />Thank you for listening<br />Any Questions?<br />
Upcoming SlideShare
Loading in …5
×

From RAE to REF

1,050 views

Published on

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

From RAE to REF

  1. 1. Research Assessment<br />David Clay<br />Playing the game of Bibliometrics, DIT<br />20th November 2009<br />
  2. 2. Outline<br />Research Funding in England<br />Research Assessment Exercise (RAE)<br />Research Evaluation Framework<br />Impact of the RAE/REF on research outputs<br />Open access<br />Conclusions<br />
  3. 3. RIN (2008) Making sense of research funding in UK higher education. London: RIN. http://www.rin.ac.uk/resources/factsheets/making-sense-research-funding-uk<br />
  4. 4. Research Assessment Exercise 2008 <br />The purpose of the RAE is:<br />“to provide authoritative and comprehensive quality ratings for research in all disciplines carried out in universities and colleges across the UK, to inform UK higher education funding bodies’ allocation of grant for research.”<br />
  5. 5. RAE 2008<br />HEIs made submissions to the UOAs in which they specialised<br />All submissions were reviewed by expert panels, who made judgements against a pre-defined quality assessment.<br />Each panel issued a statement detailing its criteria and working methods. <br />Each HEI was provided with a quality profile for each of the UOAs to which they submitted<br />
  6. 6. Elements of assessment<br />Research outputs: 4 per person (70%)<br />Research<br />environment (20%) <br />Esteem & impact indicators (10%)<br />Quality profile<br />e.g.<br />Research income<br />PhD students<br />Staff development<br />Weighted and aggregated across each submission<br />Source: Thorpe, R (2009) Reflections on RAE 2008. http://www.ufhrd.co.uk/wordpress/wp-content/.../reflections-on-rae-2008.ppt <br />
  7. 7. Selection of staff<br />HEFCE (2009) Selection of staff for inclusion in RAE2008. London: HEFCE. http://www.hefce.ac.uk/pubs/hefce/2009/09_34/09_34.pdf<br />
  8. 8. RAE 2008 quality criteria for outputs<br />* To knowledge, theory, policy or practice<br />Source: Thorpe, R (2009) Reflections on RAE 2008. http://www.ufhrd.co.uk/wordpress/wp-content/.../reflections-on-rae-2008.ppt <br />
  9. 9. RAE 2008 Quality Profile<br />http://www.rae.ac.uk/Submissions/submission.aspx?id=23&type=uoa&subid=1587<br />
  10. 10. From RAE to Research Evaluation Framework (REF)<br />In 2006 DfES proposed moving to a metrics-based system of allocating research funding<br />HEFCE developed proposals for a system based on:<br />a bibliometric indicator of quality, research income and research student data for STEM subjects<br />light-touch peer review informed by metrics for the other subjects <br />2007-08: consultation on bibliometrics<br />2009: Announced a more unified framework, combining metrics and expert review across all subjects<br />2009/10: consultation on Impact<br />
  11. 11. Outputs (60%)<br />REF Framework____________________________________________________________________________________________________________<br />Impact (25%)<br />Environment (15%)<br />Quality of all types of research<br />Economic, social, cultural and quality of life benefits<br />Quality and sustainability of the research environment<br />Expert review of selected outputs (informed by citation information in appropriate UoAs)<br />Narrative statement and case studies, supported by indicators<br />Narrative supported by indicators<br />Langlands, A (2009) Research Funding and Assessment: The Future. HEPI Conference. 14th October. http://www.hepi.ac.uk/files/Alan%20Langlands%20Research%20Funding%20and%20Assessment.ppt<br />
  12. 12. Impact <br />Rosenberg, G (2009) The REF – taking account of the impact of research. AURIL Conference, 10 October. http://www.auril.org.uk/modules/download_gallery/dlc.php?file=66%20-<br />
  13. 13. Outputs <br />HEIs will select staff and outputs (3) to submit for assessment<br />The criteria for assessing outputs will be ‘originality, rigour and significance’<br />Sub-panels will assess outputs through a process of expert review<br />Some sub-panels will also make use of citation information to inform their assessment of outputs <br />
  14. 14. Use of citation information<br />UOAs for which ‘robust’ data is available will make use of citation information<br />Panels will be provided with citation information about submitted outputs, and with appropriate benchmarks<br />Citation information will be used to inform and supplement the review of outputs<br />Decisions will not be made solely on the basis of citation information<br />All outputs will be treated equally, whether or not there is citation information available for them. <br />
  15. 15. REF Outcomes<br />For each UoA submitted the outcomes will be presented as:<br />a sub-profile for each of outputs, impact and environment to show the proportion of submitted work meeting each level in a five-point scale<br />an overall excellence profile, which combines the three sub-profiles.<br />
  16. 16. The influence of research assessment<br />“... for the clear majority of researchers, the RAE is the dominant concern when it comes to assessing their research.”<br />RIN (2009) Communicating knowledge: How & why UK researchers publish & disseminate their findings. London: JISC. p.33 http://www.jisc.ac.uk/publications/documents/communicatingknowledgereport.aspx#downloads<br />
  17. 17. RIN (2009) Communicating knowledge: How & why UK researchers publish & disseminate their findings. London: JISC. p.33 http://www.jisc.ac.uk/publications/documents/communicatingknowledgereport.aspx#downloads<br />
  18. 18. RIN (2009) Communicating knowledge: How & why UK researchers publish & disseminate their findings. London: JISC. p.38 http://www.jisc.ac.uk/publications/documents/communicatingknowledgereport.aspx#downloads<br />Citation behaviour <br />Major influence on citation are the perceived authority of the publication and author<br />Accessibility is a major influence on what is cited<br />Citation practice related to length of experience<br />Citation practices are clearly infuenced by disciplinary norms and the policies of individual journals<br />Emphasis on bibliometrics is likely to change citation practice. <br />
  19. 19. Beating the REF<br />Increase author self citation<br />Publish with US authors because they overcite their own papers<br />Publish controversial papers<br />Make citation arrangements<br />Cite collaborators work more often<br />
  20. 20. RAE Rules and Institutional Policies<br />Understanding of institutional policies has an important part in determining behaviour<br />Institutions are perceived to have pressured researchers to publish in high impact journals<br />Institutions adopt strategies to maximise their performance in research assessments; this may constrain researchers autonomy<br />
  21. 21. Open Access<br />The use of bibliometrics in research assessment will lead to more researchers making there work available on open access<br />The major research funders in the UK have open access mandates<br />There are now 11 institutional mandates in the UK and at least 5 more are being discussed<br />Institutions are having to building better publication databases in preparation for REF<br />Some using this as an opportunity to populate their repository<br />Others are embedding the repository into the RMS<br />
  22. 22. Conclusions<br />“Researchers are not fools. Whenever targets have changed in the past, academic’s behaviour has adjusted to the target. So I expect publishing behaviour to be changed to align with the requirements of the REF.”<br />
  23. 23. This work is licensed under a Creative Commons Licence.<br />Thank you for listening<br />Any Questions?<br />

×