Russell Group PVCs 3 Jun 09


Published on

Presentation by Michael Jubb to Russell Group Pro-Vice Chancellors, June 2009

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Russell Group PVCs 3 Jun 09

  1. 1. Research, Publication, Management …….and the REF Michael Jubb Research Information Network Russell Group PVCs (Research) Working Group 3 June 2009
  2. 2. What are we trying to measure/assess? <ul><li>productivity </li></ul><ul><ul><li>individual, institutional and national volumes and shares of outputs </li></ul></ul><ul><li>research impact </li></ul><ul><ul><li>citations </li></ul></ul><ul><ul><li>readership and usage </li></ul></ul><ul><li>networks </li></ul><ul><ul><li>who is reading, citing, linking to whom </li></ul></ul><ul><li>socio-economic impact </li></ul><ul><ul><li>but what precisely and over what time-frame? </li></ul></ul>
  3. 3. Researchers’ publication and dissemination behaviour <ul><li>study commissioned by RIN </li></ul><ul><li>broad aim to gather and analyse evidence about: </li></ul><ul><ul><li>the motivations, incentives and constraints that lead researchers in different subjects and disciplines to publish and disseminate their work in different ways and at different times; </li></ul></ul><ul><ul><li>how and why researchers cite other researchers’ work; </li></ul></ul><ul><ul><li>how researchers’ decisions on publication and citation are influenced (or not) by considerations arising from research assessment. </li></ul></ul>
  4. 4. What kinds of outputs? <ul><li>journals dominant across all disciplines (ex arts) </li></ul><ul><li>diverse range of other outputs </li></ul><ul><ul><li>patents, instrument building, databases, web-based resources, working papers……… </li></ul></ul>sciences, 0-1% soc sci, arts & humanities, 0-1% ? reviews, reports……… sciences, 0% soc sci, arts & humanities, 0-35% x exhibitions, performances etc sciences, 0-14% soc sci, arts & humanities, 0-5% ? conference papers sciences, 0-3% soc sci, arts & humanities 22-88% ? books, chapters sciences, 79-99% soc sci, arts & humanities, 22-88%  journal articles RAE 2008 %
  5. 5. Some measures of productivity Values shown for UK percentage presence in different reports
  6. 6. Why the differences? <ul><li>data sources </li></ul><ul><ul><li>WoS </li></ul></ul><ul><ul><ul><li>SCI, SSCI, AHCI </li></ul></ul></ul><ul><ul><ul><li>database version </li></ul></ul></ul><ul><ul><ul><li>include letters and conference proceedings? </li></ul></ul></ul><ul><ul><li>SCOPUS </li></ul></ul><ul><li>year to be counted </li></ul><ul><ul><li>year published in print or online? </li></ul></ul><ul><li>counting method </li></ul><ul><ul><li>integer counting </li></ul></ul><ul><ul><li>fractional counting </li></ul></ul>
  7. 7. So what are researchers telling us? where, when and how to publish <ul><li>key motivation is recognition by peers </li></ul><ul><ul><li>peer review critically important </li></ul></ul><ul><ul><li>recognition measured by citation </li></ul></ul><ul><ul><li>career advancement </li></ul></ul><ul><li>secondary motivation is maximising dissemination </li></ul><ul><ul><li>tension between targeting best audience and highest quality journal </li></ul></ul><ul><li>increasing collaboration more co-authorship </li></ul><ul><ul><li>significant rise in proportion of multi-authored works between 2003 and 2008 </li></ul></ul><ul><li>research assessment affects choices </li></ul><ul><li>signs of increase in productivity </li></ul><ul><ul><li>small rise in no. of articles per author 2003-2008 </li></ul></ul>
  8. 8. Disciplinary differences? <ul><li>books/book chapters equal in importance to journals in humanities </li></ul><ul><li>conference papers important in engineering </li></ul><ul><li>concerns about practice-led outputs </li></ul><ul><ul><li>creative and performing arts </li></ul></ul><ul><ul><li>applied disciplines such as (aspects of) psychology, nursing and midwifery </li></ul></ul>
  9. 9. So what are researchers telling us? citation behaviour (citations out) <ul><li>some evidence of increase in volume of citations </li></ul><ul><li>varying reasons for citing others/types of citation </li></ul><ul><li>associated with types of output </li></ul><ul><ul><li>articles, books, conference papers etc </li></ul></ul><ul><li>motivated by </li></ul><ul><ul><li>authority of cited material (64%) or of author (44%) </li></ul></ul><ul><ul><li>requirement to reference a method/theory/argument (53%) </li></ul></ul><ul><ul><li>guidance from others: mainly reviewers and editors (29%) </li></ul></ul><ul><li>self citation of multi-authored papers </li></ul><ul><li>disciplinary differences </li></ul><ul><ul><li>medical sciences tend not to cite conference papers </li></ul></ul><ul><ul><li>humanities cite personal communications and anecdotal refs </li></ul></ul>
  10. 10. Some citation measures (citations in) <ul><li>no. of citations </li></ul><ul><ul><li>including and excluding self-citations </li></ul></ul><ul><li>average no. of citations per publication </li></ul><ul><li>% of publications not cited </li></ul><ul><li>normalised worldwide average no. of citations per publication for a specific field </li></ul><ul><ul><li>percentile breakdowns </li></ul></ul><ul><li>comparisons of individuals, groups, departments, institutions etc against normalised worldwide average for a specific field </li></ul><ul><li>quality profiles and percentile breakdowns </li></ul>
  11. 11. Some normalised worldwide averages 0.55 0.51 history 1.12 1.06 sociology 2.25 2.06 psychology 1.34 1.13 economics 1.11 1.21 energy 0.71 0.82 civil engineering 0.88 0.74 maths 3.18 2.52 chemistry 5.25 4.23 astronomy 4.83 4.02 biomedical sciences 3.47 2.69 biological sciences 2.51 2.24 agriculture and food UK average (without self-citations) worldwide average 2003-06
  12. 12. Some problems: coverage of the literature Internal WoS coverage Soc Sci in Medicine Physics & Astronomy Geosciences Clinical Medicine Engineering Psychology & Psychiatry Chemistry Humanities & Arts Economics Bio Science – animals & plants Bio Science – humans Other Soc Sci Mathematics App Physics & Chemistry Molecular Biology & Biochemistry <40% 40-60% 60-80% 80-100%
  13. 13. Some problems: timing and citation half-life <ul><li>Time-lags and skewed distributions </li></ul><ul><li>Disciplinary, and sub-disciplinary, differences </li></ul>
  14. 14. Some problems: multiple authorships
  15. 15. Other measures? the Hirsch index From: Lutz Bornmann (2006) H Index
  16. 16. Other measures? <ul><li>network and page rank analysis </li></ul><ul><ul><li>weighted transfers of prestige from one journal/researcher/institution to another </li></ul></ul><ul><ul><li>eigenfactor, SCImago journal ranking </li></ul></ul><ul><li>usage measures </li></ul><ul><ul><li>COUNTER metrics </li></ul></ul><ul><ul><li>network analysis </li></ul></ul>
  17. 17. Do they tell the same story?
  18. 18. Do they tell the same story? Earth and planetary sciences in selected institutions
  19. 19. So what are researchers telling us? <ul><li>Has dissemination behaviour been influenced by RAE? Will it be influenced by REF? </li></ul><ul><ul><li>across all disciplines, Yes </li></ul></ul><ul><ul><li>senior academics less affected, early-to-mid career academics more so </li></ul></ul><ul><ul><li>focus on publication </li></ul></ul><ul><ul><li>institutional strategies </li></ul></ul><ul><li>quarter of all researchers think the RAE excluded important research outputs </li></ul><ul><li>if REF based on citations, they will employ open access publishing more often (42%) </li></ul>
  20. 20. So what are researchers telling us? <ul><li>Has citation behaviour been influenced by RAE? </li></ul><ul><ul><li>across all disciplines (ex physical sciences), No or Not Sure </li></ul></ul><ul><li>Will it be influenced by REF? </li></ul><ul><ul><li>across all disciplines (ex economics), Yes, or Might </li></ul></ul><ul><ul><li>likely to cite collaborators’ work more often (38%) and competitors’ work less (13%) </li></ul></ul>
  21. 21. Issues for REF <ul><li>coverage of different kinds of outputs </li></ul><ul><ul><li>disciplinary differences </li></ul></ul><ul><li>accuracy of data </li></ul><ul><ul><li>original citation </li></ul></ul><ul><ul><li>publication databases (publishers’ and institutions’) </li></ul></ul><ul><li>definition of fields </li></ul><ul><ul><li>interdisciplinarity </li></ul></ul><ul><li>costs………. </li></ul><ul><li>selectivity or not? </li></ul><ul><li>who owns the publication or citation? </li></ul><ul><ul><li>institution </li></ul></ul><ul><ul><li>individual </li></ul></ul><ul><li>bottom up or top down analysis? </li></ul><ul><li>different metrics give different results </li></ul>
  22. 22. Relationship between metrics and peer review <ul><li>“ The future of research assessment exercises lies in the intelligent combination of metrics and peer review” </li></ul><ul><li>Henk Moed, CWTS, Leiden University </li></ul><ul><li>peer review informed by bibliometrics, </li></ul><ul><li>or </li></ul><ul><li>bibliometrics moderated by peer review? </li></ul><ul><li>possibilities </li></ul><ul><ul><li>let the type of peer review depend on the outcomes of the bibliometrics </li></ul></ul><ul><ul><li>use citation analysis for initial rankings and explicitly justify any subsequent deviations from them </li></ul></ul>
  23. 23. Lessons for institutions? <ul><li>even simple bibliometrics are not simple </li></ul><ul><ul><li>and they are rapidly becoming more complex and sophisticated </li></ul></ul><ul><ul><li>need for bibliometric expertise to understand and be able to employ a range of measures </li></ul></ul><ul><ul><li>local, central or commercial services? </li></ul></ul><ul><li>researchers’ motivations and behaviours are complex </li></ul><ul><ul><li>need for assessments by others is implicit in all their motivations </li></ul></ul><ul><ul><li>rewards come from assessments; RAE/REF part of a wider ecology </li></ul></ul><ul><ul><li>disciplinary differences are real </li></ul></ul><ul><ul><li>institutional policies and strategies must take account of them </li></ul></ul>
  24. 24. Lessons for institutions? <ul><li>staff awareness and consultation </li></ul><ul><li>lessons from pilots </li></ul><ul><ul><li>comprehensive research information systems </li></ul></ul><ul><ul><li>publications databases </li></ul></ul><ul><ul><ul><li>not necessarily the same as the repository </li></ul></ul></ul><ul><ul><li>accurate bibliographic data </li></ul></ul><ul><li>author ID systems? </li></ul><ul><li>other lessons once current study completed? </li></ul>
  25. 25. <ul><li>Questions??? </li></ul><ul><li>Michael Jubb </li></ul><ul><li> </li></ul>