BIBLIOMETRICS AND THE REF: WHAT DO RESEARCHERS WANT  OR NEED FOR THE FUTURE? <ul><li>Tim Wales, Associate Director (E-Stra...
BRIEF BIO <ul><li>Pro A&I </li></ul>
Source: http://www.flickr.com/photos/25552808@N04/2847690523/ University College London
King’s College London (former Pharmacy Library) Source: http:// http://www.flickr.com/photos/albedo/541536327
Source: http://www.flickr.com/photos/25552808@N04/2847690523/ City University
London Business School Source: http://www.flickr.com/photos/ashishaggarwal/2983190165/
http://www.flickr.com/photos/dannybull/4017080101/ Open University
Trinity College, Bridgeford University Source: http:// www.flickr.com/photos/bathintime/3726752408/ Royal Holloway, Univer...
REF (BIBLIOMETRIC) UPDATE
Source: http://www.flickr.com/photos/treacle/66010614/
Red card to bibliometrics? <ul><li>HEFCE Pilot exercise 2008-09 concluded: </li></ul><ul><li>“ ...citation information is ...
More of a yellow card... <ul><li>Details of citation data use </li></ul><ul><li>Sub-panels decide in advance </li></ul><ul...
...with final warning(s) <ul><li>Clear guidelines on using the data robustly to take account of the known limitations and ...
REF BIBLIOMETRICS PILOT
Methodology <ul><li>Thomson Reuters Web of Science and Elsevier Scopus citation data used </li></ul><ul><li>22 HEIs took p...
Bibliometric models <ul><li>Publication data by institutional address </li></ul><ul><li>Publication data by staff, all out...
Model summary <ul><li>Institutional address </li></ul><ul><ul><li>Papers assigned to HEIs on basis of author address in Wo...
Data gathering issues <ul><li>There were difficulties for some in drawing together data from disparate sources, some of wh...
UoA publishing variations Tend to publish in conference proceedings? Source: HEFCE (2009a), p.154
Coverage variations Source: HEFCE (2009a), p.158 (Education 85% not indexed)
WHAT ABOUT RESEARCHERS?
 
Journals dominate Source: Communicating knowledge, p.17
Monographs less so Source: Communicating knowledge, p.18
RAE distorts behaviour Source: Communicating knowledge, p.34
UK researchers need TLC <ul><li>Many researchers are confused by the mixed messages they are receiving as to how best to c...
RESEARCHERS NEED E-A&I <ul><li>ISB research </li></ul>
Power browsers Source: Tennopir & King, 2008 Printed TOCs, indices etc c.75%
Tennopir & King ISB summary <ul><li>US scientists increased number of readings (from libraries) via searching & citations ...
 
Digital ISBs <ul><li>Horizontal information seeking </li></ul><ul><li>Viewers  </li></ul><ul><li>Power browsers </li></ul>...
Nicholas et al. study Source: Nicholas et al., 2007
OK, WHAT’S MY POINT?
Given that... <ul><li>The HEFCE Bibliometric pilot found that HEIs had major difficulties in sourcing and providing accura...
It follows that... <ul><li>HEIs and researchers need a REF compliant system that: </li></ul><ul><li>contains all of an HEI...
RESEARCH INFORMATION SYSTEMS (RIS) <ul><li>The answer? </li></ul>
Source: http://www.flickr.com/photos/roadsidepictures/244926428/
[email_address] Web of Science PubMed etc Research Managers Institutional Repository Importing of bibliographic data Resea...
RIS products... Assisted HEFCE REF Pilot PURE CONVERIS (Develop in-house solutions such as at University College Dublin)
 
 
 
Standard RIS A&I data ingest <ul><li>PURE </li></ul><ul><li>ISI Web of Science * </li></ul><ul><li>Scopus * </li></ul><ul>...
EVENTS DEAR BOY, EVENTS <ul><li>(Harold Macmillan 1894-1986) </li></ul>
2009 Merger <ul><li>Royal Holloway </li></ul>St George’s  Medical School Merge..err?
PURE V3 SCREENSHOTS <ul><li>RIS example </li></ul>
Researcher’s view
Import A&I data
Author matching
Full-text/OA archiving
Bibliometric reporting
Citation analysis
Conclusion – Long live A&I <ul><li>A&I services have a new and vital role to play in HEI REF/general research administrati...
<ul><li>“ ...one of the publishers joked that given the evidence showing the key role of abstracts in today’s crowded info...
References & Thanks <ul><li>CIBER (2008) Information behaviour of the researcher of the future. Available from:  http://ti...
http://www.flickr.com/photos/nickbush/450151862/
Upcoming SlideShare
Loading in …5
×

JIBS 2009 Bibliometrics And The REF 2009-11-13

1,190 views
1,114 views

Published on

Presentation to JIBS User Group A&I workshop, Manchester, UK. 13th November 2009

Published in: Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,190
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

JIBS 2009 Bibliometrics And The REF 2009-11-13

  1. 1. BIBLIOMETRICS AND THE REF: WHAT DO RESEARCHERS WANT OR NEED FOR THE FUTURE? <ul><li>Tim Wales, Associate Director (E-Strategy) </li></ul><ul><li>Royal Holloway, University of London </li></ul>13 th November 2009
  2. 2. BRIEF BIO <ul><li>Pro A&I </li></ul>
  3. 3. Source: http://www.flickr.com/photos/25552808@N04/2847690523/ University College London
  4. 4. King’s College London (former Pharmacy Library) Source: http:// http://www.flickr.com/photos/albedo/541536327
  5. 5. Source: http://www.flickr.com/photos/25552808@N04/2847690523/ City University
  6. 6. London Business School Source: http://www.flickr.com/photos/ashishaggarwal/2983190165/
  7. 7. http://www.flickr.com/photos/dannybull/4017080101/ Open University
  8. 8. Trinity College, Bridgeford University Source: http:// www.flickr.com/photos/bathintime/3726752408/ Royal Holloway, University of London
  9. 9. REF (BIBLIOMETRIC) UPDATE
  10. 10. Source: http://www.flickr.com/photos/treacle/66010614/
  11. 11. Red card to bibliometrics? <ul><li>HEFCE Pilot exercise 2008-09 concluded: </li></ul><ul><li>“ ...citation information is not sufficiently robust to be used formulaically or as a primary indicator of quality, but there is consider scope for it to inform and enhance the process of peer review.” </li></ul>HEFCE Pilot exercise 2008-09 concluded: “ ...citation information is not sufficiently robust to be used formulaically or as a primary indicator of quality , but there is consider scope for it to inform and enhance the process of peer review.” Source: HEFCE (2009b), p.3
  12. 12. More of a yellow card... <ul><li>Details of citation data use </li></ul><ul><li>Sub-panels decide in advance </li></ul><ul><li># of citations for each output provided </li></ul><ul><li>With “appropriate benchmarks” </li></ul><ul><li>To “inform and supplement” review </li></ul><ul><li>“ Robust” citation data UoAs </li></ul><ul><li>Medicine </li></ul><ul><li>Science </li></ul><ul><li>Engineering </li></ul><ul><li>Arts </li></ul><ul><li>Humanities </li></ul><ul><li>“ others” </li></ul>Source: HEFCE (2009b), p.3
  13. 13. ...with final warning(s) <ul><li>Clear guidelines on using the data robustly to take account of the known limitations and to avoid bias </li></ul><ul><li>Panels will not make judgements about quality of outputs solely on the basis of citation information; expert information must be applied </li></ul><ul><li>All submitted outputs will be treated equally, whether or not citation information available </li></ul>Source: HEFCE (2009b), p.3 <ul><li>Clear guidelines on using the data robustly to take account of the known limitations and to avoid bias </li></ul><ul><li>Panels will not make judgements about quality of outputs solely on the basis of citation information; expert information must be applied </li></ul><ul><li>All submitted outputs will be treated equally , whether or not citation information available </li></ul>
  14. 14. REF BIBLIOMETRICS PILOT
  15. 15. Methodology <ul><li>Thomson Reuters Web of Science and Elsevier Scopus citation data used </li></ul><ul><li>22 HEIs took part </li></ul><ul><li>35 UoAs from 2008 RAE used </li></ul><ul><ul><li>(Coverage greater than 40% in citation databases) </li></ul></ul><ul><li>Publications between 2001 and 2006 inclusive + citation data to 2007 </li></ul><ul><li>3 data gathering models tested </li></ul><ul><li>Journal articles + review papers only </li></ul>
  16. 16. Bibliometric models <ul><li>Publication data by institutional address </li></ul><ul><li>Publication data by staff, all outputs </li></ul><ul><li>Publication data by staff, selected outputs </li></ul>
  17. 17. Model summary <ul><li>Institutional address </li></ul><ul><ul><li>Papers assigned to HEIs on basis of author address in WoS/Scopus </li></ul></ul><ul><ul><li>Also assigned to subject categories based on destination journal </li></ul></ul><ul><li>Staff, all outputs </li></ul><ul><ul><li>All papers by staff submitted to RAE2008 for the pilot UoAs </li></ul></ul><ul><li>Staff, selected outputs </li></ul><ul><ul><li>6 most highly cited outputs of staff in 2. </li></ul></ul> Too ‘burdensome’  Not individual/UoA specific 4 
  18. 18. Data gathering issues <ul><li>There were difficulties for some in drawing together data from disparate sources, some of which were not held electronically. </li></ul><ul><li>Many institutions also found it difficult to comprehensively identify all published research produced by their staff (especially former staff). </li></ul><ul><li>...further issues arose due to the need to disambiguate papers written by different people with the same name (especially for larger institutions) </li></ul>
  19. 19. UoA publishing variations Tend to publish in conference proceedings? Source: HEFCE (2009a), p.154
  20. 20. Coverage variations Source: HEFCE (2009a), p.158 (Education 85% not indexed)
  21. 21. WHAT ABOUT RESEARCHERS?
  22. 23. Journals dominate Source: Communicating knowledge, p.17
  23. 24. Monographs less so Source: Communicating knowledge, p.18
  24. 25. RAE distorts behaviour Source: Communicating knowledge, p.34
  25. 26. UK researchers need TLC <ul><li>Many researchers are confused by the mixed messages they are receiving as to how best to communicate their findings.... </li></ul><ul><li>Funders, learned societies and publishers may also wish to consider whether they might take more of a lead in helping to devise guidelines on good practice [in attribtion & listing of authors] </li></ul><ul><li>Research timescales need to be carefully considered in any arrangements for the assessment of performance. </li></ul>Source: Communicating knowledge, pp.6-7
  26. 27. RESEARCHERS NEED E-A&I <ul><li>ISB research </li></ul>
  27. 28. Power browsers Source: Tennopir & King, 2008 Printed TOCs, indices etc c.75%
  28. 29. Tennopir & King ISB summary <ul><li>US scientists increased number of readings (from libraries) via searching & citations </li></ul><ul><li>Broader range of readings thanks to Library subscriptions to online collections </li></ul><ul><li>80% of articles >10 years found via online searching or citation (links) </li></ul><ul><li>US scientists read many articles for every one that they cite </li></ul><ul><li>Choosing the best article to cite may be subject to peer pressure in the form of choosing more often to cite those that are cited by others. </li></ul><ul><li>Following citation links in e-journal articles may have proportionately more influence on citation behavior than reading behavior </li></ul>(3 decade analysis)
  29. 31. Digital ISBs <ul><li>Horizontal information seeking </li></ul><ul><li>Viewers </li></ul><ul><li>Power browsers </li></ul><ul><li>“ 60% of e-journal users view no more than 3 pages” </li></ul><ul><li>“ Average time on e-book or e-journal sites: 4-8 mins” </li></ul><ul><li>“ Power browse horizontally through titles, contents pages and abstracts going for quick wins” </li></ul>horizontal bouncing checking viewing promiscuous diverse volatile Source: CIBER, 2008
  30. 32. Nicholas et al. study Source: Nicholas et al., 2007
  31. 33. OK, WHAT’S MY POINT?
  32. 34. Given that... <ul><li>The HEFCE Bibliometric pilot found that HEIs had major difficulties in sourcing and providing accurate publication data efficiently </li></ul><ul><li>There is still a limited need for citation data for REF </li></ul><ul><li>RAE submission trends and external ISB research indicate an increase in journal article publishing and use across most disciplines </li></ul><ul><li>Today’s and tomorrow’s researchers value A&I data from libraries to aid power browsing and identifying articles to cite </li></ul>
  33. 35. It follows that... <ul><li>HEIs and researchers need a REF compliant system that: </li></ul><ul><li>contains all of an HEI’s publication data for current and former staff </li></ul><ul><li>Provides citation and benchmarking data for those UoAs that will be using it in the REF </li></ul><ul><li>Harnesses researchers use of Library A&I databases to good effect </li></ul><ul><li>Interfaces with existing bibliographic and web systems, e.g. OA repository to help promote individual outputs and raise profiles </li></ul>
  34. 36. RESEARCH INFORMATION SYSTEMS (RIS) <ul><li>The answer? </li></ul>
  35. 37. Source: http://www.flickr.com/photos/roadsidepictures/244926428/
  36. 38. [email_address] Web of Science PubMed etc Research Managers Institutional Repository Importing of bibliographic data Researchers Validation of bibliographic data Submission of research artefacts Depositing research artefacts Research Management System Library staff Querying of research output data HEFCE HR ID Mgt CMS Querying RHUL Systems Submission of bibliographic data WWW Updating researchers’ websites Grants REF
  37. 39. RIS products... Assisted HEFCE REF Pilot PURE CONVERIS (Develop in-house solutions such as at University College Dublin)
  38. 43. Standard RIS A&I data ingest <ul><li>PURE </li></ul><ul><li>ISI Web of Science * </li></ul><ul><li>Scopus * </li></ul><ul><li>PubMed </li></ul><ul><li>ArXiv.org </li></ul><ul><li>DBLP </li></ul><ul><li>Google Books </li></ul><ul><li>Google Scholar </li></ul><ul><li>British Library </li></ul><ul><li>OCLC WorldCat * </li></ul><ul><li>ZETOC * </li></ul><ul><li>Symplectic </li></ul><ul><li>ISI Web of Science * </li></ul><ul><li>Scopus * </li></ul><ul><li>PubMed </li></ul><ul><li>ArXiv.org </li></ul><ul><li>DBLP </li></ul><ul><li>Google Books </li></ul><ul><li>Google Scholar </li></ul><ul><li>British Library </li></ul><ul><li>OCLC WorldCat * </li></ul><ul><li>ZETOC * </li></ul>* = Institutional subscription required
  39. 44. EVENTS DEAR BOY, EVENTS <ul><li>(Harold Macmillan 1894-1986) </li></ul>
  40. 45. 2009 Merger <ul><li>Royal Holloway </li></ul>St George’s Medical School Merge..err?
  41. 46. PURE V3 SCREENSHOTS <ul><li>RIS example </li></ul>
  42. 47. Researcher’s view
  43. 48. Import A&I data
  44. 49. Author matching
  45. 50. Full-text/OA archiving
  46. 51. Bibliometric reporting
  47. 52. Citation analysis
  48. 53. Conclusion – Long live A&I <ul><li>A&I services have a new and vital role to play in HEI REF/general research administration thanks to the new market for RIS </li></ul><ul><ul><li>Reduce manual ingest of bibliographic data </li></ul></ul><ul><ul><li>Sometimes are the only means of identifying past HEI research </li></ul></ul><ul><ul><li>Facilitate output benchmarking thanks to controlled vocabularies and added-value metadata </li></ul></ul>
  49. 54. <ul><li>“ ...one of the publishers joked that given the evidence showing the key role of abstracts in today’s crowded information environment, maybe they should reverse their business model and give full-text away and charge for abstracts. They were only half joking.” </li></ul>Final thought Source: Nicholas et al. (2007) A&I research
  50. 55. References & Thanks <ul><li>CIBER (2008) Information behaviour of the researcher of the future. Available from: http://tinyurl.com/c377rk </li></ul><ul><li>HEFCE (2009a) Report on the pilot exercise to develop bibliometric indicators for the Research Excellence Framework. Available from: http://tinyurl.com/yhru5so </li></ul><ul><li>HEFCE (2009b) The Research Excellence Framework: A brief guide to the proposals. Available from: http://tinyurl.com/yljnw44 </li></ul><ul><li>Nicholas, D., Huntingdon, P. & Jamali, H.R. (2007) The Use, Users, and Role of Abstracts in the Digital Scholarly Environment. Journal of Academic Librarianship Vol.33(4),pp.446-453. </li></ul><ul><li>RIN (2009) Communicating knowledge: how and why researchers publish their findings. Available from: http://tinyurl.com/ydumxyt </li></ul><ul><li>Tennopir, C. & King, D. (2008) Electronic Journals and Changes in Scholarly Article Seeking and Reading Patterns. D-Lib Magazine , Vol.14(11/12). Available from: http://tinyurl.com/5sd3td </li></ul><ul><li>Thanks to Adrian Joyce (Business Analyst, RHUL) for RIS material </li></ul><ul><li>PURE screenshots (c) copyright Atira A/S 2009 </li></ul>[email_address] @timwales
  51. 56. http://www.flickr.com/photos/nickbush/450151862/

×