A proposal and initiative to develop metrics for impact of research on professional practice, especially with regards to clinical guidelines.

1. Professional impact
of clinical research
Gustaf Nelhans
Swedish School of Library and Information Science
STI 2016, Valencia 2016-09-15
Research sponsored by MINSO Solutions, Borås, Sweden

2. Professional impact
- Academic impact
- Traditional journal based citation analysis measure “inter-academic impact”
(disciplinary impact).
- Professional impact:
- Publications that have a direct impact on professional practice (e.g. clinical
guidelines, patents, standards)
- Social impact (altmetrics):
- E.g. ResearchGate, Academia.edu, Twitter mentions (–relevance for practice?;
academia?)
• This presentation focuses on:
“Professional practice with regards to
clinical practice in the health sector”
Gustaf Nelhans, Swedish School of Library and Information
Science, University of Borås, gustaf.nelhans@hb.se 2

3. Agreement on Medical Education and
Research (“ALF-medel”)
• REF based model (SRC 2013)
• Panels evaluating:
• Scientific quality: 60 %
• (e.g. bibliometric data, external funding)
• Clinical importance: 30 %
• (e.g. self evaluations with case studies)
• Conditions for research 10 %
• (e.g.governance, personel with academic degree etc)
Gustaf Nelhans, Swedish School of Library and Information
Science, University of Borås, gustaf.nelhans@hb.se 3

4. Active research on clinical guideline
references
• [c]ounting of citations in the serial literature […] may measure academic
influence, but the funders of such research are usually more concerned
to see if it has had a practical benefit, especially to patients. (Lewison &
Sullivan, 2008).
• [n]ational guidelines, those based on systematic reviews and those using
explicit evidence-linked methods for deriving recommendations and
those being regularly updated and featuring implementation strategies
were more likely to be of high quality, than their counterparts. (Andersen,
2013)
• [c]itations from clinical guidelines [...] give evidence that research findings
have been used to inform the day-to-day practice of medical staff
(Thelwall & Maflahi, 2016)
Gustaf Nelhans, Swedish School of Library and Information
Science, University of Borås, gustaf.nelhans@hb.se 4

5. Citation database for clinical guidelines
Fig. borrowed from Hans Eriksson, Minso ABGustaf Nelhans, Swedish School of Library and Information
Science, University of Borås, gustaf.nelhans@hb.se 5

6. Database and empirical data
Sources No.
guidelines
No of refs
(with PubMed-id)
SBU 343 27.570
National board of
health and welfare
(Socialstyrelsen)
49 8.002
- Clin guidelines 13 6.610
• Presently >50 000 references.
Of these around 3/4 (72,9 %)
are matched to a PubMed-id.
Clinical guidelines TYPE Year Refs
Strokesjukvård 2009 CARDIO 2009 645
Vuxentandvård 2011 DENTA 2011 1301
Diabetesvård DIABE 2015 523
Sjukdomsförebyggande metoder GENER 2011 316
Antipsykotisk läkemedelsbehandling vid schizofreni och
schizofreniliknande tillstånd MENTH 2014 25
Demenssjukdom 2010 MENTH 2010 528
Depression och ångestsyndrom 2010 MENTH 2010 567
Missbruk och beroende MENTH 2015 537
Schizofreni eller schizofreniliknande tillstånd 2011 MENTH 2011 106
Bröst-, prostata-, tjocktarms och ändtarmscancervård
2014 ONCOL 2014 1.226
Lungcancervård 2011 ONCOL 2011 277
Palliativ vård i livets slutskede PALLIA 2013 78
Rörelseorganens sjukdomar 2012 REUMA 2012 481
Total 6.610
Gustaf Nelhans, Swedish School of Library and Information
Science, University of Borås, gustaf.nelhans@hb.se 6

7. Descriptive stats of refs (WoS data)
Record type
Article Review
Article; Proceedings Paper Letter
Editorial Material Note
Review; Book Chapter News Item
Total of 406 references older
than 20 years (max= 57)
Mean = 7.8 years
0
100
200
300
400
500
600
700
800
-10 -5 0 5 10 15 20
Noofrefs
Reference age
.
Reference age
Gustaf Nelhans, Swedish School of Library and Information
Science, University of Borås, gustaf.nelhans@hb.se 7

8. Citation data / share of “highly cited”
# Journal C.G. refs Citations Cit/C.G ref
1 JOURNAL OF CLINICAL ONCOLOGY 187 54591 291.9
2 NEW ENGLAND JOURNAL OF MEDICINE 152 167521 1102.1
3 STROKE 144 24435 169.7
4 JOURNAL OF CLINICAL PERIODONTOLOGY 120 7368 61.4
5 COCHRANE DATABASE OF SYSTEMATIC REVIEWS 112 6077 54.3
6 LANCET 109 68433 627.8
7 JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 101 58214 576.4
8 JOURNAL OF PERIODONTOLOGY 81 3373 41.6
9 BRITISH MEDICAL JOURNAL 75 18110 241.5
10 AMERICAN JOURNAL OF PSYCHIATRY 73 11070 151.6
(raw data, not field normalized)
Gustaf Nelhans, Swedish School of Library and Information
Science, University of Borås, gustaf.nelhans@hb.se 8

9. Frequently publishing countries (n=85)
LinLog mode normalizationGustaf Nelhans, Swedish School of Library and Information
Science, University of Borås, gustaf.nelhans@hb.se 9

10. Frequently publishing institutions
Of 6631 institutions, 665 have at least 5
published papers in the set. (Swedish
institution names are harmonized)
Gustaf Nelhans, Swedish School of Library and Information
Science, University of Borås, gustaf.nelhans@hb.se 11

11. Gustaf Nelhans, Swedish School of Library and Information
Science, University of Borås, gustaf.nelhans@hb.se
Keyword occurrences Keyword occurrences Keyword occurrences
depression 105 prostate cancer 137 meta-analysis 135
treatment 71 breast cancer 93 stroke 84
randomized controlled trial 51 survival 61 type 2 diabetes 42
pain 41 colorectal cancer 53 epidemiology 33
alcohol 38 radiotherapy 52 osteoarthritis 26
systematic review 106 cost-effectiveness 105 children 22
clinical trial 54 dementia 93 cytotoxicity 19
periodontitis 30 alzheimer's disease 66 prevalence 14
chlorhexidine 29 quality of life 63 allergic contact dermatitis 11
dental implants 29 rehabilitation 57 biocompatibility 9
12

12. Professional Impact: Goals and sources for data
• Clinical guidelines and Treatment recommendations
• …also patents (Wilsdon et al 2015)
• Climate policy documents (Bornmann et al. 2016)
• Standards documents (e.g. ISO, SIS)
• Government White Papers (e.g. Swedish SOU: “Statens Offentliga Utredningar”)
• Basically any issued document having cited references
Gustaf Nelhans, Swedish School of Library and Information
Science, University of Borås, gustaf.nelhans@hb.se
13

13. Open questions to relate to if clinical impact
for health care should be analysed:
• Professional/clinical impact – theoretic relevance?
• ≠ academic impact, should not be treated within a single indicator
• Citation window?
• Slightly longer delay before
references are found in
clinical guidelines ~8 years:
• How to handle references to non-clinical research?
• E.g. references to other clinical guidelines or reviews and not to
underlying clinical research -> follow through iteratively?
• Coverage for source items in PubMed, WoS ~3/4
• Is it worth including other sources?
Cited references
Clinical guidelines
Gustaf Nelhans, Swedish School of Library and Information
Science, University of Borås, gustaf.nelhans@hb.se
14

14. Conclusions
• In sum, it is argued professional impact, here
designated as impact of clinical research within
clinical guidelines issued at the national or
international level, provides grounds for measuring
impact closer to the professional expertise that
actually uses the research. Professional impact, as
opposed to intra-scientific impact within the
academic literature and “social impact”, based on
mentions in social media or inclusion in on-line
reference databases should be viewed as a distinct
instance of research impact that operates on its
own specific premises.
Gustaf Nelhans, Swedish School of Library and
Information Science, University of Borås,
gustaf.nelhans@hb.se
15

15. Thank you! / References
• Andersen, J. P. (2013). Conceptualising research quality in medicine for evaluative
bibliometrics. Københavns Universitet: Det Humanistiske Fakultet.
• Bornmann, L., Haunschild, R., & Marx, W. (2015). Policy documents as sources for
measuring societal impact: How is climate change research perceived in policy documents?
Unpublished manuscript, (arXiv:1512.07071).
• Lewison, G., & Sullivan, R. (2008). The impact of cancer research: how publications
influence UK cancer clinical guidelines. British Journal of Cancer, 98(12), 1944-1950.
• Swedish Research Council. (2013). En utvecklad modell för kvalitetsutvärdering av klinisk
forskning finansierad av ALF-medel. Stockholm: Swedish Research Council
• Thelwall, M., & Maflahi, N. (2016). Guideline references and academic citations as evidence
of the clinical value of health research. Journal of the Association for Information Science
and Technology, 67(4), 960-966.
• Wilsdon, J., et al.,. (2015). The Metric Tide: Report of the Independent Review of the Role
of Metrics in Research Assessment and Management.
Gustaf Nelhans, Swedish School of Library and Information
Science, University of Borås, gustaf.nelhans@hb.se
16