The document discusses whether key performance indicators (KPIs) focusing on recruitment numbers for the National Institute for Health Research Clinical Research Network (NIHR CRN) may have unintentionally reduced the quality of research conducted in the NHS in England from 2008 to 2016. It notes that while recruitment numbers have increased, the proportion of interventional to observational studies has decreased. This could indicate less emphasis on higher quality interventional research. Data is presented showing trends in commercial vs. non-commercial research and relationships between recruitment numbers and quality metrics. The discussion examines how KPIs focused solely on quantity may have shifted priorities away from quality and calls for new KPIs that reward both quality and quantity to better recognize high quality, embedded

1. Is there any evidence that ‘NIHR HLO1’ has had a
counterproductive impact upon the intrinsic quality of
research being delivered through the NHS in England
between 2008 & 2016?
Discussion Session – Theme 3 ‘Working Better Together’

3. Today’s Session
1. Background
2. What prompted the question?
3. What does the data look like?
4. Delivering quality & quantity
5. Working better together
6. What’s been left unsaid

4. NIHR Clinical Research Network
• [1] Set up in 2006 to “Improve health and wealth of nation”
• 15 LCRNs, Central Coordinating Centre
• All NHS Trusts research active
• ~650,000 participants, ~4,500 studies p.a.
• UK more attractive to Sponsors
• Retained & increased commercial research activity
• Research training (~100,000 GCP)
• Fund a large workforce (~40,000 staff)
• [2] Economic and health benefits (KPMG LLG, 2016)
Key to success, several key performance indicators [3]…

5. Increase the number of
participants recruited
into NIHR CRN
Portfolio studies
Increase the proportion
of studies in the NIHR
CRN Portfolio delivering
to recruitment
target to time
Increase the number of
commercial contract
studies delivered
through the NIHR CRN
Reduce the time taken
for eligible studies to
achieve set-up in the
NHS
Reduce the time taken to
recruit to recruit first
participant into NIHR
Portfolio studies
Increase NHS
participation in NIHR
CRN Portfolio studies
Increase the number of
participants recruited
into Dementia and
Neurodegeneration
studies on the Portfolio

6. NIHR CRN – Founding Aims
“More patients and health professionals
participating in health research with high-
quality protocols and early access to new
intervention and prevention strategies”
‘Five year aims’, page 6
[1] ‘Best Research for Best Health’ DoH, January 2006
• Q/ What HLO do we have for quality?
• Q/ What HLO do we have to support non-
commercial interventional studies?

7. Increase the number of
participants recruited
into NIHR CRN
Portfolio studies
Increase the proportion
of studies in the NIHR
CRN Portfolio delivering
to recruitment
target to time
Increase the number of
commercial contract
studies delivered
through the NIHR CRN
Reduce the time taken
for eligible studies to
achieve set-up in the
NHS
Reduce the time taken to
recruit to recruit first
participant into NIHR
Portfolio studies
Increase NHS
participation in NIHR
CRN Portfolio studies
Increase the number of
participants recruited
into Dementia and
Neurodegeneration
studies on the Portfolio

8. What is QUALITY in Clinical Research?
Does not have to be a
perfect solution…

9. What is QUALITY in Clinical Research to Me
NIHR Interventional Studies (Highest Quality)
• Changing patient care. Greatest potential to quickly translate into practice
• Studies are resource heavy, larger team, longer visits, follow-up visits
• Hard slog (time & effort), some work goes unrecognised
NIHR Observational Studies (Quality)
• Observing patient care. Quickly change practice?
• Less resource heavy; often without follow-up
• Quick wins. Can deliver more with less
My proxy for Quality = Participants in interventional studies
Total research participants

10. What Prompted the Question?
• Promoting and recognising quality and interventional studies?
• Price of everything (VfM) but value of nothing (QfM?)
• Success based on numbers (Partnership Board, League Tables)
• Celebrate the quick and easy, but less so the hard and difficult
• Sense that money does not follow activity (CRN>LCRN>?>Trust)
• Studies set-up to run slowly in a LCRN, rather than fast across many
• Trust ‘Sleeper Studies’ to bump up recruitment
• Obsession on numbers permeating to all staff
Do any of the above put the interests of science
and patients at the fore?

11. Delivering More With Less
• Funding is going down (in real terms)
• Based on anecdotal observations
• Questions:
1. Has the pursuit of numbers (HLO1) had an
impact on the quality (% interventional) of
research?
2. Has the pursuit of HLO1 (£) and HLO3 (£)
impacted on non-commercial interventional
studies?

12. NIHR Portfolio Studies (2008-2016)
[4] Data was collected from NIHR’s ‘Open Data Platform’ (data cut 5th Jan 2017)

13. NIHR Portfolio Studies by Type (2008-2016)
[4] Data was collected from NIHR’s ‘Open Data Platform’ (data cut 5th Jan 2017)

14. Non-Commercial Interventional Studies
• Q/Should we be concerned by 26% shift in Interventional
trials from non-commercial to commercial?
• Maybe so?
– Stepping stone for researchers (obs.>int.>commercial)
– Charity, University, NHS & NIHR sponsored
– What the UK is known for
– Not profit led, often looks at using existing medicines in new
combinations, conditions
• Maybe not?
– Is it not a consequence of commercial interventional doing so well?
– Actual number of non-commercial interventional is still increasing

15. NIHR Portfolio Recruitment (2008-2016)

16. NIHR Portfolio Recruitment by Type (2008-2016)

18. % Participants in Interventional Trials (2015/16)
[4] Data was collected from NIHR’s ‘Open Data Platform’ (data cut 5th Jan 2017)

19. Research at Typical Acute Trust
• Treats 731,114 patients p.a.
• 2,865 participate in research
• 117 studies
– Interventional ~40%
– Observational ~40%
– Commercial ~20%
Most Typical Trust:-
South Tees Hospitals NHS
Foundation Trust

20. % Participants in Interventional Trials (2015/16)
Top
Recruiting
DGH
Top
Recruiting
NHS Trust
[4] Data was collected from NIHR’s ‘Open Data Platform’ (data cut 5th Jan 2017)
Highest Quality
NHS Trust
(91st for recruits)

21. Is High Recruitment linked to Low Interventional Studies?
[4] Data was collected from NIHR’s ‘Open Data Platform’ (data cut 5th Jan 2017)

22. Quality & Quantity – Acute NHS Trusts 2015/16
Higher Quality
Higher Quantity
Higher Quality
Lower Quantity
Lower Quality
Higher Quantity
[5 Data was collected from NIHR’s ‘Open Data Platform’ (data cut 5th Jan 2017)
[5,6,7,8] Total patients treated (inpatient, outpatient, emergency & maternity)

23. Trust ‘Quality & Quantity’ Prescription
Q/ How can my local hospital go from
‘Lower Quality, Lower Quantity’ to ‘Higher
Quality, Higher Quantity’?
Currently 127 studies, 1,635 recruits
Require 70 Int. recruits pcm
[4] Data was collected from NIHR’s ‘Open Data Platform’ (data cut 5th Jan 2017)
[5,6,7,8] Total patients treated (inpatient, outpatient, emergency & maternity)

24. Quality* x Quantity* = Embedded Research Culture?
[5 Data was collected from NIHR’s ‘Open Data Platform’ (data cut 5th Jan 2017)
[5,6,7,8] Total patients treated (inpatient, outpatient, emergency & maternity)
* Need to ensure
Quality and Quantity
are equally weighted

25. NIHR Research Activity League Table [9]
• Published each year (2016/17 due soon!)
• Every NHS Trust taking part in research
• Currently recognises ‘participants’ & ‘studies’
• Celebrates performance at largest Trusts?
• Is it motivating to all Trusts?
• Used as a basis to make statements about
‘research culture’!
• Simple areas to make table more informative
Quality of Research = Interventional Participants / Total Participants
Quantity of Research = Research Participants / Total Patients Treated
• Multiply Quality by Culture then rank for ‘Embedded Research Culture’

26. Alternative–Research Activity League Table for 2015/2016
NHS Trust
(Acute)
Number of
Studies
Recruiting
Participants in
Studies
Relative
Quality1 of
Research
Relative
Quantity2 of
Research
Embedded
Research
Culture3
National Trust
Ranking
BIRMINGHAM WOMEN'S NHS FT 59 4,536 60.6% 2.6% 75.2% 1
GUY'S AND ST THOMAS' NHS FT 515 28,407 45.1% 1.8% 39.0% 2
UNIVERSITY HOSPITAL SOUTHAMPTON NHS FT 401 21,051 20.9% 2.4% 23.6% 3
ROYAL LIVERPOOL AND BROADGREEN UNI. HOSP. 161 8,866 44.7% 0.9% 18.6% 4
IMPERIAL COLLEGE HEALTHCARE NHS TRUST 437 13,973 29.6% 1.0% 13.7% 5
NORTH BRISTOL NHS TRUST 164 4,022 45.4% 0.6% 13.3% 6
SOUTH TYNESIDE NHS FT 35 953 80.6% 0.3% 11.9% 7
OXFORD UNIVERSITY HOSPITALS NHS FT 479 21,444 15.3% 1.6% 11.5% 8
THE ROYAL MARSDEN NHS FT 212 2,890 28.9% 0.8% 10.7% 9
BARTS HEALTH NHS TRUST 348 14,264 29.0% 0.7% 10.1% 10
THE WALTON CENTRE NHS FT 45 1,524 17.4% 1.2% 10.1% 11
MID ESSEX HOSPITAL SERVICES NHS TRUST 67 1,118 56.5% 0.4% 9.4% 12
HEART OF ENGLAND NHS RT 169 6,022 46.8% 0.4% 9.0% 13
SHEFFIELD TEACHING HOSPITALS NHS FT 378 8,799 34.2% 0.6% 8.9% 14
LIVERPOOL WOMEN'S NHS FT 42 1,336 35.0% 0.5% 8.8% 15
PORTSMOUTH HOSPITALS NHS TRUST 178 3,768 42.6% 0.4% 8.2% 16
PAPWORTH HOSPITAL NHS FT 54 1,138 16.4% 1.0% 8.1% 17
LEEDS TEACHING HOSPITALS NHS TRUST 491 11,893 24.0% 0.7% 8.1% 18
GREAT ORMOND STREET HOSPITAL FOR CHILDREN 176 3,926 11.8% 1.4% 7.8% 19
UNIVERSITY COLLEGE LONDON HOSPITALS NHS FT 410 10,473 25.3% 0.7% 7.7% 20
“Recognising NHS acute Trusts (regardless of size) for embedding a
research culture based on the number of their patients recruited into
research studies and the quality of the research studies they support”

27. Alternative LCRN Research League Table (Acute 2015/2016)
“Recognising LCRNs for embedding a research culture based on the
number of Trust patients recruited into research studies and the
quality of the research studies they support”
Local Research Network
(Acute NHS Trusts)
Number of
Studies
Recruiting
Participants in
Studies
Relative Quality1
of Research
Relative
Quantity2 of
Research
Embedded
Research
Culture3
National LCRN
Ranking4
South London 1,152 62,145 28.4% 0.55% 7.4 1 (2)
Wessex 821 42,883 23.5% 0.59% 6.6 2 (4)
North East and North Cumbria 851 33,961 39.9% 0.32% 6.0 3 (8)
North West Coast 742 28,787 28.0% 0.44% 5.9 4 (8)
North West London 770 27,765 26.8% 0.45% 5.7 5 (8)
Thames Valley and South Midlands 724 46,853 12.7% 0.82% 5.0 6 (1)
West Midlands 1,059 57,272 27.4% 0.38% 4.9 7 (3)
North Thames 1,386 74,064 30.0% 0.34% 4.8 8 (7)
West of England 646 26,772 31.8% 0.32% 4.8 9 (5)
East Midlands 868 42,823 31.2% 0.26% 3.9 10 (3)
South West Peninsula 587 24,670 25.3% 0.31% 3.7 11 (4)
Yorkshire and Humber 1,134 73,814 22.3% 0.33% 3.5 12 (10)
Eastern 855 37,580 22.6% 0.32% 3.4 13 (3)
Greater Manchester 986 40,126 17.5% 0.31% 2.5 14 (6)
Kent, Surrey and Sussex 639 35,215 27.8% 0.18% 2.4 15 (5)

28. Summing up – Working Better Together
• HLOs have been a success in getting us to where we are
• Indirectly, HLOs are now becoming counterproductive:
– Trusts having to do more with less resources
– Consciously/subconsciously prioritising quantity, as this is
what gets recognised
• Time for a new HLO1 metric that rewards quality & recognises
‘embedded research culture’ independent of Trust size
HLO1
Increase the number of
participants recruited
into NIHR CRN
Portfolio studies
HLO1
Increase the research
culture within NHS:
Quality of research
supported and number
of patients participating
in research.
• Recommend changes to ‘Research Activity League Table’
– Quality of Research (ODP)
– Quantity of Research (ODP & NHS Digital)
– Rank by ‘Embedded Research Culture’

29. Thank You &
Safe Journey Home

30. References
1. ‘Best research for best health: a new national health research strategy.’ Policy Paper, Department of
Health, 25th January 2006
2. ‘NIHR Clinical Research Network: Impact and Value Assessment.’ KPMG LLG, Public Version,
September 2016.
3. ‘NIHR CRN High Level Objectives Year End Performance Report 2015/16.’ NIHR Clinical Research
Network, Public Version: Version 1, 13th June 2016.
4. NIHR ‘Open Data Platform’ (data cut 5th January 2017
5. ‘Hospital Outpatient Activity 2015-16 in English NHS hospitals’. NHS Digital, 1st December 2016.
6. ‘Admitted patient care activity 2015-16 in English NHS hospitals’. NHS Digital, 9th November 2016.
7. ‘Hospital Accident and Emergency Activity, 2015-16 in English hospitals’. NHS Digital, 10th January.
2017.
8. ‘Hospital Maternity Activity, 2015-16 in English hospitals’. NHS Digital, November 9th 2016.
9. ‘NIHR Research Activity League Table’ 2015/16’