Existing benchmarks might not be capturing the information that decision-makers need. published June 2015

1. See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/304276418
The Limitations Inherent in using “benchmark” outcomes to estimate NHS
health service safety
Presentation · June 2016
DOI: 10.13140/RG.2.1.2468.3121
CITATIONS
0
READS
115
4 authors, including:
Elena A Baker-Glenn
Hertfordshire Partnership NHS Foundation Trust
11 PUBLICATIONS 219 CITATIONS
SEE PROFILE
Stephen Shervais
Eastern Washington University
26 PUBLICATIONS 97 CITATIONS
SEE PROFILE
All content following this page was uploaded by Stephen Shervais on 23 June 2016.
The user has requested enhancement of the downloaded file.

2. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 1Shervais, Baker-Glenn, Dickerson and Spencer June 2016
Presentation 2
The Limitations Inherent in using “benchmark”
outcomes to estimate NHS health service safety

3. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 2
Background – People involved in the study
Dr Jennifer Spencer
Healthcare Fellow University of Cambridge, Cambridge UK
Dual CCT RCPsych CAMH and ID psychiatry, MRCPsych, MB, BAO, BCh, BMedSci, BA,
Dr Elena Baker-Glenn
Dual CCT in training RCPsych General & Old Age psychiatry, MRCPsych, BSC MBBCHIR MMEDSCI
Cambridgeshire and Peterborough NHS Foundation Trust
Dr Terry Dickerson
Assistant Director EDC, University of Cambridge, Cambridge UK
PhD, MiMechE, CEng, BSc
Professor Stephen Shervais
Associate Professor of Management Information Systems
Accounting and Information Systems, College of Business and Public Administration, Eastern Washington U
PhD, MS, MA, BA

4. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 3
Background – What motivated the study
• Clinicians have been complaining that it is difficult to ensure people with
mental health disorders obtain the care they need when they are acutely
physically ill.
• Benchmark goals have shifted numerous times over the years, thus long term
monitoring of appropriate outcome measures has been difficult for NHS trusts
to accomplish.
• We wished to see if any appropriate long term outcome measures
demonstrated an association with governmental policies regarding the NHS
over time.

5. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 4
Theory/Framework
Design Research Methodology
Blessing et al, 2009

6. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 5
Mental illness
Encompasses a range of diagnoses including dementia, substance misuse,
depression, anxiety, mania, psychosis, eating disorders, and personality
disorders.
• Patients with mental illness have lower life expectancy than the rest of the
population
• Differences are more marked in younger adults
5

7. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 6
Benchmark outcomes
• Benchmarking is intended to help managers implement best practice at best
cost
• In the UK bemnchmarks are currently used as tools to monitor impact of
governance, management, clinical outcomes and logistics
6

8. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 7
Importance of the Measure
• Relevance to stakeholders
• Health importance
• Applicability to measuring the equitable
distribution of health care (for health
delivery measures)
• or of health (for population health
measures)
• Potential for improvement
• Susceptibility to being influenced by the
health care system
Scientific Soundness: Clinical Logic
• Explicitness of evidence
• Strength of evidence
• Scientific Soundness: Measure Properties
• Reliability
• Validity
• Allowance for patient/consumer
factors as required
• Comprehensible
• Feasibility
• Explicit specification of numerator and
denominator
• Data availability
Desirable attributes of a Quality Measure
https://www.qualitymeasures.ahrq.gov/tutorial/attributes.aspx

9. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 8
General Health Care Delivery
Measures
Clinical quality Measures
• Process
• Access
• Outcome
• Structure
• Patient Experience
Related Health Care Delivery
Measures
• User-Enrollee Health State
• Management
• Use of Services
• Clinical Efficacy Measures
• Efficiency
Population Health Measure
Domains
Domains of Measurement

10. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 9
NHS Outcomes Framework

11. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 10
NHS Outcomes Framework Goals
Domain 1: Preventing People from Dying Prematurely
• Maximising the contribution that the NHS can make to
preventing disease
• Finding the ‘missing millions’ and diagnosing earlier and more
accurately
• Treating people in an appropriate and timely way
• Addressing unwarranted variation in mortality and
survival rates
• Reducing deaths in babies and young children

12. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 11
NHS outcome framework for mental illness
2015/16
• Percentage of adults receiving secondary mental health services living
independently
• Proportion of all people in prison who have a mental illness
• Percentage of adults in contact with secondary mental health services in
paid employment
• Excess mortality rate in adults with serious mental illness, aged
under 75, per 100,000 population
• Age–standardised mortality rate from suicide and injury of
undetermined intent per 100,000 population (in development)
11

13. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 12
Methodology
1. Benchmarks used by mental health NHS trusts were identified from the National Quality
Measures website
2. A highly respected “Big Data” database was selected (The World Health Organization Mortality
Database)
3. Mental Health mortality and self harm rates as well as general population mortality rate were
collected from the UK between the years 1990-2014.
4. Data was graphically depicted to look for trends
5. Further data was then collected from the WHO site from countries with similar economic and
health care systems
6. Statistical analyses were conducted to determine the statistical differences between similar
parameters. Tests were kept to a minimum to avoid Type II error.

14. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 13
Results – Initial Graph
UK Mental Health Mortality rate, Self Harm Mortality
rate and NHS Healthcare Policy changes
0
5000
10000
15000
20000
25000
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
UK total deaths
attributable to mental ill
health
UK female deaths
attributable to mental ill
health
UK male deaths
attributable to mental ill
health
UK total deaths caused by
Intentional self-harm
UK male deaths caused by
Intentional self-harm
UK female deaths caused
by Intentional self-harm
Sir Liam Donaldson publishes
"An organisation with a Memory"
Deloitte, Monitor and Parliament
implement the Foundation Trust
programme
Care in the community
implemented & atypical
antipsychotics on the market.
13

15. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 14
Further Data Collection and Preparation
• WHO mortality data was obtained for 36 countries in Europe including the
UK from 1991– 2014 using ICD-10 diagnoses for:
• Total deaths from all causes
• Deaths due to mental and behavioural disorders
• Deaths due to self harm
• Data was converted to the rate per 100,000 population and then graphed

16. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 15
Total Population Mortality Rates in Europe
0
500
1000
1500
2000
2500
3000
1990
1991
1992
1993
1994
1994
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
Deaths per 100,000
Total Population Mortality Rates in Europe (UK Mortality Rate in purple)
Austria
Belarus
Belgium
Bulgaria
Croatia
Cyprus
Czech Republic
Denmark
Estonia
Finland
France
Georgia
Germany
Greece
Hungary
Iceland
Ireland
Italy
Latvia
Lithuania
Luxembourg
Malta
Netherlands
Norway
Poland
Portugal
Romania
Russian Federation
Serbia
Slovakia
Slovenia
Spain
Sweden
Switzerland
Ukraine
United Kingdom

17. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 16
European Mortality Rates for
People with Mental Health Conditions
0
20
40
60
80
100
120
140
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
Mortality Rate for people with a Mental Health
Condition (per 100,000 total population)
Mortality Rate for people with a Mental Health Condition (UK in purple)
Austria
Belarus
Belgium
Bulgaria
Croatia
Cyprus
Czech Republic
Denmark
Estonia
Finland
France
Georgia
Germany
Greece
Hungary
Iceland
Ireland
Italy
Latvia
Lithuania
Luxembourg
Malta
Netherlands
Norway
Poland
Portugal
Romania
Russian Federation
Serbia
Slovakia
Slovenia
Spain
Sweden
Switzerland
Ukraine
United Kingdom

18. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 17
European Mortality Rates secondary to Self Harm
0
10
20
30
40
50
60
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
Mortality Rate due to Self Harm
(per 100,000 total population)
European Mortality Rates due to Self Harm (UK is in purple)
Austria
Belarus
Belgium
Bulgaria
Croatia
Cyprus
Czech Republic
Denmark
Estonia
Finland
France
Georgia
Germany
Greece
Hungary
Iceland
Ireland
Italy
Latvia
Lithuania
Luxembourg
Malta
Netherlands
Norway
Poland
Portugal
Romania
Russian Federation
Serbia
Slovakia
Slovenia
Spain
Sweden
Switzerland
Ukraine
United Kingdom

19. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 18
Statistical Analyses
We performed a series of one-tail t-tests on the years prior to and following each
policy change. Sample size was adjusted based on the number of years available.
Policy Year Years Before Years After n t-test result
1993 1990-1992 1994-1996 3 0.0037
2000 1994-1999 2001-2006 6 0.00003
2008 2003-2007 2009-2013 5 0.0146
We also averaged the mortality rates for the years prior to and after each policy
change and performed a paired t-test (n=3) on the result. The result was 0.216,
which is not statistically significant. We are extending the range of sample
measurements and repeating the test.

20. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 19
Conclusion
• There appears to be an association between government
mandated changes in the way NHS services operate, and an
increase in the mortality rate of people with mental health
disorders
• The way benchmark outcomes are currently used may not be
providing decision makers with enough information to create and
design safe services.

21. IHC 2016, Abstract Ref 0282 Shervais, Baker-Glenn, Dickerson and Spencer June 2016 Slide 20
References
• Agency for Healthcare Research and Quality US Department of Health and Human Services
National Quality Measures Clearinghouse
(https://www.qualitymeasures.ahrq.gov/tutorial/index.aspx , page last viewed 18 June 2016
and https://www.qualitymeasures.ahrq.gov/tutorial/selecting.aspx page last reviewed 22 June
2016)
• Krousel-Wood. Practical Considerations in the Measurement of Outcomes in Healthcare. 187-
194. October 1999.
• NHS Outcomes Framework Measurement (https://www.england.nhs.uk/resources/resources-
for-ccgs/out-frwrk/dom-1/Porter)
• What Is Value in Health Care? New England Journal of Medicine. 2477-2481. December 2010
• World Health Organisation Mortality Database, accessed November 2015– June 2016
(http://apps.who.int/healthinfo/statistics/mortality/whodpms/ )
View publication statsView publication stats