The document discusses issues with current approaches to measuring health care quality and proposes alternatives. It summarizes the evolution of performance measurement efforts in the US from public reporting to pay-for-performance programs. However, these approaches have shown limited evidence of improved population health or driving meaningful change. The document proposes resetting the measurement agenda by shifting the focus to outcomes achieved per dollar spent, reimagining quality measurement based on patient goals and preferences, and leveraging emerging data sources to support innovative models of care.

1. Eric C. Schneider MD, FACP
Senior Vice President for Policy and Research
@ericschneidermd
The Mis-measure of
Health Care
Can Measurement, Improvement, and Cost
Reduction be Reunited?

2. 2

3. 3https://www.wired.com/2014/05/fantastically-wrong-martian-
canals/

4. 4
1.The U.S. health care challenge and evolution
of performance measurement
2.Limits of current measurement efforts
3.Resetting the measurement agenda
Agenda

5. 5
Source: Schneider et al. Mirror, Mirror 2017:
Note: See the methodology appendix for a description of how the performance score is
calculated.
U.S. health care system performance
is not optimal
UK AUS
NETH
NZ NOR
SWIZ SWE GER
CAN
FRA
US
Eleven-country average
Higher performing
Lower performing

6. 6
0
2
4
6
8
10
12
14
16
18
20
1980 1985 1990 1995 2000 2005 2010 2015
US (17.2%)
SWIZ (12.3%)
FRA (11.5%)
GER (11.3%)
SWE (10.9%)
CAN (10.4%)
NOR (10.4%)
NETH (10.1%)
UK (9.7%)
AUS (9.1%)
NZ (9.0%)
Notes: Current expenditures on health per capita, adjusted for current US$ purchasing power parities
(PPPs). Based on System of Health Accounts methodology, with some differences between country
methodologies (Data for Australia uses narrower definition for long-term care spending than other
countries). *2017 data are provisional or estimated.
Source: OECD Health Data 2018.
SPENDING & COSTS
Health Care Spending as a Percent of GDP,
1980–2017
Adjusted for Differences in Cost of Living
Percent (%) of GDP 2017* data:
2017*

7. 7
Kocher R, Sahni NR. N Engl J Med 2011;365:1370-1372.
• Real Sector Growth (Compound Annual Growth Rate), Broken into Labor Productivity
Growth and Employment Growth: U.S. Economy 1990–2010
Growth of labor productivity:
Health care lags other sectors

8. 8
1. MONITORING: NCQA develops performance
measurement with public reporting to counter
capitation incentives of HMOs (1990s)

9. 9
•Market transparency, consumer choice
• Consumers/purchasers will select plans and providers
• Demand for quality will motivate competition
•Organizational and professional improvement
• Reputation and brand
• Intrinsic motivation
2. IMPROVEMENT MOTIVATOR:
Performance reporting will drive
improvement via two paths (2003)
Berwick et al, Med Care 2003

10. 10
3. PAY-FOR-PERFORMANCE: payment incentives
create stronger motivation than public reporting
alone
Lindenauer et al, NEJM 2007

11. 11
4. HOSPITAL VALUE-BASED-PURCHASING: Bonuses
and penalties can drive high-quality, cost-conscious
care (2012)
16
Outcome
5%
25%20%
25%
25%
Outcomes
Process
Efficiency and
Cost ReductionSafety
Patient and
Caregiver
Centered
Experience
of Care/Care
Coordination
Patient and Caregiver Centered Experience of Care/Care
Coordination
Hospital Consumer Assessment of Healthcare
Providers and Systems (HCAHPS) Survey
Clinical Care
Outcomes Process
MORT-30-AMI
MORT-30-HF
MORT-30-PN
AMI-7a
IMM-2
PC-01*
Safety
CLABSI
CAUTI
SSI: Colon & Abdominal Hysterectomy
MRSA Infections*
C-difficile Infections*
AHRQ PSI-90
Efficiency and Cost Reduction
MSPB-1
Domain Weights (2017)
Clinical Care
An asterisk (*) indicates a newly adopted measure for the Hospital VBP Program.

12. 12Zuckerman RB et al. N Engl J Med 2016;374:1543-1551
5. CONDITION-TARGETED PENALTIES: Hospital
Readmissions Reduction Program (HRRP)

13. 13
6. PHYSICIAN P4P: Medicare Quality Payment
Program (QPP)
MIPS

14. 14
1.The U.S. health care challenge and evolution of
performance measurement
2.Limits of current measurement efforts
3.Resetting the measurement agenda
Agenda

15. 15
• Consumers not using the results to inform
choice
• Limited evidence of improved population
health
• Professional skepticism about results
• Technical issues: risk adjustment, coding,
selection, patient preferences
• Limited utility in the daily work of clinicians
• Burden
• Redundant, misaligned measures
• Data collection and reporting requirements
Performance Measurement and Reporting:
The Bad and the Ugly

16. 16
• Measures too technical, not enough about interpersonal
quality and communication
• Some highly salient data not available to consumers at
the point of care (cost and price information)
• Few measure development efforts adequately consider
the consumer experience
Consumers not using performance reports
to select hospitals or physicians
Concannon T, et al
(https://www.rand.org/pubs/research_reports/RR1760.html)
Rogut L, et al. (https://nyshealthfoundation.org/wp-
content/uploads/2017/12/empowering-new-yorkers-with-quality-
measures-dec-2017.pdf)

17. 17
0
10
20
30
40
50
60
70
80
90
100
2009 2010 2011 2012 2013 2014 2015 2016
%
Axis Title
Adult BMI Assessment Rate, by Payer Type, 2009-2016
Commercial - HMO Commercial - PPO Medicaid - HMO Medicare - HMO Medicare - PPO
Source: National Committee for Quality Assurance, State of Health Care Quality 2017:
http://www.ncqa.org/report-cards/health-plans/state-of-health-care-quality/2017-table-of-
contents/adult-bmi
“Streetlamp” measures suggest progress

18. 18
Prevalence of Obesity and Extreme Obesity in US Children and Adolescents Aged 2 to 19 Years From 1988-1994
Through 2013-2014. Data from National Health and Nutrition Examination Surveys. The error bars indicate 95%
confidence intervals. The prevalence estimates are weighted.
aDefined as at or above the sex-specific 95th percentile on the US Centers for Disease Control and Prevention
(CDC) BMI-for-age growth charts.
bDefined as at or above 120% of the sex-specific 95th percentile on the CDC BMI-for-age growth charts.
But health impact is not apparent
Trends in Obesity Prevalence Among Children and Adolescents in the United
States, 1988-1994 Through 2013-2014 JAMA. 2016;315(21):2292-2299.
doi:10.1001/jama.2016.6361

19. 19
• U.S. General Accounting Office, 2015
“High-stakes” measures top out quickly

20. 20
Bonfrer I, et al. Impact of Financial Incentives on Early and Late Adopters among US Hospitals:
observational study. BMJ Open 2018
Process-outcome
links are weak and
not clearly related
to payment
incentives

21. 21
• Readmissions reduction weakly correlated with mortality
reductions (Dharmarajan K et al, JAMA, 2017)
• But 30-day and 1-year risk-adjusted mortality among
heart failure patients increased after HRRP
implemented (Gupta A et al, JAMA Cardiology, 2018)
• Penalties were larger for safety net hospitals and under-
resourced hospitals than other hospitals (Joynt-Maddox
et al, HSR, 2019)
• Pre-penalty phase decline in readmissions due to
electronic data-reporting modification that altered risk
adjustment (Ody C et al, Health Affairs, 2019)
Readmissions penalties: a cautionary
tale

22. 22Zuckerman RB et al. N Engl J Med 2016;374:1543-1551
5. CONDITION-TARGETED PENALTIES: Hospital
Readmissions Reduction Program (HRRP)

23. 23
Bardach N et al, Effect of Pay-for-Performance Incentives on Quality of Care in Small Practices With
Electronic Health Records: A Randomized Trial, JAMA, 2013
Maybe measurement + incentives
could work in ambulatory care?

24. 24Mendelson A et al. The effects of pay-for-performance programs on health,
health care use, and processes of care. Ann Intern Med 2017:166:341-353
Evidence for effects of pay-for-performance in
ambulatory care settings is weak overall
Measure
Domain
Study Designs Study
Limitations
Strength of
Evidence
Summary of Findings
Ambulatory
Process 1 RCT
7 ITS
23 controlled B-A
13 uncontrolled B-A
Medium Low Much of the evidence for positive effects comes from
the QOF program. Little evidence of long-term
effects; biggest improvements seen in areas with
poor baseline performance.
Health 8 controlled B-A
2 uncontrolled B-A
High Insufficient Most of the controlled studies have significant
selection bias, and the 2 uncontrolled studies do not
provide sufficient information to draw conclusions.
Utilization 11 controlled B-A
1 uncontrolled B-A
Medium Low Stronger study designs showed no effect.
Intermediate 2 RCTs
2 ITS
1 controlled B-A
7 uncontrolled B-A
Medium Low No consistently large effects; stronger observational
studies showed no effect; 2 trials produced
conflicting results
ITS = interrupted time series; B-A = Before-After; RCT = Randomized, controlled trial ;QOF = Quality-Outcomes Framework

25. 25
Minchin M et al, N Engl J Med 2018;379:948-957
Removal of financial incentives reveals the
weaknesses of P4P

26. 26Minchin M et al, N Engl J Med 2018;379:948-957
Removal of financial incentives reveals the
weaknesses of P4P

27. 27
“It is common sense to take a
method and try it: If it fails, admit
it frankly and try another. But
above all, try something.”
President Franklin Delano Roosevelt: Address to Oglethorpe University,
May 1932
Is value-based purchasing diverting
health care from quality and
affordability?

28. 28
Diagnosis Potential Solutions
Poor program design • Technical fixes (risk-adjustment,
attribution, gaming)
Wrong measures • Fewer measures
• Move away from process and toward
outcome measures
• New measure development efforts
• Better electronic data (EHRs)
Weak incentives • Put more money at risk in P4P
• Global caps and risk-based contracting
Inability of professionals to
adapt and change
• Stronger financial incentives
• More competition
• Learning collaboratives
Inherent uncertainty of
medicine, measurement,
and motivation
• Reduce financial incentives
• Reinvigorate professionalism and intrinsic
motivation
Diagnoses and potential solutions

29. 29
1.The U.S. health care challenge and evolution of
performance measurement
2.Limits of current measurement efforts
3.Resetting the measurement agenda
Agenda

30. 30
1. Shift focus to measuring health outcomes
2. Reimagine performance measurement
3. Invest in research and development on novel
uses of emerging data sources
4. Repurpose measurement to support disruptive
innovations
Measurement Reset Pathways

31. 31
•Health outcomes achieved per dollar spent
• Numerator defined as condition-specific, multidimensional
health outcomes
• Denominator is aggregate spending for a ‘cycle of care’ for each
condition
• Requires longitudinal measurement of episodes
• Defined for patient groups with similar needs
• Agnostic to process of care
• Challenges: Which outcomes matter? How are they
measured?
1. To get to value, focus on outcomes
Porter ME, NEJM, 2009

32. 32
• Heterogeneity and variability of biology
• Limited number of conditions highly responsive to care
• Ambiguous attribution to professionals and organizations
• Case-mix differences across providers
• Risk adjustment models impossible to perfect
• Experience with PROMs has been disappointing to date
Despite intuitive appeal, “health
outcomes” face several challenges as
performance measures

33. 33
•The purpose of care is to select interventions
that optimize the patient’s wellbeing
•The effectiveness of clinical interventions
matters
•But patient goals and preferences are equally
crucial input
2. Reimagining quality measurement:
From guideline adherence to patient-
informed treatment
McGlynn EM, Schneider EC, Kerr EA; NEJM 2014

34. 34
1. Comprehensive inventory of each patient’s health status,
risks, and health care needs
2. Analytics for matching potential evidence-based
interventions to the documented patient needs
3. Structured record of each patients’ health-related goals
and preferences to inform the priority of interventions
Value Metric: An aggregate estimate of the effectiveness of
clinicians and systems at delivering appropriate and effective
care to the right individuals based on jointly-developed
individual goals and preferences…
Reimagined quality measurement
system has three components
McGlynn EM, Schneider EC, Kerr EA; NEJM 2014

35. 35McGlynn EA et al. N Engl J Med 2014;371:2150-2153.
Illustrative Quality-Measurement and Care Delivery
Matrix for Two Women between the Ages of 45 and 64
Years with the Same Medical Conditions.

36. 36
•“Passive” data collection with consent
• Consumer transaction, social media, and sensor data
•Personal interactive digital assistants
• Real-time elicitation of goals, preferences, and needs
•Large data computational methods
• Enable complex predictions, monitoring, and feedback
loops
Digital advances may be setting the stage
for “reimagined” performance
measurement

37. 37
Jared B Hawkins et al. BMJ Qual Saf 2016;25:404-413
Sentiment and mortality among hospitals that
have ≥50 patient experience tweets (n=297).

38. 38
• Retire pay-for-performance applications of
measurement
• Retain measurement and reporting for aggregated
populations (insured beneficiaries, large delivery
systems, geographic areas)
• Targeted analyses that inform policymakers, regulators, and
managers of large systems (rather than routine reporting)
• Focus on key policy objectives (population health, access to
care, strengthening primary care services, improved experience,
reducing administrative burden)
• Actively measure and report disparities in care
Principles

39. 39
• Innovative cost-saving care
models are percolating
• Current uses of quality
measurement in pay-for-
performance programs divert
resources and may suppress
innovation
• Use quality measurement
judiciously to achieve specific
improvement goals
• Invest in R&D to develop novel
measurement approaches and
tools
Conclusions

40. Discussion
University of Chicago
May 2019
Eric C. Schneider, MD, MSc, FACP
Senior Vice President for Policy and Research
The Commonwealth Fund
@ericschneidermd