EDM ForumEDM Forum CommunityeGEMs (Generating Evidence & M.docx

EDM Forum
EDM Forum Community
eGEMs (Generating Evidence & Methods to
improve patient outcomes) Publish
4-20-2017
Reducing Healthcare Costs Through Patient
Targeting: Risk Adjustment Modeling to Predict
Patients Remaining High-Cost
Jonathan A. Wrathall
Intermountain Healthcare, [email protected]
Tom Belnap
Intermountain Healthcare, [email protected]
Follow this and additional works at: http://repository.edm-
forum.org/egems
Part of the Other Medicine and Health Sciences Commons, and
the Social Statistics Commons
This Methods Case Study is brought to you for free and open
access by the the Publish at EDM Forum Community. It has
been peer-reviewed and
accepted for publication in eGEMs (Generating Evidence &
Methods to improve patient outcomes).
The Electronic Data Methods (EDM) Forum is supported by the
Agency for Healthcare Research and Quality (AHRQ), Grant
1U18HS022789-01.
eGEMs publications do not reflect the official views of AHRQ
or the United States Department of Health and Human Services.

Recommended Citation
Wrathall, Jonathan A. and Belnap, Tom (2017) "Reducing
Healthcare Costs Through Patient Targeting: Risk Adjustment
Modeling to
Predict Patients Remaining High-Cost," eGEMs (Generating
Evidence & Methods to improve patient outcomes): Vol. 5: Iss.
2, Article 4.
DOI: https://doi.org/10.13063/2327-9214.1279
Available at: http://repository.edm-forum.org/egems/vol5/iss2/4
Reducing Healthcare Costs Through Patient Targeting: Risk
Adjustment
Modeling to Predict Patients Remaining High-Cost
Abstract
Context: The transition to population health management has
changed the healthcare landscape to identify
high risk, high cost patients. Various measures of patient risk
have attempted to identify likely candidates for
care management programs. Pre-screening patients for outreach
has often required several years of data.
Intermountain Healthcare relied on cost-ranking algorithms
which had limited predictive ability. A new risk-
adjusted algorithm shows improvements in predicting patients’
future cost status to facilitate identifying
patient eligibility for care management.
Case Description: A retrospective cohort study design was used
to evaluate high-cost patient status for two
of the next three years. Modeling was developed using logistic
regression and tested against other decision tree
methods. Key variables included those readily available in
electronic health records supplemented by
additional clinical data and estimates of socio-economic status.

Findings: The risk-adjusted modeling correctly identified 79.0%
of patients ranking among the top 15% of
costs in one of the next three years. In addition, it correctly
estimated 48.1% of the patients in the top 15% cost
group in two of the next three years. This method identified
patients with higher medical costs and more
comorbid conditions than previous cost-ranking methods.
Major Themes: This approach improves the predictive accuracy
of identifying high cost patients in the future
and increases the sensitivity of identifying at-risk patients. It
also shortened data requirements to identify
eligibility criteria for case management interventions.
Conclusion: Risk-adjustment modeling may improve
management programs’ interface with patients thus
decreasing costs. This method may be generalized to other
healthcare settings.
Acknowledgements
Acknowledgements: The authors would like to thank Andy
Merrill, MS for his contributions.
Keywords
Value/Cost, Care Coordination, Population Health
Disciplines
Other Medicine and Health Sciences | Social Statistics
Creative Commons License
This work is licensed under a Creative Commons Attribution-
Noncommercial-No Derivative Works 3.0
License.

This case study is available at EDM Forum Community:
http://repository.edm-forum.org/egems/vol5/iss2/4
Reducing Health Care Costs Through Patient
Targeting: Risk Adjustment Modeling to
Predict Patients Remaining High Cost
Jonathan Wrathall, PhD;i Tom Belnap, MS
iIntermountain Healthcare
Context: The transition to population health management has
changed the healthcare landscape to
identify high risk, high cost patients. Various measures of
patient risk have attempted to identify likely
candidates for care management programs. Pre-screening
patients for outreach has often required
several years of data. Intermountain Healthcare relied on cost-
ranking algorithms which had limited
predictive ability. A new risk-adjusted algorithm shows
improvements in predicting patients’ future cost
status to facilitate identifying patient eligibility for care
management.
Case Description: A retrospective cohort study design was used
to evaluate high-cost patient status
for two of the next three years. Modeling was developed using
logistic regression and tested against

other decision tree methods. Key variables included those
readily available in electronic health records
supplemented by additional clinical data and estimates of socio-
economic status.
Findings:
and more comorbid conditions than previous cost-ranking
methods.
Major Themes: This approach improves the predictive accuracy
of identifying high cost patients in the
future and increases the sensitivity of identifying at-risk
patients. It also shortened data requirements to
identify eligibility criteria for case management interventions.
Conclusion: Risk-adjustment modeling may improve
management programs’ interface with patients
thus decreasing costs. This method may be generalized to other
healthcare settings.
ABSTRACT
Generating Evidence & Methods
to improve patient outcomes
eGEMs
1
Wrathall and Belnap: Predicting Patients Remaining High-Cost

Published by EDM Forum Community, 2017
Introduction
In the face of rising health care costs, many voices
within the health care industry have called for
changes toward a more sustainable approach to
health care with emphasis on population health
management.1 In this paper, we describe modeling
techniques used to improve identification of high-
cost patients likely to benefit from care management
interventions. The modeling techniques described
below do not require a resource intensive approach
and may provide a means for other health systems
to improve their own patient-intervention targeting.
One objective of population health management
at Intermountain Healthcare is to facilitate the
transition from a traditional “fee-for-service” system
that compensates providers for services rendered,

with a “fee-for-value” approach in which providers
promote health among a defined patient cohort.
This approach emphasizes improving outcomes and
quality of service, and lowering overall health care
costs.2 This new health care climate requires changes
to existing delivery systems in order to meet the
needs of the community in ways that focus on the
triple aim of improving the experience of care, the
health of the population, and the cost of health care.3
Case Description
Intermountain Healthcare is an integrated delivery
system based in Salt Lake City, Utah consisting of 22
hospitals and over 185 clinics. Intermountain has been
actively engaged in developing programs designed
to improve outcomes for defined patient populations
that may require additional resources beyond the
standard of care provided through a patient-centered
medical home. One of these programs, known as

Community Care Management (CCM), is designed
to provide high intensity care management to high-
cost, complex patients. This program is designed
to help patients navigate the health care system
with the goal of preventing avoidable utilization
and slowing the progression of chronic conditions.
The CCM teams specialize in in-home assessments,
interdisciplinary care, intensive care coordination,
and community integration. This program was
designed to decrease catastrophic health episodes
through patient education, disease management, and
connection to community resources. To accomplish
this, CCM teams are expected to improve the
timeliness of care, improve medical coordination
to reduce complications, and foster community
relationships. These initiatives are intended to
decrease overall health care costs primarily through
avoiding unnecessary care or overutilizations.

In order for CCM programs to be successful, it
is critical to identify and target the right patient
population. To accomplish this, the stakeholders
originally created a list of eligible patients via
a ranking methodology, or Rank Algorithm,
centered on reasonably simple inclusion criteria.
In order to be eligible for the program, patients
must be at least 18 years old, live within 30 miles
of the program location, not already be enrolled
in a care management program, be insured by
Intermountain’s affiliated health plan or be uninsured,
and have health care costs in the top 10 percent of
patients for the last year and in the top 15 percent
of patients in one of the preceding two years.
Patients meeting the inclusion criteria were then
ranked equally based on the four following inclusive
factors; prior year health care spending, the Charlson
Comorbidity Index Score,4 and two proprietary risk

scores available within the organization—the IndiGO
Expected Benefit Score5 and the Optum Prospective
Risk Score6. Patients were ranked independently
by each factor, then rankings were averaged across
the factors to get an overall rank. The patient with
the lowest overall score was prioritized first, and the
CCM staff was expected to invite patients into the
program based on the order of the prioritized list.
2
eGEMs (Generating Evidence & Methods to improve patient
outcomes), Vol. 5 [2017], Iss. 2, Art. 4
DOI: 10.13063/2327-9214.1279
Volume 5
The goal of this approach was to provide an
objective enrollment process that was likely to enroll
patients who would both benefit from the program
and have enough cost savings potential to make the

program viable. While the original approach was
largely based on past health care spending, it did
provide an objective approach to enrolling patient in
the CCM program. These elements were used to rank
patients based on historical data in order to guide
patient outreach in the upcoming year. As a result,
there were limitations to the Rank Algorithm that
became apparent in the program over time.
The implementation team worked closely with the
CCM clinical staff to implement the use of the Rank
Algorithm. Over time there was ongoing feedback
and refinement to the tool in order to ensure it was
meeting the program’s needs. The Rank Algorithm
resulted in clinical staff taking significant time
to review patient charts and appraise potential
candidates. Many patients were considered ineligible,
they declined to participate or their high cost
episodes had resolved. As a result, there was a need

to revisit the approach and methods used to identify
patients and put in place something that better
identified patients for the CCM program.
The team undertook an evaluation of the original
patient selection process and tried to identify how
the process had been used and how it could be
improved moving forward. This evaluation identified
several drawbacks to the ranking method, which held
two important consequences. First, retrospective
patient identification meant the system was less able
to introduce appropriate health care interventions
until after a health crisis, thus patients were able
to be candidates for care management only when
they had already experienced an acute episode.
Second, a retrospective targeting method required
significant time before patients accumulated enough
utilization and cost to be identified as eligible
for additional services. Additionally, this ranking

method relied somewhat on opaque, third-party
proprietary algorithms to establish clinical risk. These
algorithms could not be calculated on all patients
and were difficult for the clinical staff to interpret.
Going forward, a predictive algorithm was needed
to identify rising risk patients before they became
medically complex and high cost. To accomplish this,
a new algorithm has been developed to incorporate
an approach that better predicts future patient costs
and refines patient targeting. With these changes,
there is an increasing ability to identify at-risk
patients and to better engage them in their care.
Recent discussions of high-cost patient prediction
have included debate as to the importance of
administrative or clinical data sources.7 As part
of the recommendations made by Cucciare et al.,
the revised prediction methodology was modified
to take advantage of gains introduced by both

administrative and clinical data. In recent years,
high-cost patient prediction has increasingly
included an element of prior years’ cost data as a
means of predicting future patient costs. Doing so
leads to better predictions than those obtained by
patient demographics alone.8,9 Alternative studies
have shown that a combination of clinical and
demographic data has also proved useful as a means
to predict future patient costs.10,11,12,13
A retrospective cohort study design was used with
logistic regression to evaluate high-cost patient
status for two of the next three years, and was
termed the “Logistic Model.” The study sample
consisted of patients in the top 15 percent of health
care costs from January 1 to December 31, 2011
comprising 26,173 unique patients. Training data
consisted of a random selection of 75 percent of
the total sample while the remainder were reserved

for the test data set. Because of the emphasis on
patient enrollment in a Care Management program,
similar inclusion criteria were adopted from the Rank
Algorithm that included living adults over age 18,
3
patients not already enrolled in a care management
program, uninsured patients or those covered
by Intermountain Healthcare’s insurance arm,
SelectHealth, and patients living within 30 miles of a
care management clinic. SelectHealth customers and
the uninsured were included as a group of patients
for which Intermountain Healthcare has assumed
financial risk.
Health care costs for the study excluded
chemotherapy, dialysis, intravenous (IV) therapy,

spinal fusion, and knee and hip replacement.
However, patients with these procedures could
still be included if they had significant health
care costs in other areas. It was determined that
these conditions could not be impacted by the
interventions provided by care management teams.
Key predictors used in logistic regression modeling
included age with gender and marital status derived
from patient records. Socioeconomic factors
included Average Household Income in the patient
ZIP code based on the 2010 U.S. Census and the
Area Deprivation Index (ADI) score in the patient
Census block.14 Dummy variables were used for ADI
values greater than 115. Supplementary indicators
were used for behavioral health conditions,
additional comorbidities including obstructive sleep
apnea, morbid obesity, coronary artery disease,
hyperlipidemia, hypertension, and the count of

Charlson Comorbidities.15,16 Charlson Comorbidities
and behavioral health conditions included in the
analysis are shown in Table 1. Summary statistics
on the training sample are included in Table 2. All
analyses were performed using R software for
statistical modeling and computing.17
Table 1. Charlson Comorbidities and Behavioral Health
Conditions Included In Logistic Regression
Modeling
CHARLSON COMORBIDITIES BEHAVIORAL HEALTH
CONDITIONS
Myocardial Infarction
Cancer
Connective Tissue Disease-Rheumatic Disease
Chronic Pulmonary Disease
Cerebrovascular Disease
Metastatic Carcinoma
Dementia
Moderate or Severe Liver Disease

Diabetes with complications
Diabetes without complications
Mild Liver Disease
Peripheral Vascular Disease
AIDS/HIV
Peptic Ulcer Disease
Congestive Heart Failure
Renal Disease
Paraplegia and Hemiplegia
Schizophrenic Disorders
Depression Disorders
Bipolar Disorders
Affective Disorders
Organic Psychotic Conditions
Nonorganic Psychoses
Neurotic Disorders
Personality Disorders
Alcohol/Drug Dependence

Eating Disorders
Childhood/Adolescence Disorders
Intellectual Disability
4
DOI: 10.13063/2327-9214.1279
Volume 5
Findings
The results presented here were aimed at predicting
the likelihood of a patient being in the highest 15th
percentile for costs in two of the next three years
for patients already in the top 15th percentile in the
last year as described in the Logistic Model. Many of
these metrics for the number of identified patients
are somewhat arbitrary. If we identify the likelihood
of being a high-cost patient in the future, there

could be a range of optimal likelihoods that could be
used. For example, determining the likelihood of a
50 percent chance of being in the top 15 percent of
costs in the next two years would result in a larger
patient cohort compared to those with a 95 percent
chance. Due to constraints of the CCM teams, the
number of manageable patients was estimated to
be about 2,000. These results reflect an optimal
match between the predicted likelihood of patient
targeting and the number of patients with whom
CCM teams might intervene. We report the results of
targeting patients with likelihood score greater than
0.85 based on the fitted population being in the top
15th percentile of high-cost patients in two of the next
three years. The original Rank Algorithm utilized by
CCM predicted 63 percent of patients from the prior
year would remain in the top 15th cost percentile for
one of the next three years. Using logistic regression

and additional sociodemographic covariates, the
Logistic Model increased the predicted likelihood
from 63 to 79 percent. Additionally, the Logistic
Model demonstrated increases in the predicted
likelihood of prescreening patients remaining in
the top 15th percentile of cost for two of the next
three years from 31 to 48 percent. The C-statistic,
representing the “goodness of fit” of each model,
also increased from .54 under the ranking model to
.71 using logistic regression. Estimates indicate the
patient cohort overlap to be less than 10 percent
between the two models. Additional results of
patient targeting methods are presented in Table
3. The Logistic Model shows gains in identifying
medically complex patients, namely among those
with additional chronic comorbidities, behavioral
health conditions, obesity, and hypertension.
Alternative validation analyses were also conducted

using decision tree methods including Classification
and Regression Tree (CART) and Random Forest
methodologies. CART is built on logical if-then
Table 2. Summary Statistics of Training Sample
VARIABLE PERCENT VARIABLE MEAN (SD)
Percent Female 66.34% Comorbidity
Count
1.29 (1.26)
Percent White 92.48% Age 46.8 (14.9)
Percent Married 71.21% Mean ZIP Income $63,311 ($16,372)
Percent with Hypertension 40.35% Prior Year Health
Care Costs
$13,213 ($16,711)
Percent with Obesity 30.22%
Percent with Behavioral Health Condition 55.95%
Percent with Area Deprivation Index > 115 10.57%
5

conditions that partition data based on different
predictors. Predictions in CART are based on
stratifying the predictor space into regions and
making predictions based on the mean of the
total observations in each region. Random Forest
methodology utilizes bootstrapping to stabilize
the pathways of possible alternative outcomes. For
this test, the number of bootstrap iterations was
500. Both methods are considered alternatives to
regression methods in tuning variable importance
and selection used in predictive modeling. However,
neither test was found to increase the C-statistic of
outcome prediction.
Major Themes
In the search for sustainable health care, many
health care systems are turning to data for help
in understanding the health of their population.

The approaches used here demonstrate gains
in identifying the patients most likely to benefit
from patient intervention programs. The Logistic
Model described above relies primarily on patient
demographics, including the socioeconomic context
of the patient and patient health care cost in the last
year, to predict the future likelihood of being a high-
cost patient in two of the next three years. We claim
that the use of widely available patient demographic
information in combination with rudimentary clinical
data may be more predictive of high-cost patients
beyond alternative ranking methods such as the
Rank Algorithm, which rely on lengthy accumulated
cost history and third-party clinical risk-adjustment
indices.
Because of the cyclical nature of care episodes,
many high-cost patients will have decreasing health
care spending over time. As episodes resolve, there

is significant “regression to the mean” that occurs
within this patient population. Consequently it
becomes increasingly important to identify the
subset of the population that is likely to remain
high cost in the future. The Rank Algorithm relied
too heavily on past cost and was not designed
to effectively predict future health care spending
beyond relying on past trends. Since the Logistic
Model has been implemented, CCM clinic staff have
become more efficient in selecting the right patients,
Table 3. Results of Patient Targeting Methods
MEASURE RANK ALGORITHM LOGISTIC MODEL
Average, SD Prior Year Cost $38,700 ($27,256) $44,000
($61,125)
Average, SD Number of Charlson Comorbidities 3.6 (1.9) 5.0
(2.4)
Average, SD Number of Behavioral Health Conditions 1.7 (1.3)
2.2 (1.84)
Average, SD Number of Other Comorbidities 1.4 (.98) 2.3 (1.2)

Percent of Patients with Area Deprivation Index > 115
(Top Quintile)
16.9 18.0
Percent of Patients Diagnosed With Behavioral
Health Condition
63.2 82.8
Percent of Patients Diagnosed With Obesity 27.8 54.9
Percent of Patients Diagnosed With Hypertension 59.3 80.3
6
DOI: 10.13063/2327-9214.1279
Volume 5
which has resulted in a reduced overall burden of
vetting patients.
Additionally, the gains from a regression-based
patient targeting model provide the advantage
that engagement with future high-risk patients

could occur in multiple ways. For example, patient
outreach could happen at the point of care, in
proactive outreach settings such as the CCM case
setting described above, or by delivery systems or
payers with access to the necessary data used in the
statistical modeling itself. These data are relatively
common to most electronic medical record systems
and reduce the data requirements from three years
to one year of retrospective patient history. Using
one year of data to make predictions is beneficial
because it allows systems to more accurately target
the segment of the at-risk population most likely to
benefit from additional services and support. More
precise allocation of services can reduce waste and
improve access to care, which is particularly valuable
throughout the population health transition many
health systems are currently facing. Conversely,
in the era of “big data” there may be common

acceptance of the assumption that more data is
better to use in predicting overall health outcomes.
In this instance, health systems struggling to
make use of emergent data systems need not feel
overwhelmed by a lack of large or highly fine-tuned
data systems. Our Logistic Model was developed on
relatively few predictors on open-sourced software.
Furthermore, we found, at least for the time being,
that regression tree methods that rely on large data
sets were less effective in obtaining greater modeling
accuracy than traditional regression methods.
This study has several limitations. First, we claim
to have increased the ability to target high-
cost patients by using predictive methods over
a rudimentary ranking system in the pursuit of
reducing health care costs and improving patient
outcomes. We do not claim to show that predictive
methods can account for all these changes. Because

the study relied upon retrospective data for the
use of future cost prediction, we merely speak to
methodological updates in patient identification
and leave additional research to quantify how much
downstream interventions may be able to reduce
costs. Second, this modeling may not account for
all the health conditions that may cause patients
to be high risk in the future. The approach shown
here represents a parsimonious prediction strategy,
having compared multiple predictor variables and
methods. Due to its parsimony, the Logistic Model
may prove to be a useful starting point for alternative
health care systems to engage in their own high-
cost patient targeting intervention strategies.
However, data training and testing was performed
on a sample of patients with relatively homogenous
demographics living in the intermountain western
United States. This sample may represent a patient

population with inherently different risk factors
and health care needs than patients in another
geographic location. While the Logistic method was
not explicitly tested against the IndiGO or Optum
indices directly, the lower performance of both
indices combined as included in the Rank Algorithm
did not warrant additional independent testing. The
unanticipated finding surrounding the limited utility
of third-party algorithms underscores the need for
health systems pursuing population health initiatives
to be sensitive to the unique characteristics of their
population. In the present study, we found that
third-party predictive algorithms trained on other
populations were less helpful than training data on
our own population.
Conclusion
Many strategies have been implemented in the
search for health care delivery strategies that help

patients manage illness and reduce waste. High-
cost patient targeting can aid care management
teams to effectively focus their efforts on those
7
in the most need of intervention. Compared to
alternative modeling techniques, our Logistic Model,
based on administrative and basic socioeconomic
context data as well as information on chronic
health conditions, increases the predictive ability to
target at-risk patients. Using this model can shorten
the time requirements to identify patients who
are most likely to benefit from case management
interventions, thus decreasing cost burdens to
hospitals and patients alike. It is possible that this
approach may prove helpful to other health care

settings seeking to establish patient intervention
programs of their own.
Acknowledgements
The authors would like to thank Andy Merrill, MS for
his contributions.
References
1. Washington A, Coye MJ, Boulware L. Academic health
systems’ third curve: Population health improvement. JAMA.
2016; 315(5): p. 459-460.
2. Intermountain Healthcare. Shared Accountability and
Population Health. Salt Lake City: Intermountain Healthcare,
Decision Support; 2016.
3. Berwick DM, Nolan TW, Whittington J. The Triple Aim:
Care,
Health, And Cost. Health Affairs. 2008 May; 27(3): p. 759-769.
4. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new
method of classifying prognostic comorbidity in longitudinal
studies: Development and validation. Journal of Chronic
Diseases. 1987; 40(5): p. 373-383.
5. Eddy DM, Adler J, Morris M. The ‘Global Outcomes Score’:
a
quality measure, based on health outcomes, that compares
current care to a target level of care. Health Affairs. 2012 Nov;
31(11): p. 2441-2450.
6. Dunn DL, Celebi D. System and method for assessing

healthcare risks. 2015 Jun 16. US Patent 9,058,629.
7. Cucciare MA, O’Donohue W. Predicting future healthcare
costs: how well does risk-adjustment work? Journal of health
organization and management. 2006: p. 150-162.
8. Chechulin Y, Nazerian A, Rais S, Malikov K. Predicting
patients
with high risk of becoming high-cost healthcare users in
Ontario (Canada). Health Policy. 2014 Feb; 9(3): p. 68-79.
9. Lieberthal RD. Analyzing the Health Care Cost Curve: A
Case
Study. Population Health Management. 2013 May; 16(5): p.
341-
348.
10. Nyce S, Grossmeier J, Anderson DR, Terry PE, Kelley B.
Association between changes in health risk status and
changes in future health care costs: a multiemployer study.
Journal of Occupational and Environmental Medicine. 2012
Nov: p. 1364-73.
11. Fishman PA, Goodman MJ, Hornbrook MC, Meenan RT,
Bachman DJ, O’Keeffe Rosetti MC. Risk Adjustment Using
Automated Ambulatory Pharmacy Data: The RxRisk Model.
Medical Care. 2003; 41(1).
12. Otani K, Baden WW. Healthcare Cost and Predictive
Factors:
High- and Low-Utilization Model Development. Health
Marketing Quarterly. 2009 Oct; 26(3): p. 198-208.
13. Moturu ST, Johnson WG, Liu H. Predictive risk modelling
for
forecasting high-cost patients: a real-world application using

Medicaid data. Int. J. Biomedical Engineering and Technology.
2010: p. 114-132.
14. Singh GK. Area Deprivation and Widening Inequalities in
US
Mortality, 1969-1998. Am J Public Health. 2003 Jul; 93(7): p.
1137-1143.
15. Sundararajan V, Quan H, Halfon P, Fushimi K, Luthi JC,
Burnand
B, et al. Cross-National Comparative Performance of Three
Versions of the ICD-10 Charlson Index. Medical Care. 2007;
45(12): p. 1210-1215.
16. Sundararajan V, Henderson T, Perry C, Muggivan A, Quan
H,
Ghali WA. New ICD-10 version of the Charlson comorbidity
index predicted in-hospital mortality. Journal of Clinical
Epidemiology. 2004 Dec; 57(12): p. 1288-1294.
17. R Core Team. R: A Language and Environment for
Statistical
Computing Vienna, Austria: R Foundation for Statistical
Computing; 2015.
8
DOI: 10.13063/2327-9214.1279

RESEARCH ARTICLE Open Access
A new instrument to measure high value,
cost-conscious care attitudes among
healthcare stakeholders: development of
the MHAQ
Serge B. R. Mordang1* , Karen D. Könings1, Andrea N. Leep
Hunderfund2, Aggie T. G. Paulus3,
Frank W. J. M. Smeenk1,4 and Laurents P. S. Stassen1,5
Abstract
Background: Residents have to learn to provide high value,
cost-conscious care (HVCCC) to counter the trend of
excessive healthcare costs. Their learning is impacted by
individuals from different stakeholder groups within the
workplace environment. These individuals’ attitudes toward
HVCCC may influence how and what residents learn.
This study was carried out to develop an instrument to reliably
measure HVCCC attitudes among residents, staff
physicians, administrators, and patients. The instrument can be
used to assess the residency-training environment.
Method: The Maastricht HVCCC Attitude Questionnaire
(MHAQ) was developed in four phases. First, we conducted
exploratory factor analyses using original data from a
previously published survey. Next, we added nine items to
strengthen subscales and tested the new questionnaire among
the four stakeholder groups. We used exploratory
factor analysis and Cronbach’s alphas to define subscales, after
which the final version of the MHAQ was
constructed. Finally, we used generalizability theory to
determine the number of respondents (residents or staff
physicians) needed to reliably measure a specialty attitude
score.

Results: Initial factor analysis identified three subscales.
Thereafter, 301 residents, 297 staff physicians, 53
administrators and 792 patients completed the new
questionnaire between June 2017 and July 2018. The best
fitting subscale composition was a three-factor model.
Subscales were defined as high-value care, cost incorporation,
and perceived drawbacks. Cronbach’s alphas were between 0.61
and 0.82 for all stakeholders on all subscales.
Sufficient reliability for assessing national specialty attitude
(G-coefficient > 0.6) could be achieved from 14
respondents.
Conclusions: The MHAQ reliably measures individual attitudes
toward HVCCC in different stakeholders in health
care contexts. It addresses key dimensions of HVCCC,
providing content validity evidence. The MHAQ can be used
to identify frontrunners of HVCCC, pinpoint aspects of
residency training that need improvement, and benchmark
and compare across specialties, hospitals and regions.
Keywords: High-value cost-conscious care, Attitudes,
Instrument development, Learning environment, Post-
graduate medical training
© The Author(s). 2020 Open Access This article is distributed
under the terms of the Creative Commons Attribution 4.0
International License
(http://creativecommons.org/licenses/by/4.0/), which permits
unrestricted use, distribution, and
reproduction in any medium, provided you give appropriate
credit to the original author(s) and the source, provide a link to
the Creative Commons license, and indicate if changes were
made. The Creative Commons Public Domain Dedication waiver
(http://creativecommons.org/publicdomain/zero/1.0/) applies to
the data made available in this article, unless otherwise stated.

* Correspondence: [email protected]
1Department of Educational Development and Research, School
of Health
Professions Education, Maastricht University, P. O. Box 616,
6200 MD,
Universiteitssingel 60, 6229, ER, Maastricht, the Netherlands
Full list of author information is available at the end of the
article
Mordang et al. BMC Health Services Research (2020)
20:156
https://doi.org/10.1186/s12913-020-4979-z
Background
Providing high value, cost-conscious care (HVCCC)
is critical to improve the value of health care and at
the same time counter rising costs, eliminate wasted
spending, and reduce overuse (provision of health-
care services with no medical basis or for which
harms equal or exceed benefit) [1–5]. Value in this
context can be understood as quality divided by cost
over time [6]. Cost-conscious refers to the aware-
ness an individual has on the specific expenses and
cost-effectiveness of an intervention, as well as
negative consequences as a result of providing – or
not providing - an intervention, like patient dissatis-
faction [7, 8]. Providing HVCCC requires physicians
to balance the potential benefits and harms of a test
or treatment, while simultaneously considering costs
and possible drawbacks [7]. Physician practice pat-
terns influence the number and type of healthcare
services patients receive [9]. The post-graduate
training appears to be particularly formative in
shaping residents’ current and future behaviors re-

lated to high-value care, such as during exposure to
faculty discussions on patient care [10]. Medical
education thus has an obligation to ensure that
stakeholders within the post-graduate learning en-
vironment support the development of HVCCC
practice patterns [11–17].
Learning environments are complex, involving per-
sonal, social, organizational, physical, and virtual compo-
nents [18]. Multiple individuals from different
stakeholder groups contribute to the creation of work-
place environments, and the attitudes of these individ-
uals may influence an organizations’ culture regarding
how and what residents learn [19–23]. Attitudes are also
important (albeit imperfect) predictors of individual be-
havior [24], as evidenced by multiple studies showing as-
sociations between physician attitudes and beliefs and
their utilization of healthcare services [25–28]. Under-
standing the attitudes of key stakeholders thus has the
potential to offer valuable insights into the post-graduate
training environment [29], but there is a scarcity of reli-
able tools to measure individual attitudes on all dimen-
sions of HVCCC.
In post-graduate medical training, staff physicians, ad-
ministrators and patients shape residents’ recognition
and understanding of HVCCC’s necessity [15, 17, 30–
32]. While different stakeholders can have different pref-
erences regarding the provision of HVCCC, measuring
all stakeholders’ attitudes can give insight in the resi-
dent’s workplace environment regarding the different di-
mensions of providing HVCCC. Prior studies have tried
to measure the attitudes of particular stakeholder groups
with respect to specific dimensions of HVCCC [8, 10,
23, 32–39]. However, a single reliable instrument to

measure the individual attitudes of all these stakeholder
groups toward multiple dimensions of providing
HVCCC has not yet been developed. Such an instrument
could both assess attitudes at the individual level and
compare attitudes between stakeholders on distinct di-
mensions. It also enables comparisons among different
units, organizations, and specialties on the dimensions of
providing HVCCC.
This study aims to a) develop an instrument, the
Maastricht HVCCC-Attitudes Questionnaire (MHAQ),
to measure resident, staff physician, administrator and
patient attitudes toward HVCCC and b) determine,
using generalizability (G) theory [40], how many respon-
dents are needed to reliably measure a specialty attitude
score on a national level.
Method
We reviewed the literature to identify existing instru-
ments for assessing individual attitudes toward HVCCC.
From these, we selected items from the questionnaire
used by Leep Hunderfund et al. [36] in their study of
medical student attitudes toward cost-conscious care.
These items were based on previously published surveys
of practicing physicians and focus groups interviews with
physicians, who gave input and suggestions on the items,
as well as on reviews of the literature on cost-conscious
care with input from various field experts [8, 33–35],
supporting its content validity [41]. For more details on
the development of the items, see the study by Leep
Hunderfund et al. [36]. However, the concept of
HVCCC consists of three key dimensions. Next to cost-
conscious care and potential drawbacks, containing both
the direct cost-effectiveness and downstream conse-
quences of including cost-effectiveness, also the
provision of value needs to be addressed [7]. Further-

more, because results were reported on an item level,
underlying constructs needed to be explored in order to
methodologically interpret and compare results of differ-
ent stakeholders.
We developed the MHAQ through a four-phase
process (Fig. 1):
1) Investigating subscales of cost-conscious care, using
items and original data from the survey conducted
by Leep Hunderfund, et al. [36].
2) Adding items, which include the value dimension, to
strengthen subscales, and adapting items for use by
residents, staff physicians, administrators, and patients.
3) Testing items among four samples of these
stakeholders and developing the final version of the
MHAQ.
4) Assessing the number of respondents per specialty
on a national level needed to reliably measure a
specialty attitude score through generalizability
analysis.
20:156 Page 2 of 10
Phase 1: investigating subscales
Questionnaire and data
We used items from the aforementioned published sur-
vey of U.S. medical students as the starting point for
questionnaire development, as this survey derived their
21 items assessing individual attitudes toward cost-

conscious care, on recently published surveys for prac-
ticing physicians [36]. The authors used a four-point
Likert scale (1 = strongly disagree to 4 = strongly agree).
Analysis
Since we developed a new scale without having a priori
hypotheses about the structure of the variables, we used
exploratory factor analysis (principle component ana-
lysis, PCA) to examine the structure of these 21 survey
items and to define subscales. PCA maximizes explained
variance of the items [42] and is considered suitable
when examining new constructs [43, 44]. Varimax rota-
tion was performed to maximize spread of all factors,
resulting in better interpretable factors [42]. We used a
parallel analysis, the Kaiser Guttman criterion (eigen-
values > 1) and inspection of the scree plot, to identify
the optimal number of factors [45]. We tested internal-
consistency reliability of constructs using Cronbach’s
alpha [46].
Phase 2: preparing the MHAQ
Additional items
Based on the internal-consistency reliability of identified
subscales (which were around 0.6) and to tailor the
MHAQ to new stakeholders and a new context, we
added nine items to the original questionnaire. Because
the initial 21 items focused primarily on costs, new items
focused on value (e.g., risks and benefits of treatment,
consideration of patient values) given the importance of
value in HVCCC. These items were based on items de-
scribed in the context of validated surveys on high-value
originating from experts in the field [10, 23, 39, 47].
Fig. 1 Overview of the four-phase process to develop the
MHAQ

20:156 Page 3 of 10
Different stakeholders
We developed a parallel questionnaire for medical resi-
dents, staff physicians and administrators. Items for pa-
tients were identical in content, but formulated for a lay
audience. Additionally, we added a fifth answering op-
tion (‘I don’t know’) for patients, to prevent random an-
swering when questions were not well understood.
These items were pilot-tested with 56 patients in 4 cy-
cles to refine formulations.
Different context
For usage in a Dutch context, we translated all items
into Dutch. A professional translator translated all items
back into English to evaluate similarity between the ori-
ginal source and translated items [48].
Phase 3: administering the MHAQ and developing the
final version
Data collection
To recruit respondents, we approached hospital educa-
tional committees from all academic training regions
(n = 8) in the Netherlands. Willing members of the hos-
pital educational committees recruited medical residents
and staff physicians to participate in the study. Addition-
ally, we approached residents and staff physicians
through the periodic newsletter of the ‘Bewustzijnspro-
ject’, a Dutch project promoting HVCCC on a national
level. The last authors (F.S. and L.S.) approached admin-
istrators (policy and/or financial) in several hospitals.
We approached patients before and after patient con-

sults, after gaining (ethical) approval by the relevant hos-
pital and the physician in charge of the department, and
via several patient platforms. We sent all invitations to
complete the MHAQ between June 2017 and July 2018.
Participants received an information letter, after which
they signed an informed consent form before answering
the questionnaire. Medical residents, staff physicians and
administrators filled out the questionnaire online via
Qualtrics, a survey software program. Patients also had
the option to answer the questionnaire on hardcopy.
Analysis
We analyzed data following the same procedure as in
Phase 1. We analyzed data from all stakeholder groups
separately, after which an optimal solution was deter-
mined through a parallel analysis, as well as examination
of each of the scree-plots and the Kaiser-Guttman criter-
ion, followed by an inspection of the factor loadings. We
calculated internal consistency reliability of constructs
separately for all subscales and all stakeholders using
Cronbach’s alpha. Since we developed new scales, a
Cronbach’s alpha > 0.6 was considered acceptable [49].
Phase 4: generalizability analysis
We conducted a generalizability analysis [50] to assess
the number of respondents needed to reliably measure a
shared attitude score toward HVCCC of residents and
staff physicians by specialty on a national level. We used
Levene’s homogeneity tests to determine equal variances
between specialties of different hospitals. In terms of
generalizability theory, we performed a single facet ana-
lysis with attitude scores nested within specialties. We
carried out a variance component analysis, using spe-
cialty as random factor and attitude score as dependent
factor. We estimated the variance associated with spe-
cialties and the variance of attitude scores nested within

specialties using the following formula:
G ¼ Vs
Vsþ Vp : s
Np
in which Vs is the associated variance of specialties, Vp:s
is the associated variance of a participants’ attitude score
within specialties, and Np is the number of participants
attitude scores. We used results from G-study variance
components to estimate SEM and conduct D-studies to
project reliability estimates for varying numbers of re-
spondents. For feasibility, we accepted a G-coefficient
greater than 0.6 [50]. All data were analyzed using IBM
SPSS statistics for Windows, version 25.0 (Armonk, NY:
IBM Corp.).
Results
Phase 1
The dataset from the published study on cost-conscious
care included responses from students at 10 medical
schools geographically distributed across the U.S.. Nine
of these schools granted permission to use de-identified
data from their students for the purposes of this study
(3195 responses of 5992 total students surveyed). No
student identifiers were collected and we removed
school identifiers prior to sharing. Results of PCA indi-
cated a three subscale-model. All factors had eigenvalues
above 1.5. The first subscale contained five items about
the responsibility of physicians to provide/promote
HVCCC (Table 1); the second subscale contained five
items about the relationship of physicians and patients
when implementing HVCCC; the final subscale con-
tained four items about considering costs in clinical de-
cision making. Cronbach’s alphas of the subscales were

between 0.64 and 0.66. Seven items had factor loadings
< .4, representing a low communality for these items,
and were not included in these subscales. These items,
however, were still included in phases 2 and 3.
20:156 Page 4 of 10
Phase 2
Table 3 shows the nine new items we added in phase 2,
indicated with an asterisk. After translation into Dutch
language, content of the original source items and the
translated items was identical. The resulting question-
naires for all stakeholder groups contained 30 items, in-
cluding 21 items from the original questionnaire and
nine newly added items.
Phase 3
In total, 301 residents and 297 staff-physicians com-
pleted the MHAQ. Residents and staff physicians
worked in 31 different specialties and 32 hospitals, geo-
graphically distributed across the Netherlands. Fifty-
three administrators and 521 patients completed the
MHAQ. Administrators and patients came from five
hospitals in the South of the Netherlands (Table 2).
Data analyses
To develop a questionnaire that is applicable to mul-
tiple stakeholders in postgraduate medical education
and enables reliable comparisons between stake-
holders, grouping of items per subscale has to be the
same for all stakeholders. S.M. and K.K. determined a
best-fitting subscale composition for all stakeholders,

based on the inspection of factor structures for each
of the stakeholders. When compromises were neces-
sary, factor analyses of residents and staff-physicians
were prioritized when creating optimal subscales for
all stakeholders, since these groups are most central
Table 1 Original items per subscale
Survey item Cronbach’s alpha
Subscale 1 α = .65
Physician clinical practices (e.g., ordering, prescribing) are key
drivers of high health care costs.
Cost to society should be important in physician decisions to
use or not to use an intervention.
Cost-effectiveness data should be used to determine what
treatments are offered to patients.
Trying to contain costs is the responsibility of every physician.
Managing health care resources for all patients is compatible
with physicians’ obligation to serve individual patients.
Subscale 2 α = .64
Patients will be less satisfied with the care they receive from
physicians who discuss costs when choosing tests and
treatments.
Doctors are too busy to worry about the costs of tests and
procedures.
It is easier to order a test than to explain to the patient why a
particular test is unnecessary.
Practicing cost-conscious care will undermine patients’ trust in
physicians.
Ordering fewer tests and procedures will increase physicians’
risk of medical malpractice litigation.

Subscale 3 α = .66
Physicians should take a more prominent role in limiting use of
unnecessary tests.
Physicians should be aware of the costs of the tests or
treatments they recommend.
Physicians should talk to patients about the costs of care when
discussing treatment options.
Physicians should change their clinical practices (eg, ordering,
prescribing) if the cost of care they provide is higher than
colleagues
who care for similar patients.
Table 2 Demographics of each stakeholder group
Characteristics Residents Staff physicians Administrators
Patients
N respondents 301 297 53 521
N female respondents (%) 191 (65) 151 (51) 27 (51) 241 (46)
Age in years, Mean 30.6 45.9 51.7 59
Medical specialty (%) 296 (98.3) 295 (99.3) - -
Non-Surgical 172 (57.1) 166 (55.9) - -
Surgical 89 (29.6) 70 (23.6) - -
Supportive 35 (11.6) 59 (19.9) - -
Type of administrator (%)
Department administrator - - 17 (32.1) -

Division administrator - - 13 (24.5) -
Hospital administrator - Board level - - 7 (13.2) -
Other Administrator - - 16 (30.2) -
20:156 Page 5 of 10
in post-graduate medical training. The best-fitting
subscale composition for all stakeholders was a three-
factor model. All factors had eigenvalues above 1.
Four of five items of subscale 1 in phase 1 again clus-
tered on the same factor, together with three add-
itional items from the original subscale 3, as well as
two items that had a low factor loading in phase 1
and one new item. The four items of subscale 2 in
phase 1 again loaded all on the same factor. Three
new items also loaded on this factor. The remaining
item from subscale 3 loaded on a third factor, which
also included one item from subscale 1, two items
with low factor loadings in phase 1, and four new
items. Thus, eight of the nine items added in phase 2
strengthened the subscales. All items in phase 1 fo-
cused on cost-conscious care, but in phase 3 some of
these items loaded on high value care. This is due to
the content of these items, which do contain a cost
component, but are in essence statements on high value
care. Because in phase 1 high value care was not evaluated,
these items loaded in this phase on a different subscale. For
the final subscale composition, we optimized Cronbach’s al-
phas for each stakeholder group, considering all subscales
had to fit every stakeholder.

Final MHAQ
The aforementioned analyses resulted in 25 items distrib-
uted among three subscales, each covering an important di-
mension of HVCCC in clinical environments. We defined
the labels of subscales in our team of experts, based on the
main focus of the consisting items. Subscale 1, defined as
high-value care, contained eight items about physicians’
provision of high value care (Cronbach’s alphas ranging
from 0.61 for staff physicians to 0.77 for administrators).
Subscale 2, defined as cost incorporation, contained 10
items about the integration of healthcare costs in physi-
cians’ daily practice (Cronbach’s alphas ranging from 0.69
for staff physicians to 0.80 for patients). Subscale 3, defined
as perceived drawbacks, contained seven items about per-
ceived drawbacks of practicing HVCCC (Cronbach’s alphas
ranging from 0.67 for residents to 0.82 for patients).
Table 3 presents the final version of the MHAQ. (The
survey instrument is available as supplementary file.)
Phase 4
Generalizability
This reliability estimation was performed separately
for medical residents and staff physicians and for each
subscale. Levene’s homogeneity tests indicated equal
Table 3 An overview of the MHAQ, viewing all items per
subscale. (R) Reversed items.
Survey item Cronbach’s alpha
Residents Staff-physicians Administrators Patients
(1) High-value care α = .65 α = .61 α = .77 α = .67
Physicians should take a more prominent role in limiting use of
unnecessary tests.

The cost of a test or medication is only important if the patient
has to pay for it out of pocket. (R)
Managing health care resources for all patients is compatible
with physicians’ obligation to serve individual patients.
Eliminating unnecessary tests and procedures will improve
patient safety.
Physicians should consider a patient’s doubts and values in their
clinical decisions.a
Physicians should offer patients choices of care, taking
advantages, disadvantages and costs into account.a
Physicians should limit waste of care in their own
hospital/clinic.a
Physicians should have sufficient knowledge of the interplay
between advantages/disadvantages and costs of common tests.a
(2) Cost incorporation α = .71 α = .69 α = .74 α = .80
Physicians should try not to think about the cost to the health
care system when making treatment decisions. (R)
Physicians should be aware of the costs of the tests or
treatments they recommend.
Physicians should talk to patients about the costs of care when
discussing treatment options.
Physicians should change their clinical practices (e.g., ordering,
prescribing) if the costs of care they provide is higher than
colleagues who care for
similar patients.
Physician clinical practices (e.g., ordering, prescribing) are key
drivers of high health care costs.
Costs to society should be important in physician decisions to
use or not to use an intervention.
It is unfair to ask physicians to be cost-conscious and still keep
the welfare of their patients foremost in their minds. (R)

Cost-effectiveness data should be used to determine what
treatments are offered to patients.
Trying to contain costs is the responsibility of every physician.
Physicians should discuss cost efficiency of care with their
patients.a
(3) Perceived drawbacks α = .67 α = .70 α = .79 α = .82
Patients will be less satisfied with the care they receive from
physicians who discuss costs when choosing tests and
treatments.
Doctors are too busy to worry about the costs of tests and
procedures.
Practicing cost-conscious care will undermine patients’ trust in
physicians.
Ordering fewer tests and procedures will increase physicians’
risk of medical malpractice litigation.
Ordering more tests reduces a physicians’ diagnostic
uncertainty.a
Ordering fewer tests and procedures will lead to more
complications.a
Patients find it unpleasant to talk about costs of tests or
treatments.a
aNew items that were added in phase 2. The item “if a
physicians’ medical practices have a direct influence on a
physicians’ salary, it will obstruct a physicians’
cost-conscious care approach” did not cluster on any of the
subscales
20:156 Page 6 of 10

variances between specialties (e.g., cardiology, internal
medicine) across different hospitals. Results from D-
studies indicated the number of respondents needed
to reliably measure (G-score ≥ 0.6) residents’ attitude
score per specialty on a national level is 28 for the
subscale high value care, 52 for the subscale cost in-
corporation, and 15 for the subscale perceived draw-
backs. For staff physicians, the number of respondents
needed was respectively 14 for the subscale high value
care, 21 for the subscale cost incorporation, and 32
for the subscale perceived drawbacks. Figures 2 and 3
display an overview of the G-score per subscale for
residents and staff physicians.
Discussion
This study describes the development of the MHAQ and
provides reliability evidence supporting its use to measure
attitudes toward HVCCC among important stakeholders
in the post-graduate clinical learning environment. The
MHAQ assesses three key dimensions of HVCCC and
may be used to identify frontrunners who endorse
and prioritize HVCCC, to pinpoint aspects of HVCCC
Fig. 2 D-study projecting MHAQ reliability of resident
respondents. Note: value of 0.6 is considered reliable
Fig. 3 D-study projecting MHAQ reliability of staff physician
respondents. Note: value of 0.6 is considered reliable
20:156 Page 7 of 10

that need to be improved or changed to better sup-
port HVCCC in the post-graduate learning environ-
ment, and to facilitate comparisons among different
stakeholder groups, specialties, regions, and potentially
hospitals or departments. The MHAQ includes three
subscales relating to provision of high-value care (8
items), integration of costs (10 items), and perceived
drawbacks of HVCCC (7 items). These subscales en-
compass all key dimensions of providing HVCCC in
clinical practice [7], hence supporting the content val-
idity of MHAQ scores.
Scores on high-value care reflect the degree to
which individuals believe physicians should be respon-
sible for limiting unnecessary testing, reducing waste,
considering risks, benefits, and patient preferences
when making diagnostic or therapeutic intervention
decisions. High scores on this subscale can identify
proponents of HVCCC who believe physicians should
be frontrunners in the provision of high-value care.
When key individuals within the clinical learning en-
vironment advocate high-value care, corresponding
role modelling can help to shape future physicians’
HVCCC practice patterns [17, 30, 51].
Scores on cost incorporation reflect individual beliefs
about the degree to which physicians should integrate
costs in their daily clinical practice, for example when
making treatment decisions or when discussing options
with patients. Although physicians assume they contrib-
ute minimally to healthcare costs [35], they actually dir-
ect up to 87% of all healthcare spending [52]. Knowing
physicians’ view on the incorporation of costs in their
daily practice, together with patients’ view on the incorp-
oration of costs, can be important starting points for
transformation efforts to educate future physicians about

providing HVCCC [14].
Scores on perceived drawbacks reflect individual be-
liefs about potential drawbacks of HVCCC, like patient
dissatisfaction or risks of malpractice. Perceptions like
these are known barriers to the implementation of
HVCCC in practice [53] and drivers of unnecessary test-
ing [54]. When individuals within the same organization
have different perceptions of the drawbacks, incorpor-
ation of HVCCC in daily clinical practices is unsustain-
able. Pinpointing organizations as such could initiate
aligned education programs for all stakeholders in that
organization on the benefits of HVCCC, to create a
common understanding and support of the delivery of
HVCCC [17, 55].
Internal consistency reliability was sufficient for all
stakeholders on all subscales. The internal consistency re-
liability for subscale scores was lower for residents and
staff physicians than for patients and administrators. This
could suggest that residents and physicians have more nu-
anced views on the provision of high-value care,
integration of costs into clinical practice, and potential
drawbacks of HVCCC. Alternatively, items formulated for
a lay audience may be more evident in meaning and there-
fore clearer to answer than items used in the question-
naires for residents, staff physicians, and administrators.
The patient version of the MHAQ thus has the potential
to inform future improvement of subscale reliability for
other stakeholders when developing the MHAQ further.
The MHAQ can not only be used to measure attitudes
toward HVCCC at the individual level, but also to com-
pare attitudes among larger groups, e.g. specialties, hos-
pitals, regions. Our D-study results predict 14 to 52

respondents would be required to reliably assess
HVCCC attitudes among resident or staff physicians,
supporting the feasibility of group comparisons at the
national, specialty level.
Strengths and limitations
This study has certain strengths and limitations. First, the
MHAQ is based on a previously published questionnaire
informed by a literature review on HVCCC, which was
further enhanced through the addition of items (also
based on the literature) that emphasized value as an im-
portant dimension in addition to cost and drawbacks. Fu-
ture studies could provide additional content validity
evidence for MHAQ scores by presenting items to subject
matter experts, for example in a Delphi-study [56]. Sec-
ond, while we are the first, to our knowledge, to simultan-
eously survey resident, staff physician, administrator, and
patient attitudes toward HVCCC, our study did not in-
clude all potential stakeholders. Future studies could ex-
tend our work by including other relevant groups, such as
nurses and other allied health professionals, who contrib-
ute to the clinical learning environment. Third, we used
the same items in the U.S. and the Netherlands, which
strengthens the broad usability of the MHAQ. However,
healthcare delivery systems vary by country and MHAQ
items may not be equally applicable in all settings. Fourth,
while the final version of the MHAQ showed promising
reliabilities, and D-studies support the feasibility of reliable
assessments at the specialty level, there were too few re-
sults from a single department within a single hospital to
calculate a reliable G-score at the department level. Fur-
ther studies are needed to assess the number of respon-
dents needed for a reliable department-level attitude
score, which may most closely approximate the clinical
learning environment experience by residents.

Conclusion
The MHAQ is a new instrument capable of reliably
measuring attitudes toward HVCCC among individuals
within multiple relevant stakeholder groups - residents,
staff physicians, administrators, and patients - with sub-
scales that address key dimensions of HVCCC. The
20:156 Page 8 of 10
MHAQ can be used to identify frontrunners who en-
dorse and prioritize HVCCC, to pinpoint aspects of
HVCCC that need to improved or changed to better
support HVCCC in the post-graduate learning environ-
ment, and to facilitate comparisons among different
stakeholder groups, specialties, regions, and potentially
hospitals or departments.
Supplementary information
Supplementary information accompanies this paper at
https://doi.org/10.
1186/s12913-020-4979-z.
Additional file 1. The Maastricht HVCCC Attitude
Questionnaire (MHAQ).
Abbreviations
D-studies: Decision studies; G-coefficient: Generalizability
coefficient; G-
score: Generalizability score; G-studies: Generalizability
studies; HVCCC: High-
Value, Cost Conscious Care; MHAQ: Maastricht HVCCC
Attitude
Questionnaire; PCA: Principal Component Analysis; SEM:

Standard Error of
Measurement; U.S.: United States
Acknowledgements
The authors would like to thank Angelique van Bijsterveld and
Corry den
Rooyen for their support and assistance with connecting to key
individuals
across the Netherlands, who were able to help with data
collection. In
addition, the authors are grateful for the help of all project
leaders in
different Dutch educational regions in administering the MHAQ
to potential
participants. This research study was presented as a research
paper at the
2019 Annual AMEE (Association for Medical Education in
Europe)
Conference, Vienna, Austria, August 26, 2019 [57].
Authors’ contributions
S.M. contributed to conception and design of the study, to the
acquisition,
analysis and interpretation of the data in this study, and drafting
and revising
of the paper. He approves submission and publication of the
paper and
agrees with being accountable for all aspects thereof. K.K.
contributed to
conception and design of the study, to the acquisition, analysis
and
interpretation of the data in this study, and substantial revising
of different
versions of the paper. She approves submission and publication
of the paper
and agrees with being accountable for all aspects thereof.

A.L.H. contributed
to the acquisition of the data in this study, and substantintial
revising of
different versions (of parts) of the paper. She approves
submission and
publication of the paper and agrees with being accountable for
all aspects
thereof. A.P. contributed to the analysis used in the paper and
reviewing of
different versions (of parts) of the paper. She approves
submission and
all aspects
thereof. F.S. contributed to conception and design of the study,
to the
acquisition and interpretation of the data in this study, and
substantial
revising of different versions of the paper. He approves
submission and
all aspects
thereof. L.S. contributed to conception and design of the study,
to the
acquisition and interpretation of the data in this study, and
substantial
revising of different versions of the paper. He approves
submission and
all aspects
thereof. The author(s) read and approved the final manuscript.
Funding
Not applicable.
Availability of data and materials
The Dutch dataset collected during the current study is available

from the
corresponding author on reasonable request.
Ethics approval and consent to participate
The Ethical Review Board (ERB) of the Netherlands
Association for Medical
Education (NVMO) approved this study (no. NERB814 and
amendment no.
NERB817) before launch.
Informed consent was asked from all participants in this study
and all were
given the opportunity to withdraw from participating in the
study.
Consent for publication
All participants consented to their data being used
anonymously.
Competing interests
The authors declare that they have no competing interests.
Author details
1Department of Educational Development and Research, School
of Health
Professions Education, Maastricht University, P. O. Box 616,
6200 MD,
Universiteitssingel 60, 6229, ER, Maastricht, the Netherlands.
2Department of
Neurology, Mayo Clinic, Rochester, MN, USA. 3Department of
Health Services
Research, Care and Public Health Research Institute, Maastricht
University,
Maastricht, the Netherlands. 4Department of Pulmonary
Medicine, Catharina
Hospital, Eindhoven, the Netherlands. 5Department of Surgery,

Maastricht
University Medical Center, Maastricht, the Netherlands.
Received: 4 November 2019 Accepted: 11 February 2020
References
1. American College of Physicians. Controlling health care
costs while promoting
the best possible health outcomes. Philadelphia: American
College of
Physicians; 2009: Policy Monograph. (Available from American
College of
Physicians, 190 N. Independence Mall West, Philadelphia, PA
19106.).
2. Fischer ES, Welch GH. Avoiding the unintended
consequences of growth in
medical care: how might more be worse? JAMA.
1999;281(5):446–53.
3. Sociaal Cultureel Planbureau. Meebetalen aan de zorg. In:
Het ministerie van
Volksgezondheid WeS, editor. Den Haag: Textcetera; 2012.
4. College Geneeskundige Specialismen. Bewustzijnsproject:
Bewust
kosteneffectief kwaliteit van zorg leveren in geneeskundig-
specialistische
vervolgopleidingen. 2015.
5. Morgan DJ, Brownlee S, Leppin AL, Kressin N, Dhruva SS,
Levin L, et al.
Setting a research agenda for medical overuse. BMJ.
2015;351:h4534.

6. Smoldt RK, Cortese DA. Pay-for-performance or pay for
value? Mayo Clin
Proc. 2007;82:210–3.
7. Owens DK, Qaseem A, Chou R, Shekelle P. High-value, cost-
conscious health
care: concepts for clinicians to evaluate the benefits, harms, and
costs of
medical interventions. Ann Intern Med. 2011;154:174–80.
8. Goold SD, Hofer T, Zimmerman M, Hayward RA. Measuring
physician
attitudes toward cost, uncertainty, malpractice, and utilization
review. J Gen
Intern Med. 1994;9:544–9.
9. Sirovich BE, Gallagher PM, Wennberg DE, Fisher ES.
Discretionary decision
making by primary care physicians and the cost of U.S. health
care. Health
Aff. 2008;27:813–23.
10. Ryskina KL, Smith CD, Weissman A, Post J, Dine JC,
Bollman K, et al. U.S.
internal medicine residents’ knowledge and practice of high-
value care: a
national survey. Acad Med. 2015;90(10):1373–9.
11. Cooke M. Cost consciousness in patient care-what is
medical education's
responsibility? N Engl J Med. 2010;362:1253–5.
12. King BC, Abramson E, DiPace J, Gerber L, Hammad H,
Naifeh M. High value,
cost-conscious care: perspective of pediatric faculty and
residents. Acad

Pediatr. 2016;16(6):e7.
13. Korenstein D. Charting the route to high-value care the role
of medical
education. JAMA. 2015;314(22):2359–61.
14. Weinberger SE. Educating trainees about appropriate and
cost-conscious
diagnostic testing. Acad Med. 2011;86(11):1352.
15. Weinberger SE. Providing high value cost-conscious care: A
critical seventh
general competency for physicians. Ann Intern Med.
2011;155:386–8.
16. Ryskina KL, Korenstein D, Weissman A, Masters P, Alguire
P, Smith CD.
Development of a high-value care subscore on the internal
medicine in-
training examination. Ann Intern Med. 2014;161(10):733–9.
17. Stammen LA, Stalmeijer RE, Paternotte E, Oudkerk Pool A,
Driessen EW,
Scheele F, et al. Training physicians to provide high-value,
cost-conscious
care: a systematic review. JAMA. 2015;314(22):2384–400.
18. Foundation JMJ. Improving environments for learning in the
health
professions. Atlanta: Macy Foundation Conference; 2018.
19. Cruess RL, Cruess SR, Boudreau JD, Snell L, Steinert Y. A
schematic
representation of the professional identity formation and
socialization of

20:156 Page 9 of 10
medical students and residents: a guide for medical educators.
Acad Med.
2015;90(6):718–25.
20. Jarvis-Selinger S, Pratt DD, Regehr G. Competency is not
enough:
integrating identity formation into the medical education
discourse. Acad
Med. 2012;87(9):1185–90.
21. Ajzen I. The theory of planned behavior. Organ Behav Hum
Decis Process.
1991;50:179–211.
22. Ajzen I. The theory of planned behaviour: reactions and
reflections. Psychol
Health. 2011;26(9):1113–27.
23. Gupta R, Moriates C, Harrison JD, Valencia V, Ong M,
Clarke R, et al.
Development of a high-value care culture survey: a modified
delphi process
and psychometric evaluation. BMJ Qual Saf. 2016;0:1–9.
24. Armitage CJ, Conner M. Efficacy of the theory of planned
behaviour: a
meta-analytic review. Br J Soc Psychol. 2001;40:471–99.
25. Cutler D, Skinner JS, Stern AD, Wennberg DE. Physician
beliefs and patient
preferences: a new look at regional variation in health care

spending. Am
Econ J: Econ Policy. 2019;11(1):192–221.
26. Pearson SD, Goldman L, Orav EJ, Guadagnoli E, Garcia TB,
Johnson PA, et al.
Triage decisions for emergency department patients with chest
pain: do
physicians’ risk attitudes make the difference? J Gen Intern
Med. 1995;
10(10):557–64.
27. Tubbs EP, Broeckel Elrod JA, Flum DA. Risk taking and
tolerance of
uncertainty: implications for surgeons. J Surg Res. 2006;131:1–
6.
28. Zaat JOM, van Eijk JTM. General Practitioners’ uncertainty,
risk preference,
and use of laboratory tests. Med Care. 1992;30(9):846–54.
29. Bhatia RS, Levinson W, Shortt S, Pendrith C, Fric-Shamji
E, Kallewaard M,
et al. Measuring the effect of choosing wisely: an integrated
framework to
assess campaign impact on low-value care. BMJ Qual Saf.
2015;24(8):523–31.
30. Jochemsen-van der Leeuw HG, van Dijk N, van Etten-
Jamaludin FS,
Wieringa-de Waard M. The attributes of the clinical trainer as a
role model:
a systematic review. Acad Med. 2013;88(1):26–34.
31. Finset A. Patient participation, engagement and activation:
increased emphasis
on the role of patients in healthcare. Patient Educ Couns.

2017;100(7):1245–6.
32. Parand A, Dopson S, Renz A, Vincent C. The role of
hospital managers in
quality and patient safety: a systematic review. BMJ.
2014;4(9):e005055.
33. Hurst S, Slowther AM, Forde R, Pegoraro R, Reiter-Theil S,
Perrier A, et al.
Prevalence and determinants of physician bedside rationing. J
Gen Intern
Med. 2006;21:1138–43.
34. Kirchhoff AC, Hart G, Campbell EG. Rural and urban
primary care physician
professional beliefs and quality improvement behaviors. J Rural
Health.
2014;30:235–43.
35. Tilburt JC, Wynia MK, Sheeler RD, Thorsteinsdottir B,
James KM, Egginton JS,
et al. Views of US physicians about controlling health care
costs. JAMA.
2013;310(4):380–8.
36. Leep Hunderfund AN, Dyrbye LN, Starr SR, Mandrekar J,
Naessens JM, Tilburt
JC, et al. Role modeling and regional health care intensity: U.S.
medical
student attitudes toward and experiences with cost-conscious
care. Acad
Med. 2017;92(5):694–702.
37. Detsky AS. What patients really want from health care.
JAMA. 2011;306(22):
2500–1.

38. Weiner BJ, Shortell SM, Alexander J. Promoting clinical
involvement in
hospital quality improvement efforts: the effects of top
management,
board, and physician leadership. Health Serv Res.
1997;32(4):491–510.
39. Colla CH, Kinsella EA, Morden NE, Meyers DJ, Rosenthal
MB, Sequist TD.
Physician perceptions of choosing wisely and drivers of
overuse. Am J
Manag Care. 2016;22(5):337–43.
40. Hall C. Comment: generalizability theory and assessment in
medical
training. Neurology. 2015;85:1628.
41. Association AER, Association AP, Education NCoMi.
Standards for
educational and psychological testing. Washington, United
States of
America: American Educational Research Association; 2014.
42. Field A. Discovering statistics using IBM SPSS statistics:
sage; 2013.
43. McCoach DB, Gable RK, Madura JP. Instrument
development in the affective
domain: school and corporate applications. 3rd ed. New York:
Springer; 2013.
44. Boateng GO, Neilands TB, Frongillo EA, Melgar-Quinonez
HR, Young SL. Best
practices for developing and validating scales for health, social,
and

behavioral research: a primer. Front Public Health. 2018;6:149.
45. O'Connor BP. SPSS and SAS programs for determining the
number of
components using parallel analysis and Velicer’s MAP test.
Behav Res
Methods Instrum Comput. 2000;32(3):396–402.
46. Etchegaray JM, Fischer W. Survey research: be careful
where you step. BMJ
Qual Saf. 2006;15(3):154–5.
47. American Board Of Internal Medicine Foundation. National
physician survey.
2014.
48. Brislin RW. Back-translation for cross-cultural research. J
Cross-Cult Psychol.
1970;1(3):185–216.
49. Loewenthal K, Lewis CA. An introduction to psychological
tests and scales.
Philadelphia: Psychology press; 2015.
50. Bloch R, Norman G. Generalizability theory for the
perplexed: a practical
introduction and guide: AMEE Guide No. 68. Med Teach.
2012;34(11):960–92.
51. Colla CH, Sequist TD, Rosenthal MB, Schpero WL, Gottlieb
DJ, Morden NE.
Use of non-indicated cardiac testing in low-risk patients:
choosing wisely.
BMJ Qual Saf. 2014;24(2):149–53.
52. Tartaglia KM, Kman N, Ledford C. Medical student

perceptions of cost-
conscious care in an internal medicine clerkship: a thematic
analysis. J Gen
Intern Med. 2015;30(10):1491–6.
53. Kanzaria HK, Hoffman JR, Probst MA, Caloyeras JP, Berry
SH, Brook RH.
Emergency physician perceptions of medically unnecessary
advanced
diagnostic imaging. Acad Emerg Med. 2015;22(4):390–8.
54. Bishop TF, Federman AD, Keyhani S. Physician views on
defensive medicine
a national survey. Arch Intern Med. 2010;170(12):1081–3.
55. Gupta R, Sehgal N, Arora VM. Aligning Delivery System
and Training Missions in
Academic Medical Centers to Promote High-Value Care. Acad
Med. 2018.
56. Michels NRM, Denekens J, Driessen EW, Van Gaal LF,
Bossaert LL, De Winter
YD. A Delphi study to construct a CanMEDS competence based
inventory
applicable for workplace assessment. BMC Med Educ.
2012;12(1):86.
57. Mordang SBR, Könings KD, Leep Hunderfund AN, Paulus
ATG, Smeenk
FWJM, Stassen LPS. A new instrument to measure attitudes
regarding high
value, cost-conscious care of healthcare stakeholders:
development of the
MHAQ. Vienna: Oral presentation at Association for Medical
Education in
Europe Annual Conference; 2019.

Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional
claims in
published maps and institutional affiliations.
20:156 Page 10 of 10
BioMed Central publishes under the Creative Commons
Attribution License (CCAL). Under
the CCAL, authors retain copyright to the article but users are
allowed to download, reprint,
distribute and /or copy articles in BioMed Central journals, as
long as the original work is
properly cited.
RESEARCH ARTICLE
Geographic variation in the delivery of high-
value inpatient care
John RomleyID
1,2☯*, Erin Trish2☯, Dana Goldman1,2☯, Melinda Beeuwkes
Buntin3☯,
Yulei He4☯, Paul Ginsburg1,5☯
1 Price School of Public Policy, University of Southern
California, Los Angeles, California, United States of

America, 2 School of Pharmacy, University of Southern
California, Los Angeles, California, United States of
America, 3 Vanderbilt University, Nashville, Tennessee, United
States of America, 4 University of Maryland
University College, Adelphi, Maryland, United States of
America, 5 Brookings Institution, Washington D.C.,
United States of America
☯ These authors contributed equally to this work.
* [email protected]
Abstract
Objectives
To measure value in the delivery of inpatient care and to
quantify its variation across U.S.
regions.
Data sources / Study setting
A random (20%) sample of 33,713 elderly fee-for-service
Medicare beneficiaries treated in
2,232 hospitals for a heart attack in 2013.
Study design
We estimate a production function for inpatient care, defining
output as stays with favorable

patient outcomes in terms of survival and readmission. The
regression model includes hos-
pital inputs measured by treatment costs, as well as patient
characteristics. Region-level
effects in the production function are used to estimate the
productivity and value of the care
delivered by hospitals within regions.
Data collection / Extraction methods
Medicare claims and enrollment files, linked to the Dartmouth
Atlas of Health Care and Inpa-
tient Prospective Payment System Impact Files.
Principal findings
Hospitals in the hospital referral region at the 90th percentile of
the value distribution deliv-
ered 54% more high-quality stays than hospitals at the 10th
percentile could have delivered,
after adjusting for treatment costs and patient severity.
PLOS ONE | https://doi.org/10.1371/journal.pone.0213647
March 25, 2019 1 / 11
a1111111111
a1111111111
a1111111111

a1111111111
a1111111111
OPEN ACCESS
Citation: Romley J, Trish E, Goldman D, Beeuwkes
Buntin M, He Y, Ginsburg P (2019) Geographic
variation in the delivery of high-value inpatient care.
PLoS ONE 14(3): e0213647. https://doi.org/
10.1371/journal.pone.0213647
Editor: Ravishankar Jayadevappa, University of
Pennsylvania, UNITED STATES
Received: July 4, 2018
Accepted: February 26, 2019
Published: March 25, 2019
Copyright: © 2019 Romley et al. This is an open
access article distributed under the terms of the
Creative Commons Attribution License, which
permits unrestricted use, distribution, and
reproduction in any medium, provided the original

author and source are credited.
Data Availability Statement: The primary data
source for the project is CMS Medicare claims
data. The CMS data used in this project cannot be
shared with other researchers under the terms of
our Data Use Agreement (DUA). A researcher can
request access to the same data and obtain their
own DUA through the CMS Data Request Center
(https://urldefense.proofpoint.com/v2/url?u=https-
3A__www.resdac.org_cms-2Ddata_request_cms-
2Ddata-2Drequest-2Dcenter&d=DwIGaQ&c=
clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7
CSfnc_gI&r=Dq3XCqc5W3cxuMIbAN95iFjFR
gaCqQatH6Y8Kxmh84s&m=tzfyZNw6S7D9HV2tb
Conclusions
Variation in the delivery of high-value inpatient care points to
opportunities for better quality

and lower costs.
Introduction
The Institute of Medicine has taken the position that “the only
sensible way to restrain costs is
to enhance the value of the health care system.”[1] Value is an
elusive term in health care, but
good value tends to mean high quality in relation to cost [2],
and an array of initiatives in the
private and public sectors seek to improve quality while
containing costs. For example, the
Centers for Medicare and Medicaid Services implemented its
Hospital-Value Based Purchas-
ing and Hospital Readmissions Reduction Programs in 2013,
and has recently been rolling out
Advanced Alternative Payment Models.[3]
This growing emphasis on value has outpaced the development
of practical metrics of value
performance.[4] For therapeutic drugs, cost-effectiveness has
long been the standard to evalu-
ate treatments. Nevertheless, a very lively dialogue about the
appropriate framework for assess-
ing value in pharmaceuticals has re-emerged. The measurement
of value is still more unsettled

in other settings, such as hospital care, even as reimbursement
has been tied to indicators of
quality and cost.
There are many reasons to suspect important variation in the
value of care that is delivered.
As scholars at the Dartmouth Institute first discovered and
others have since confirmed, health
care utilization and spending vary markedly throughout the U.S.
Quality of care is also highly
variable [5–7]; for example, among Medicare beneficiaries
undergoing surgery in hospitals in
2009–2010, the 30-day risk-adjusted readmission rate was more
than seventy percent higher at
the 75th percentile of its distribution than at the 25th
percentile.[8]
Such variability in both quality and cost—the core elements of
value—is strongly suggestive
of similar variation in value. Yet information about quality and
cost is not directly informative
about value in care delivery. If hospitals in one region have
better quality but higher cost than
those in another region, the formers’ care can be higher or
lower-value than the latters’. If qual-
ity were higher but costs were the same, one could reach the

qualitative conclusion that value
is higher, but not the quantitative conclusion as to how much
higher.
This study uses a production function framework to develop a
value metric for inpatient
care. Focusing on a high-prevalence medical condition - - heart
attacks - - we assessed the
value of the care delivered to Medicare beneficiaries
hospitalized in 2013, and examined how
value varies across regions.
Methods
Providers deliver high-value care by producing good quality in
relation to their costs.[2]
Accordingly, we specify and analyze a production function for
inpatient care; the output and
inputs of our production function are detailed below. This
analytical framework, and the
closely related framework for cost functions, have been applied
extensively to hospitals.[9–23]
The primary data source for our analysis was the Medicare
Inpatient File from 2013. The
medical claims in this file report patient diagnoses and
procedures, demographic characteris-

tics, charges and payments, dates of service, and the identity of
the short-stay hospital. The
Variation in high-value hospital care
March 25, 2019 2 / 11
G9wdjzW-eWfRS094roJc6vSnYA&s=
dfhmalHLUmHDB0mYWEArnnK-
up1DQr682tZB3VZvOIA&e=). The researcher
should request Research Identifiable Files. See
https://urldefense.proofpoint.com/v2/urlu=https-
3A__www.resdac.org_cms-2Ddata_request_
research-2Didentifiable-2Dfiles&d=DwIGaQ&c=
clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p
7CSfnc_gI&r=Dq3XCqc5W3cxuMIbAN95iFjFR
gaCqQatH6Y8Kxmh84s&m=tzfyZNw6S7D9H
V2tbG9wdjzW-eWfRS094roJc6vSnYA&s=
ZPDgim_PD2mCQw6Cm3gIBm38f482K89
AVhrtLz8BuD0&e=. Assistance for accessing and
using these data is made available by the Research

Data Assistance Center (ResDAC). ResDAC is a
consortium of faculty and staff from the University
of Minnesota, Boston University, Dartmouth
Medical School, and the Morehouse School of
Medicine. ResDAC provides free assistance to
academic and non-profit researchers interested in
using Medicare, Medicaid, SCHIP, and Medicare
Current Beneficiary Survey (MCBS) data for
research. We will make available the code that is
used to generate our analytic data files and
conduct the analyses, and anyone will be able to
download the code from the repository hosted
here: https://healthpolicy.box.com/s/
emfiwrf4c6nc11zotyocig9hnvzorwly. Also included
will be a “readme” file that explains how a
researcher can get access to the data and a
description of the files that will guide a researcher
through use of the code.

Funding: This research was supported by the
Commonwealth Fund and the National Institute on
Aging. The funders had no role in study design,
data collection and analysis, decision to publish, or
preparation of the manuscript.
Competing interests: The authors have declared
that no competing interests exist.
research-identifiable 20 percent sample file that we used also
reports patient ZIP codes. Where
necessary, multiple claims were “rolled up” into a hospital stay.
We identified heart attack patients according to ICD-9 codes
used in the Inpatient Quality
Indicator (IQI) for risk-adjusted mortality from the Agency for
Healthcare Research and Qual-
ity (AHRQ).[24] We then applied a number of additional
criteria to create our final heart
attack cohort. For example, patients who were transferred to
other hospitals were excluded;
complete criteria are shown in an appendix. We further limited
the cohort in this study to

elderly fee-for-service beneficiaries.[23]
To analyze the delivery of hospital care, we must define the
output produced and the inputs
used to produce it. We defined output to include not only
quantity—as is common in studies
of production—but also quality.[25] Specifically, following
prior work [23], we measured the
total number of “high-quality” stays in which the patient
survived at least 30 days beyond the
admission, and avoided an unplanned readmission within 30
days of discharge. Death dates
were available from the Medicare Beneficiary Summary File;
unplanned readmissions were
identified based on the algorithm used by CMS for reporting
and payment purposes.[26]
These favorable outcomes are publicly reported and
incorporated into current Medicare reim-
bursement; for example, mortality has been included in CMS’s
Hospital Value-Based Purchas-
ing Program since its introduction in fiscal year 2013.[27]
Under our approach, only high-
quality stays count toward the output that hospitals produce.
In a supplemental analysis, we also accounted for patient
experience, multiplying the num-

ber of survivors without a readmission by the percentage of
survey respondents who would
have definitely recommended a hospital to friends and family
from the Hospital Consumer
Assessment of Healthcare Providers and Systems
(HCAHPS).[28]
Our output measure makes an assumption about the tradeoff
between the quantity and
quality of hospital stays. In particular, output is unchanged if
quality increases by one percent
while the quantity of stays decreases by one percent. To assess
the robustness of our findings to
this assumption, we performed a sensitivity analysis that used
the number of stays (regardless
of outcomes) as the dependent variable, and included mortality,
readmission and satisfaction
rates as explanatory variables in the production model. In
addition to health care, hospitals
produce graduate medical education, and so all models included
variables for residents-per-
bed thresholds used in the literature and reported in the
Inpatient Prospective Payment System
(PPS) Impact File.[29–32] To address the provision of tertiary
care, all analyses also included

indicator variables for delivery of advanced cardiac and
neurological procedures, as defined in
the Dartmouth Atlas of Health Care.[33]
To characterize hospital inputs—the key explanatory variable in
the production model - -
we followed the literature on inpatient care in using an
aggregate measure.[5, 6, 20, 21, 34]
Specifically, we measured the total cost to each hospital of
treating patients in the heart attack
cohort (including patients with unfavorable outcomes.) To do
so, we first converted total hos-
pital charges covered by Medicare to costs based on the cost-to-
charge ratios submitted by hos-
pitals to CMS as part of their cost accounting reports, which are
reported in the CMS Impact
File. We then adjusted for geographic differences in labor
prices using the hospital wage index,
also from the Impact File; this adjustment was applied to the
labor-related portion of the base
PPS payment rate. We measured costs in 2014 US dollars, based
on the medical component of
the consumer price index. In a sensitivity analysis, we did not
adjust for area wages; this analy-

sis assessed the impact of wage adjustment, as there have been
concerns about mismeasure-
ment of wages.[35]
We followed prior work in addressing patient severity.[5, 6, 23,
36] For each hospital, we
included variables for the proportions of patients with heart
attacks in specific locations based
on diagnosis codes (for example, 410.2 for acute myocardial
infarction of the inferolateral
March 25, 2019 3 / 11
wall.)[21] We also included the proportions of a hospital’s
patients with different numbers of
Charlson co-morbidities in the medical claims for heart attack
stays [37], as well as the average
socio-demographic characteristics of patients’ zip codes from
the 2009–2013 American Com-
munity Survey (for example, the poverty rate and the percentage
of elderly residents with dis-
abilities.)[38] To further address patient severity, we adjusted
for the likelihood of death

during the hospital stay, using the risk adjustment model
developed by clinical experts as an
input into AHRQ’s heart attack mortality IQI.[24] The AHRQ
risk model predicts the proba-
bility that a patient dies based on her age and sex, transfer from
another hospital, and All Payer
Refined-Diagnosis Related Group (APR-DRG); each APR-DRG
includes its own mortality-
risk scale.[39] We included covariates for average age and
proportion female, which could be
related to treatment costs as well as patient severity. We also
adjusted for race and ethnicity. In
a sensitivity analysis, we excluded all diagnosis-based
covariates while adding the proportion
of patients admitted from the emergency room or transferred
from another hospital, because
there is some evidence of regional differences in how conditions
are diagnosed.[40, 41]
In studying inpatient treatment of heart attack in 2013, we focus
on the value of care deliv-
ered within areas defined by hospital referral regions (HRRs)
from the Dartmouth Atlas of
Health Care.[33] Thus, a high-value HRR is one whose
hospitals tended to produce more

stays - - or a better rate of high-quality stays than expected - -
given its levels of treatment costs
and patient severity. We implemented our model by assuming
that HRR-level value was nor-
mally distributed and applying the method of maximum
likelihood.[42, 43] For representative-
ness, each hospital-level observation was weighted by the
number of patients treated. Our
approach produced an estimate of the proportion of
(unmeasured) variation in output result-
ing from differences between HRRs in the average performance
of their hospitals, compared
to the differences around the average among the hospitals
within the HRRs (this latter varia-
tion reflects hospital-level value as well as randomness.) This
approach did, however, make the
assumption that value was systematically unrelated to other
factors, such as patient severity
across areas. In a sensitivity analysis, we relaxed this
assumption using fixed-effects regression
to assess HRRs.
These analyses produced estimates of value for each HRR,
adjusted for the reliability of the
value performance signal based on the size of the area. We

transformed these HRR-specific
estimates into a value index with a national mean of 100.
We explored the relationship between quality, cost and value.
While our production frame-
work analyzed total costs in relation to the total number of
high-quality stays, it is natural and
commonplace to assess provider cost and quality based on
average performance. We therefore
compared our value index to cost per stay and the rate of high-
quality stays, adjusting each for
the patient and hospital characteristics noted above in
independent regressions. The appendix
provides further information on the data and analyses, including
additional robustness checks.
Results
In our 2013 sample, 33,713 elderly fee-for-service beneficiaries
were admitted with a heart
attack to 2,232 hospitals in 304 hospital referral regions (HRRs)
with at least 11 heart-attack
stays in our database of Medicare claims. Fifty-one percent of
these patients were female, and
the average age was 80 years. The cost of these hospital stays
averaged $14,900 in 2014 dollars.

In terms of outcomes, 87% of patients survived at least 30 days
beyond the admission, while
86% of these survivors avoided an unplanned readmission
within 30 days of discharge. The
overall rate of high-quality hospital stays (survival without
readmission) was 74%.
Based on quality of care, treatment cost, patient severity and
hospital characteristics
(including teaching status), our analytic framework quantifies
value in inpatient heart attack
March 25, 2019 4 / 11
care across the U.S. The national map in Fig 1 shows the value
of care delivered in each HRR,
with dark green indicating the highest quintile of value.
Compared to the U.S. average of 100,
Miami’s score on our value index was 87. Thus, hospitals in
Miami produced 13% fewer high-
quality hospital stays (87%—100% = -13%) than hospitals in
the average U.S. region would
have been expected to produce if their costs and patients had
been the same. As another exam-

ple, Everett, Washington performed better than the national
average, with a value index score
of 122. Both of these scores were statistically distinguishable
(with 95% confidence) from the
national average; among all HRRs, 71% were significantly
different from 100.
The range of value index scores is shown in the histogram in
Fig 2. About one in 8 U.S.
regions had a value index in excess of 120, thus delivering at
least 20% more value than the
national average, that is, 20% more high-quality heart attack
stays than the average region after
adjusting for treatment cost and patient severity. The value
index for the HRR at the 90th per-
centile of the distribution, compared to the score at the 10th
percentile, exhibited a ratio of
1.54:1. That is, value in care delivery was 54% higher for the
region whose performance
exceeded 9 out of 10 of all regions, compared to the region
whose performance exceeded only
1 out of 10 regions. For the components of value, adjusted costs
and quality of care, the corre-
sponding 90–10 ratios were 1.42:1 and 1.36:1, respectively. In
terms of value in care delivery,

hospitals in the median HRR would have to increase their
performance by 22% to reach the
top decile (i.e., the 90–50 ratio was 1.22.) These differences
between HRRs accounted for 32%
of the unmeasured variation in hospital output.
Fig 3 shows quality, cost and value in the delivery of inpatient
care for heart attack. Specifi-
cally, HRRs are characterized as above- or below-average in
value, and are located within
quadrants defined by average cost and quality. In the upper left
quadrant, adjusted cost is
below the national average, while adjusted quality is above
average. Within this quadrant, 78%
of HRRs were above-average in value, with value index scores
exceeding 100. In the bottom
right quadrant, cost is above average and quality below average.
Here only 13% of HRRs are
above average in value. When costs and quality are above
average—the upper right quadrant—
31% of regions deliver above average value, with higher quality
than would have been expected
given the high costs. When both costs and quality are below
average - - the bottom left quad-

rant—some HRRs (specifically, 63%) are also above average.
Among all regions with above-
average value, 55% were below average in terms of adjusted
quality or above average in cost.
The regression results (reported in the appendix) imply that a
region with 10% higher cost
Fig 1. Value index for inpatient heart attack care in 2013, by
hospital referral region grouped into quintiles. Note:
Darker green indicates higher value.
https://doi.org/10.1371/journal.pone.0213647.g001
March 25, 2019 5 / 11
than another region lies on the same production function - - and
thus delivers equivalent
value - - if the higher-cost region also delivers 8% more quality
(in terms of the rate of high-
quality stays).
In a sensitivity analysis, we relaxed the assumption that HRR-
level value was independent
of factors such as patient severity. The resulting (“fixed
effects”) value index scores for HRRs

were quite similar to the scores from the primary analysis (ρ =
+0.789, p< 0.001.) In another
sensitivity analysis, we redefined the dependent variable of
hospital output as the number of
heart attack stays and included the rates of 30-day survival and
unplanned readmission as
regression covariates, and again found similar value index
scores for HRRs (ρ = +0.848,
p< 0.001.) The value index scores were also similar when we
incorporated the patient experi-
ence into hospital output (ρ = +0.942, p< 0.001.) Finally, we
found that the value scores were
not highly sensitive to the adjustment of costs for area wages or
to the measurement of patient
severity based on recorded diagnoses; for both of these
sensitivity analysis, the correlation coef-
ficient with the results of our primary analysis exceeded +0.90.
The scores were also insensitive
to a number of other robustness tests described in the appendix.
Discussion
This study has used a framework for the production of high-
quality health care to develop and
implement a measure of the value of inpatient care among

EDM ForumEDM Forum CommunityeGEMs (Generating Evidence & M.docx

EDM ForumEDM Forum CommunityeGEMs (Generating Evidence & M.docx

Recommended

Recommended

More Related Content

Similar to EDM ForumEDM Forum CommunityeGEMs (Generating Evidence & M.docx

Similar to EDM ForumEDM Forum CommunityeGEMs (Generating Evidence & M.docx (20)

More from greg1eden90113

More from greg1eden90113 (20)

Recently uploaded

Recently uploaded (20)

EDM ForumEDM Forum CommunityeGEMs (Generating Evidence & M.docx