Speaker Presentation from U.S. News Healthcare of Tomorrow leadership summit, November 2-4, 2016 in Washington, DC. Find out more about this forum at www.usnewshot.com.

1. Inside the U.S. News Best
Children’s Hospitals Rankings
Avery Comarow & co l l e a g u e s
N O V E M B E R 3 , 2 0 1 6
W A S H I N G T O N , D . C .

2. 2
LEAD INVESTIGATORS
Avery Comarow
Health Rankings Editor
Murrey Olmsted, Ph.D.
Project Director, RTI
International
Ben Harder
Chief of Health Analysis |
Managing Editor

3. 3
PERSPECTIVE ON PUBLIC REPORTING
Best Children’s Hospitals Mission:
To provide decision support to families and referring
physicians for children whose medical needs call for the
highest quality of care.
Quality improvement and public accountability have
important societal benefits but are secondary to this
mission.

4. 4
DISCLOSURES
Our Group:
• Avery Comarow and Ben Harder are employed by U.S. News & World Report. U.S. News is
sole sponsor of the Best Children’s Hospitals analysis and receives revenues from mulRple
adverRsers including health systems.
• Dr. Murrey Olmsted is employed by RTI InternaRonal, U.S. News contractor for producing
the Best Children’s Hospitals rankings.
Avery Comarow, Murrey Olmsted: No disclosures
Ben Harder:
• Part-Rme Senior Fellow at GuideStar
• Wife is MedStar Health-employed physician
• Sister is Brigham & Women’s-employed physician

5. 5
TODAY’S DISCUSSION AGENDA
Looking ahead to 2017
Ø Timely inclusion of all evaluated hospitals
Ø TransiRoning from ICD-9 to ICD-10
Ø Rethinking domain weights in one or more specialRes
Addressing burden of survey on hospitals
Ø CompleRon Rme
Ø Enhanced hospital ve]ng
Ø Specific survey pain points
Data challenges
Ø Prelaunch data review
Ø Catching quesRonable values
Ø Lack of risk adjustment
Ø PrevenRng errors
Under discussion
Ø Introduce adviser transparency
Ø Split medical/surgical specialRes
Ø Rework survey and interface

6. 6
1. Timely inclusion of all evaluated hospitals
Following the most recent launch in June, hospitals were
added that were evaluated but did not achieve a top-50
ranking. Scores are displayed in all variables other than
overall score or ranking.
In 2017, such hospitals will be displayed at launch along
with ranked hospitals. Results in individual metrics but
without rank or overall score will again be included.
LOOKING AHEAD TO 2017

7. 7
2. TransiLoning from ICD-9 to ICD-10
We appreciate the impact on hospitals. We are working to
smooth the transiRon and to minimize large volume shies
by wording quesRons clearly and defining codes
appropriately.
If transiRon-related complicaRons threaten Rmely survey
compleRon, we *may* be able to allow extra Rme case by
case. Advance noRce would be helpful.
LOOKING AHEAD TO 2017

8. 8
3. Rethinking domain weights
Outcomes may receive addiRonal weight in cardiology &
heart surgery, for example, because most outcomes are
taken from registries.
Only specialRes with a variety of robust outcomes would
be eligible for significant weighRng changes. Most have a
few good outcomes, but they are rarely risk-adjusted.
LOOKING AHEAD TO 2017

9. 9
SURVEY BURDEN: COMPLETION TIME
1. Adding compleLon Lme
Hospitals have requested more than the current 4-6
weeks they have historically received aeer the survey is in
the field in early January.
U.S. News response:
To give hospitals six weeks to prepare for the January
release, we will post a working drae of the survey
quesRons online later this month. QuesRons or concerns
with specific measures should be raised during this
period.

10. 10
SURVEY BURDEN: IN-HOUSE VETTING
2. Enhancing hospital veSng
Survey responsibility at some hospitals is assigned to
individuals who would benefit from assistance provided
by clinical leaders.
U.S. News response: A higher degree of clinical
engagement would speed processing and improve QC
both within hospitals and by RTI and U.S. News.

11. 11
SURVEY BURDEN: IN-HOUSE VETTING
2. Enhancing hospital veSng (cont.)
We require CMO signoff on the completed survey and
encourage service chiefs to be involved in collecRng,
reviewing and signing off on responses for their specialty.
We also collect their contact informaRon for possible
follow-up. We may make formal signoff a hard
requirement in the future.

12. 12
SURVEY BURDEN: PAIN POINTS
1. Imprecise quesLons
Some quesRons may be open to misinterpretaRon.
U.S. News response: We ask hospitals to idenRfy such
quesRons asap during the survey period so we can issue
clarificaRons.
Post-survey feedback is also helpful for working groups to
consider revisions in following year.

13. 13
SURVEY BURDEN: PAIN POINTS
1. Imprecise quesLons (cont.)
Example of clarificaRon in place for 2017:
In cancer secRon (Q. B27), idenRficaRon of brain tumors
will be improved by placing them into 5 categories, with
ICD-10 diagnosis and procedure codes.

14. 14
HOSPITAL SURVEY BURDEN: PAIN POINTS
2. ReporLng volume data
Hospitals say generaRng volumes is extremely Rme-
consuming and individual volume metrics have low
weight. OutpaRents from mulRple clinics oeen must be
combined and the numbers may not be readily available
from the EMR or through the HIT system.

15. 15
SURVEY BURDEN: PAIN POINTS
2. ReporLng volume data (cont.)
U.S. News response: EffecRve outpaRent care lowers the
number of sick, high-cost inpaRents. Hospitals agree.
Therefore outpaRent care should be evaluated.
Our advisers have suggested that for some quesRons a
hospital could simply indicate whether or not it sees a
threshold number of paRents, such as 25 or more. The
premise is that having the funcRon is more important
than a high volume of paRents. We are considering this.

16. 16
SURVEY BURDEN: PAIN POINTS
3. QuesLons that change year to year.
Some hospitals set up survey-specific data queries to their
EMRs. Changes may require reprogramming.
U.S. News response: We try to make changes only when
they lead to improved measurement or reduce the
potenRal for misinterpretaRon.

17. 17
SURVEY BURDEN: PAIN POINTS
4. InfecLon irrelevance
CLABSI and CAUTI are immaterial in some specialRes.
U.S. News response: Both are now tracked only in cancer,
gastroenterology & GI surgery, cardiology & heart surgery,
nephrology and pulmonology. In neonatology only NICU-
specific infecRons are tracked. Urology only tracks CAUTI.
Neither is tracked in diabetes & endocrinology, neurology
& neurosurgery, or orthopedics because of fewer ICU
days.

18. 18
SURVEY BURDEN: PAIN POINTS
5. QuesLons requiring chart review
Many hospitals have not yet integrated chart data into
their EMRs. Responding takes an inordinate Rme.
U.S. News response:
We try to minimize quesRons requiring chart review and
reassess the value of each such quesRon every year.

19. 19
SURVEY BURDEN: PAIN POINTS
5. QuesLons requiring chart review (cont.)
For example, a quesRon that will be deleted in 2017 asked
for the % of visits in which current cancer paRents see
their primary oncologist for management and evaluaRon:
q ≥90%
q 80-89%
q 65-79%
q 50-64%
q Less than 50%
Responding was burdensome and may have led some
hospitals to supply esRmates rather than actual values.

20. 20
SURVEY BURDEN: PAIN POINTS
6. Difficulty prinLng out completed secLons of
online survey
U.S. News response: The survey can be downloaded as a
PDF and selected pages can then be printed out. Tables
may not look exactly the same as they do onscreen, but
they are readable. RTI will convert the completed online
survey to a PDF for any hospital that makes a request. This
is regularly done for about 20 hospitals.

21. 21
DATA CHALLENGES
1. Prelaunch data review
Hospitals want a formal review process to validate our
findings before they are made public.
U.S. News response: PosRng embargoed results for all
metrics on Dashboard and providing 4 weeks or so for
data reconciliaRon might be possible. Extensive technical
input and tesRng and significant commitment of
addiRonal U.S. News resources would be required and the
current schedule would be disrupted.

22. 22
DATA CHALLENGES
2. Catching quesLonable values
Examples we have noted include across-the-board 100%
followup rates, high volumes with few providers and
significant year-to-year volume changes.
U.S. News response: We target important values that can
be quickly addressed. All 100% responses, all values 2-3
SD from mean and “unlikely” values are flagged. Year-to-
year variaRon is a noisy target—mergers, new programs
and other factors create unpredictable change.

23. 23
DATA CHALLENGES
2. Catching quesLonable values (cont.)
Flagged examples from 2016 responses:
• F33.1 – Readmission rate 5 Rmes higher than all other
hospitals.
• H27 – Hospital indicated SSI rates are tracked but
provided no rates.
• G21d – Number of kidney transplant paRents exceeded
total of all kidney paRents.
• J24e – Median weight-for-length percenRle was 3 S.D.
below the mean.

24. 24
DATA CHALLENGES
3. Lack of risk adjustment
Risk-adjusted outcomes would beser quanRfy hospital
performance by accounRng for paRent acuity.
U.S. News response: It would be theoreRcally possible for
hospitals to provide anonymized paRent records directly
to U.S. News. We would then have the data necessary to
measure risk-adjusted performance.
This would be a target, not a next step. Planning would
take several years or more.

25. 25
DATA CHALLENGES
4. PrevenLng errors
We are intensifying the effort to minimize the need for
post-launch correcRons, which create difficulRes for both
affected hospitals and U.S. News.
U.S. News response:
AddiRonal checks include the following—
Ø We have extended the scope of unlikely values by
building addiRonal “if…then” logic checks to evaluate
data as it is entered and alerRng hospitals to
implausible entries.

26. 26
DATA CHALLENGES
4. PrevenLng errors (cont.)
Ø We anRcipate increasing the number of specific
hospitals we will flag to request that they validate or
corroborate their submised data.
Ø We will step up scruRny of large year-to-year data
changes. As noted, such shies do not always represent
a reporRng problem, but they invite a closer look.
Ø We will consider showing each hospital’s enRre
scorecard in the Dashboard—not just its rank—during
the embargo period.

27. 27
UNDER DISCUSSION
1. Introduce adviser transparency
Hospitals and various organizaRons and individuals have
asked us to idenRfy working group members and their
hospitals.
U.S. News response: We endorse transparency but do not
want to subject advisers to pressure or distracRons (e.g.
unsolicited calls and emails), especially decisions on
methodology are not theirs but rest with U.S. News.

28. 28
UNDER DISCUSSION
2. Split medical/surgical specialLes
Separately evaluaRng specialRes in combinaRons such as
cardiology & heart surgery would provide families with
informaRon that is beser focused.
U.S. News response: Concerns include relaRvely low
volumes in the divided specialRes, lack of non-surgical
outcomes in some medical specialRes, and interlocking
clinical relaRonships that cannot be readily separated.

29. 29
UNDER DISCUSSION
3. Beyond clarifying and pruning, can the survey
and its interface be significantly improved to
expedite compleLon?
U.S. News response: Both U.S. News and RTI welcome
suggesRons on how to accomplish this worthwhile goal.

30. 30
YOUR COMMENTS ARE WELCOME
Please email suggesLons to:
besthospitals@RTI.org
acomarow@usnews.com
bharder@usnews.com