The U.S. News Rankings: Future Role of the Reputation Survey (Jeffery Jacobs)

Jeffrey P. Jacobs, M.D., FACS, FACC, FCCP
Professor of Surgery and Pediatrics, Johns Hopkins University
Co-Director, Johns Hopkins All Children’s Heart Institute
Chief, Division of Cardiovascular Surgery
Director, Andrews/Daicoff Cardiovascular Program
Surgical Director of Heart Transplantation
Johns Hopkins All Children’s Heart Institute
Johns Hopkins All Children’s Hospital and Florida Hospital for Children
The U.S. News Rankings:
Future Role of the Reputation Survey
U.S. News & World Report
Healthcare of Tomorrow
November 3, 2017

•  Chair, STS National Database Workforce
•  Chair, CHSS Committee on Quality Improvement
and Outcomes
•  Working Group Leader, Heart/Heart Surgery
Working Group for U.S. News America's Best
Children's Hospitals rankings
•  Editor-in-Chief, Cardiology in the Young
•  Co-Chair, World Congress of Pediatric Cardiology
and Cardiac Surgery 2021
Disclosure

Barach P, Jacobs JP, Lipshultz SE, Laussen P. (Eds.). Pediatric and Congenital Cardiac Care - Volume 1: Outcomes
Analysis. Springer-Verlag London. Pages 1 – 515. ISBN: 978-1-4471-6586-6 (Print). 978-1-4471-6587-3 (Online).
Published in 2014.
Barach P, Jacobs JP, Lipshultz SE, Laussen P. (Eds.). Pediatric and Congenital Cardiac Care - Volume 2: Quality
Improvement and Patient Safety. Springer-Verlag London. 2015, Pages 1 – 456. ISBN: 978-1-4471-6565-1 (Print).
978-1-4471-6566-8 (Online). Published in 2014.

•  2008: Introduction of pediatric specialties. Heart &
Heart Surgery was 50% reputation, 40% structure,
10% outcomes; weights varied slightly in other
specialties.
Reputation Score:
Pediatric Cardiology & Heart Surgery

•  2008: Introduction of pediatric specialties. Heart &
Heart Surgery was 50% reputation, 40% structure,
10% outcomes; weights varied slightly in other
specialties.
•  2010: Decreased weight of reputation from 50%
to 35% reputation in all specialties. 40% structure,
25% outcomes.
Reputation Score:

•  2008: Introduction of pediatric specialties. Heart & Heart
Surgery was 50% reputation, 40% structure, 10%
outcomes; weights varied slightly in other specialties.
•  2010: Decreased weight of reputation from 50% to
35% reputation in all specialties. 40% structure, 25%
outcomes.
•  2011: Decreased weight of reputation from 35% to 25%
reputation in all specialties. 40% structure, 35%
outcomes.
Reputation Score:

•  2008: Introduction of pediatric specialties. Heart & Heart
Surgery was 50% reputation, 40% structure, 10%
outcomes; weights varied slightly in other specialties.
reputation in all specialties. 40% structure, 25% outcomes.
•  2014: Decreased weight of reputation from 25% to 16.7% in
all pediatric specialties
Reputation Score:

•  2008: Introduction of pediatric specialties. Heart & Heart Surgery
was 50% reputation, 40% structure, 10% outcomes; weights
varied slightly in other specialties.
•  2011: Decreased weight of reputation from 35% to 25% reputation
in all specialties. 40% structure, 35% outcomes.
•  2014: Decreased weight of reputation from 25% to 16.7% in all
pediatric specialties
•  2016: Decreased weight of reputation from 16.7% to 15% in all
pediatric specialties, and gave pediatric hospitals credit for being
publicly transparent about their pediatric STS outcomes
Reputation Score:

•  2008: Introduction of pediatric specialties. Heart & Heart Surgery was 50%
reputation, 40% structure, 10% outcomes; weights varied slightly in other
specialties.
•  2010: Decreased weight of reputation from 50% to 35% reputation in all
specialties. 40% structure, 25% outcomes.
•  2011: Decreased weight of reputation from 35% to 25% reputation in all
specialties. 40% structure, 35% outcomes.
•  2014: Decreased weight of reputation from 25% to 16.7% in all pediatric
specialties
•  2016: Decreased weight of reputation from 16.7% to 15% in all pediatric
specialties, and gave pediatric hospitals credit for being publicly transparent
about their pediatric STS outcomes
•  2017: Decreased weight of reputation from 15% to 8.5% in pediatric Cardiology
& Heart Surgery. Remained at 15% in all other pediatric specialties.
Reputation Score:

10
Modeling Change to US News Rankings after Removal of
Reputation scores (Specialty: Cardiology & Heart Surgery)
•  Largest Posi,ve change = +7
•  Largest Nega,ve change = -19
•  Number of No change = 9 (17%)
•  Number of pos. change = 25 (49%)
•  Number of neg. change = 17 (33%)
•  Average Ranking change +/- 3.3
9 hospitals with no
change
25 hospitals with
pos. change
17 hospitals with
neg. change
Largest pos.
change (+7)
Largest neg..
change (-19)

11
9 hospitals with no
change
25 hospitals with
pos. change
17 hospitals with
neg. change
Largest pos.
change (+7)
Largest neg..
change (-19)

12
9 hospitals with no
change
25 hospitals with
pos. change
17 hospitals with
neg. change
Largest pos.
change (+7)
Largest neg..
change (-19)

13
9 hospitals with no
change
25 hospitals with
pos. change
17 hospitals with
neg. change
Largest pos.
change (+7)
Largest neg..
change (-19)

•  We obtained the Total Raw Scores for the Cardiology and Heart Surgery Specialty
from the HDI Pediatrics analytic platform (USNWR 2017 rankings)
•  We obtained the Reputation values from HDI Pediatrics analytic platform
–  Also found in “2017-18 Best Children’s Hospitals Rankings by Specialty” document
–  Field = Reputation with Physicians in Specialty
•  We calculated the Reputation Score contribution by using US News formula*
•  We calculated the New Total Raw Scores by subtracting Reputation Score
•  We found Hospital rankings based on new Total Raw Scores without the Reputation
factor
•  We selected the top 50 Hospitals from original ranking and compared their new
ranking without reputation to their original ranking
•  We charted the differences between original and new rankings.
•  (see chart on next slide)
Methodology for obtaining USNWR 2017 ranking without
reputation score using US News Hospital Data Insights (HDI)
Pediatrics analytic platform:

* Reputa,on Contribu,on to Total Raw Score =
(((Log(R+10) -1)*10 )/ P )* W
R = ques,on result P = possible points W = weight

Methodology: Calculation example
Total Score
Original Ranking 33/50 P = 10.41 Possible points
Total Raw Score 74.3 W = 8.48 Weight
Reputa,on Value 2.1 R = 2.1 Ques,on Value
Formula to ﬁnd
Reputa,on Score
Contribu,on to Total
Raw Score
(((Log(R+10)
-1)*10 )/ P )* W

(((Log(2.1+10)
-1)*10 )/ 10.41 )*
8.48
= .675
Possible Points =
P = 10.41
Weight = W =
8.48
Result Value = R =
2.1

Reputa,on Score .675
New Total Raw Score 73.35 74.0 - .675 =
73.35

New Ranking based on
New score
31/50

•  how good or bad something is
•  a characteristic or feature that someone or
something has : something that can be noticed
as a part of a person or thing
•  a high level of value or excellence.
Definition of Quality
[http://www.merriam-webster.com/dictionary/quality].
Accessed November 10, 2015

Donabedian’s Triad
Donabedian A. Evalua.ng the quality of medical care.
Milbank Mem Fund Q. 1966;44(Suppl):166–206.

Michael Porter
value deﬁned as the health outcomes achieved per dollar spent
Michael E. Porter, Ph.D. Perspec,ve. What Is Value in Health Care?
N Engl J Med 2010; 363:2477-2481

Congenital Heart Disease
Meaningful
Multi-institutional Outcomes Analysis
Accomplishments
1)  Common Language = Nomenclature
2)  Mechanism of Data Collection (Database - Registry)
3)  Mechanism of Evaluating Case Complexity
4)  Mechanism to Verify Data Validity and Accuracy
5)  Collaboration Between Subspecialties
6)  Longitudinal Follow-Up and Linked Databases
7)  Quality Improvement
Barach P, Jacobs JP, Lipshultz SE, Laussen P. (Eds.). Pediatric
and Congenital Cardiac Care - Volume 1: Outcomes Analysis.
Springer-Verlag London. Pages 1 – 515. ISBN:
978-1-4471-6586-6 (Print). 978-1-4471-6587-3 (Online).

The validity of coding of lesions seen in
the congenitally malformed heart via the
International Classification of Diseases
(ICD) is poor
1.  Cronk CE, Malloy ME, Pelech AN, et al. Completeness of state administrative databases for
surveillance of congenital heart disease. Birth Defects Res A Clin Mol Teratol 2003;67:597-603.
2.  Frohnert BK, Lussky RC, Alms MA, Mendelsohn NJ, Symonik DM, Falken MC. Validity of hospital
discharge data for identifying infants with cardiac defects. J Perinatol 2005;25:737-42.
3.  Strickland MJ, Riehle-Colarusso TJ, Jacobs JP, Reller MD, Mahle WT, Botto LD, Tolbert PE, Jacobs
ML, Lacour-Gayet FG, Tchervenkov CI, Mavroudis C, Correa A. The importance of nomenclature
for congenital cardiac disease: implications for research and evaluation. In: 2008 Cardiology
in the Young Supplement: Databases and The Assessment of Complications associated with The
Treatment of Patients with Congenital Cardiac Disease, Prepared by: The Multi-Societal Database
Committee for Pediatric and Congenital Heart Disease, Jeffrey P. Jacobs, MD (editor). Cardiology in
the Young, Volume 18, Issue S2 (Suppl. 2), pp 92–100, December 9, 2008.
4.  Pasquali SK, Peterson ED, Jacobs JP, He X, Li JS, Jacobs ML, Gaynor JW, Hirsch JC, Shah SS,
Mayer JE. Differential case ascertainment in clinical registry versus administrative data and
impact on outcomes assessment for pediatric cardiac operations. Ann Thorac Surg. 2013 Jan;
95(1):197-203. doi: 10.1016/j.athoracsur.2012.08.074. Epub 2012 Nov 7. PMID: 23141907.

International Paediatric and Congenital Cardiac Code
(IPCCC)
and
Eleventh Iteration of the International Classification of
Diseases
(ICD-11)
www.ipccc.net

The Report of the 2015 STS
Congenital Heart Surgery
Practice Survey
  undertaken by the Society of Thoracic
Surgeons Workforce on Congenital Heart
Surgery
  125 centers in the United States of
America perform pediatric and congenital
heart surgery
  8 centers in Canada perform pediatric
and congenital heart surgery
Morales DL, Khan MS, Turek JW, Biniwale R, Tchervenkov CI, Rush M, Jacobs JP, Tweddell
JS, Jacobs ML. Report of the 2015 Society of Thoracic Surgeons Congenital Heart
Surgery Practice Survey. Ann Thorac Surg. 2017 Feb;103(2):622-628. doi: 10.1016/
j.athoracsur.2016.05.108. Epub 2016 Aug 20. PMID: 27553498.

Jacobs JP, Jacobs ML, Mavroudis C, Tchervenkov CI, Pasquali SK. Executive Summary: The Society of Thoracic
Surgeons Congenital Heart Surgery Database – Twenty-sixth Harvest – (January 1, 2013 – December 31, 2016).
The Society of Thoracic Surgeons (STS) and Duke Clinical Research Institute (DCRI), Duke University Medical
Center, Durham, North Carolina, United States, Spring 2017 Harvest.
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Par,cipa,ng Centers 18 21 34 47 58 68 79 93 101 105 111 113 117 116
18
21
34
47
58
68
79
93
101
105
111
113
117 116
0
20
40
60
80
100
120
140
Growth in the STS Congenital Heart Surgery Database
Par.cipa.ng Centers Per Harvest

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Opera,ons 16,461 28,351 37,093 45,635 61,014 72,002 91,639 103,664 114,041 130,823 136,617 143,842 153,558 157,357
16,461
28,351
37,093
45,635
61,014
72,002
91,639
103,664
114,041
130,823
136,617
143,842
153,558
157,357
0
20,000
40,000
60,000
80,000
100,000
120,000
140,000
160,000
180,000
Opera.ons per averaged 4 year data collec.on cycle

2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Cumula,ve Opera,ons 9,747 16,537 26,404 39,988 58,181 79,399 98,406 119,266 148,110 179,697 213,416 257,932 292,828 331,672 394,980 435,373
9,747 16,537
26,404
39,988
58,181
79,399
98,406
119,266
148,110
179,697
213,416
257,932
292,828
331,672
394,980
435,373
0
50,000
100,000
150,000
200,000
250,000
300,000
350,000
400,000
450,000
500,000
Cumula.ve opera.ons over .me

STS Database
Penetrance in USA
  The STS Congenital Heart Surgery Database (STS-CHSD) is the largest
clinical database in the world for congenital and pediatric cardiac
surgery.
  The Report of the 2010 STS Congenital Heart Surgery Practice and
Manpower Survey, undertaken by the STS Workforce on Congenital Heart
Surgery, documented that 125 hospitals in the United States of America
and 8 hospitals in Canada perform pediatric and congenital heart
surgery.
  The STS-CHSD contains data from 120 of the 125 hospitals (96%
penetrance by hospital) in the United States of America and 3 of the 8
centers in Canada.

STS Database
Penetrance in USA
  The STS Congenital Heart Surgery Database (STS-CHSD) is the largest
clinical database in the world for congenital and pediatric cardiac
surgery.
  The Report of the 2010 STS Congenital Heart Surgery Practice and
Manpower Survey, undertaken by the STS Workforce on Congenital Heart
Surgery, documented that 125 hospitals in the United States of America
and 8 hospitals in Canada perform pediatric and congenital heart
surgery.
  The STS-CHSD contains data from 120 of the 125 hospitals (96%
penetrance by hospital) in the United States of America and 3 of the 8
centers in Canada.
REPRESENTATIVE

Adjustment for
Case Mix
“Differences in medical outcomes may result from
disease severity, treatment effectiveness, or
chance.
Because most outcome studies are observational….
risk adjustment is necessary to account for case mix”
Shahian DM, Blackstone EH, Edwards FH, Grover FL,
Grunkemeier GL, Naftel DC, Nashef SA, Nugent WC, Peterson
ED. STS workforce on evidence-based surgery. Cardiac
surgery risk models: a position article. Ann Thorac Surg.
2004;78(5):1868–77

0
5
10
15
20
% Mortality
% Mortality 0.78 2.1 3.4 8.5 19.9
1 2 3 4 5STAT Category
Combined ECHSA/EACTS and STS Congenital
Heart Surgery Databases:
111,494 index cardiac
operations
Jacobs JP, Jacobs ML, Maruszewski B, Lacour-Gayet FG, Tchervenkov CI, Tobota Z, Stellin G, Kurosawa H,
Murakami A, Gaynor JW, Pasquali SK, Clarke DR, Austin EH 3rd, Mavroudis C. Initial application in the EACTS
and STS Congenital Heart Surgery Databases of an empirically derived methodology of complexity
adjustment to evaluate surgical case mix and results. Eur J Cardiothorac Surg. 2012 Nov;42(5):775-80. doi:
10.1093/ejcts/ezs026. Epub 2012 Jun 14. PMID: 22700597.

STS Congenital Heart Surgery
Database Mortality Risk Model
Variable
Age a
Primary procedure b
Weight (neonates and infants)
Prior cardiothoracic operation
Any non-cardiac congenital anatomic abnormality (except ‘Other noncardiac congenital abnormality’
with code value = 990)
Any chromosomal abnormality or syndrome (except ‘Other chromosomal abnormality’ with code
value = 310 and except ‘Other syndromic abnormality’ with code value = 510)
Prematurity (neonates and infants)
Preoperative Factors
• Preoperative/Preprocedural mechanical circulatory support (IABP, VAD, ECMO, or CPS) c
• Shock, Persistent at time of surgery
• Mechanical ventilation to treat cardiorespiratory failure
• Renal failure requiring dialysis and/or Renal dysfunction
• Preoperative neurological deficit
• Any other preoperative factor (except ‘Other preoperative factors’ with code value = 777) d
a Modeled as a piecewise linear function with separate intercepts and slopes for each STS-defined age group
(neonate, infant, child, adult).
b The model adjusts for each combination of primary procedure and age group. Coefficients obtained via
shrinkage estimation with The Society of Thoracic Surgeons–European Association for Cardio-Thoracic
Surgery (STS-EACTS [STAT]) Mortality Category as an auxiliary variable.
c CPS = cardiopulmonary support; ECMO =extracorporeal membrane oxygenation; IABP = intraaortic balloon
pump; VAD = ventricular assist device
d Any other preoperative factor is defined as any of the other specified preoperative factors contained in the list
of preoperative factors in the data collection form of the STS Congenital Heart Surgery Database, exclusive of
777 = ‘Other preoperative factors’.

• All index cardiac operations in the STS-CHSD
(January 1, 2010–December 31, 2013) were
eligible for inclusion.
• Isolated PDA closures in patients <2.5kg were
excluded, as were centers with >10%
missing data and patients with missing data
for key variables.

52,224 operations
from 86 centers were
included

Model

Covariates
Development
Sample C-Stat
Valida.on
Sample C-Stat
1 STAT Levels
C = 0.772 C = 0.787
2 STAT Levels +
age and weight C = 0.818 C = 0.817
3 STAT Levels +
age and weight +
pa,ent factors
C = 0.862 C = 0.852
4 Primary procedure +
age and weight C = 0.846 C = 0.831
(Final Model) Primary procedure +
age and weight +
pa,ent factors
C = 0.875 C = 0.858

42
Fig 1. Distribution of hospital-specific observed-to-expected (O/E)
ratios for operative mortality with 95% confidence intervals (gray
lines).

43
lines).

44
lines).

45
lines).

46
lines).

Congenital Heart Disease
Meaningful
Multi-institutional Outcomes Analysis
Accomplishments
1)  Common Language = Nomenclature
2)  Mechanism of Data Collection (Database - Registry)
3)  Mechanism of Evaluating Case Complexity
4)  Mechanism to Verify Data Validity and Accuracy
5)  Collaboration Between Subspecialties
6)  Longitudinal Follow-Up and Linked Databases
7)  Quality Improvement
Barach P, Jacobs JP, Lipshultz SE, Laussen P. (Eds.). Pediatric and
Congenital Cardiac Care - Volume 1: Outcomes Analysis.
Springer-Verlag London. Pages 1 – 515. ISBN: 978-1-4471-6586-6
(Print). 978-1-4471-6587-3 (Online). Published in 2014.

STS CHSD Data
Verification
10% of sites audited each year
Analysis of general variables
–  data completeness rate of 99.94% and
–  overall data agreement rate of 98.05%
Analysis of mortality variables
–  data completeness rate of 100% and
–  overall data agreement rate of 99.09%

Jacobs JP. (Editor). 2008 Cardiology in the Young
Supplement: Databases and The Assessment of
Complications associated with The Treatment of Patients
with Congenital Cardiac Disease, Prepared by: The Multi-
Societal Database Committee for Pediatric and Congenital
Heart Disease, Cardiology in the Young, Volume 18, Supplement
S2, pages 1 –530, December 9, 2008.

Collaboration Between Subspecialties
Accomplishments
1)  STS Congenital Heart Surgery Database
2)  IMPACT Database of the American College of
Cardiology (Interventional Cardiology)
3)  MAP-IT: Multicenter Pediatric and Adult Congenital EP
Common Language = Nomenclature
4)  Pediatric Cardiac Critical Care Consortium (PC4)
5)  Congenital Cardiac Anesthesia Society Database
(CCAS)

“Science tells us what we can do;
Guidelines what we should do; &
Registries what we are actually doing.”

Outcomes Analysis
Pa.ent Safety
Quality
Improvement

The U.S. News Rankings: Future Role of the Reputation Survey (Jeffery Jacobs)

Recommended

Recommended

More Related Content

Similar to The U.S. News Rankings: Future Role of the Reputation Survey (Jeffery Jacobs)

Similar to The U.S. News Rankings: Future Role of the Reputation Survey (Jeffery Jacobs) (20)

More from U.S. News Healthcare of Tomorrow

More from U.S. News Healthcare of Tomorrow (20)

Recently uploaded

Recently uploaded (20)

The U.S. News Rankings: Future Role of the Reputation Survey (Jeffery Jacobs)