Sills MR. Leveraging SAFTINet resources to enhance value in performance measurement. Annual Convocation of the Scalable Architecture for Federated Translational Inquiries Network (SAFTINet). Aurora, Colorado, February 2014. For more information on SAFTINet, please see http://www.ucdenver.edu/academics/colleges/medicalschool/programs/outcomes/COHO/saftinet/Pages/default.aspx

1. Marion R. Sills, MD, MPH
SAFTINet co-Investigator
LEVERAGING SAFTINET
RESOURCES TO
ENHANCE VALUE IN
PERFORMANCE
MEASUREMENT

2. Questions
• In general, what makes a performance measure more/less
actionable? (or valuable)?
• What makes UDS adult weight measure, specifically, more/less actionable?
• How generalizable to other UDS, meaningful use measures?
• What does actionable usually mean?
• What kinds of actions would you take?
• How would we detect action/follow-up?
• What features of a measure or a report would affect actionability?

3. Value of Performance Measures
• Attributes of performance measures related to their value* include:
• the relevance of the measure to a topic of importance* that will inform
important* healthcare decisions
• the scientific soundness of the measure, including validity, accuracy and
reproducibility
• the feasibility of the measure, including logistical and financial burden
• relevance
• scientific soundness
• feasibility*to stakeholders

4. Limitations of Current Measure
• Relevance
• Is the measure easily interpreted?
• What is the significance of the measure to stakeholders?
• What are the policy and financial implications of implementing the
measure? Does it encourage activities that use resources efficiently to
maximize health?
• What is the prevalence and overall impact (health, financial) of the
condition in the population? {selected for this in choosing measure}
• What control does the measured entity have over the condition?
• Will there be wide variations across systems?
• How much room is available for measured entities to improve
performance?
• relevance
• scientific soundness
• feasibility

5. Limitations of Current Measure
• Scientific soundness
• Does the measure precisely evaluate what is actually happening? What
measures of data quality are reported?
• What is the strength of the evidence linking the clinical processes and
outcomes that the measure addresses?
• Does the measure produce reproducible results when repeated in the
same population and setting?
• Does the measure make sense logically and clinically? (validity)
• Is it appropriate to risk-adjust the measure by age or some other variable?
• Are accuracy, reproducibility and validity consistent across different data
systems and settings?
• relevance
• scientific soundness
• feasibility

6. Limitations of Current Measure
• Feasibility
• Does the measure impose an inappropriate burden on health care systems?
• Does the measure have clear specifications for data sources and methods
for data collection and reporting?
• Does the data collection violate accepted standards of member
confidentiality?
• Is the required data logistically feasible to access?
• Is the measure susceptible to manipulation that would be undetectable in
an audit?
• relevance
• scientific soundness
• feasibility

7. Selected UDS Measure, Table 6B

8. Limitations of Current Measure
• Relevance
• Is the measure easily interpreted?
• relevance
• scientific soundness
• feasibility
UDS process measure
Composite measure obfuscates what
portion of the compliance rate
reflects documentation of
• BMI
• follow-up
• underweight or overweight
• by age
• by type: nutrition, exercise

9. Limitations of Current Measure
• Parsing the composite measure may help stakeholders identify
which components contribute to the compliance rate
• Easier to interpret
Adults with > 1 visit
BMI
documented
BMI > 30
Follow-up
documented

10. UDS process measure
Limitations of Current Measure
• Relevance
• What is the significance of the measure to stakeholders?
• Does it target measures of value? Stakeholder-valued process measures
(hypothetical)
Adults with > 1 visit
BMI
documented
Stakeholder-valued outcome measures
BMI > 30
Follow-up
documented
?

11. UDS process measure
Limitations of Current Measure
• Relevance
• What is the significance of the measure to stakeholders?
• Does it target measures of value? Stakeholder-valued process measures
(hypothetical)
Adults with > 1 visit
BMI
documented
?
Stakeholder-valued outcome measures
BMI > 30
Follow-up
documented
Do stakeholders give
equal value to follow-up
documentation in
• a 65 year old with a
BMI of 21 as in
• a 21 year old with a
BMI of 65?

12. UDS process measure
Limitations of Current Measure
• Relevance
• What is the significance of the measure to stakeholders?
• Does it target measures of value? Stakeholder-valued process measures
(hypothetical)
Adults with > 1 visit
BMI
documented
?
Stakeholder-valued outcome measures
BMI > 30
Follow-up
documented
Measure could have
enhanced value if made
concordant with a BMI
cut-off of value to
stakeholders

13. Limitations of Current Measure
• Relevance
• Does the measure encourage activities that use resources efficiently to
maximize health?
• Is the data useful for improving improved performance?
• Aggregate nature of measure makes it hard to use for process
improvement
• cannot compare providers or practices
• thus cannot measure improvement associated with practice-level interventions
• cannot target individual patients for case management or other services
• relevance
• scientific soundness
• feasibility

14. Addressing Relevance-Related Limitations
• Addressing lack of concordance with measures of value
• Add relevant variables to measure
0
10
20
30
40
50
60
70
80
Org 1 Org 2 Org 3 Org 4
UDS Adult Weight Measure Compliance, 2012

15. Addressing Relevance-Related Limitations
• Addressing composite nature of current measure
Parameter Selection
BMI documented
 Yes
 No
Follow-up documented
 Yes
 No
Age and BMI parameters
 < age 65 AND BMI > 25
 > age 65 AND BMI > 30
 < age 65 AND BMI < 18.5
 > age 65 AND BMI < 22
0%
10%
20%
30%
40%
50%
60%
Org 1 Org 2 Org 3 Org 4
UDS Adult Weight Measure Compliance, 2012

16. Addressing Relevance-Related Limitations
• Addressing lack of concordance with measures of value
0%
10%
20%
30%
40%
50%
60%
Org 1 Org 2 Org 3 Org 4
UDS Adult Weight Measure Compliance, 2012
Parameter Selection
BMI documented
 Yes
 No
Follow-up documented
 Yes
 No
Age and BMI parameters
 < age 65 AND BMI > 25
 > age 65 AND BMI > 30
 < age 65 AND BMI < 18.5
 > age 65 AND BMI < 22
 Age _____ to _____
 BMI _____ to _____ Age _____ to _____
 BMI _____ to _____
 Number of visits in 12 months _____ to _____
 Hgb A1C _____ to _____
 Systolic BP _____ to _____
 Diastolic BP _____ to _____
 Hospitalizations in 12 months _____ to _____
 Nutrition
 Exercise

17. Addressing Utility-Related Limitations
• Addressing aggregate nature of current measure
Population
Practice
 Main St Clinic
 West Health Practice
 Whole Family Center
 Casey Cares Clinic
 Peaceful Practice
 Wellness Center
Provider
 Brett Almond
 Pat Grant
 Darby Eden
 Carson Brooke
 Harper Keegan
 Addison Keith
 Kelly Jordan
0
10
20
30
40
50
60
70
Main St Clinic West Health
Practice
Whole Family
Center
Casey Cares
Clinic
Peaceful
Practice
Wellness
Center
0
10
20
30
40
50
60
70
Brett Almond Pat Grant Darby Eden Carson
Brooke
Harper
Keegan
Addison
Keith
Kelly Jordan

18. Addressing Utility-Related Limitations
• Addressing aggregate nature of current measure: patient registry

19. Limitations of Current Measure
• Scientific soundness
• Does the measure precisely evaluate what is actually happening? What
measures of data quality are reported?
• What is the strength of the evidence linking the clinical processes and
outcomes that the measure addresses?
• Does the measure produce reproducible results when repeated in the
same population and setting?
• Does the measure make sense logically and clinically? (validity)
• Is it appropriate to risk-adjust the measure by age or some other variable?
• Are accuracy, reproducibility and validity consistent across different data
systems and settings?
• relevance
• scientific soundness
• feasibility

20. Limitations of Current Measure
• Scientific soundness: data quality
• If chart-review is used to derive UDS measure
• inter-relater variation:
• measure elements—especially deciding what constitutes
follow-up—are not simple to identify
• observation bias, misclassification
• If EHR is used to derive UDS measure
• clinicians vary
• what justifies clicking a “follow-up done’ checkbox
• how often they forget to click the checkbox
(completeness of data)
• EHRs vary in how easily data is recorded and
extracted

21. Limitations of Current Measure
• Scientific soundness
• Lack of information about data quality in the
measure reporting is a barrier to
• interpreting and using the measure
• fixing data quality issues
• relevance
• scientific soundness
• feasibility

22. Addressing Scientific Soundness-Related
Limitations
• Addressing lack of information about data quality
0
10
20
30
40
50
60
70
Main St Clinic West Health
Practice
Whole Family
Center
Casey Cares
Clinic
Peaceful
Practice
Wellness
Center
Data Quality Elements
Out-of-range values
 Height > 260 cm
 Height < 60 cm
 Weight > 200 kg
 Weight < 20 kg
 Pregnant males
Missing values
 Missing height
 Missing weight
 Missing BMI
 Missing follow-up
 Missing provider
Population
Practice
 Main St Clinic
 West Health Practice
 Whole Family Center
 Casey Cares Clinic
 Peaceful Practice
 Wellness Center
Provider
 Brett Almond
 Pat Grant
 Darby Eden
 Carson Brooke
 Harper Keegan
 Addison Keith
 Kelly Jordan

23. Limitations of Current Measure
• Feasibility
• Does the measure impose an inappropriate burden on health care systems?
• Does the measure have clear specifications for data sources and methods
for data collection and reporting?
• Does the data collection violate accepted standards of member
confidentiality?
• Is the required data logistically feasible to access?
• Is the measure susceptible to manipulation that would be undetectable in
an audit?
• relevance
• scientific soundness
• feasibility

24. Limitations of Current Measure
• Feasibility
• Chart review of 70 records is resource intensive
• What would be the resource commitment involved
in transitioning to EHR-based reporting of this
measure?
• relevance
• scientific soundness
• feasibility

25. R/QI Committee Process
• Define priorities for addressing limitations
• Refine and finalize scope-of-work for addressing key limitations of
the current UDS measure
• Develop reports that address these limitations
• Define and pursue next steps
• assess changes in relevance, scientific soundness and feasibility
• disseminate findings
• to whom?
• what format?
• possible groundwork for pragmatic trial
• relevance
• scientific soundness
• feasibility