Validating & Promoting your Program's Success Using ROI and other Evaluation Methods

1. June 1st, 2017
Karl Ensign, Chief, Performance Improvement, Research & Evaluation (PIRE)
Association of State & Territorial Health Officials (ASTHO)
Validating & Promoting your Program’s Success
Using ROI & other Evaluation Methods
2nd Annual Practical Playbook National Meeting
Westin City Center --Washington, DC

2. What’s keeping you up at
night…???
II need to show
how my programs
are performing.
How do I develop
performance
measures??

3. Things to remember
• Takes time and effort – doesn’t “just happen”
• Build in staff time and resources
• Data collection can’t be sprung on people last minute – quality
and reliability will suffer
• Make this an expectation from the beginning
• Important that measures aren’t seen as “imposed from above” –
may not reflect a full understanding of implementation
• Involve key stakeholders in all aspects of design and
interpretation
• Most successful as a learning process – knowledge development
• Not a thumbs up or thumbs down but rather continuous
improvement

4. Differences between monitoring &
evaluation
• Monitoring – Is the program, policy, or initiative functioning
as intended?
• Evaluation – What is the impact of the program, policy, or
initiative?

5. Beginning the journey
Evaluation Roadmap
Phase I: Planning and conceptualization
Phase II: Data collection and analysis
Phase III: Reporting and action

6. Phase I: Planning & Conceptualization
First Steps…
Get up to speed
Engage the input of key
stakeholders
•Policymakers
•Funders
•Program Administrators
•Data collectors
Build a logic model

7. But what exactly is a logic model…???

8. Inputs Activities Outputs Outcomes
Short-term Intermediate Long-term
Resources What we
do
What is
produced
What changes
Logic model components
Contextual Variables

9. Advantages of a
logic model
• Interactive, collaborative process
• Makes the theory of change explicit
• Highlights areas of agreement, ambiguity
• Gets everyone on the same page
• Becomes a living, orienting framework throughout the
project

10. • Slide without swoosh and logo.

11. •Process Measures (Monitoring)
•What was produced?
•Outcome Measures (Impact)
•Change in knowledge, status,
behavior, function, values
A word about process measures &
outcomes

12. Select evaluation measures &
definitions from logic model
Measure Definition Instructions
Process Measure:
Partnerships formalized for
Million Hearts initiative
# of MOUs Choose measure and each definition that
applies to your initiative. Numerator will
reflect current number of partnerships
formalized. Denominator will reflect goal.
# of contracts signed
# of informal partnerships established
Process Measure: Protocols
developed and put in place
# of state data exchange protocols developed Choose measure and each definition that
applies to your initiative. Numerator will
reflect current number of protocols developed.
Denominator will reflect goal for protocols for
each year.
# of local data exchange protocols developed
# of local client referral & follow-up protocols
developed
# of state client referral & follow protocols
developed
Outcome Measure: Patients
with reduced blood pressure
# of patients with measureable reduction in BP
over baseline (for instance, 5 mm diastolic and
10 mm systolic change). Should not include
patients with reduced and controlled BP, only
those with reduced BP.
Numerator is hypertensive patients referred to
services with reduced BP. Should not include
patients with reduced AND controlled BP, only
those with reduced BP. Denominator is
patients referred as hypertensive.

13. Phase II: Data Collection &
Analysis – Get going!
Require interim data collection
and reporting
Data collection and reporting
templates help
Need to conduct frequent
quality and consistency checks
Feedback interim results to
advisory group and questions to
data collectors

14. A good measure is SMART!
measure Example – By the end of the project
year, the 4 regional collaboratives
will include 4 – 6 states on
average.
Example – By the end of the project
year, increase estimated state
response time for certain
emergencies by 25%.
Example – By the end of the project
year, 4 states will have
implemented most aspects of the
comprehensive strategies outlined
in the environmental health toolkit.

15. A word about performance measures
or…“process measures with an expectation”
0
1
2
3
4
5
6
Category 1 Category 2 Category 3 Category 4
Chart Title
Series 1 Series 2 Series 3
Goal
Red Flag

17. “ROI is one way of
measuring and
communicating public
health effectiveness in a
manner that is particularly
salient for policymakers,
funders, administrators
and the general public”
• Helps answer the following
questions:
• Are we making the right investments?
• Are we becoming more efficient?
• What is our budget accomplishing?
• What returns do different investments
yield?
• Must be done thoughtfully and
carefully
A Word about ROI

18. ROI calculated:
 Net benefit = Benefits – Costs
 ROI = Benefits – Costs
Costs
• Hypothetical values:
̶ $5 = ($400 + $500 + $300) – ($150 + $50)
($150 + $50)
 Or …

19. “…a dollar spent on pediatric
immunization is estimated to save
$5 in treating preventable illness.”

20. ASTHO’s Web-Based ROI Tool
Let’s see how it works!

21. Tool structure –
Process improvements or project implementation
• Define process/project time periods
• Plan Phase – Pre-Implementation,
Baseline
• Do – Initial Implementation
• Study – Mid-course Corrections
• Act – Full Implementation

22. ROI tool makes comparisons over
time (PDSA)
• Investment costs
• Planning and tracking implementation of the initiative
(labor, other direct, and indirect costs)
• Routine operating costs
• Delivery of project or program that is implemented or
altered (labor, other direct, and indirect costs)
• Outputs or outcomes
• Additional benefits (health outcomes) realized through the
initiative

23. Planning and Development Costs
Planning & Tracking
Cost Category Plan
(Time1)
Do
(Time2)
Study
(Time3)
Act
(Time4)
Personnel Costs
Non-Personal
• Contracted Services
• Office Operations
• Facilities/Maint/Rent
• Communications
• Equipment
• Construction/
Renovation
Other Direct Costs
Indirect Costs

24. Routine Operating Costs
Delivery
Cost Category Plan
(Time1)
Do
(Time2)
Study
(Time3)
Act
(Time4)
Personnel Costs
Non-Personal
• Contracted Services
• Office Operations
• Facilities/Maint/Rent
• Communications
• Equipment
• Construction/
Renovation
Other Direct Costs
Indirect Costs

25. Phase III: Reporting & Action
• Engage stakeholder group in reviewing the findings, shaping
message
• Don’t bury findings in lengthy reports
• Executive Summary – short on analysis process and big on
findings and implications
• Have short, concise, stand-alone documents for key
messages and audiences
• Develop action plan for dissemination and communication
• Who
• Where
• How

26. Getting from process to impact
Target Population
Patients Identified
Patients Referred
Patients Followed-Up
General Population
Undiagnosed Patients Screened
Patients with
Reduced/Controlled
Blood Pressure
Patients Adhering to Treatment
Plan

27. Help us tell your story…
Since we began our CDC/ASTHO Million Hearts Learning
Collaborative work, we have identified _#_ people in our
target population. By implementing our intervention, _#_ of
undiagnosed patients were screened for hypertension. Of
those, _#_ were identified as hypertensive. Of those identified
as hypertensive, _#_ were referred to services. Of those
referred to services, _#_ followed-up on their referral. Of
those who followed-up, _#_ are adhering to a treatment plan.
Of those adhering to a treatment plan, _#_ have reduced
blood pressure. Of those with reduced blood pressure, _#_
have their hypertension under control.

28. Key points for the evaluation journey…
Select the strategy that is right for you and the situation
 Evaluation – impact, ROI
 Monitoring
Don’t shortchange the time and effort needed
 The 10% rule
 Build expectations into project from beginning
Involve a wide range of stakeholders early and often
 Program administrators and end-users of the
analysis
 Can help foster a dialogue about intent of
policy/program and data interpretation
Especially when starting, less can be better
 A few key measures and data collection points can
go a long ways
 Process measures form the foundation
 Outcome measures show the impact
Think about the end first!!

29. Questions and Discussion