DGHD Program Manager Retreat Evaluation Presentation

IMPROVING OUTCOME
EVALUATION
DGHD Program Managers Retreat -July 25 2013
Christopher Fink, PhD
Ohio Wesleyan University
Dept. of Health & Human Kinetics
Thursday, July 25, 13

OVERVIEW
• Importance of Evaluation
• Improving Evaluation through
Planning (Logic model)
• Evaluation Considerations and
Methods
• Application to DGHD
Priorities
• Discussion

TEN ESSENTIAL SERVICES
OF PUBLIC HEALTH
1. Monitor health status to identify
community health problems.
2. Diagnose and investigate health
problems and health hazards in the
community.
3. Inform, educate, and empower people
about health issues.
4. Mobilize community partnerships to
identify and solve health problems.
5. Develop policies and plans that
support individual and community
health efforts.
6. Enforce laws and regulations that
protect health and ensure safety.
7. Link people to needed personal health
services and assure the provision of
health care when otherwise
unavailable.
8. Assure a competent public health and
personal healthcare workforce.
9. Evaluate effectiveness, accessibility, and
quality of personal and population-
based health services.
10.Research for new insights and
innovative solutions to health
problems.
http://www.apha.org/programs/standards/performancestandardsprogram/
resexxentialservices.htm

TEN ESSENTIAL SERVICES
OF PUBLIC HEALTH
1. Monitor health status to identify
community health problems.
2. Diagnose and investigate health
problems and health hazards in the
community.
3. Inform, educate, and empower people
about health issues.
4. Mobilize community partnerships to
identify and solve health problems.
5. Develop policies and plans that
support individual and community
health efforts.
6. Enforce laws and regulations that
protect health and ensure safety.
7. Link people to needed personal health
services and assure the provision of
health care when otherwise
unavailable.
8. Assure a competent public health and
personal healthcare workforce.
9.Evaluate effectiveness,
accessibility, and quality of
personal and population-
based health services.
10.Research for new insights and
innovative solutions to health
problems.
http://www.apha.org/programs/standards/performancestandardsprogram/
resexxentialservices.htm

"It does not take much strength to do things, but it
requires a great deal of strength to decide what to do."
Elbert Hubbard
http://www.pickthebrain.com/blog/the-strength-to-overcome/

EVALUATION &
THEORETICAL FRAMEWORK

EVALUATION &
QOL
Improvement

EVALUATION &
Health
QOL
Improvement

EVALUATION &
Behavioral
Outcomes
Health
QOL
Improvement

EVALUATION &
Behavioral
Outcomes
Health
QOL
Improvement
Environmental
Outcomes

EVALUATION &
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Environmental
Outcomes

EVALUATION &
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Performance
Objectives
Environmental
Outcomes

EVALUATION &
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Performance
Objectives
Environmental
Outcomes

EVALUATION &
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Performance
Objectives
Environmental
Outcomes
Personal
Determinants

EVALUATION &
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Performance
Objectives
Environmental
Outcomes
Personal
Determinants

EVALUATION &
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Change
Objectives
Performance
Objectives
Environmental
Outcomes
Personal
Determinants

EVALUATION &
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Change
Objectives
Performance
Objectives
Environmental
Outcomes
Personal
Determinants
At-Risk Group

EVALUATION &
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Change
Objectives
Performance
Objectives
Environmental
Outcomes
Personal
Determinants
At-Risk Group
Environmental Agent

EVALUATION &
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Change
Objectives
Performance
Objectives
Environmental
Outcomes
Personal
Determinants
At-Risk Group
Environmental Agent
{

EVALUATION &
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Change
Objectives
Performance
Objectives
Environmental
Outcomes
Personal
Determinants
At-Risk Group
Environmental Agent
{
Logic of Change

EVALUATION &
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Change
Objectives
Performance
Objectives
Environmental
Outcomes
Personal
Determinants
At-Risk Group
Environmental Agent
{
Logic of Change
{Thursday, July 25, 13

EVALUATION &
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Change
Objectives
Performance
Objectives
Environmental
Outcomes
Personal
Determinants
At-Risk Group
Environmental Agent
{
Logic of Change
{Outcomes

SOCIAL ECOLOGICAL MODEL
http://www.ahrq.gov/professionals/prevention-chronic-care/resources/clinical-community-relationships-measures-atlas/ccrm-atlas3.html

http://www.louisianabreastfeedingcoalition.org/wp-content/uploads/2010/09/SEM2.png
SEM: Louisiana Breastfeeding Coalition

DETERMINANTS:ASSESSING
IMPORTANCE & CHANGEABILITY
Determinants Importance Changeability
Evidence for
Importance
Knowledge + +++
Precondition for
personal attitude
Risk perception + +
Precondition for
personal relevance
Self-efﬁcacy +++ + r=0.71, p<0.01
Personal norm +++ + r=0.78, p<0.01
Subjective norm + + r=0.29, p<0.01
Skills ++ +
Precondition for self-
efﬁcacy improvement

EVALUATION &
THEORETICAL FRAMEWORK:
LOGIC MODEL
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Change
Objectives
Performance
Objectives
Environmental
Outcomes
Personal
Determinants
At-Risk Group
Environmental Agent
{
Logic of Change
{Outcomes

MATRIX FOR CHILDREN IN
SPF PROGRAM
Behavior: Wear Sunscreen and Protective Clothing When OutdoorsBehavior: Wear Sunscreen and Protective Clothing When OutdoorsBehavior: Wear Sunscreen and Protective Clothing When OutdoorsBehavior: Wear Sunscreen and Protective Clothing When OutdoorsBehavior: Wear Sunscreen and Protective Clothing When Outdoors
Personal DeterminantsPersonal DeterminantsPersonal DeterminantsPersonal DeterminantsPersonal Determinants
Performance
Objectives
(Children)
Attitudes Skills/Self-Efﬁcacy Knowledge
Outcome
Expectations
PO 1. Cooperate
with sun
protection
practices by
parent or
preschool staff
A.1 Express
positive feeling
toward being
protected from
the sun
OE 1. Describe how
cooperating will keep
skin healthy
PO 1.1 Stand still
for application
A. 1.1 Express
positive attitude
toward being a
helper
K 1.1 Explain
how standing still
allows sunscreen
to be put on
evenly

MATRIX FOR ENVIRONMENT
IN SPF PROGRAM
Organizational Environmental Change in SPF ProgramOrganizational Environmental Change in SPF ProgramOrganizational Environmental Change in SPF ProgramOrganizational Environmental Change in SPF ProgramOrganizational Environmental Change in SPF Program
Personal DeterminantsPersonal DeterminantsPersonal DeterminantsPersonal DeterminantsPersonal Determinants
Performance
Objectives
(Agents)
Perceived
Norms
Attitudes Skills/Self-Efﬁcacy
Outcome
Expectations /
Perceived
Susceptibility
PO 1. Preschool
directors decide
to prioritize
increasing amt. of
shade areas used
by preschoolers
PN 1. Recognize
sun protection
and shade
adequacy as
concerns for
preschool
administration
A 1. Describe
assuring shade as
positive
OEPS 1. a Describe
children as
susceptible to skin
cancer caused by
lack of shade
OEPS 1. b.Argue
that increasing shade
will decrease risk

LOGIC MODEL LINKTO
EVALUATION

LOGIC MODEL LINKTO
EVALUATION
• Behavioral / Environmental Outcomes = Impact of Activities/Program

LOGIC MODEL LINKTO
EVALUATION
• Performance Objectives = Effectiveness of Activities/Program
• Speciﬁcity of objectives is key
• Uncover key factors for behavioral/environmental success
• Often help to deﬁne the behavior or environmental condition (improving
evaluation)

LOGIC MODEL LINKTO
EVALUATION
• Performance Objectives = Effectiveness of Activities/Program
• Speciﬁcity of objectives is key
• Uncover key factors for behavioral/environmental success
• Often help to deﬁne the behavior or environmental condition (improving
evaluation)
• Change Objectives = Immediate impact

What we observe is not nature itself, but nature
exposed to our method of questioning.
-Werner Heisenberg
http://resources0.news.com.au/images/2012/11/06/1226511/766824-solar-eclipse-glasses.jpg

EVALUATION BASICS

EVALUATION BASICS
• Formative: MAPP

EVALUATION BASICS
• Formative: MAPP
• Process: Implementation

PROCESS EVALUATION

PROCESS EVALUATION
• Reach: The proportion of the intended audience to whom
the program is actually delivered

PROCESS EVALUATION
• Dose Delivered:The amount of intended units of each
program component that is delivered

PROCESS EVALUATION
• Dose Received:The extent to which the participants
engage with the program

PROCESS EVALUATION
• Dose Received:The extent to which the participants
engage with the program
• Fidelity:The extent to which the program was delivered as
intended

EVALUATION BASICS
• Formative: MAPP
• Impact: Immediate effects

EVALUATION BASICS
• Formative: MAPP
• Outcome: Ultimate results

EVALUATION BASICS
• Formative: MAPP
• Outcome: Ultimate results
Key: Matching measurement to outcome of interest!

QUALITATIVE +
QUANTITATIVE METHODS

QUALITATIVE + QUANTITATIVE
COMPLIMENTARY METHODS

MEASUREMENT
Qualitative Quantitative

• Interviews with key informants
MEASUREMENT

• Observations of program activities
MEASUREMENT

• Focus groups
MEASUREMENT

• Focus groups
• Thematic analysis
MEASUREMENT

• Focus groups
• Demographics
MEASUREMENT

• Focus groups
• Demographics
• Pre-post test scores
MEASUREMENT

• Focus groups
• Demographics
• Ratings of attitude and self-efﬁcacy
MEASUREMENT

• Focus groups
• Demographics
• Existing numerical data
MEASUREMENT

• Focus groups
• Demographics
• Existing numerical data
• Statistical analysis
MEASUREMENT

Qualitative Quantitative Result
Qualitative
ResultQuantitative
Quantitative
Qualitative Result
EVALUATION METHODS:
Qualitative Result Quantitative
1
2
3
4

EXAMPLES OF QUALITATIVE +
QUANTITATIVE DATA USE
What is being
measured?
Quantitative Data Qualitative Data
Awareness
Written instrument (y/n) to
measure awareness of PSA on
healthy eating
Interviews at local supermarkets
to measure awareness of healthy
eating campaign
Knowledge
Written/oral knowledge
questionnaire (pre-post, likert) to
measure knowledge gained at
nutrition seminar
Interviews to measure
knowledge about nutritional
value of f/v, focus groups w/
children about same
Attitudes
Written instrument (pre-post,
likert) to measure attitudes and
preferences of children for f/v
Interviews with mothers of small
children to ID attitudes toward
feeding kids f/v
Behavior
Self-report w/ scaled responses
to measure freq. + type of
healthy eating, obs. counts
Interviews with individuals to
gather info on eating behaviors,
focus groups, obs.

EXAMPLES OF QUALITATIVE +
QUANTITATIVE DATA USE
What is being
measured?
Quantitative Data Qualitative Data
Policy changes
Measurement of number of
children who consume f/v before
and after school policy requiring
salad bars in school
Interviews with children in
school about what they like/
dislike on salad bar since policy
implemented
Organizational Changes
Records documenting # and
demographics of people who go
to food bank for food + now get
info on preparing nutritious food
and food stamp program
Interviews w/ food bank staff to
gather opinions about whether
food bank’s new nutrition ed and
food stamp outreach has positive
effect on clients
Environmental Changes
Survey of farmers’ markets in the
county to document the size of
the markets, types of produce
provided, cost of produce
Observations of and interviews
with people attending farmers’
markets to identify satisfaction
with market

EVALUATION:
OTHERTHOUGHTS
• Link between objective, outcome, and measurement is key
• Follow SMART objective guidelines to make evaluation easier
and more effective
• Use theory and logic model (where possible) to guide the
process to make change more likely, evaluation more effective
• But, don’t get stuck there!
• Be sure to have process, impact, outcome represented in plan

DGHD PRIORITIES:
THOUGHTS
• A healthy community, a chance to
climb even higher
• Great efforts, great people, great
community buy-in
• Excellent priorities and processes
• Where to improve?

DGHD PRIORITIES:
SOME EXAMPLES
OBJECTIVE /
Tactic
TIMEFRAME OUTCOME
MEASUREMENT
a. unit b.collection method
Determine sources
and then monitor,
evaluate and
disseminate county
wide and community
specific obesity data.
June 1, 2012 -
December 31, 2012
for determination of
data sources, then
annually each January
Determine obesity
trends for Delaware
County.
a. % of county residents that are
obese, evaluate change over time
and disseminate information to
community via agency website
b. review of standardized list of
data
Goal 1.1

DGHD PRIORITIES:
SOME EXAMPLES
OBJECTIVE /
Tactic
TIMEFRAME OUTCOME
MEASUREMENT
100% of the clients
requesting birth
certificates will be
given a safe food
handling brochure.
Ongoing
Improved knowledge
of safe food handling.
a. Number of birth certificates
requested and count of those
receiving food safety information
b. Number that receive food safety
information
Goal 3.7

DGHD PRIORITIES:
SOME EXAMPLES
OBJECTIVE /
Tactic
TIMEFRAME OUTCOME
MEASUREMENT
Utilize Wii game
system and “Impaired
Driving Fatal Vision
Mario Cart” at least 10
times, with targeted
populations, annually.
July, 2010 (with
borrowed system), and
ongoing
Raise awareness of
impaired youth driving
in a safe environment.
a. The number of events at which
an interactive impaired driving
program is utilized with high school
students
b. Tracking by staff
Goal 4.2.1

SUMMARY
• Importance of evaluation
• Importance of planning and
objectives in evaluation
• Consistency and using a
framework
• Method based on needs, goals
• Asking the right questions is
key!

Map 0
Full Generic Map
nothing is perfect. life is messy. relationships are
complex. outcomes are uncertain. people are irrational.
hugh mackay

If anyone on the verge of action should judge himself
according to the outcome, he would never begin."
Søren Kierkegaard

Thanks!
Presentation PDF available at:
http://slideshare.net/ChristopherFink
Harvard Campaign & Public Communication Evaluation Example
http://db.tt/pufWl30z
Healthy Maine Healthy Weight Awareness Campaign Evaluation
http://db.tt/zVqSQ7hP
WHO ImmunizationTracking & Data Management Info
http://db.tt/OLu3wEuE

DGHD Program Manager Retreat Evaluation Presentation

Recommended

Recommended

More Related Content

Similar to DGHD Program Manager Retreat Evaluation Presentation

Similar to DGHD Program Manager Retreat Evaluation Presentation (20)

Recently uploaded

Recently uploaded (20)

DGHD Program Manager Retreat Evaluation Presentation