IMPROVING OUTCOME
EVALUATION
DGHD Program Managers Retreat -July 25 2013
Christopher Fink, PhD
Ohio Wesleyan University
De...
OVERVIEW
• Importance of Evaluation
• Improving Evaluation through
Planning (Logic model)
• Evaluation Considerations and
...
TEN ESSENTIAL SERVICES
OF PUBLIC HEALTH
1. Monitor health status to identify
community health problems.
2. Diagnose and in...
TEN ESSENTIAL SERVICES
OF PUBLIC HEALTH
1. Monitor health status to identify
community health problems.
2. Diagnose and in...
"It does not take much strength to do things, but it
requires a great deal of strength to decide what to do."
Elbert Hubba...
EVALUATION &
THEORETICAL FRAMEWORK
Thursday, July 25, 13
EVALUATION &
THEORETICAL FRAMEWORK
QOL
Improvement
Thursday, July 25, 13
EVALUATION &
THEORETICAL FRAMEWORK
Health
QOL
Improvement
Thursday, July 25, 13
EVALUATION &
THEORETICAL FRAMEWORK
Health
QOL
Improvement
Thursday, July 25, 13
EVALUATION &
THEORETICAL FRAMEWORK
Behavioral
Outcomes
Health
QOL
Improvement
Thursday, July 25, 13
EVALUATION &
THEORETICAL FRAMEWORK
Behavioral
Outcomes
Health
QOL
Improvement
Thursday, July 25, 13
EVALUATION &
THEORETICAL FRAMEWORK
Behavioral
Outcomes
Health
QOL
Improvement
Environmental
Outcomes
Thursday, July 25, 13
EVALUATION &
THEORETICAL FRAMEWORK
Behavioral
Outcomes
Health
QOL
Improvement
Environmental
Outcomes
Thursday, July 25, 13
EVALUATION &
THEORETICAL FRAMEWORK
Behavioral
Outcomes
Health
QOL
Improvement
Environmental
Outcomes
Thursday, July 25, 13
EVALUATION &
THEORETICAL FRAMEWORK
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Environmental
Outcome...
EVALUATION &
THEORETICAL FRAMEWORK
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Environmental
Outcome...
EVALUATION &
THEORETICAL FRAMEWORK
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Performance
Objective...
EVALUATION &
THEORETICAL FRAMEWORK
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Performance
Objective...
EVALUATION &
THEORETICAL FRAMEWORK
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement
Performance
Objective...
EVALUATION &
THEORETICAL FRAMEWORK
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement...
EVALUATION &
THEORETICAL FRAMEWORK
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement...
EVALUATION &
THEORETICAL FRAMEWORK
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement...
EVALUATION &
THEORETICAL FRAMEWORK
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Health
QOL
Improvement...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
SOCIAL ECOLOGICAL MODEL
http://www.ahrq.gov/professionals/prevention-chronic-care/resources/clinical-community-relationshi...
http://www.louisianabreastfeedingcoalition.org/wp-content/uploads/2010/09/SEM2.png
SEM: Louisiana Breastfeeding Coalition
...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
EVALUATION &
THEORETICAL FRAMEWORK
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
Outcomes
Heal...
DETERMINANTS:ASSESSING
IMPORTANCE & CHANGEABILITY
Determinants Importance Changeability
Evidence for
Importance
Knowledge ...
EVALUATION &
THEORETICAL FRAMEWORK:
LOGIC MODEL
Change
Objectives
Personal
Determinants
Performance
Objectives
Behavioral
...
MATRIX FOR CHILDREN IN
SPF PROGRAM
Behavior: Wear Sunscreen and Protective Clothing When OutdoorsBehavior: Wear Sunscreen ...
MATRIX FOR ENVIRONMENT
IN SPF PROGRAM
Organizational Environmental Change in SPF ProgramOrganizational Environmental Chang...
LOGIC MODEL LINKTO
EVALUATION
Thursday, July 25, 13
LOGIC MODEL LINKTO
EVALUATION
• Behavioral / Environmental Outcomes = Impact of Activities/Program
Thursday, July 25, 13
LOGIC MODEL LINKTO
EVALUATION
• Behavioral / Environmental Outcomes = Impact of Activities/Program
• Performance Objective...
LOGIC MODEL LINKTO
EVALUATION
• Behavioral / Environmental Outcomes = Impact of Activities/Program
• Performance Objective...
What we observe is not nature itself, but nature
exposed to our method of questioning.
-Werner Heisenberg
http://resources...
EVALUATION BASICS
Thursday, July 25, 13
EVALUATION BASICS
• Formative: MAPP
Thursday, July 25, 13
EVALUATION BASICS
• Formative: MAPP
• Process: Implementation
Thursday, July 25, 13
PROCESS EVALUATION
Thursday, July 25, 13
PROCESS EVALUATION
• Reach: The proportion of the intended audience to whom
the program is actually delivered
Thursday, Ju...
PROCESS EVALUATION
• Reach: The proportion of the intended audience to whom
the program is actually delivered
• Dose Deliv...
PROCESS EVALUATION
• Reach: The proportion of the intended audience to whom
the program is actually delivered
• Dose Deliv...
PROCESS EVALUATION
• Reach: The proportion of the intended audience to whom
the program is actually delivered
• Dose Deliv...
EVALUATION BASICS
• Formative: MAPP
• Process: Implementation
Thursday, July 25, 13
EVALUATION BASICS
• Formative: MAPP
• Process: Implementation
• Impact: Immediate effects
Thursday, July 25, 13
EVALUATION BASICS
• Formative: MAPP
• Process: Implementation
• Impact: Immediate effects
• Outcome: Ultimate results
Thur...
EVALUATION BASICS
• Formative: MAPP
• Process: Implementation
• Impact: Immediate effects
• Outcome: Ultimate results
Key:...
QUALITATIVE +
QUANTITATIVE METHODS
Thursday, July 25, 13
QUALITATIVE + QUANTITATIVE
COMPLIMENTARY METHODS
Thursday, July 25, 13
QUALITATIVE + QUANTITATIVE
MEASUREMENT
Qualitative Quantitative
Thursday, July 25, 13
• Interviews with key informants
QUALITATIVE + QUANTITATIVE
MEASUREMENT
Qualitative Quantitative
Thursday, July 25, 13
• Interviews with key informants
• Observations of program activities
QUALITATIVE + QUANTITATIVE
MEASUREMENT
Qualitative Q...
• Interviews with key informants
• Observations of program activities
• Focus groups
QUALITATIVE + QUANTITATIVE
MEASUREMEN...
• Interviews with key informants
• Observations of program activities
• Focus groups
• Thematic analysis
QUALITATIVE + QUA...
• Interviews with key informants
• Observations of program activities
• Focus groups
• Thematic analysis
QUALITATIVE + QUA...
• Interviews with key informants
• Observations of program activities
• Focus groups
• Thematic analysis
QUALITATIVE + QUA...
• Interviews with key informants
• Observations of program activities
• Focus groups
• Thematic analysis
• Demographics
QU...
• Interviews with key informants
• Observations of program activities
• Focus groups
• Thematic analysis
• Demographics
• ...
• Interviews with key informants
• Observations of program activities
• Focus groups
• Thematic analysis
• Demographics
• ...
• Interviews with key informants
• Observations of program activities
• Focus groups
• Thematic analysis
• Demographics
• ...
• Interviews with key informants
• Observations of program activities
• Focus groups
• Thematic analysis
• Demographics
• ...
Qualitative Quantitative Result
Qualitative
ResultQuantitative
Quantitative
Qualitative Result
EVALUATION METHODS:
QUALITA...
EXAMPLES OF QUALITATIVE +
QUANTITATIVE DATA USE
What is being
measured?
Quantitative Data Qualitative Data
Awareness
Writt...
EXAMPLES OF QUALITATIVE +
QUANTITATIVE DATA USE
What is being
measured?
Quantitative Data Qualitative Data
Policy changes
...
EVALUATION:
OTHERTHOUGHTS
• Link between objective, outcome, and measurement is key
• Follow SMART objective guidelines to...
DGHD PRIORITIES:
THOUGHTS
• A healthy community, a chance to
climb even higher
• Great efforts, great people, great
commun...
DGHD PRIORITIES:
SOME EXAMPLES
OBJECTIVE /
Tactic
TIMEFRAME OUTCOME
MEASUREMENT
a. unit b.collection method
Determine sour...
DGHD PRIORITIES:
SOME EXAMPLES
OBJECTIVE /
Tactic
TIMEFRAME OUTCOME
MEASUREMENT
a. unit b.collection method
100% of the cl...
DGHD PRIORITIES:
SOME EXAMPLES
OBJECTIVE /
Tactic
TIMEFRAME OUTCOME
MEASUREMENT
a. unit b.collection method
Utilize Wii ga...
SUMMARY
• Importance of evaluation
• Importance of planning and
objectives in evaluation
• Consistency and using a
framewo...
Map 0
Full Generic Map
nothing is perfect. life is messy. relationships are
complex. outcomes are uncertain. people are ir...
If anyone on the verge of action should judge himself
according to the outcome, he would never begin."
Søren Kierkegaard
T...
Thanks!
Presentation PDF available at:
http://slideshare.net/ChristopherFink
Harvard Campaign & Public Communication Evalu...
Upcoming SlideShare
Loading in...5
×

DGHD Program Manager Retreat Evaluation Presentation

637

Published on

Published in: Health & Medicine, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
637
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

DGHD Program Manager Retreat Evaluation Presentation

  1. 1. IMPROVING OUTCOME EVALUATION DGHD Program Managers Retreat -July 25 2013 Christopher Fink, PhD Ohio Wesleyan University Dept. of Health & Human Kinetics Thursday, July 25, 13
  2. 2. OVERVIEW • Importance of Evaluation • Improving Evaluation through Planning (Logic model) • Evaluation Considerations and Methods • Application to DGHD Priorities • Discussion Thursday, July 25, 13
  3. 3. TEN ESSENTIAL SERVICES OF PUBLIC HEALTH 1. Monitor health status to identify community health problems. 2. Diagnose and investigate health problems and health hazards in the community. 3. Inform, educate, and empower people about health issues. 4. Mobilize community partnerships to identify and solve health problems. 5. Develop policies and plans that support individual and community health efforts. 6. Enforce laws and regulations that protect health and ensure safety. 7. Link people to needed personal health services and assure the provision of health care when otherwise unavailable. 8. Assure a competent public health and personal healthcare workforce. 9. Evaluate effectiveness, accessibility, and quality of personal and population- based health services. 10.Research for new insights and innovative solutions to health problems. http://www.apha.org/programs/standards/performancestandardsprogram/ resexxentialservices.htm Thursday, July 25, 13
  4. 4. TEN ESSENTIAL SERVICES OF PUBLIC HEALTH 1. Monitor health status to identify community health problems. 2. Diagnose and investigate health problems and health hazards in the community. 3. Inform, educate, and empower people about health issues. 4. Mobilize community partnerships to identify and solve health problems. 5. Develop policies and plans that support individual and community health efforts. 6. Enforce laws and regulations that protect health and ensure safety. 7. Link people to needed personal health services and assure the provision of health care when otherwise unavailable. 8. Assure a competent public health and personal healthcare workforce. 9.Evaluate effectiveness, accessibility, and quality of personal and population- based health services. 10.Research for new insights and innovative solutions to health problems. http://www.apha.org/programs/standards/performancestandardsprogram/ resexxentialservices.htm Thursday, July 25, 13
  5. 5. "It does not take much strength to do things, but it requires a great deal of strength to decide what to do." Elbert Hubbard http://www.pickthebrain.com/blog/the-strength-to-overcome/ Thursday, July 25, 13
  6. 6. EVALUATION & THEORETICAL FRAMEWORK Thursday, July 25, 13
  7. 7. EVALUATION & THEORETICAL FRAMEWORK QOL Improvement Thursday, July 25, 13
  8. 8. EVALUATION & THEORETICAL FRAMEWORK Health QOL Improvement Thursday, July 25, 13
  9. 9. EVALUATION & THEORETICAL FRAMEWORK Health QOL Improvement Thursday, July 25, 13
  10. 10. EVALUATION & THEORETICAL FRAMEWORK Behavioral Outcomes Health QOL Improvement Thursday, July 25, 13
  11. 11. EVALUATION & THEORETICAL FRAMEWORK Behavioral Outcomes Health QOL Improvement Thursday, July 25, 13
  12. 12. EVALUATION & THEORETICAL FRAMEWORK Behavioral Outcomes Health QOL Improvement Environmental Outcomes Thursday, July 25, 13
  13. 13. EVALUATION & THEORETICAL FRAMEWORK Behavioral Outcomes Health QOL Improvement Environmental Outcomes Thursday, July 25, 13
  14. 14. EVALUATION & THEORETICAL FRAMEWORK Behavioral Outcomes Health QOL Improvement Environmental Outcomes Thursday, July 25, 13
  15. 15. EVALUATION & THEORETICAL FRAMEWORK Performance Objectives Behavioral Outcomes Health QOL Improvement Environmental Outcomes Thursday, July 25, 13
  16. 16. EVALUATION & THEORETICAL FRAMEWORK Performance Objectives Behavioral Outcomes Health QOL Improvement Environmental Outcomes Thursday, July 25, 13
  17. 17. EVALUATION & THEORETICAL FRAMEWORK Performance Objectives Behavioral Outcomes Health QOL Improvement Performance Objectives Environmental Outcomes Thursday, July 25, 13
  18. 18. EVALUATION & THEORETICAL FRAMEWORK Performance Objectives Behavioral Outcomes Health QOL Improvement Performance Objectives Environmental Outcomes Thursday, July 25, 13
  19. 19. EVALUATION & THEORETICAL FRAMEWORK Performance Objectives Behavioral Outcomes Health QOL Improvement Performance Objectives Environmental Outcomes Thursday, July 25, 13
  20. 20. EVALUATION & THEORETICAL FRAMEWORK Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Performance Objectives Environmental Outcomes Thursday, July 25, 13
  21. 21. EVALUATION & THEORETICAL FRAMEWORK Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Performance Objectives Environmental Outcomes Thursday, July 25, 13
  22. 22. EVALUATION & THEORETICAL FRAMEWORK Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Performance Objectives Environmental Outcomes Personal Determinants Thursday, July 25, 13
  23. 23. EVALUATION & THEORETICAL FRAMEWORK Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Performance Objectives Environmental Outcomes Personal Determinants Thursday, July 25, 13
  24. 24. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Performance Objectives Environmental Outcomes Personal Determinants Thursday, July 25, 13
  25. 25. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Performance Objectives Environmental Outcomes Personal Determinants Thursday, July 25, 13
  26. 26. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants Thursday, July 25, 13
  27. 27. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants Thursday, July 25, 13
  28. 28. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants At-Risk Group Thursday, July 25, 13
  29. 29. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants At-Risk Group Environmental Agent Thursday, July 25, 13
  30. 30. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants At-Risk Group Environmental Agent { Thursday, July 25, 13
  31. 31. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants At-Risk Group Environmental Agent { Logic of Change Thursday, July 25, 13
  32. 32. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants At-Risk Group Environmental Agent { Logic of Change {Thursday, July 25, 13
  33. 33. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants At-Risk Group Environmental Agent { Logic of Change {Outcomes Thursday, July 25, 13
  34. 34. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants At-Risk Group Environmental Agent { Logic of Change {Outcomes Thursday, July 25, 13
  35. 35. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants At-Risk Group Environmental Agent { Logic of Change {Outcomes Thursday, July 25, 13
  36. 36. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants At-Risk Group Environmental Agent { Logic of Change {Outcomes Thursday, July 25, 13
  37. 37. SOCIAL ECOLOGICAL MODEL http://www.ahrq.gov/professionals/prevention-chronic-care/resources/clinical-community-relationships-measures-atlas/ccrm-atlas3.html Thursday, July 25, 13
  38. 38. http://www.louisianabreastfeedingcoalition.org/wp-content/uploads/2010/09/SEM2.png SEM: Louisiana Breastfeeding Coalition Thursday, July 25, 13
  39. 39. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants At-Risk Group Environmental Agent { Logic of Change {Outcomes Thursday, July 25, 13
  40. 40. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants At-Risk Group Environmental Agent { Logic of Change {Outcomes Thursday, July 25, 13
  41. 41. EVALUATION & THEORETICAL FRAMEWORK Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants At-Risk Group Environmental Agent { Logic of Change {Outcomes Thursday, July 25, 13
  42. 42. DETERMINANTS:ASSESSING IMPORTANCE & CHANGEABILITY Determinants Importance Changeability Evidence for Importance Knowledge + +++ Precondition for personal attitude Risk perception + + Precondition for personal relevance Self-efficacy +++ + r=0.71, p<0.01 Personal norm +++ + r=0.78, p<0.01 Subjective norm + + r=0.29, p<0.01 Skills ++ + Precondition for self- efficacy improvement Thursday, July 25, 13
  43. 43. EVALUATION & THEORETICAL FRAMEWORK: LOGIC MODEL Change Objectives Personal Determinants Performance Objectives Behavioral Outcomes Health QOL Improvement Change Objectives Performance Objectives Environmental Outcomes Personal Determinants At-Risk Group Environmental Agent { Logic of Change {Outcomes Thursday, July 25, 13
  44. 44. MATRIX FOR CHILDREN IN SPF PROGRAM Behavior: Wear Sunscreen and Protective Clothing When OutdoorsBehavior: Wear Sunscreen and Protective Clothing When OutdoorsBehavior: Wear Sunscreen and Protective Clothing When OutdoorsBehavior: Wear Sunscreen and Protective Clothing When OutdoorsBehavior: Wear Sunscreen and Protective Clothing When Outdoors Personal DeterminantsPersonal DeterminantsPersonal DeterminantsPersonal DeterminantsPersonal Determinants Performance Objectives (Children) Attitudes Skills/Self-Efficacy Knowledge Outcome Expectations PO 1. Cooperate with sun protection practices by parent or preschool staff A.1 Express positive feeling toward being protected from the sun OE 1. Describe how cooperating will keep skin healthy PO 1.1 Stand still for application A. 1.1 Express positive attitude toward being a helper K 1.1 Explain how standing still allows sunscreen to be put on evenly Thursday, July 25, 13
  45. 45. MATRIX FOR ENVIRONMENT IN SPF PROGRAM Organizational Environmental Change in SPF ProgramOrganizational Environmental Change in SPF ProgramOrganizational Environmental Change in SPF ProgramOrganizational Environmental Change in SPF ProgramOrganizational Environmental Change in SPF Program Personal DeterminantsPersonal DeterminantsPersonal DeterminantsPersonal DeterminantsPersonal Determinants Performance Objectives (Agents) Perceived Norms Attitudes Skills/Self-Efficacy Outcome Expectations / Perceived Susceptibility PO 1. Preschool directors decide to prioritize increasing amt. of shade areas used by preschoolers PN 1. Recognize sun protection and shade adequacy as concerns for preschool administration A 1. Describe assuring shade as positive OEPS 1. a Describe children as susceptible to skin cancer caused by lack of shade OEPS 1. b.Argue that increasing shade will decrease risk Thursday, July 25, 13
  46. 46. LOGIC MODEL LINKTO EVALUATION Thursday, July 25, 13
  47. 47. LOGIC MODEL LINKTO EVALUATION • Behavioral / Environmental Outcomes = Impact of Activities/Program Thursday, July 25, 13
  48. 48. LOGIC MODEL LINKTO EVALUATION • Behavioral / Environmental Outcomes = Impact of Activities/Program • Performance Objectives = Effectiveness of Activities/Program • Specificity of objectives is key • Uncover key factors for behavioral/environmental success • Often help to define the behavior or environmental condition (improving evaluation) Thursday, July 25, 13
  49. 49. LOGIC MODEL LINKTO EVALUATION • Behavioral / Environmental Outcomes = Impact of Activities/Program • Performance Objectives = Effectiveness of Activities/Program • Specificity of objectives is key • Uncover key factors for behavioral/environmental success • Often help to define the behavior or environmental condition (improving evaluation) • Change Objectives = Immediate impact Thursday, July 25, 13
  50. 50. What we observe is not nature itself, but nature exposed to our method of questioning. -Werner Heisenberg http://resources0.news.com.au/images/2012/11/06/1226511/766824-solar-eclipse-glasses.jpg Thursday, July 25, 13
  51. 51. EVALUATION BASICS Thursday, July 25, 13
  52. 52. EVALUATION BASICS • Formative: MAPP Thursday, July 25, 13
  53. 53. EVALUATION BASICS • Formative: MAPP • Process: Implementation Thursday, July 25, 13
  54. 54. PROCESS EVALUATION Thursday, July 25, 13
  55. 55. PROCESS EVALUATION • Reach: The proportion of the intended audience to whom the program is actually delivered Thursday, July 25, 13
  56. 56. PROCESS EVALUATION • Reach: The proportion of the intended audience to whom the program is actually delivered • Dose Delivered:The amount of intended units of each program component that is delivered Thursday, July 25, 13
  57. 57. PROCESS EVALUATION • Reach: The proportion of the intended audience to whom the program is actually delivered • Dose Delivered:The amount of intended units of each program component that is delivered • Dose Received:The extent to which the participants engage with the program Thursday, July 25, 13
  58. 58. PROCESS EVALUATION • Reach: The proportion of the intended audience to whom the program is actually delivered • Dose Delivered:The amount of intended units of each program component that is delivered • Dose Received:The extent to which the participants engage with the program • Fidelity:The extent to which the program was delivered as intended Thursday, July 25, 13
  59. 59. EVALUATION BASICS • Formative: MAPP • Process: Implementation Thursday, July 25, 13
  60. 60. EVALUATION BASICS • Formative: MAPP • Process: Implementation • Impact: Immediate effects Thursday, July 25, 13
  61. 61. EVALUATION BASICS • Formative: MAPP • Process: Implementation • Impact: Immediate effects • Outcome: Ultimate results Thursday, July 25, 13
  62. 62. EVALUATION BASICS • Formative: MAPP • Process: Implementation • Impact: Immediate effects • Outcome: Ultimate results Key: Matching measurement to outcome of interest! Thursday, July 25, 13
  63. 63. QUALITATIVE + QUANTITATIVE METHODS Thursday, July 25, 13
  64. 64. QUALITATIVE + QUANTITATIVE COMPLIMENTARY METHODS Thursday, July 25, 13
  65. 65. QUALITATIVE + QUANTITATIVE MEASUREMENT Qualitative Quantitative Thursday, July 25, 13
  66. 66. • Interviews with key informants QUALITATIVE + QUANTITATIVE MEASUREMENT Qualitative Quantitative Thursday, July 25, 13
  67. 67. • Interviews with key informants • Observations of program activities QUALITATIVE + QUANTITATIVE MEASUREMENT Qualitative Quantitative Thursday, July 25, 13
  68. 68. • Interviews with key informants • Observations of program activities • Focus groups QUALITATIVE + QUANTITATIVE MEASUREMENT Qualitative Quantitative Thursday, July 25, 13
  69. 69. • Interviews with key informants • Observations of program activities • Focus groups • Thematic analysis QUALITATIVE + QUANTITATIVE MEASUREMENT Qualitative Quantitative Thursday, July 25, 13
  70. 70. • Interviews with key informants • Observations of program activities • Focus groups • Thematic analysis QUALITATIVE + QUANTITATIVE MEASUREMENT Qualitative Quantitative Thursday, July 25, 13
  71. 71. • Interviews with key informants • Observations of program activities • Focus groups • Thematic analysis QUALITATIVE + QUANTITATIVE MEASUREMENT Qualitative Quantitative Thursday, July 25, 13
  72. 72. • Interviews with key informants • Observations of program activities • Focus groups • Thematic analysis • Demographics QUALITATIVE + QUANTITATIVE MEASUREMENT Qualitative Quantitative Thursday, July 25, 13
  73. 73. • Interviews with key informants • Observations of program activities • Focus groups • Thematic analysis • Demographics • Pre-post test scores QUALITATIVE + QUANTITATIVE MEASUREMENT Qualitative Quantitative Thursday, July 25, 13
  74. 74. • Interviews with key informants • Observations of program activities • Focus groups • Thematic analysis • Demographics • Pre-post test scores • Ratings of attitude and self-efficacy QUALITATIVE + QUANTITATIVE MEASUREMENT Qualitative Quantitative Thursday, July 25, 13
  75. 75. • Interviews with key informants • Observations of program activities • Focus groups • Thematic analysis • Demographics • Pre-post test scores • Ratings of attitude and self-efficacy • Existing numerical data QUALITATIVE + QUANTITATIVE MEASUREMENT Qualitative Quantitative Thursday, July 25, 13
  76. 76. • Interviews with key informants • Observations of program activities • Focus groups • Thematic analysis • Demographics • Pre-post test scores • Ratings of attitude and self-efficacy • Existing numerical data • Statistical analysis QUALITATIVE + QUANTITATIVE MEASUREMENT Qualitative Quantitative Thursday, July 25, 13
  77. 77. Qualitative Quantitative Result Qualitative ResultQuantitative Quantitative Qualitative Result EVALUATION METHODS: QUALITATIVE + QUANTITATIVE Qualitative Result Quantitative 1 2 3 4 Thursday, July 25, 13
  78. 78. EXAMPLES OF QUALITATIVE + QUANTITATIVE DATA USE What is being measured? Quantitative Data Qualitative Data Awareness Written instrument (y/n) to measure awareness of PSA on healthy eating Interviews at local supermarkets to measure awareness of healthy eating campaign Knowledge Written/oral knowledge questionnaire (pre-post, likert) to measure knowledge gained at nutrition seminar Interviews to measure knowledge about nutritional value of f/v, focus groups w/ children about same Attitudes Written instrument (pre-post, likert) to measure attitudes and preferences of children for f/v Interviews with mothers of small children to ID attitudes toward feeding kids f/v Behavior Self-report w/ scaled responses to measure freq. + type of healthy eating, obs. counts Interviews with individuals to gather info on eating behaviors, focus groups, obs. Thursday, July 25, 13
  79. 79. EXAMPLES OF QUALITATIVE + QUANTITATIVE DATA USE What is being measured? Quantitative Data Qualitative Data Policy changes Measurement of number of children who consume f/v before and after school policy requiring salad bars in school Interviews with children in school about what they like/ dislike on salad bar since policy implemented Organizational Changes Records documenting # and demographics of people who go to food bank for food + now get info on preparing nutritious food and food stamp program Interviews w/ food bank staff to gather opinions about whether food bank’s new nutrition ed and food stamp outreach has positive effect on clients Environmental Changes Survey of farmers’ markets in the county to document the size of the markets, types of produce provided, cost of produce Observations of and interviews with people attending farmers’ markets to identify satisfaction with market Thursday, July 25, 13
  80. 80. EVALUATION: OTHERTHOUGHTS • Link between objective, outcome, and measurement is key • Follow SMART objective guidelines to make evaluation easier and more effective • Use theory and logic model (where possible) to guide the process to make change more likely, evaluation more effective • But, don’t get stuck there! • Be sure to have process, impact, outcome represented in plan Thursday, July 25, 13
  81. 81. DGHD PRIORITIES: THOUGHTS • A healthy community, a chance to climb even higher • Great efforts, great people, great community buy-in • Excellent priorities and processes • Where to improve? Thursday, July 25, 13
  82. 82. DGHD PRIORITIES: SOME EXAMPLES OBJECTIVE / Tactic TIMEFRAME OUTCOME MEASUREMENT a. unit b.collection method Determine sources and then monitor, evaluate and disseminate county wide and community specific obesity data. June 1, 2012 - December 31, 2012 for determination of data sources, then annually each January Determine obesity trends for Delaware County. a. % of county residents that are obese, evaluate change over time and disseminate information to community via agency website b. review of standardized list of data Goal 1.1 Thursday, July 25, 13
  83. 83. DGHD PRIORITIES: SOME EXAMPLES OBJECTIVE / Tactic TIMEFRAME OUTCOME MEASUREMENT a. unit b.collection method 100% of the clients requesting birth certificates will be given a safe food handling brochure. Ongoing Improved knowledge of safe food handling. a. Number of birth certificates requested and count of those receiving food safety information b. Number that receive food safety information Goal 3.7 Thursday, July 25, 13
  84. 84. DGHD PRIORITIES: SOME EXAMPLES OBJECTIVE / Tactic TIMEFRAME OUTCOME MEASUREMENT a. unit b.collection method Utilize Wii game system and “Impaired Driving Fatal Vision Mario Cart” at least 10 times, with targeted populations, annually. July, 2010 (with borrowed system), and ongoing Raise awareness of impaired youth driving in a safe environment. a. The number of events at which an interactive impaired driving program is utilized with high school students b. Tracking by staff Goal 4.2.1 Thursday, July 25, 13
  85. 85. SUMMARY • Importance of evaluation • Importance of planning and objectives in evaluation • Consistency and using a framework • Method based on needs, goals • Asking the right questions is key! Thursday, July 25, 13
  86. 86. Map 0 Full Generic Map nothing is perfect. life is messy. relationships are complex. outcomes are uncertain. people are irrational. hugh mackay Thursday, July 25, 13
  87. 87. If anyone on the verge of action should judge himself according to the outcome, he would never begin." Søren Kierkegaard Thursday, July 25, 13
  88. 88. Thanks! Presentation PDF available at: http://slideshare.net/ChristopherFink Harvard Campaign & Public Communication Evaluation Example http://db.tt/pufWl30z Healthy Maine Healthy Weight Awareness Campaign Evaluation http://db.tt/zVqSQ7hP WHO ImmunizationTracking & Data Management Info http://db.tt/OLu3wEuE Thursday, July 25, 13

×