Review of Evaluation Methods


Published on

Slide show presented at May 2007 in service for UNHCE Extension Educators

Published in: Economy & Finance, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Review of Evaluation Methods

    1. 1. Review ---Evaluation Data Collection Methods:
    2. 2. <ul><li>Considerations in selecting data collection methods </li></ul><ul><li>Commonly used data collection methods </li></ul><ul><li>Small group work on methods selection </li></ul>Objectives
    3. 3. <ul><li>Outcomes/Outputs – what are you looking to measure? </li></ul><ul><ul><li>Outputs (Activities) – numbers & effectiveness </li></ul></ul><ul><ul><li>Learning Outcomes </li></ul></ul><ul><ul><li>Action Outcomes </li></ul></ul><ul><li>Target audience characteristics </li></ul><ul><li>Cost </li></ul><ul><li>How accurate do you need to be? (validity) </li></ul>Considerations in Selecting Data Collection Methods
    4. 4. <ul><ul><li>Size of population </li></ul></ul><ul><ul><li>Age </li></ul></ul><ul><ul><li>Literacy level </li></ul></ul><ul><ul><li>Cultural practices </li></ul></ul><ul><ul><li>Accessibility </li></ul></ul><ul><ul><li>Anonymity requirements </li></ul></ul>Target Audience Characteristics
    5. 5. <ul><li>Surveys – web, mail, telephone, face-to-face </li></ul><ul><ul><li>Retrospective </li></ul></ul><ul><ul><li>Post only </li></ul></ul><ul><ul><li>Follow-up </li></ul></ul><ul><li>Interviews </li></ul><ul><li>Focus Groups </li></ul><ul><li>Tests – including pre/post tests </li></ul><ul><li>Direct Observation </li></ul><ul><ul><li>Rubrics/Score sheets </li></ul></ul><ul><li>Document Analysis </li></ul>Commonly Used Data Collection Methods
    6. 6. <ul><ul><li>Retrospective – reflective of pre-event state - perceptions </li></ul></ul><ul><ul><li>Post only – no reflection requested </li></ul></ul><ul><ul><li>Follow-up – longer term change measured; usually affects distribution methods </li></ul></ul>Surveys
    7. 7. <ul><li>Facts and perceptions can be queried </li></ul><ul><li>Ability to clarify questions if confusion possible </li></ul><ul><li>Difficult for geographically dispersed population </li></ul><ul><li>Response bias can be affected by perceived lack of anonymity </li></ul><ul><li>Interviewer might bias responses </li></ul><ul><li>Time intensive </li></ul>Surveys – Face-to-Face
    8. 8. <ul><li>Facts and perceptions can be queried </li></ul><ul><li>Can clarify questions if confusion more than mail /web </li></ul><ul><li>Useful for geographically dispersed </li></ul><ul><li>Fastest response rate </li></ul><ul><li>Let’s clientele know we care </li></ul><ul><li>Question complexity limited </li></ul><ul><li>Response bias increased if perceived lack of anonymity </li></ul><ul><li>Leaves out people with no phone or unlisted numbers </li></ul><ul><li>May be time consuming if audience is large </li></ul>Surveys – Telephone
    9. 9. <ul><li>Can be done anonymously </li></ul><ul><li>Useful for geographically dispersed </li></ul><ul><li>Low control of reader response sequence </li></ul><ul><li>Less control over non-response to questions </li></ul><ul><li>Low success with open-ended questions </li></ul><ul><li>Question complexity somewhat limited </li></ul>Surveys – Mail
    10. 10. <ul><li>Same as mail except: </li></ul><ul><li>Good for audiences with web access </li></ul><ul><li>Must know technological accessibility and utilization by audience </li></ul><ul><li>Low cost per respondent </li></ul>Surveys – Web
    11. 11. <ul><li>Can fully explore impressions and motivations </li></ul><ul><li>Can record unintentional findings easily </li></ul><ul><li>Can overcome literacy/ culturally-oriented sample biases </li></ul><ul><li>Usually avoid sensitive topics </li></ul><ul><li>Difficult for geographically dispersed population </li></ul><ul><li>High potential for interview/setting bias </li></ul><ul><li>Cost and time intensive per subject </li></ul>One-on-One Interviews
    12. 12. <ul><li>Can explore process and outcomes in-depth </li></ul><ul><li>Fewer questions; more diverse opinions than interviews </li></ul><ul><li>Rapid feedback possible for common impressions </li></ul><ul><li>Can overcome literacy and accessibility obstacles </li></ul><ul><li>Sampling error may be high </li></ul><ul><li>Response bias affected by perceived social expectations </li></ul><ul><li>Facilitator bias can be high </li></ul><ul><li>Hard to generalize data </li></ul><ul><li>Need 6-10 per event (can be logistically challenging) </li></ul>Focus Groups
    13. 13. <ul><li>Provide “hard”, measurable data </li></ul><ul><li>Allow comparisons with other populations </li></ul><ul><li>Instrument may already be tested – norm or criterion referenced </li></ul><ul><li>May be unsuitable for some populations (adults) </li></ul><ul><li>Language or vocabulary might be a difficult </li></ul><ul><li>May be anxiety over how results are used </li></ul>Tests
    14. 14. <ul><li>Can record information about process and product </li></ul><ul><li>Can record unintentional findings easily </li></ul><ul><li>Unobtrusive </li></ul><ul><li>May perceive lack of anonymity </li></ul><ul><li>Categorizing behaviors can be difficult </li></ul><ul><li>High observer bias; may influence behaviors </li></ul><ul><li>Doesn’t measure knowledge or feelings well </li></ul><ul><li>Time and personnel costs can be high </li></ul>Direct Observations
    15. 15. Rubrics/Score sheets <ul><li>Focuses an observer </li></ul><ul><li>Quantifies data that might not be easily identified that way </li></ul><ul><li>Assess what people do – not what they know </li></ul><ul><li>High degree of subjectivity </li></ul><ul><li>Development can be time consuming </li></ul>
    16. 16. <ul><li>Factual; can provide history (longitudinal data) </li></ul><ul><li>Unobtrusive – can be built into existing routine </li></ul><ul><li>Provides immediate data </li></ul><ul><li>May not be time and cost-effective </li></ul><ul><li>Inflexible; restricted to what exists </li></ul><ul><li>Privacy issues can limit access to necessary data </li></ul><ul><li>Must trust data sources selected </li></ul><ul><li>Need to be clear up front, what you’re looking for </li></ul>Document Analysis
    17. 17. Mixed Methods <ul><li>Using more than one method </li></ul><ul><li>Often combines qualitative with quantitative methods </li></ul><ul><li>May help in increasing validity </li></ul><ul><li>Useful in writing more rich impact statements </li></ul>
    18. 18. Exercise <ul><li>Small groups –Discuss which data collection method(s) would be best for your evaluation </li></ul><ul><li>Begin to fill out the evaluation plan table </li></ul>