Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11

  • 502 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
502
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
4
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Measuring Financial EducationSuccess: Evaluation Methods and Assessment Tools Barbara O’Neill, Ph.D, CFP®, CRPC® Professor II, Rutgers University oneill@aesop.rutgers.edu
  • 2. Are You Evaluating Your Financial Education Programs? If You Are, How?
  • 3. The Million Dollar “So What?” Question… At the end of the day…did your financial education program make a difference?How do you know?
  • 4. The Current State of Financial Education Program Evaluation• Current evaluation efforts are still far from satisfactory• General lack of evaluation capacity• Evaluation is often treated as an after-thought• Outcomes (e.g., measures of changed behavior) are often confused with program outputs (e.g., number of participants)
  • 5. We are in an “Accountability Era”• What gets measured gets funded• With evaluation results, you can – assess impact of programs on learners – see if you accomplished what you planned – know if a program was “worth it” – celebrate success and learn from failure – make informed decisions to improve, hold, or fold programs – promote your program and win public support
  • 6. Introducing the Logic Model• See http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html• Begins with the end in mind• Explains what a program is and what it will accomplish• Shows relationships between inputs, outputs, and outcomes
  • 7. Everyday SituationHUNGR Get food Get food Eat food Eat food Feel better Feel betterY
  • 8. Building a Strong Financial Education ProgramINPUTS OUTPUTS OUTCOMES Long- Activities Participation Short Medium term Programinvestments What What Who We What Results We We Do Reach Invest SO WHAT?? What is the VALUE?
  • 9. INPUTS Staff Money Time Volunteers PartnersEquipment/Technology Policies Research
  • 10. OUTPUTS What We Do Who We Reach ACTIVITIES PARTICIPATIONAssess needs and assets ParticipantsDesign curriculum ClientsEducate students CustomersConduct workshops UsersFacilitate learning groups GroupsSponsor conferencesWork with the media Reactions - SatisfactionPartner – collaborate
  • 11. OUTCOMES What Results for Individuals, Organizations, Communities..… SHORT MEDIUM LONG-TERM Learning Action ConditionsAwareness Behavior HumanKnowledge Practice EconomicAttitudes Decisions CivicSkills Policies EnvironmentOpinion Social actionAspirationsMotivation
  • 12. Commonly Measured Items That Are Not Outcomes• Participant satisfaction• Number of people taught• Units of education completed OUTCOMES• Number of events held• Time and money spent• Level of effort
  • 13. Impact Evaluation Data Collection Methods • Surveys – Post-evaluation only (short programs) – Pre- and post-evaluation – Follow-up (e.g., 3 months later) • Focus groups • Interviews • Observations • Tests of knowledge/ability • RARE: Control groups and longitudinal studies
  • 14. Typical Survey Questions• General reactions to the program• Changes in knowledge• Changes in motivation, confidence, and abilities• Intended changes in behavior• Actual changes in behavior• Future programming needs and preferences• Demographics• Qualitative/open-ended responses
  • 15. Post-Then-Pre Evaluation Surveys• Also known as a “Retrospective Pre-Test”• Helps identify changes in knowledge, attitudes, and behavior
  • 16. Another Post-Then-Pre ExampleBig Advantage: Avoids the problem of learners under-estimating what they don’t know before a program
  • 17. NEFE Evaluation Toolkithttp://toolkit.nefe.org
  • 18. Downloadable Evaluation Manual
  • 19. Build Your Own Evaluations Online
  • 20. One Last Step:Share Your Program Evaluation Results WHAT? SO WHAT? NOW WHAT?
  • 21. Tips for “Telling Your Story”• Use simple descriptive statistics (e.g., counts, percentages, and averages)• Don’t use jargon• Don’t overstate your results (e.g., causality)• Blend quantitative and qualitative data• Clearly describe who the results represent (i.e., demographic characteristics of participants)• Be honest about your program’s strengths and weaknesses, while highlighting the positive
  • 22. Impact Statements: Intentions As a result of participating in this financial education program, X% of participants reported that they…• plan to do/use/adopt…• are more knowledgeable about…• are more confident in their ability to…• are more likely than before to do/use/adopt…• will do/use/adopt……a particular attitude, piece of information, or behavior.
  • 23. Impact Statements: Actions As a result of participating in this financial education program, X% of participants reported that they…• are now doing…• did…• used…• increased their knowledge of…• adopted…• changed…… a particular attitude, piece of information, or behavior.
  • 24. Questions and Comments?Barbara ONeill, Ph.D., CFP®, CRPCExtension Specialist in Financial Resource Management andProfessor IIRutgers UniversityPhone: 732-932-9155 Extension 250E-mail: oneill@aesop.rutgers.eduInternet: http://njaes.rutgers.edu/money2000/Twitter: http://twitter.com/moneytalk1