Mentor Program Evaluation

1. Quality in Action Mentor Program Evaluation May 4, 2011 Mentoring Partnership of Minnesota

3. Overview Introduction & Best Practices Elements of Effective Practice Evaluation Plan & Logic Model Tips for Selecting Instruments Oregon Mentors Evaluation Toolkit Important Rules for Administering Instruments Additional Resources 3

4. What are Programs Measuring? 4

5. What are Programs Measuring? 5

6. What are Programs Measuring? 6

7. What are Programs Measuring? 7

9. meeting frequency and relationship duration; and

11. Select appropriate instruments to measure outcomes

13. Develop and deliver reports to program constituents, funders and the media8

14. Panelists Mike Garringer Advisor National Mentoring Center Education Northwest Celeste Janssen Program Director Oregon Mentors 9

15. Measuring Effectiveness 10

16. Tips for Evaluating Mentoring Programs Let’s look at the mechanisms of mentoring… 11

17. Missed Opportunities in Mentoring Program Evaluation Service delivery Did they get the mentoring promised? Participant satisfaction Mentoring relationship Did they form the bond needed to make any of this work? Intermediary outcomes are OK! 12

18. What is a Logic Model? Systematic picture of how you believe your program works “…helps you visualize and understand how human and financial investments can contribute to achieving your intended program goals.” 13

19. Sample Vacation Logic Model 14

20. Sample YouthBuild Mentoring Logic Model 15

21. Tips for Selecting Good Instruments Try to use instruments that have been tested for reliability and validity Make sure they fit your logic model, especially in terms of proximal and distal outcomes Note the complexity of the tool: Age range Reading level Concepts Can you provide a comparison group of some kind? 16

22. Quantitative Tools Indicator: a variable with the purpose to measure change in a phenomena or process Valid: Does the tool measure what it claims to measure? Reliable: Does the tool give consistent results? Pre and Post Test Example Pretest Posttest Pretest Posttest 17

23. Live Demo - Oregon Mentors Evaluation Instrument Toolkit Browse for specific instruments (19 “domains”) Simple format for each instrument “record” Tips for using Links to other online repositories 18

24. Toolkit Domains of InterestLive Demo of Toolkit & Instruments Personal Attitudes and Beliefs: Antisocial attitude scale Decision making skills scale Goal Setting Skills Scale Prosocial Involvement, Opportunities and Rewards Mentoring relationship quality: Match Characteristics Questionnaire (MCQ) Youth Mentoring Survey (YMS) Youth Survey: Measuring the Quality of Mentor-Youth Relationships Comprehensive Surveys: Youth Outcome Survey Youth Risk Behavior Surveillance System Connectedness: Hemingway Scale of Adolescent Connectedness School and Academics: Education Expectations and Aspirations Scale Motivated Strategies for Learning Questionnaire (MSLQ) 19

25. Tips for Administering Instruments Plan for problems Don’t hand-pick which questions to use Don’t reword questions Get parent permission Provide a private space to administer Don’t provide guidance Administer the instruments in a consistent, uniform way Keep the answers anonymous/ sort out any coding before administration Plan for data storage and safety 20

26. Resources 21

27. Resources 22

28. Resources W.K. Kellogg Foundation Evaluation Handbook W.K. Kellogg Foundation Logic Model Development Guide Stand by Me: The Risks and Rewards of Mentoring Today's Youth (The Family and Public Policy) by Jean E. Rhodes Buying InformationReviewsEndorsements 23

29. Handoutshttp://www.slideshare.net/traininginstitute Using Standardized Instruments Comparing Program Outcomes to National Standards (Appendix B – 36) Blank Logic Model Template Sample YouthBuild Mentoring Logic Model Using Your Logic Model to Focus the Evaluation Checklist for Selecting an Evaluator Sample Contact & Information Release Form Sample Evaluation Press Release Evaluating Mentoring Programs – P/PV Brief Additional Reading and Resources for Evaluating Your Mentoring Program Glossary of Measurement and Evaluation Terms 24

30. Other Resources This presentation & otherswww.slideshare.net/traininginstitute MPM Trainingwww.mpmn.org/traininginstitute Web sites & PDFswww.delicious.com/traininginstitute 25

31. Next Quality in Action Webinar June 1 – Eat Well, Stay Active, Have Fun A new resource from the Harvard School of Public Health, "Eat Well, Stay Active, Have Fun: A Guide for Mentors,"provides mentors with information and guidance on how to make healthy eating and physical activity natural parts of the mentor-mentee relationship. Join this webinar to learn more about the guide and how local mentors are using it. Mentoring Program Evaluation & Goals Monday, May 16 - 10:00-11:15am Pacific Time Go to Friends for Youth's Mentoring Instituteto register online for this FREE webinar! 26