Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Quality In Action - May 2011


Published on

Mentor Program Evaluation

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

Quality In Action - May 2011

  1. 1. Quality in Action<br />Mentor Program Evaluation<br />May 4, 2011<br />Mentoring Partnership of Minnesota<br />
  2. 2. Webinar Logistics<br /> Asking Questions & Sharing Comments During the Webinar<br />“Raise your hand” & MPM Organizers will unmute you<br />Or, type questions (and comments) in the question/answer section and submit; we will respond directly to you or possibly share your question with all attendees<br /><ul><li>When unmuted, please monitor your background noise</li></ul>April Riordan, Director of Training and Community Partnerships<br />Recordings of past webinars are available on our YouTubeChannel.<br />2<br />2<br />2<br />
  3. 3. Overview<br />Introduction & Best Practices<br />Elements of Effective Practice<br />Evaluation Plan & Logic Model<br />Tips for Selecting Instruments<br />Oregon Mentors Evaluation Toolkit<br />Important Rules for Administering Instruments<br />Additional Resources<br />3<br />
  4. 4. What are Programs Measuring?<br />4<br />
  5. 5. What are Programs Measuring?<br />5<br />
  6. 6. What are Programs Measuring?<br />6<br />
  7. 7. What are Programs Measuring?<br />7<br />
  8. 8. Elements of Effective Practice<br />Ensure program quality and effectiveness:<br />Develop a plan to measure program process:<br /><ul><li>Select indicators such as training hours,
  9. 9. meeting frequency and relationship duration; and
  10. 10. Develop a system for collecting and managing specified data.</li></ul>Develop a plan to measure expected outcomes:<br /><ul><li>Specify expected outcomes;
  11. 11. Select appropriate instruments to measure outcomes
  12. 12. Select and implement an evaluation design.</li></ul>Create a process to reflect on and disseminate evaluation findings:<br /><ul><li>Refine the program design and operations based on the findings; and
  13. 13. Develop and deliver reports to program constituents, funders and the media</li></ul>8<br />
  14. 14. Panelists<br />Mike Garringer<br />Advisor<br />National Mentoring Center Education Northwest <br />Celeste Janssen<br />Program Director<br />Oregon Mentors<br />9<br />
  15. 15. Measuring Effectiveness<br />10<br />
  16. 16. Tips for Evaluating Mentoring Programs<br />Let’s look at the mechanisms of mentoring…<br />11<br />
  17. 17. Missed Opportunities in Mentoring Program Evaluation<br />Service delivery<br />Did they get the mentoring promised?<br />Participant satisfaction<br />Mentoring relationship<br />Did they form the bond needed to make any of this work?<br />Intermediary outcomes are OK! <br />12<br />
  18. 18. What is a Logic Model?<br />Systematic picture of how you believe your program works<br />“…helps you visualize and understand how human and financial investments can contribute to achieving your intended program goals.”<br />13<br />
  19. 19. Sample Vacation Logic Model<br />14<br />
  20. 20. Sample YouthBuild Mentoring Logic Model<br />15<br />
  21. 21. Tips for Selecting Good Instruments<br />Try to use instruments that have been tested for reliability and validity<br />Make sure they fit your logic model, especially in terms of proximal and distal outcomes<br />Note the complexity of the tool: <br />Age range<br />Reading level<br />Concepts<br />Can you provide a comparison group of some kind?<br />16<br />
  22. 22. Quantitative Tools<br />Indicator: <br />a variable with the purpose to measure change in a phenomena or process<br />Valid:<br />Does the tool measure what it claims to measure?<br />Reliable:<br />Does the tool give consistent results?<br />Pre and Post Test<br />Example<br />Pretest Posttest<br />Pretest Posttest<br />17<br />
  23. 23. Live Demo - Oregon Mentors Evaluation Instrument Toolkit<br />Browse for specific instruments (19 “domains”)<br />Simple format for each instrument “record”<br />Tips for using<br />Links to other online repositories<br />18<br />
  24. 24. Toolkit Domains of InterestLive Demo of Toolkit & Instruments<br />Personal Attitudes and Beliefs:<br />Antisocial attitude scale<br />Decision making skills scale<br />Goal Setting Skills Scale<br />Prosocial Involvement, Opportunities and Rewards<br />Mentoring relationship quality:<br />Match Characteristics Questionnaire (MCQ)<br />Youth Mentoring Survey (YMS)<br />Youth Survey: Measuring the Quality of Mentor-Youth Relationships<br />Comprehensive Surveys:<br />Youth Outcome Survey<br />Youth Risk Behavior Surveillance System<br />Connectedness:<br />Hemingway Scale of Adolescent Connectedness<br />School and Academics:<br />Education Expectations and Aspirations Scale<br />Motivated Strategies for Learning Questionnaire (MSLQ)<br />19<br />
  25. 25. Tips for Administering Instruments<br />Plan for problems<br />Don’t hand-pick which questions to use<br />Don’t reword questions<br />Get parent permission<br />Provide a private space to administer<br />Don’t provide guidance<br />Administer the instruments in a consistent, uniform way<br />Keep the answers anonymous/ sort out any coding before administration<br />Plan for data storage and safety<br />20<br />
  26. 26. Resources<br />21<br />
  27. 27. Resources<br />22<br />
  28. 28. Resources<br />W.K. Kellogg Foundation Evaluation Handbook <br />W.K. Kellogg Foundation Logic Model Development Guide<br />Stand by Me: The Risks and Rewards of Mentoring Today's Youth (The Family and Public Policy) by Jean E. Rhodes <br />Buying InformationReviewsEndorsements<br />23<br />
  29. 29. Handouts<br />Using Standardized Instruments<br />Comparing Program Outcomes to National Standards (Appendix B – 36)<br />Blank Logic Model Template<br />Sample YouthBuild Mentoring Logic Model<br />Using Your Logic Model to Focus the Evaluation<br />Checklist for Selecting an Evaluator<br />Sample Contact & Information Release Form<br />Sample Evaluation Press Release<br />Evaluating Mentoring Programs – P/PV Brief<br />Additional Reading and Resources for Evaluating Your Mentoring Program<br />Glossary of Measurement and Evaluation Terms<br />24<br />
  30. 30. Other Resources<br />This presentation &<br />MPM<br />Web sites &<br />25<br />
  31. 31. Next Quality in Action Webinar<br />June 1 – Eat Well, Stay Active, Have Fun<br />A new resource from the Harvard School of Public Health, "Eat Well, Stay Active, Have Fun: A Guide for Mentors,"provides mentors with information and guidance on how to make healthy eating and physical activity natural parts of the mentor-mentee relationship. Join this webinar to learn more about the guide and how local mentors are using it.<br />Mentoring Program Evaluation & Goals<br />Monday, May 16 - 10:00-11:15am Pacific Time<br />Go to Friends for Youth's Mentoring Instituteto register online for this FREE webinar! <br />26<br />