Quality In Action - May 2011


Published on

Mentor Program Evaluation

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Quality In Action - May 2011

  1. 1. Quality in Action<br />Mentor Program Evaluation<br />May 4, 2011<br />Mentoring Partnership of Minnesota<br />
  2. 2. Webinar Logistics<br /> Asking Questions & Sharing Comments During the Webinar<br />“Raise your hand” & MPM Organizers will unmute you<br />Or, type questions (and comments) in the question/answer section and submit; we will respond directly to you or possibly share your question with all attendees<br /><ul><li>When unmuted, please monitor your background noise</li></ul>April Riordan, Director of Training and Community Partnerships<br />Recordings of past webinars are available on our YouTubeChannel.<br />2<br />2<br />2<br />
  3. 3. Overview<br />Introduction & Best Practices<br />Elements of Effective Practice<br />Evaluation Plan & Logic Model<br />Tips for Selecting Instruments<br />Oregon Mentors Evaluation Toolkit<br />Important Rules for Administering Instruments<br />Additional Resources<br />3<br />
  4. 4. What are Programs Measuring?<br />4<br />
  5. 5. What are Programs Measuring?<br />5<br />
  6. 6. What are Programs Measuring?<br />6<br />
  7. 7. What are Programs Measuring?<br />7<br />
  8. 8. Elements of Effective Practice<br />Ensure program quality and effectiveness:<br />Develop a plan to measure program process:<br /><ul><li>Select indicators such as training hours,
  9. 9. meeting frequency and relationship duration; and
  10. 10. Develop a system for collecting and managing specified data.</li></ul>Develop a plan to measure expected outcomes:<br /><ul><li>Specify expected outcomes;
  11. 11. Select appropriate instruments to measure outcomes
  12. 12. Select and implement an evaluation design.</li></ul>Create a process to reflect on and disseminate evaluation findings:<br /><ul><li>Refine the program design and operations based on the findings; and
  13. 13. Develop and deliver reports to program constituents, funders and the media</li></ul>8<br />
  14. 14. Panelists<br />Mike Garringer<br />Advisor<br />National Mentoring Center Education Northwest <br />Celeste Janssen<br />Program Director<br />Oregon Mentors<br />9<br />
  15. 15. Measuring Effectiveness<br />10<br />
  16. 16. Tips for Evaluating Mentoring Programs<br />Let’s look at the mechanisms of mentoring…<br />11<br />
  17. 17. Missed Opportunities in Mentoring Program Evaluation<br />Service delivery<br />Did they get the mentoring promised?<br />Participant satisfaction<br />Mentoring relationship<br />Did they form the bond needed to make any of this work?<br />Intermediary outcomes are OK! <br />12<br />
  18. 18. What is a Logic Model?<br />Systematic picture of how you believe your program works<br />“…helps you visualize and understand how human and financial investments can contribute to achieving your intended program goals.”<br />13<br />
  19. 19. Sample Vacation Logic Model<br />14<br />
  20. 20. Sample YouthBuild Mentoring Logic Model<br />15<br />
  21. 21. Tips for Selecting Good Instruments<br />Try to use instruments that have been tested for reliability and validity<br />Make sure they fit your logic model, especially in terms of proximal and distal outcomes<br />Note the complexity of the tool: <br />Age range<br />Reading level<br />Concepts<br />Can you provide a comparison group of some kind?<br />16<br />
  22. 22. Quantitative Tools<br />Indicator: <br />a variable with the purpose to measure change in a phenomena or process<br />Valid:<br />Does the tool measure what it claims to measure?<br />Reliable:<br />Does the tool give consistent results?<br />Pre and Post Test<br />Example<br />Pretest Posttest<br />Pretest Posttest<br />17<br />
  23. 23. Live Demo - Oregon Mentors Evaluation Instrument Toolkit<br />Browse for specific instruments (19 “domains”)<br />Simple format for each instrument “record”<br />Tips for using<br />Links to other online repositories<br />18<br />
  24. 24. Toolkit Domains of InterestLive Demo of Toolkit & Instruments<br />Personal Attitudes and Beliefs:<br />Antisocial attitude scale<br />Decision making skills scale<br />Goal Setting Skills Scale<br />Prosocial Involvement, Opportunities and Rewards<br />Mentoring relationship quality:<br />Match Characteristics Questionnaire (MCQ)<br />Youth Mentoring Survey (YMS)<br />Youth Survey: Measuring the Quality of Mentor-Youth Relationships<br />Comprehensive Surveys:<br />Youth Outcome Survey<br />Youth Risk Behavior Surveillance System<br />Connectedness:<br />Hemingway Scale of Adolescent Connectedness<br />School and Academics:<br />Education Expectations and Aspirations Scale<br />Motivated Strategies for Learning Questionnaire (MSLQ)<br />19<br />
  25. 25. Tips for Administering Instruments<br />Plan for problems<br />Don’t hand-pick which questions to use<br />Don’t reword questions<br />Get parent permission<br />Provide a private space to administer<br />Don’t provide guidance<br />Administer the instruments in a consistent, uniform way<br />Keep the answers anonymous/ sort out any coding before administration<br />Plan for data storage and safety<br />20<br />
  26. 26. Resources<br />21<br />
  27. 27. Resources<br />22<br />
  28. 28. Resources<br />W.K. Kellogg Foundation Evaluation Handbook <br />W.K. Kellogg Foundation Logic Model Development Guide<br />Stand by Me: The Risks and Rewards of Mentoring Today's Youth (The Family and Public Policy) by Jean E. Rhodes <br />Buying InformationReviewsEndorsements<br />23<br />
  29. 29. Handoutshttp://www.slideshare.net/traininginstitute<br />Using Standardized Instruments<br />Comparing Program Outcomes to National Standards (Appendix B – 36)<br />Blank Logic Model Template<br />Sample YouthBuild Mentoring Logic Model<br />Using Your Logic Model to Focus the Evaluation<br />Checklist for Selecting an Evaluator<br />Sample Contact & Information Release Form<br />Sample Evaluation Press Release<br />Evaluating Mentoring Programs – P/PV Brief<br />Additional Reading and Resources for Evaluating Your Mentoring Program<br />Glossary of Measurement and Evaluation Terms<br />24<br />
  30. 30. Other Resources<br />This presentation & otherswww.slideshare.net/traininginstitute<br />MPM Trainingwww.mpmn.org/traininginstitute<br />Web sites & PDFswww.delicious.com/traininginstitute<br />25<br />
  31. 31. Next Quality in Action Webinar<br />June 1 – Eat Well, Stay Active, Have Fun<br />A new resource from the Harvard School of Public Health, "Eat Well, Stay Active, Have Fun: A Guide for Mentors,"provides mentors with information and guidance on how to make healthy eating and physical activity natural parts of the mentor-mentee relationship. Join this webinar to learn more about the guide and how local mentors are using it.<br />Mentoring Program Evaluation & Goals<br />Monday, May 16 - 10:00-11:15am Pacific Time<br />Go to Friends for Youth's Mentoring Instituteto register online for this FREE webinar! <br />26<br />