Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mentoring Program Evaluation & Goals


Published on

This presentation was delivered on Monday, May 16, 2011, as part of the free monthly webinar series from Friends for Youth's Mentoring Institute.

One of the most important aspects of running a successful mentoring program is the evaluation of both participant outcomes and the quality of your service delivery. But, despite its importance, program evaluation in a mentoring context is often misunderstood, misapplied, or even feared. This webinar will attempt to take some of the mystery and insecurity out of mentoring program evaluation and goals, featuring a live demo of the Oregon Mentors Evaluation Instrument Toolkit, a collection of downloadable evaluation instruments, surveys, scales, and questionnaires designed to provide youth mentoring programs with increased access to reliable evaluation tools. This comprehensive site was developed by two organizations and our webinar features key staff from each: Mike Garringer, Resource Advisor with the National Mentoring Center, and Celeste Janssen, Program Director with Oregon Mentors.

Designed for program coordinators and other staff who may be new to evaluation concepts and resources or for those who would like a refresher, the presentation will address aligning evaluation with the true goals and activities of your program, explore new tools that can help you collect meaningful data, and review some common pitfalls that can complicate mentoring program evaluation. This presentation was featured as a pre-conference course at Friends for Youth’s January 2011 annual mentoring conference. Presented as a webinar, you will have the opportunity to learn more about free or inexpensive tools available to your program in case you missed this session.

This webinar was also offered by the Mentoring Partnership of Minnesota’s Quality in Action Webinar Series on Wednesday, May 4, 2011.

Mentoring Program Evaluation & Goals

  1. 1. Mentoring Program Evaluation andTransforming lives through the power of mentoring Goals Friends for Youth’s Mentoring Institute May 2011 Webinar In collaboration with the Mentoring Partnership of Minnesota
  2. 2. Webinar Logistics: Adding Comments•  All attendees muted for best sound•  Type questions and comments in the question box; responses will either be direct to you or shared with all attendees•  “Raise your hand” to be unmuted at end to ask question live during webinar •  Works best for telephone or headset-to-computer connections •  Please monitor background noise
  3. 3. PanelistsSarah Kremer, ATR-BC Michael Garringer Program Director Resource Advisor Friends for Youth’s April Riordan Education Northwest/ Celeste Janssen Mentoring Institute Director of Training & National Mentoring Center Program Director Partnerships Oregon Mentors Mentoring Partnership of Minnesota
  4. 4. •  Slides posted to SlideShare•  Recording of webinar will be posted to http:// Webinars.html•  Link and brief survey included in follow-up email
  5. 5. Recommended Best Practices•  Mission Statement and Long Range Plan•  Recruitment Plan for Mentors and Mentees•  Customized Orientations for Mentors and Mentees•  Eligibility Screening/Application Process for Mentors and Mentees•  Training Curriculum for Mentors and Mentees•  Matching Strategy•  Monitoring Process•  Support, Recognition, and Retention Component•  Relationship Closure Steps•  Evaluation Process California Governor’s Mentoring Partnership, 2002
  6. 6. Friends for Youth’s Impact•  Nearly 1,800 matches•  88% overall success rate of mentors and mentees reaching one year together•  Program Coordinator contacts –  1x/week for first 3 months + 2x/weeks for remainder (Senior Friends) –  1x/month (Junior Friends + parents/caregivers)•  Junior Friend Evaluation outcomes –  Developed by Cindy Sipe, Ph.D. –  Evaluation by William Lapp, Ph.D. –  Over 10 years of data on matches
  7. 7. 5 Outcome Areas•  Increase in involvement in school (attendance, truancy, attitude, grades, suspensions)•  Reduce risk behaviors (drug/alcohol use, police involvement, probation)•  Increase positive behaviors (community service, new activities)•  Increase opportunities to influence future potential (office visits, talking about college & career with adult)•  Improve self-concept (trusting relationships, self-assuredness, self- reliance, handle unexpected problems)
  8. 8. Friends for Youth’s ImpactAfter 12-month period, youth made significant changes•  Improved school attendance –  81% reported not skipping school at all –  75% reported avoiding suspensions•  Improved attitude toward school –  100% showed positive change in at least 1 aspect –  69% showed positive change in 2 or more aspects•  Reduction in risk behaviors –  reduced self-reported incidents of police involvement (13% vs. 28%), stealing (4% vs. 23%), hurting others (12% vs. 32%) –  alcohol & drug use decreased in time & showed delay in starting use Lapp 2009 (2007-2009 results)
  9. 9. Friends for Youth’s ImpactAfter 12-month period, youth made significant changes•  Increase in positive behaviors –  87% participated in community service project vs. 35% baseline –  significant increase in new activities•  Improved self esteem –  95% reported increase in at least 1 aspect of self-esteem –  63%reported increase in all 3 aspects•  Increase in number of trusting relationships with family, peers, & other adults –  nearly 90% reported positive overall change Lapp 2009 (2007-2009 results)
  10. 10. Friends for Youth Transforming lives through the power of mentoring . Youth-in-Need “My mentor Friends for Youth participants are: Lisa doesn’t * mostly middle-school age Societal Cost scream at me * mainly low income and two-thirds are below the poverty level California spends or hit me. She helps me with * often from single parent homes $234,000 per year for stu , and she teaches me things. * sometimes have a parent who is each youth in a Juvenile I’ve stopped hurting people. I’ve incarcerated Justice Facility. stopped beating up people.” * all at-risk of not reaching their full potential and may be susceptible to: Using Mentee Pedro For each youth who Substance Abuse avoids dropping out of The very foundation of mentoring Violence & Gang Involvement school for a life of drugs is that if a caring, concerned adult is Academic Failure and crime, the country involved in the life of a young person, These youth have little adult support, saves between $2.6 to the youth will be more likely to and now schools are cutting back $5.3 million.Evaluation become a successful adult. after-school programs. Measurable Impact of Our Program Friends for Youth puts !"#$%&(&$)#*+",-%.)/012"13%# signi cant emphasis on program evaluation and Results !"#$"%&(")*+,#-."+"%&)/0"#)1)2"#)3%)&4")5#3"%67)8-#)2-9&4)!#-(#+) its impact on our youth. (""# Ten years of statistical anaylsis con rm the "# positive impact of our program. &"# %"# “In conclusion, the Friends for Youth program is $"# functioning at an extremely high degree of e ciency !"# and has shown a positive # # # # # C# # F# B;# 0# # F# well ,+ 3 6;0 20 +3 02 ,3 ,52 *,; -/ ,1 28 235 How0 Mentoring 52Helps;F#.26* 01 23 ,5# 9: + *2 #> 67 - ?6 +@ 7A #) @/ -# .; 27 B #) 5A #I =) A +3 0M B NO9 0;2 20 ,B, as helping to address negative in uences that ./ 8 + -# ;7 3 +, /3 , 2I B2C2 2B7 2#.2 02 #L -# -2 16 +, @0 )* #H +B ;#L could otherwise cause +, 57 =48 )* 8 F#7 70 27 )* #) ,3 #./ +8 G6 0#E 7623 62 43 56 problems in [mentees] 02 ;7 +, - #E /0 43 5 27 )* D+ 05 6#? immediate lives and 56 7,3 D, K2 43 < J2 possibly into adulthood.” William Lapp, Ph.D. “My mentor Mike came to conferences and spoke to my teachers about my potential and about how smart and promising I am. It was nice to have someone tell them that I am a smart kid.” Mentee Diego
  11. 11. Overview•  Introduction & Best Practices –  Elements of Effective Practice –  Evaluation Plan & Logic Model•  Tips for Selecting Instruments –  Oregon Mentors Evaluation Toolkit•  Important Rules for Administering Instruments•  Additional Resources
  12. 12. Measuring Effectiveness How well  Youth How well  Quality of  Mentee’s  your  behavior  you run  match  a;tudes  volunteers  (outcomes  your  rela7on‐ on certain  mentor the  you wish program  ships  topics  youth  to achieve) 
  13. 13. Tips for Evaluating Mentoring Programs Let’s look at the mechanisms of mentoring…
  14. 14. Missed Opportunities in Mentoring Program Evaluation1.  Service delivery Did they get the mentoring promised?2.  Participant satisfaction3.  Mentoring relationship Did they form the bond needed to make any of this work?4.  Intermediary outcomes are OK!
  15. 15. What is a Logic Model?•  Systematic picture of how you believe your program works•  “…helps you visualize and understand how human and financial investments can contribute to achieving your intended program goals.”
  16. 16. Sample Vacation Logic Model
  17. 17. Sample YouthBuild Mentoring Logic Model
  18. 18. Tips for Selecting Good Instruments•  Try to use instruments that have been tested for reliability and validity•  Make sure they fit your logic model, especially in terms of proximal and distal outcomes•  Note the complexity of the tool: –  Age range –  Reading level –  Concepts•  Can you provide a comparison group of some kind?
  19. 19. Quantitative Tools•  Indicator: Example –  a variable with the purpose to measure change in a phenomena or process•  Valid: –  Does the tool measure Pretest Posttest what it claims to measure?•  Reliable: –  Does the tool give consistent results?•  Pre and Post Test Pretest Posttest
  20. 20. Live Demo - Oregon Mentors Evaluation Instrument Toolkit•  Browse for specific instruments (19 “domains”)•  Simple format for each instrument “record”•  Tips for using•  Links to other online repositories
  21. 21. Toolkit Domains of Interest Live Demo of Toolkit & InstrumentsComprehensive Surveys: Personal Attitudes and Beliefs:•  Youth Outcome Survey •  Antisocial attitude scale•  Youth Risk Behavior Surveillance •  Decision making skills scale System •  Goal Setting Skills Scale •  Prosocial Involvement,Connectedness: Opportunities and Rewards•  Hemingway Scale of Adolescent Connectedness Mentoring relationship quality:School and Academics: •  Match Characteristics•  Education Expectations and Questionnaire (MCQ) Aspirations Scale •  Youth Mentoring Survey (YMS)•  Motivated Strategies for Learning •  Youth Survey: Measuring the Questionnaire (MSLQ) Quality of Mentor-Youth Relationships
  22. 22. Tips for Administering Instruments1.  Plan for problems 6.  Don’t provide guidance2.  Don’t hand-pick which 7.  Administer the questions to use instruments in a consistent, uniform way3.  Don’t reword questions 8.  Keep the answers anonymous/ sort out any4.  Get parent permission coding before administration5.  Provide a private space to administer 9.  Plan for data storage and safety
  23. 23. Tips for Selecting External Evaluator1.  Consider his or her philosophical assumptions about evaluation and how appropriate they are to addressing the questions you want answered.2.  Invite finalists to meet project staff and others with whom they will be working closely to see who best fits with individual styles and your organizational culture.3.  An important part of an evaluator’s job is to assist in building the skills, knowledge, and abilities of other staff and stakeholders.4.  Think of evaluation as everyone’s responsibility. Be careful not to delegate all evaluation decision making to your evaluator. See handout: Hiring an Evaluator Checklist
  24. 24. Tips for Working with External Evaluator1.  Don’t try and fake serious evaluation2.  Set aside appropriate funds (5-10 %)3.  Try and get a grad student from local college or university - departments of education, child psychology, social work, health, etc.4.  Remember that you are still in charge of your evaluation!
  25. 25. Resources
  26. 26. Resources
  27. 27. Resources•  W.K. Kellogg Foundation Evaluation Handbook Foundation-Evaluation-Handbook.aspx•  W.K. Kellogg Foundation Logic Model Development Guide Foundation-Logic-Model-Development-Guide.aspx•  Stand by Me: The Risks and Rewards of Mentoring Todays Youth by Jean E. Rhodes
  28. 28. Handouts  Using Standardized Instruments  Comparing Program Outcomes to National Standards  Blank Logic Model Template  Sample YouthBuild Mentoring Logic Model  Using Your Logic Model to Focus the Evaluation  Checklist for Selecting an Evaluator  Sample Contact & Information Release Form  Sample Evaluation Press Release  Evaluating Mentoring Programs – P/PV Brief  Additional Reading and Resources for Evaluating Your Mentoring Program  Glossary of Measurement and Evaluation Terms
  29. 29. Questions?
  30. 30. Thank you!•  Slides posted to SlideShare•  Recording of webinar will be posted to http:// Webinars.html•  Link and brief survey included in follow-up email
  31. 31. Next Webinar•  Tuesday, June 14 Helping Young  People Find Their Spark: A New  Gateway into Mentor‐Mentee  Rela?onships with April Riordan Mentoring Partnership of Minnesota •  Wednesday, June 1: Eat Well,      Stay Ac7ve, Have Fun: A new      resource from the Harvard School      of Public Health  Register online:
  32. 32. 650-559-0200 105093182858863