Mentoring Program Evaluation & Goals

Mentoring Program Evaluation andTransforming lives through the power of mentoring Goals Friends for Youth’s Mentoring Institute May 2011 Webinar In collaboration with the Mentoring Partnership of Minnesota

Webinar Logistics: Adding Comments•  All attendees muted for best sound•  Type questions and comments in the question box; responses will either be direct to you or shared with all attendees•  “Raise your hand” to be unmuted at end to ask question live during webinar •  Works best for telephone or headset-to-computer connections •  Please monitor background noise

PanelistsSarah Kremer, ATR-BC Michael Garringer Program Director Resource Advisor Friends for Youth’s April Riordan Education Northwest/ Celeste Janssen Mentoring Institute Director of Training & National Mentoring Center Program Director Partnerships Oregon Mentors Mentoring Partnership of Minnesota

•  Slides posted to SlideShare•  Recording of webinar will be posted to http:// www.friendsforyouth.org/ Webinars.html•  Link and brief survey included in follow-up email

Recommended Best Practices•  Mission Statement and Long Range Plan•  Recruitment Plan for Mentors and Mentees•  Customized Orientations for Mentors and Mentees•  Eligibility Screening/Application Process for Mentors and Mentees•  Training Curriculum for Mentors and Mentees•  Matching Strategy•  Monitoring Process•  Support, Recognition, and Retention Component•  Relationship Closure Steps•  Evaluation Process California Governor’s Mentoring Partnership, 2002

Friends for Youth’s Impact•  Nearly 1,800 matches•  88% overall success rate of mentors and mentees reaching one year together•  Program Coordinator contacts –  1x/week for first 3 months + 2x/weeks for remainder (Senior Friends) –  1x/month (Junior Friends + parents/caregivers)•  Junior Friend Evaluation outcomes –  Developed by Cindy Sipe, Ph.D. –  Evaluation by William Lapp, Ph.D. –  Over 10 years of data on matches

5 Outcome Areas•  Increase in involvement in school (attendance, truancy, attitude, grades, suspensions)•  Reduce risk behaviors (drug/alcohol use, police involvement, probation)•  Increase positive behaviors (community service, new activities)•  Increase opportunities to influence future potential (office visits, talking about college & career with adult)•  Improve self-concept (trusting relationships, self-assuredness, self- reliance, handle unexpected problems)

Friends for Youth’s ImpactAfter 12-month period, youth made significant changes•  Improved school attendance –  81% reported not skipping school at all –  75% reported avoiding suspensions•  Improved attitude toward school –  100% showed positive change in at least 1 aspect –  69% showed positive change in 2 or more aspects•  Reduction in risk behaviors –  reduced self-reported incidents of police involvement (13% vs. 28%), stealing (4% vs. 23%), hurting others (12% vs. 32%) –  alcohol & drug use decreased in time & showed delay in starting use Lapp 2009 (2007-2009 results)

Friends for Youth’s ImpactAfter 12-month period, youth made significant changes•  Increase in positive behaviors –  87% participated in community service project vs. 35% baseline –  significant increase in new activities•  Improved self esteem –  95% reported increase in at least 1 aspect of self-esteem –  63%reported increase in all 3 aspects•  Increase in number of trusting relationships with family, peers, & other adults –  nearly 90% reported positive overall change Lapp 2009 (2007-2009 results)

Friends for Youth Transforming lives through the power of mentoring . Youth-in-Need “My mentor Friends for Youth participants are: Lisa doesn’t * mostly middle-school age Societal Cost scream at me * mainly low income and two-thirds are below the poverty level California spends or hit me. She helps me with * often from single parent homes $234,000 per year for stu , and she teaches me things. * sometimes have a parent who is each youth in a Juvenile I’ve stopped hurting people. I’ve incarcerated Justice Facility. stopped beating up people.” * all at-risk of not reaching their full potential and may be susceptible to: Using Mentee Pedro For each youth who Substance Abuse avoids dropping out of The very foundation of mentoring Violence & Gang Involvement school for a life of drugs is that if a caring, concerned adult is Academic Failure and crime, the country involved in the life of a young person, These youth have little adult support, saves between $2.6 to the youth will be more likely to and now schools are cutting back $5.3 million.Evaluation become a successful adult. after-school programs. Measurable Impact of Our Program Friends for Youth puts !"#$%&(&$)#*+",-%.)/012"13%# signi cant emphasis on program evaluation and Results !"#$"%&(")*+,#-."+"%&)/0"#)1)2"#)3%)&4")5#3"%67)8-#)2-9&4)!#-(#+) its impact on our youth. (""# Ten years of statistical anaylsis con rm the "# positive impact of our program. &"# %"# “In conclusion, the Friends for Youth program is $"# functioning at an extremely high degree of e ciency !"# and has shown a positive # # # # # C# # F# B;# 0# # F# improvement...as well ,+ 3 6;0 20 +3 02 ,3 ,52 *,; -/ ,1 28 235 How0 Mentoring 52Helps;F#.26* 01 23 ,5# 9: + *2 #> 67 - ?6 +@ 7A #) @/ -# .; 27 B #) 5A #I =) A +3 0M B NO9 0;2 20 ,B, as helping to address negative in uences that ./ 8 + -# ;7 3 +, /3 , 2I B2C2 2B7 2#.2 02 #L -# -2 16 +, @0 )* #H +B ;#L could otherwise cause +, 57 =48 )* 8 F#7 70 27 )* #) ,3 #./ +8 G6 0#E 7623 62 43 56 problems in [mentees] 02 ;7 +, - #E /0 43 5 27 )* D+ 05 6#? immediate lives and 56 7,3 D, K2 43 < J2 possibly into adulthood.” William Lapp, Ph.D. “My mentor Mike came to conferences and spoke to my teachers about my potential and about how smart and promising I am. It was nice to have someone tell them that I am a smart kid.” Mentee Diego

Overview•  Introduction & Best Practices –  Elements of Effective Practice –  Evaluation Plan & Logic Model•  Tips for Selecting Instruments –  Oregon Mentors Evaluation Toolkit•  Important Rules for Administering Instruments•  Additional Resources

Measuring Effectiveness How well Youth How well Quality of Mentee’s your behavior you run match a;tudes volunteers (outcomes your rela7on‐ on certain mentor the you wish program ships topics youth to achieve)

Tips for Evaluating Mentoring Programs Let’s look at the mechanisms of mentoring…

Missed Opportunities in Mentoring Program Evaluation1.  Service delivery Did they get the mentoring promised?2.  Participant satisfaction3.  Mentoring relationship Did they form the bond needed to make any of this work?4.  Intermediary outcomes are OK!

What is a Logic Model?•  Systematic picture of how you believe your program works•  “…helps you visualize and understand how human and financial investments can contribute to achieving your intended program goals.”

Sample Vacation Logic Model

Sample YouthBuild Mentoring Logic Model

Tips for Selecting Good Instruments•  Try to use instruments that have been tested for reliability and validity•  Make sure they fit your logic model, especially in terms of proximal and distal outcomes•  Note the complexity of the tool: –  Age range –  Reading level –  Concepts•  Can you provide a comparison group of some kind?

Quantitative Tools•  Indicator: Example –  a variable with the purpose to measure change in a phenomena or process•  Valid: –  Does the tool measure Pretest Posttest what it claims to measure?•  Reliable: –  Does the tool give consistent results?•  Pre and Post Test Pretest Posttest

Live Demo - Oregon Mentors Evaluation Instrument Toolkit•  Browse for specific instruments (19 “domains”)•  Simple format for each instrument “record”•  Tips for using•  Links to other online repositories

Toolkit Domains of Interest Live Demo of Toolkit & InstrumentsComprehensive Surveys: Personal Attitudes and Beliefs:•  Youth Outcome Survey •  Antisocial attitude scale•  Youth Risk Behavior Surveillance •  Decision making skills scale System •  Goal Setting Skills Scale •  Prosocial Involvement,Connectedness: Opportunities and Rewards•  Hemingway Scale of Adolescent Connectedness Mentoring relationship quality:School and Academics: •  Match Characteristics•  Education Expectations and Questionnaire (MCQ) Aspirations Scale •  Youth Mentoring Survey (YMS)•  Motivated Strategies for Learning •  Youth Survey: Measuring the Questionnaire (MSLQ) Quality of Mentor-Youth Relationships

Tips for Administering Instruments1.  Plan for problems 6.  Don’t provide guidance2.  Don’t hand-pick which 7.  Administer the questions to use instruments in a consistent, uniform way3.  Don’t reword questions 8.  Keep the answers anonymous/ sort out any4.  Get parent permission coding before administration5.  Provide a private space to administer 9.  Plan for data storage and safety

Tips for Selecting External Evaluator1.  Consider his or her philosophical assumptions about evaluation and how appropriate they are to addressing the questions you want answered.2.  Invite finalists to meet project staff and others with whom they will be working closely to see who best fits with individual styles and your organizational culture.3.  An important part of an evaluator’s job is to assist in building the skills, knowledge, and abilities of other staff and stakeholders.4.  Think of evaluation as everyone’s responsibility. Be careful not to delegate all evaluation decision making to your evaluator. See handout: Hiring an Evaluator Checklist

Tips for Working with External Evaluator1.  Don’t try and fake serious evaluation2.  Set aside appropriate funds (5-10 %)3.  Try and get a grad student from local college or university - departments of education, child psychology, social work, health, etc.4.  Remember that you are still in charge of your evaluation!

Resourceshttp://oregonmentors.org/library/evaluationtools/

Resources http://educationnorthwest.org/nmchttp://mentoringforums.educationnorthwest.org/

Resources•  W.K. Kellogg Foundation Evaluation Handbook http://www.wkkf.org/knowledge-center/resources/2010/W-K-Kellogg- Foundation-Evaluation-Handbook.aspx•  W.K. Kellogg Foundation Logic Model Development Guide http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg- Foundation-Logic-Model-Development-Guide.aspx•  Stand by Me: The Risks and Rewards of Mentoring Todays Youth by Jean E. Rhodes

Handouts  Using Standardized Instruments  Comparing Program Outcomes to National Standards  Blank Logic Model Template  Sample YouthBuild Mentoring Logic Model  Using Your Logic Model to Focus the Evaluation  Checklist for Selecting an Evaluator  Sample Contact & Information Release Form  Sample Evaluation Press Release  Evaluating Mentoring Programs – P/PV Brief  Additional Reading and Resources for Evaluating Your Mentoring Program  Glossary of Measurement and Evaluation Terms

Questions?

Thank you!•  Slides posted to SlideShare•  Recording of webinar will be posted to http:// www.friendsforyouth.org/ Webinars.html•  Link and brief survey included in follow-up email

Next Webinar•  Tuesday, June 14 Helping Young People Find Their Spark: A New Gateway into Mentor‐Mentee Rela?onships with April Riordan Mentoring Partnership of Minnesota •  Wednesday, June 1: Eat Well, Stay Ac7ve, Have Fun: A new resource from the Harvard School of Public Health Register online: http://www.mpmn.org/Training/MPMTRAINING/WebinarSeries.aspx

www.mentoringinstitute.org 650-559-0200 http://www.facebook.com/pages/Friends-for-Youth/ 105093182858863 http://twitter.com/friendsforyouth http://www.friendsforyouth.blogspot.com/ http://www.youtube.com/user/FriendsforYouthOrg

Mentoring Program Evaluation & Goals

by Friends for Youth, Inc. on May 12, 2011

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Mentoring Program Evaluation & Goals — Presentation Transcript