Development and evaluation of mentoring programs for youth. Prof. David DuBois. June 2013


Published on

Why youth mentoring as an intervention strategy?
Why be systematic/rigorous about developing (and improving) mentoring intervention strategies and evaluating their effectiveness?
What is “best practice” when developing mentoring intervention strategies?
What are the most rigorous and informative methods for evaluating youth mentoring intervention strategies?

1 Comment
  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Development and evaluation of mentoring programs for youth. Prof. David DuBois. June 2013

  1. 1. Development and Evaluation of Mentoring Programs for Youth
  2. 2. Strategies for Fostering Success in the Next Generation David L. DuBois University of Illinois at Chicago Colloquium Presentation, University of Girona, Barcelona, Spain June 20, 2013
  3. 3.  Why youth mentoring as an intervention strategy?  Why be systematic/rigorous about developing (and improving) mentoring intervention strategies and evaluating their effectiveness?  What is “best practice” when developing mentoring intervention strategies?  What are the most rigorous and informative methods for evaluating youth mentoring intervention strategies?  Concluding thoughts
  4. 4.  Fundamental role of mentoring in human development  Evolutionary basis • Mechanism for intergenerational knowledge transfer (Wong, 2009) • Adaptive advantage of altruism toward other group members (Bowles, 2006) • Value of shared responsibilities for caregiving: allo-parenting (Diamond, 2012) and age-mixing (Ellis et al., 2012)  Development and learning powered by dyadic relationships with 4 key features (Bronfenbrenner, 2005; see also Li & Julian, 2012) • Emotional attachment • Reciprocity • Progressively more complex activities • Gradual shifting of power in favor of developing person
  5. 5.  Lack of Access to Mentoring in Contemporary Society  ~1 in 5 youth in the U.S. estimated be lacking supportive relationships with caring adults • Least access among youth experiencing economic disadvantage  Evidence suggests proportion of youth without any “go to” adult may be increasing Source: DuBois (2013).
  6. 6.  Empirical Support for Youth Mentoring Supports  Capacity of mentoring programs to promote youth outcomes demonstrated through several meta-analyses of rigorous (quasi- experimental and experimental) evaluation studies (DuBois et al., 2002, 2011; Tolan et al., 2008; Wheeler et al., 2010)  Evidence of: • Benefits in multiple areas: social, behavioral, emotional, academic
  7. 7. 0 0.2 0.4 EffectSize Youth Mentoring (Pre-1999) Youth Mentoring (1999-2010) Source: DuBois et al. (2002, 2011)
  8. 8.  Evidence of: • Benefits in multiple areas: social, behavioral, emotional, academic • Magnitude of impacts that are comparable to those achieved by other forms of youth intervention, but may be more holistic/multi-faceted 0 0.1 0.2 0.3 EffectSize Youth Mentoring (1999-2010) Other Youth Interventions Source: DuBois et al. (2011)
  9. 9. -0.5 -0.3 -0.1 0.1 0.3 0.5 Pre-Test Post-Test YouthOutcomes Non-Mentored Youth Mentored Youth Promotion Prevention  Evidence of: • Benefits in multiple areas: social, behavioral, emotional, academic • Magnitude of impacts that are comparable to those achieved by other forms of youth intervention, but may be more holistic/multi-faceted • Both preventive and promotive effects (DuBois et al., 2011)
  10. 10.  Evidence that systematic intervention development strategies (e.g., theory- and research-basis) tend to result in more effective programs  Systematic development likely to produce more replicable and scalable programs
  11. 11.  Limited magnitude of current program effects indicates need to search for stronger programs/strategies  Not safe to assume effectiveness of a strategy/program based on prior evaluations due to potential differences in implementation quality, targeted youth population, local conditions, etc.  Need to grow knowledge base for the field and fill in gaps (e.g., long-term effects)
  12. 12.  Consider drawing on one or more existing program planning frameworks (e.g., PRECEDE-PROCEED; Greene & Kreuter, 1999) 1. Assess Needs & Capacities of Population 2. Assess Causes, Set Priorities & Objectives 3. Design & Implement Program 4. Evaluate Program Reassess causes Redesign Source: Green & Kreuter (2005)
  13. 13.  Assess needs, assets, and priorities of the community to be served and design strategies accordingly.  Potential sources of information: • Existing: Research, public data bases • New: Key informant interviews, focus groups, surveys  Potential questions: • Who has least access to mentoring? • Who stands to benefit most from mentoring? • What kinds of mentoring are most needed? • What types of mentoring are already occurring? • What assets (human, organizational, etc.) could be leveraged to provide effective mentoring?
  14. 14.  Search for “leads” on promising intervention strategies  Review scholarly literature: • Theoretical models • Observational research • Evidence-supported strategies – SPECIAL EMPHASIS – Priority on high quality experimental and quasi-experimental evaluations – Draw on findings of systematic reviews of such evaluations, especially meta-analyses DuBois & Karcher (2014) Herrera, DuBois, & Grossman (2013)
  15. 15. • A Few Recent Examples (among many!) – Youth-nominated mentoring associated with improved educational and employment outcomes for marginalized older youth (Schwartz et al., in press) – Beneficial effects of community-based mentoring on depressive symptomatology among higher-risk youth (Herrera et al.. 2013) – Emphasis on similarity of interests when matching mentors and youth predictive of greater program effectiveness (DuBois et al., 2011) – Support for mentor assuming teaching and advocacy roles also linked to enhanced effectiveness (DuBois et al., 2011)
  16. 16. Empirically-Supported Best Practices -0.1 0 0.1 0.2 0.3 0.4 0.5 0 1 2 3 4 5 6 7 Number of Practices SizeofEffectonYouth Small Effect Medium Effect (Based on DuBois et al., 2002)
  17. 17.  Outreach to practice community • Program models and intervention strategies already in use • Seek input and ideas on directions for program/strategy design  Obtain input from stakeholders • Acceptability, usability, feasibility, utility • Youth, mentors, parents, staff  Develop and pilot strategies/program components  Build program logic model / theory of change  Iteratively refine through multiple cycles of implementation  Collect and analyze process evaluation data to guide process  Carefully document and “manualize” all components/activities
  18. 18.  Example: GirlPOWER! (DuBois et al., 2008)  Mentoring program for ethnic minority girls ages 10-13 • Designed as enhancement of Big Brothers Big Sisters (BBBS) community-based mentoring (CBM) program • 12 monthly, 3-hour group sessions in 10-15 mentor-youth pairs  Assessment • Informed by: – Interviews with youth development professionals, mentors, parents, and youth – Review of research • Findings: – Needs -- Lack of evidence-based mentoring programs for girls – Assets -- Girls group-based programming model already in use by local BBBS agency – Priorities – Girls’ need for support particularly great during transition to adolescence
  19. 19.  Research and theory basis • BBBS CBM program model is empirically supported • Incorporated practices linked to greater effectiveness in meta- analysis of mentoring program effectiveness (DuBois et al., 2002) – Structured activities for mentors and youth – Ongoing mentor training – Parent support and involvement • Built in theory-/research-based gender-sensitive program elements – Relational focused group format – Counter limiting stereotypes of girls – Focus on salient concerns for girls during transition to adolescence
  20. 20.  Program design process • Informed by interviews/focus groups with stakeholders (youth, mentors, parents, program staff) – Example: Girls interested in competition, but did not want to resemble school • Iterative piloting and refinement of group sessions and other program components – Example: Build in more time for informal socializing based on input of mentors and youth • Ongoing manualization of all aspects of program – Examples: Mentor training, group sessions, out-of-session activity guides, parent communications
  21. 21.  Example: Step-It-Up-2-Thrive (research in progress)  Mentoring framework for youth ages 10-16 • Designed as enhancement of BBBS CBM program • Based on “Road Map” for thriving: Sparks, Growth Mindset, Thriving Indicators, Goal Management/Pursuit Skills • Resources available on-line (  Assessment • Informed by: – Dialogue with BBBS national program staff – Review of research • Findings: – Needs – Lack of evidence-based programming to promote youth thriving through mentoring – Assets -- BBBS program model includes regularly occurring support contacts with mentors = important mechanism for program delivery – Priorities – Growing interest in moving beyond “prevention” model to “promotion” model focused on helping youth to reach their full potential
  22. 22.  Research and theory basis • Recent research findings point toward contribution of abilities and experiences in each of the four areas of the “Road Map” to outcomes important for youth (e.g., academic success, avoiding involvement in problem/delinquent behavior) Informed by: – Associations for most part relatively modest, suggesting that combination of strengths in all areas may be needed for optimal outcomes – Theoretically, effects of factors in one area also may be dependent on those in others (e.g., need growth mindset to effectively pursue sparks) • Emphasis on teaching/guidance role for mentor supported by findings of recent meta-analysis of mentoring program effectiveness (DuBois et al., 2011)
  23. 23. Positive effects of intervention teaching incremental theory (growth mindset) to 7th graders (Blackwell, Trzesniewski, & Dweck, 2007)2
  24. 24.  Program design process • Drew heavily on existing resource materials from Thrive Foundation for Youth • Driven by national BBBS staff and input from staff of 3 “lead design” agencies (BBBS affiliates) • Piloting of mentor training and group activities • Manualization of all aspects of program
  25. 25.  Planning  Decide on primary purposes of evaluation  Ensure adequate resources and technical expertise are available  Consult an accepted set of standards for program evaluation (e.g., Program Evaluation Standards of Joint Committee on Standards for Education Evaluation)  Obtain approval from research review board *See DuBois (2014) for a more detailed treatment of issues involved with evaluation of mentoring programs.
  26. 26.  Process Evaluation  Major purposes • Determine whether program happens as planned • Obtain information to guide further development/refinement of program • Accountability to stakeholders • Support good program delivery – Monitoring procedures likely to improve implementation – “what gets measured, gets done” – and linked to better mentoring program effects (DuBois et al., 2002)
  27. 27.  Should assess both fidelity of implementation and dosage • Fidelity = extent to which program/strategy was implemented as planned – Assess for both staff- and mentor-implemented activities – Key components to assess: » Adherence: extent to which services/activities are carried out according to design » Exposure: level of intended services/activities provided » Quality of Delivery: how well services/activities are implemented » Responsiveness: how participants experience services/activities » Differentiation: extent to which the services/activities differ from those associated with other similar programs • Dosage = levels of exposure/participation for individual participants – Assess with respect to services provided to both youth and their mentor  Important to gather both quantitative and qualitative data • Quantitative: May be particularly helpful for gauging whether implementation benchmarks are met and whether these improve over multiple implementations • Qualitative: May be particularly helpful for identifying ways to improve program content and for detecting unexpected experiences of participants
  28. 28.  Outcome Evaluation  Major purposes • Determine whether program has intended effects on participants • Test program’s theory of change • Obtain information needed to support economic evaluation of program’s “return on investment”
  29. 29.  Keys • Need credible comparison group of non-participants to obtain accurate and unbiased estimate of program effects – Random assignment to program vs. comparison group = “gold standard” • Need sufficiently large sample to detect effects • Need reliable and valid measures of program outcomes – Use program logic model / theory of change as guide – Ideally measure key outcomes through multiple methods – “triangulate” • Incorporate follow-up assessment of outcomes beyond the period of program involvement • Analyze data for both overall program effects and effects on different subgroups of youth
  30. 30.  Example: GirlPOWER!  Small-scale (n = 40) randomized controlled trial comparing GirlPOWER! to standard BBBS community-based mentoring  Process Evaluation • Agency records, mentor reports (Adherence, Exposure, Differentiation, Dosage) • Direct observation (Quality of delivery) • Youth, mentor, parent feedback surveys (Responsiveness)  Outcome Evaluation • Youth (and their mentors) randomly assigned to participate in GirlPOWER! or BBBS standard programming • Pre-test and one-year (end of program) • Youth, parent, and mentor surveys • School and juvenile system records
  31. 31.  Selected Findings • Process Evaluation – Strengthening mentor-youth relationships » Mutual sharing » “we got to learn more about each other” [mentor] » “we communicated more” [youth] » “Seeding” discussions and activities » “… it will be easier for us to discuss peer pressure & drugs” [mentor] » “… it gave me some ideas on how we can stay fit together” [youth] – Facilitating group support and cohesion » Supportive group experience for youth » “she really looks forward to … hanging with the Littles” [mentor] » “[performing in talent show helped me] break out of my shell” [youth]; » Mentor sharing and support » “meeting similar Matches makes the entire BBBS experience better” [mentor] » “… the Bigs find it helpful to share info, concerns, etc.” [mentor] » Match connections » “Bigs can share ideas & make plans together …” [mentor]
  32. 32. – Program improvements guided by findings » POWERSessions » make activities more interactive, less didactic; free up time for casual socializing » greater utilization of mentors as teachers and role models » POWERBuilders » Encourage matches to design their own activities; be liberal in “crediting” activities as addressing different program topics » Resource materials » Reduce length and organize in program binder
  33. 33. – Improvements evident in youth and mentor experiences of the program sessions Spring 2005 Fall 2005 AverageRatingsfor"FUN" 1 2 3 4 5 Little(Self) Big(Little) Big(Self) Respondent Rating AverageRatingsfor"HELPFUL" 1 2 3 4 5 Little(Self) Big(Little) Big(Self) Respondent Rating
  34. 34. • Outcome Evaluation – Improvements in mentoring relationship quality » More frequent mentor-youth contact » Stronger instrumental/goal-oriented dimension of relationships » Mentor seen by youth as more disapproving of problem behaviors (e.g., unhealthy eating, substance use) – Improved youth outcomes (greater improvement than youth in standard community-based mentoring) » Health behavior » Health knowledge & beliefs » Exercise » Emotional well-being and resilience » Perceived coping efficacy » Domain-specific and global self-esteem » Anxiety/depression » Academic » Intrinsic motivation for learning » Educational aspirations
  35. 35.  Example: Step-It-Up-2-Thrive  Randomized controlled trial comparing BBBS community- based mentoring with and without integration of Step-It-Up- 2-Thrive framework  10 BBBS agencies, ~800 youth ages 10-16 with one or more risk factors for engaging in delinquent behavior  Process Evaluation • Staff reports, agency records (Adherence, Exposure, Differentiation) • Implementation checklist completed by staff (Dosage) • Staff self-assessment, participant feedback (Quality of delivery) • Youth, mentor, parent feedback surveys (Responsiveness)  Outcome Evaluation • Youth (and their mentors) randomly assigned to participate in BBBS CBM with addition of “Thrive” integration or standard programming • Pre-test and 15 months • Youth, parent, and mentor surveys
  36. 36.  Benefits of “real time” analysis of process evaluation data • Unexpectedly low rates of participation in group activities for matches (mentor-youth pairs) – Adjustment: » Development of alternative formats » Activity guides for mentor-youth pairs to work through on their own » On-line format • Challenges with having matches move through different stages of intervention activity in planned sequence – Adjustment: » Allow flexibility in sequencing where fits with needs and interests of youth (and mentor) • Lack of case manager comfort with supporting intervention content – Adjustments: » Increase their role as “trainers” of mentors » Provide greater opportunities for technical assistance from national staff
  37. 37.  Promise of mentoring is arguably as well-established as any form of youth intervention  It would appear, however, that only a surprisingly small portion of this potential has been realized  More systematic and rigorous approaches to development and evaluation of mentoring intervention strategies could help to narrow this gap  Important not to overlook need for complementary focus on improving extent & equity of youth access to effective forms of mentoring
  38. 38.  Impact + Reach = Healthier Future Generations and Greater Global Prosperity
  39. 39. Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78, 246-263. Bowles, J. (2006). Group competition, reproductive leveling, and the evolution of human altruism. Science, 314(5805), 1569–1572. Bronfenbrenner, U. (2005). Making human beings human: Bioecological perspectives on human development. Thousand Oaks, CA: Sage. Diamond, J. (2012). The world until yesterday. New York: Viking Press. DuBois, D. L. (2013). Trends and correlates of youth reports of not having a parent or other adult available to provide support with personal problems. Manuscript in prepartion DuBois, D. L. (2014). Program evaluation. In D. L. DuBois & M. J. Karcher (Eds.), Handbook of youth mentoring (2nd ed., pp. 481-498). Thousand Oaks, CA: Sage. DuBois, D. L., Holloway, B. E., Valentine, J. C., & Cooper, H. (2002). Effectiveness of mentoring programs for youth: A meta-analytic review. American Journal of Community Psychology, 30, 157– 197. DuBois, D. L., & Karcher, M. J. (Eds.). (2014). Handbook of youth mentoring (2nd ed.). Thousand Oaks, CA: Sage. DuBois, D. L., Portillo, N., Rhodes, J. E., Silverthorn, N., & Valentine, J. C. (2011). How effective are mentoring programs for youth? A systematic assessment of the evidence. Psychological Science in the Public Interest, 12, 57–91.
  40. 40. DuBois, D. L., Silverthorn, N., Pryce, J., Reeves, E., Sanchez, B., Silva, A., . . . Takehara, J. (2008). Mentorship: The GirlPOWER! program. In C. LeCroy, Craig Winston, & J. E. Mann (Eds.), Handbook of prevention and intervention programs for adolescent girls (pp. 326–336). Hoboken, NJ: Wiley. Ellis, B. J., Del Giudice, M., Dishion, T. J., Figueredo, A. J., Gray, P., Griskevicius, V., . . . Wilson, D. S. (2012). The evolutionary basis of risky adolescent behavior: Implications for science, policy, and practice. Developmental Psychology, 48, 598-623. Green, L.W.,&Kreuter, M.W. (2005). Health promotion planning:An educational and ecological approach (4th ed.). New York: McGraw-Hill. Herrera, C., DuBois, D. L., & Grossman, J. B. (2013). The role of risk: Mentoring experiences and outcomes for youth with varying risk profiles. New York: A Public/Private Ventures project published by MDRC. Li, J., & Julian, M. (2012). Developmental relationship as the active ingredient: A unifying working hypothesis of ―what works‖ across intervention settings. American Journal of Orthopsychiatry, 82, 157-166. Schwartz, S., Rhodes, J., Spencer, R., & Grossman, J. (in press). American Journal of Community Psychology. Youth initiated mentoring: Investigating a new approach to working with vulnerable adolescents. American Journal of Community Psychology. Tolan, P., Henry, D., Schoeny, M., & Bass, A. (2008). Mentoring interventions to affect juvenile delinquency and associated problems. Campbell Systematic Reviews, 16. Retrieved from Wheeler, M. E., Keller, T. E., & DuBois, D. L. (2010). Review of three recent randomized trials of school-based mentoring: Making sense of mixed findings. Social Policy Report, 24(3). Wong, K. (2009). Twilight of the Neandertals. Scientific American, 301(2), 32–37.