Mentoring 418
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Mentoring 418

on

  • 471 views

 

Statistics

Views

Total Views
471
Views on SlideShare
471
Embed Views
0

Actions

Likes
0
Downloads
5
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Mentoring 418 Document Transcript

  • 1. How to Build A Successful Mentoring Program Using the Elements of Effective Practice TM A STEP-BY-STEP TOOL KIT FOR PROGRAM MANAGERS
  • 2. Don’t Forget the Free Tools! Please note that this PDF does not include all of the tools available on the CD-ROM. You can review and download the tools you want directly from the web site: Tools for Designing and Planning http://www.mentoring.org/find_resources/tool_kit/design/ Tools to Manage a Program for Success http://www.mentoring.org/find_resources/tool_kit/management/ Tools to Structure Effective Program Operations http://www.mentoring.org/find_resources/tool_kit/operations/ Tools to Establish Evaluation Criteria and Methods http://www.mentoring.org/find_resources/tool_kit/eval/
  • 3. The final mentoring program element listed in the sec- ond edition of the Elements of Effective Practice is pro- gram evaluation. The following are among the many reasons1 programs should conduct evaluations: • To increase understanding of effective practices in youth mentoring relationships and programs; • To make the programs accountable to the entities that support them; • To promote effective resource allocation (i.e. to identify the most deserving recipients of scarce funds); • To avoid unintended harmful effects of interventions; • To increase the effectiveness of programs through a feedback/continuous quality improvement process; and • To provide direct benefits for case managers, mentors and youth when evaluation of individual relationships is built into the evaluation plan. To ensure the quality and effectiveness of your pro- gram, you’ll need to do the following: • Develop a plan to measure program processes; • Develop a plan to measure expected outcomes; and • Create a process to reflect on and disseminate evaluation findings. The ultimate success of your program depends on how well you are able to assess its effectiveness, address any weaknesses and demonstrate that it is meeting estab- lished goals and objectives. With a comprehensive evaluation process in place, you can do the following:2 • Provide objective feedback to program staff and participants about whether they’re meeting their goals; • Identify achievements and milestones that warrant praise and increase motivation; • Pinpoint problems early enough to correct them; • Assure funders and supporters of your program’s accountability; • Build credibility in the community that your program is vital and deserves support; and • Quantify experiences so that your program can help others. MEASURE PROGRAM PROCESS Your plan for measuring program process should include the following: • Selecting indicators of program implementation viability and volunteer fidelity, such as training hours, meeting frequency and relationship duration; and • Developing a system for collecting and managing specified data. MEASURE EXPECTED OUTCOMES Your plan for measuring expected outcomes should include the following: • Specifying expected outcomes; • Selecting appropriate instruments to measure outcomes, such as questionnaires, surveys and interviews; and • Selecting and implementing an evaluation design. 163HOW TO ESTABLISH EVALUATION CRITERIA AND METHODS Section VII. How to Establish Evaluation Criteria and Methods
  • 4. CREATE A PROCESS TO REFLECT ON AND DISSEMINATE FINDINGS The final stage of program evaluation includes the fol- lowing activities: • Refining the program design and operations based on the findings; and • Developing and delivering reports to program constituents, funders and the media (at a minimum, yearly; optimally, each quarter). The article “Gauging the Effectiveness of Youth Mentoring,” written by Dr. Jean Rhodes for MENTOR’s Research Corner, is reprinted below. (The text of the article has been edited to meet the needs of the tool kit.) It analyzes the components out- lined above for conducting a thorough process and outcome evaluation of a mentoring program. It will be an invaluable reference for you as you determine how best to develop an evaluation plan for your mentoring program. “GAUGING THE EFFECTIVENESS OF YOUTH MENTORING” BY DR. JEAN RHODES The practice of evaluating one’s own efforts is as natural as breathing. Cooks taste their own gravy and sauce, cab- inetmakers run their hands over the wood to decide when a piece is smooth enough, and basketball players watch to see whether their shots go in. Indeed, it would be most unwise after turning on the hot water to neglect to check the water temperature before stepping into a shower stall. — Posavac & Carey (1997) Although program evaluation is not as natural or spontaneous as this sort of self-evaluation, most pro- grams engage in some form of monitoring. Sometimes it’s as simple as asking mentees and mentors about their experiences; in other cases, it involves large-scale, rigorous experimental designs. Of course, programs are more apt to launch the former, less complicated types of evaluation. Such evaluations do not require the same level of expertise, are far less expensive, place minimal burden on partic- ipants and staff, and can yield useful findings. For example, simple exit interviews can provide staff with important and immediate feedback about programs. So, you might ask, why not stop there? A primary rea- son is that funders need more convincing evidence that programs are actually reaching their objectives. Thus, accountability has increasingly involved moving beyond simple descriptions to demonstrating that spe- cific goals have been met. Knowing your options will help you make informed decisions about the scope and rigor of your design. Determining the Impact of Your Program We’ll cover several options, ranging from the simple to the more complicated.3 We’ll begin with a strategy that relies on comparing your program with others (i.e., using benchmarks) to determine whether you are having an effect. Some of the more intensive evalua- tion approaches (e.g., quasi-experimental designs), on the other hand, might require the expertise of an out- side evaluator, such as a graduate student or a profes- sor from a local university. The cost of an outside eval- uation tends to vary according to its intensity, but programs should budget between $5,000 and $10,000 for the expertise. Using Benchmarks Without actually conducting an evaluation, programs can sometimes draw on findings that have been linked to outcomes in similar programs. In other words, find- ings from other studies can be used as benchmarks against which to gauge a program’s relative effective- ness.4 This approach is feasible when your program has these characteristics: • It is targeting similar youth to the evaluated program; • It is reasonably similar in terms of relationship structure and content to the evaluated program; and • It has met or exceeded the evaluated program’s quality standards. 164 HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE
  • 5. SECTION VII 165HOW TO ESTABLISH EVALUATION CRITERIA AND METHODS What can we infer from other evaluations? DuBois and his colleagues conducted a meta-analysis of 55 evalua- tions of one-to-one youth mentoring programs.5 The analysis summarized the results of each study and cal- culated effect sizes (the magnitude of impact) across the entire group of studies. Modest effects of mentor- ing programs were found across fairly diverse pro- grams, but larger effect sizes emerged under the fol- lowing conditions: • Youth were somewhat vulnerable but had not yet succumbed to severe problems. • Relationships were characterized by 1. More frequent contact; 2. Emotional closeness; and 3. A duration of six months or longer. • Programs were characterized by practices that increased relationship quality and longevity, includ- ing these: 1. Intensive training for mentors; 2. Structured activities for mentors and youth; 3. High expectations for frequency of contact; 4. Greater support and involvement from parents; and 5. Monitoring of overall program implementation. Since greater numbers of these practices predicted more positive outcomes for youth in mentoring pro- grams, one-to-one programs that have met these crite- ria can assume positive outcomes. Additionally, research by Roffman, Reddy and Rhodes on one-to- one programs has provided two relatively simple benchmarks against which similar one-to-one mentor- ing programs can measure themselves to ensure that relationships will have positive effects:6 • Duration. Because duration tends to imply strong relationships and programs, it may be the single best benchmark of program effectiveness. Across several studies, longer durations have been associated with stronger effects. • Relationship quality. Although duration is probably the single best benchmark, research found that the quality of a mentoring relationship can predict positive outcomes above and beyond how long the relationship lasts. When responses to a ques- tionnaire used in this research indicated a posi- tive, nonproblematic rela- tionship, that relationship tended to last longer and have more positive effects. Although benchmarks can be enormously useful, they may not provide the level of detail or rigor that programs or funders desire. Moreover, at this stage, benchmarks can only be applied to one-to-one programs. Thus, it is often necessary to conduct a structured evaluation. The Nuts and Bolts of Evaluating Mentoring Programs Types of Program Evaluation There are two major types of program evaluation: process evaluations and outcome eval- uations. • Process evaluations focus on whether a program is being implemented as intended, how it is being experienced, and whether changes are needed to address any problems (e.g., difficulties in recruiting and retaining mentors, high turnover of staff, high cost of administering the program). • Outcome evaluations focus on what, if any, effects programs are having. Designs may compare youth who were mentored to those who were not or may examine the differences between mentoring approaches. Information of this sort is essential for self-monitoring and can address key questions about programs and relationships. See “Gauging the Effectiveness of Youth Mentoring Questionnaire” in Section VII on the CD. MENTORING TOOL See program evaluation tools in Section VII on the CD. MENTORING TOOL
  • 6. Process evaluations of mentoring programs usually involve data from interviews, surveys and/or program records that shed light on the following areas: • Number of new matches; • Types of activities; • Length of matches; • Frequency and duration of meetings; and • Perceptions of the relationship. Information of this sort is essential for self-monitoring and can answer key questions about programs and relationships. Despite the importance of such information, outcome evaluations have become increasingly important for accountability; therefore, the rest of this section will focus on the issues and decisions involved in conduct- ing an outcome evaluation. Outcome evaluations of mentoring programs usually involve data from surveys, interviews, records and so forth, including the following: • Mentees’ reports of their grades, behavior and psychological functioning; • Teachers’ reports of mentees’ classroom behavior; • Mentors’ reports of their well-being; • Parent-child relationships; and • High-school graduation rates. Tips for and traps in conducting an outcome evaluation Measuring outcomes • Select outcomes that are most: a. Logically related to (and influenced by) the program; b. Meaningful to you; and c. Persuasive to your funders. • Be realistic. You are better off building a record of modest successes, which keep staff and funders motivated, than focusing on “big wins,” which may be unrealistic and, when not achieved, demoralizing. • Collect outcome data after the youth and mentors have been meeting for some time, long enough to expect that some changes in the youth have occurred. Determining sources of data • Obtain information from multiple sources, including reports from mentees, mentors, parents, caseworkers and so on. • Select multiple criteria rather than just one outcome (e.g., grades, drug use, attitudes). • Use standardized questionnaires. a. Questionnaires that have been scientifically validated are more convincing to funders—and provide a better basis for cross-program comparisons—than surveys you might develop on your own. b. Such surveys are available for public use through tool kits. The Search Institute has one available (What’s Working: Tools for Evaluating Your Mentoring Program) for purchase (see list of Additional Resources below) and The Mentor Center links to several free online resources. c. The Juvenile Justice Evaluation Center provides links to questionnaires that are likely to be of interest to mentoring programs, including questionnaires about delinquency, drug and alcohol use, ethnic identity, peer relations, and psychological measures. Selecting an outcome evaluation Outcome evaluations generally are of two major types: single-group and quasi-experimental designs. • Single-group designs are the simplest and most common types of evaluation. They are less intrusive and costly and require far less effort to complete than the more ambitious methods we will 166 HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE MENTORING TOOL See forms to track program activity and participant feedback in Sections V and VI on the CD.
  • 7. describe. An example of a single-group evaluation is a questionnaire administered to participants at the completion of the program (post-test only) or before and after the program (pre-test/post-test). • Quasi-experimental designs help evaluators identify whether the program actually causes a change in program participants, using controls to eliminate possible biases. An example of a quasi-experimental design is a pre-test administered at the beginning of a program and a post-test at the completion of the program to both the target mentoring group and a matched comparison group that does not receive mentoring. Single-group designs • Post-test only a. Programs commonly use this design to determine how mentees are doing at the end of a mentoring program. Post-test evaluations can help determine whether the mentees have achieved certain goals (e.g., not dropping out of school) that match the program’s implicit or explicit goals. b. Such evaluations also help discover whether mentors are satisfied with the program. c. Such an evaluation cannot indicate whether the participant has changed during the program, only how the participant is functioning at the end of the program. • Pre-test/post-test designs a. Programs use this design when they want to determine whether mentees actually improved while they were in the program. With this type of evaluation, program staff survey how each participant is doing at the time of enrollment in the mentoring program and then after completion of the program (e.g., 6 or 12 months after the pre-test). By comparing the results of the pre-test and post-test, staff can see whether the mentee improved. b. This evaluation cannot indicate whether the program caused the improvement. Many viable alternative interpretations could explain the change, including these: • Maturation—natural change that occurred simply as a result of the passage of time; and • History—events that occurred between the time the participants took the pre-test and the post-test. c. Other problems with interpreting findings from this design include the following: • Self-selection—The experimental group might differ from the comparison group in some systematic way. For example, quite possibly only the mentees who benefited most remained in the program long enough to take the post-test. • Regression to the mean—A mentee who is functioning extremely poorly at the program’s onset might improve naturally over time. Mentees might enlist in programs when they are most distressed and then naturally return to a higher level of functioning as time passes. d. Even if one cannot identify the cause of a mentee’s improvement, a pre-test design can be useful in other ways: • The evaluator can look at differences within the group. For instance, do youth who receive more frequent or enduring mentoring benefit most? • The evaluator can determine whether certain mentee characteristics are related to achieving program goals. For instance, do boys benefit more than girls? Do minorities in same-race matches benefit more than those in cross-race matches? SECTION VII 167HOW TO ESTABLISH EVALUATION CRITERIA AND METHODS MENTORING TOOL See pre- and post-tests for mentors and mentees in Section VII on the CD. See MENTOR’s Research Corner for the article on “Mentoring and Race” by Dr. Rhodes: www.mentoring.org/ research_corner
  • 8. Quasi-experimental designs Despite their potential benefits, single-design evalua- tions seldom help evaluators identify whether the pro- gram is the cause of change in program participants. To determine that, one needs to conduct evaluations of slightly greater complexity. Such designs are called quasi-experimental because, if carefully planned, they can control for many of the biases described above. This kind of evaluation comes in a variety of types, such as time-series. We will focus on one common type of pro- gram evaluation: one that uses a comparison group. Comparison group designs • The most direct way to rule out alternative explanations is to observe additional youth who have not been part of the program but are similar in other ways to the program youth. By including a comparison group, evaluators can isolate the effects of the program from the effects of other plausible interpretations of change. • A comparison group design also helps put in perspective modest improvements or unexpected declines. Take, for example, the landmark evaluation of Big Brothers Big Sisters of America’s mentoring program.7 Although youth in both the mentored and control groups showed increases in academic, social-emotional, behavioral and relationship problems over the period of time being studied, the problems of the mentored group increased at a slower rate. • One vexing problem with comparison group studies is finding a comparison group that is sufficiently similar to the mentored group. Parents who seek out mentoring programs for their children may devote more attention to their kids at home than do parents of youth who are not mentored. Similarly, young people who willingly enlist in a mentoring program may differ (in terms of motivation, compliance, etc.) from those who do not enlist. The Big Brother Big Sisters study got around this potential problem by selecting both groups from the organization’s waiting list. Unfortunately, many programs either do not keep a waiting list or are not willing to deliberately withhold their program from eligible and motivated participants. The Bottom Line People in the mentoring field tend to believe implicitly that mentoring benefits young people and that, there- fore, expensive evaluations are an unnecessary drain on precious resources. Given the choice of spending money on evaluation or extending their services, many mentoring programs will gladly choose the latter. Although understandable, such choices may be short- sighted. We should not necessarily assume that all mentoring programs are equally beneficial—and we still have a lot to learn about the many newer types of mentoring programs (e.g., site-based, group, peer, e- mentoring). Convincing evaluations are needed to assess the effectiveness of both traditional one-to-one mentoring programs and newer approaches. Such work will play an important role in the expansion of high-quality mentoring programs. 168 HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE
  • 9. SECTION VII 169HOW TO ESTABLISH EVALUATION CRITERIA AND METHODS As you work to ensure program quality and effective- ness as outlined in the Elements of Effective Practice, use the checklist below to gauge your progress. Checking off the items on this list indicates that you are putting the proper components in place to grow a quality, sus- tainable program. If your program is already well established, you can use the checklist to gauge the soundness of your current policies, procedures and organizational structure. Note: The design, focus and structure of your program may mean that some of these components will not be applicable or will need to be modified to match your spe- cific program structure. 1. Develop a Plan to Measure Program Process 2. Develop a Plan to Measure Expected Outcomes Design and implementation of program evaluation ❑ Our program understands the importance of conducting a program evaluation. ❑ We have identified the processes and outcomes that we would like to measure in our evaluation. ❑ We have developed a plan to measure program process. ❑ We have selected indicators of program implementation viability and volunteer fidelity, such as training hours, meeting frequency and relationship duration. ❑ We have developed a system for collecting and managing specific data. ❑ We have specified expected outcomes. ❑ We have selected appropriate instruments to measure outcomes, such as questionnaires, surveys and interviews. ❑ Our program has carefully considered whether to use an outside evaluator or our staff. ❑ We have selected and implemented an evaluation design. ❑ We have established a timeline for conducting the evaluation. ❑ Our evaluation is being implemented and we are collecting and analyzing evaluation data. 3. Create a Process to Reflect on and Disseminate Evaluation Findings Use of evaluation data for program enhancement ❑ Our program uses evaluation results to improve our internal systems and procedures. ❑ Our program uses evaluation results to improve and enhance the desired outcomes for youth. ❑ We use evaluation results in marketing the program to prospective volunteers and community partners. ❑ We use evaluation results to increase the funding and sustainability of the program. ❑ Our program interprets and uses our evaluation results honestly. ❑ We have refined the program design and operations based on the findings. ❑ We developed and delivered reports to program constituents, funders and the media at least annually. Checklist of Program Progress: PROGRAM EVALUATION Adapted from Checklist of Program Progress, Oregon Mentors, Youth Mentoring: A Primer for Funders, The Connecticut Mentoring Partnership and Elements of Effective Practice, Second Edition, MENTOR/National Mentoring Partnership.
  • 10. SECTION VII 171HOW TO ESTABLISH EVALUATION CRITERIA AND METHODS Additional Resources Program Outcomes Evaluation • Analyzing Outcome Information, Harry Hatry, Jake Cowan and Michael Hendricks, 2003, The Urban Institute www.urban.org • An Evaluation Study of Mentoring Programs in Connecticut, The Connecticut Mentoring Partnership, 2003 Entire Report: www.preventionworksct.org/pdf/FINALREPORT.pdf Executive Summary: www.preventionworksct.org/pdf/ExecSumm.pdf • Connections Newsletter on Evaluation, Vol. 5, Issue 3, Summer 2003, Friends for Youth Mentoring Institute www.homestead.com/prosites-ffy/resourcesinfo.html • Evaluating Your Program: A Beginner’s Self Evaluation Workbook for Mentoring Programs, Information Technology International and Pacific Institute for Research and Evaluation, 2000) www.itiincorporated.com/sew_dl.htm • Focus Groups: A Practical Guide for Applied Research, third edition, Richard A. Krueger and Mary Anne Casey, 2000 www.sagepub.com/book.aspx?pid=3591 • Getting to Outcomes 2004: Promoting Accountability Through Methods and Tools for Planning, Implementation, and Evaluation, Matthew Chinman, Pamela Imm and Abraham Wandersman, 2004 www.rand.org/publications/TR/TR101/ • Handbook of Youth Mentoring, the SAGE Program on Applied Developmental Science, edited by David L. DuBois and Michael J. Karcher, 2005 i. Program Evaluation chapter, Jean Grossman ii Assessment of Mentoring Relationships chapter, John Harris and Mike Nakkula www.mentoring.org/youth_mentoring_handbook • Juvenile Justice Evaluation Center Online, Office of Juvenile Justice and Delinquency Prevention and Justice Research and Statistics Association www.jrsa.org/jjec • Key Steps in Outcome Management, Harry P. Hatry and Linda M. Lampkin, The Urban Institute, 2003 www.urban.org • “Learning from Logic Models in Out-of-School Time,” Harvard Family Research Project, 2000 www.gse.harvard.edu/hfrp/projects/afterschool/resources/learning_logic_models.html • Measuring Program Outcomes: A Practical Approach, United Way of America, 1996 http://national.unitedway.org/outcomes/resources/mpo • Online Outcome Measurement Resource Network, United Way of America, http://national.unitedway.org/outcomes/resources/What/OM_What.cfm
  • 11. 172 HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE • Outcome Measurement: What and Why—An Overview, United Way of America, 2002 http://national.unitedway.org/outcomes/files/TPsOMWhatandWhy.pdf • Performance Measures in Out-of-School Time Evaluation, outlines the academic, youth development, and prevention performance measures currently used by out of school time (OST) programs to assess their progress and the corresponding data sources for these measures. Out-of-School Time Evaluation Snapshots series of Harvard Family Research Project www.gse.harvard.edu/hfrp/projects/afterschool/resources/snapshot3.html • Research Corner from Dr. Jean Rhodes, Mentoring.org www.mentoring.org/research_corner • Surveying Clients About Outcomes, Martin D. Abravanel, The Urban Institute, 2003 www.urban.org • What’s Working: Tools for Evaluating Your Mentoring Program, Search Institute, 2001 www.mentoring.org/whats_working • W. K. Kellogg Foundation Evaluation Handbook, 1998 www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf
  • 12. Program Evaluation • How to Select a Survey to Assess Your Adult-Youth Mentoring Program*...........................................173 • Mentor Program Evaluations (Program Coordinator, Mentor, Mentee) • Teacher Report on the Match • Pre-Post Teacher Survey • Gauging the Effectiveness of Youth Mentoring Questionnaire . • Youth Pre-Program Survey • Youth Follow-Up Survey • Mentor Evaluation Form (Mentee Impact) • Mentor Evaluation Form (Mentor Impact) • Mentor Program: Parent Survey • Logic Model*....................................177 SECTION VII TOOLS ON CD Note: The reliability/validity of some of the sample evaluation tools contained herein is unknown. * Select tools denoted with an asterisk also appear in the print version of the tool kit.
  • 13. PROGRAMEVALUATIONTOOLS
  • 14. HOW TO SELECT A SURVEY TO ASSESS YOUR ADULT–YOUTH MENTORING PROGRAM1 By John Harris, Applied Research Consulting and Michael Nakkula, Harvard Graduate School of Education The assessment of mentoring relationship quality (MRQ) is fundamentally important to your mentoring pro- gram. In addition to helping you demonstrate the efficacy of your services, assessments of MRQ can help you identify and maintain best practices for the youth you serve and the mentors you support. Timely and appropri- ate assessment can inform match supervision and ongoing mentor training, assist with the detection of problems in a match or simply provide evidence of success to funders and mentors (who frequently fail to appreciate the difference they make). Effective use of assessments may facilitate the development and maintenance of more durable and high-quality matches. Match advisors in many programs conduct regular check-ins with participants to informally assess MRQ, and this personal supervision is critical to the maintenance of successful matches. However, a survey can be a useful addition to such check-ins (e.g., to satisfy a formal evaluation requirement). It also may be integrated into pro- gramming processes in ways that augment match supervision. To be a useful addition, a survey must generate (at a minimum) meaningful, accurate data that touches on important aspects of the match, such as closeness or instrumentality (the degree to which a match fosters growth for the served youth). To yield more meaningful insight, a survey should assess a broader array of perspectives on MRQ. If you want to integrate a survey more fully into your program’s processes, you should choose a survey that conforms particularly closely to your pro- gram’s goals and assesses the broadest variety of perspectives on MRQ. So, what should you look for in a survey that measures MRQ? First and foremost, it should be supported by scientific proof of its usefulness or validity evidence—evidence that it really measures what it says it measures. The best test of this criterion is whether an instrument has been incorporated into a study that was published in a peer-reviewed journal. Only a handful of existing instruments meet this criterion, and we have provided brief notes about them below. A survey can have strong validity evidence without being published, but if you consider an unpublished instrument, you will need to contact the author to find out about its validity evidence. The fact that a survey is used widely does not mean it was designed with sufficient scientific rigor. If an instrument has sufficient validity evidence, you need to determine whether it assesses a useful range of MRQ indicators and whether the ones it assesses are important to your program. Existing research and our own experience have convinced us that to fully understand MRQ in a given relationship it is important to consider three categories of indicators: those that pertain only to what goes on between a mentor and a child, including relational/experiential indicators (e.g., compatibility, closeness); instrumental/goal-oriented indicators (e.g., degree of focus on personal and academic growth, satisfaction with received support); and external, environmental indi- cators (e.g., programmatic influence, parental influence). Surveys can assess these indicators from a variety of per- spectives: subjective indicators that reflect how participants feel about their match; objective indicators that reflect actual match activities; positive reflections of MRQ (e.g., youth is satisfied with the match); or negative reflections of MRQ (e.g., youth is dissatisfied). Finally, the survey you choose should feel useful to you. It should ask questions that seem important to you and match your program’s mentoring model (e.g., community-based, school-based), its goals (e.g., academically 173HOW TO SELECT A SURVEY TO ASSESS YOUR ADULT–YOUTH MENTORING PROGRAM 1 Note: This is a synopsis (with some verbatim passages) of sections from Nakkula, M. J., & Harris, J. T. (in press). Assessment of Mentoring Relationships. In DuBois, D. L., & Karcher, M. J. (Eds.), Handbook of Youth Mentoring (pp. 100–117). Thousand Oaks, CA: Sage Publications Space limitations preclude a more in-depth consideration of some points, but these are covered in detail within the chapter.
  • 15. focused, career focused or purely relationship focused) and its constituents (e.g., age, gender, and literacy level). Other things to consider include the survey’s use of clear and age-appropriate language, the amount of time need- ed to administer it and the amount of insight it yields after it has been administered. NOTES ON INSTRUMENTS WITH READILY AVAILABLE VALIDITY EVIDENCE The following surveys are among those with the strongest available validity evidence. We provide only a few notes about each to help you begin your consideration of which survey to use. If you would like more information about any of them, you can read about them in the cited articles or contact the authors directly. Also, each is reviewed in detail in the chapter of the Handbook of Youth Mentoring cited above. Youth–Mentor Relationship Questionnaire (YMRQ; Roffman et al.)2 • Designed for primary- and secondary school students (ages 9–16) (15 items in 4 subscales). • Strengths: validity evidence published in peer-reviewed journal; correlates with length of match and academic performance; derived from sample of items used in Public/Private Ventures’ landmark study of mentoring (Grossman & Tierney, 1998). • Limitations: negativity tendency among the survey’s items may limit its usefulness. • Scope: assesses positive and negative subjective perspectives on relational–experiential and instrumental indicators; does not measure objective or environmental dimensions. The Youth Survey (Public/Private Ventures, 2002)3 • Designed for primary and secondary school students (ages 9–16) (19 items in 3 subscales). • Strengths: derived from the same sample of items as the YMRQ; comes closest to offering standardized norms. • Limitations: no published information about validation efforts or reliability of subscales. • Scope: measures positive and negative subjective aspects of relational–experiential dimensions of the match; does not assess objective, instrumental or environmental dimensions. Match Characteristics Questionnaire v2.0 (Harris & Nakkula, 2003a)4 • Designed for mentors of primary and secondary school students (62 items, 15 subscales). • Strengths: validity evidence of earlier version (v1.1) published in a peer-reviewed journal;5 is completed by mentors; broad scope; has been successfully integrated into match supervision processes at the Yavapai (Arizona) Big Brothers Big Sisters agency; correlates with academic outcomes. • Limitations: validity evidence supporting version 2.0 not yet published. • Scope: assesses positive, negative, subjective and objective perspectives on relational–experiential, instrumental and environmental indicators. Youth Mentoring Survey (Harris & Nakkula, 2003b)6 • Designed for mentors of primary and secondary school students (45 items, 9 subscales). • Strengths: broad scope; complements, correlates with Match Characteristics Questionnaire; has been successfully integrated into match supervision processes at the Yavapai Big Brothers Big Sisters agency; correlates with academic outcomes. • Limitations: validity evidence not yet published. • Scope: assesses positive and negative, subjective and objective, relational–experiential and instrumental dimensions of MRQ; does not assess environmental indicators. Relational Health Indices–Mentoring Scale (RHI-M) (Liang et al., 2002)7 • Designed for female college students (11 items in 3 subscales).8 • Strengths: validity evidence published in peer-reviewed journal; unique theoretical perspective; provides an assessment of natural mentoring relationships. 174 HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE
  • 16. • Limitations: difficult to generalize findings from study involving female college students at liberal arts women’s college. • Scope: assesses subjective relational–experiential dimensions with some items related to instrumentality; does not measure negative, objective or environmental dimensions. Unnamed Mentoring Scale (Darling et al., 2002)9 • Designed for college students (4 items in 1 subscale).10 • Strengths: validity evidence published in peer-reviewed journal; demonstrated to be useful in two diverse cultures (U.S./Japan); provides an assessment of natural mentoring relationships. • Limitations: narrow scope; use of dichotomous (yes or no) ratings. • Scope: assesses subjective ratings of instrumentality; does not measure negative, objective, relational–experiential or environmental dimensions. Instruments other than those reviewed above could be applied to MRQ assessment, but they lack sufficient validity evidence to support their widespread use. For instance, Information Technology International (Mertinko et al., 2000)11 and Big Brothers Big Sisters of America (Lyons & Curtis, 1998)12 have developed brief youth and adult instruments that assess elements of relationship quality but are not supported by reliability and validity evidence. A handful of researchers have developed qualitative designs to augment or complement their quantitative work. DuBois et al. (2002)13 and Keller, Pryce and Neugebauer (2003)14 have made important contributions that could inform your decisions about qualitative data collection. SUMMARY Given the free and easily accessible nature of the instruments described here, it may not be necessary to use all of the subscales of specific instruments or even to use only one instrument. While longer instruments that assess more constructs can generate more complete insight on relationship quality, this comprehensiveness may come at a cost. Both youth and adults can become bored or frustrated by scales if they are too long, particularly if they require multiple administrations or appear to contain undue overlap between items in the subscales. Because the utility of MRQ assessments may be greatest when incorporated into regular programming infrastructure, it is important to encourage participants’ buy-in. In such cases, participants should be made aware at the outset that they will be asked to complete surveys regularly and should be helped to understand why this process is important. You will want to think carefully about when you administer the surveys. Although baseline data are prized in program evaluation, it does not make sense to assess match quality before a relationship has had a chance to develop. We believe it is most advantageous to administer MRQ assessments after the match has been meeting regularly for about four months, to allow the match to progress beyond the initial awkwardness or honeymoon stage. The interval between the initial and follow-up assessments should likewise allow sufficient time for the relationship to evolve, likely about six months for the second administration and another six months for the third. Thus, a typical administration schedule might be 4, 10 and 16 months after the match is made. For matches that are still meeting after 18 months, a longer interval is likely to suffice. Finally, survey instruments such as those described here may be easily administered but require the summation and interpretation of scores, which will be enhanced by the involvement of trained researchers/evaluators. Such external support for analysis ensures accuracy and lends credibility to interpretations of the data. While professional evaluation support can be difficult for programs to afford, partnership with external evaluators is vital to ensure that the interpretations upon which programming decisions and funding may be based have been drawn accurately and responsibly from the data. 175HOW TO SELECT A SURVEY TO ASSESS YOUR ADULT–YOUTH MENTORING PROGRAM
  • 17. 2 Roffman, J., Ranjini, R., Rhodes, J., & Grossman, J. B. (in press). Promoting successful youth mentoring relationships: A preliminary screening questionnaire. Journal of Primary Prevention. 3 Public/Private Ventures. (2002). Technical assistance packet #8: Measuring the quality of mentor-youth relationships: A tool for mentoring programs. Portland, OR: Northwest Regional Educational Laboratory. Retrieved April 23, 2003, from http://www.nwrel.org/mentoring/packets.html. 4 Harris, J. T., & Nakkula, M. J. (2003a). Match Characteristics Questionnaire v2.0. Unpublished measure, Harvard Graduate School of Education, Cambridge, MA. Findings associated with the previous version (v1.1) may be found in Karcher, M. J., Nakkula, M. J., & Harris, J. T. (in press). Developmental mentoring match characteristics: The effects of mentors’ efficacy and mentees’ emotional support seeking on the perceived quality of mentoring relationships. Journal of Primary Prevention. 5 Karcher, M. J., Nakkula, M. J., & Harris, J. T. (in press). Developmental mentoring match characteristics: The effects of mentors’ efficacy and mentees’ emotional support seeking on the perceived quality of mentoring relationships. Journal of Primary Prevention. 6 Harris, J. T., & Nakkula, M. J. (2003b). Youth Mentoring Survey. Unpublished measure, Harvard Graduate School of Education, Cambridge, MA. 7 Liang, B., Tracy, A., Taylor, C. A., Williams, L. M., Jordan, J. V., & Miller, J. B. (2002). The relational health indices: A study of women’s relationships. Psychology of Women Quarterly, 26, 25–35. 8 The RHI has been adapted for school-aged students, though no validity evidence for that instrument is yet available. 9 Darling, N., Hamilton, S., Toyokawa, T., & Matsuda, S. (2002). Naturally occurring mentoring in Japan and the United States: Roles and correlates. American Journal of Community Psychology, 30(2), 245–270. 10 The RHI has been adapted for school-aged students, though no validity evidence for that instrument is yet available. 11 Mertinko, E., Novotney, L., Baker, T., & Lang, J. (2000). Evaluating your program: A beginner’s self-evaluation workbook for mentoring programs. Washington, DC: Office of Juvenile Justice and Delinquency Prevention. 12 Lyons, M., & Curtis, T. (1998). Program Outcome-Based Evaluation (POE). Philadelphia, PA: BBBSA. 13 DuBois, D. L., Neville, H. A., Parra, G. R., & Pugh-Lilly, A. O. (2002). Testing a new model of mentoring. New Directions for Youth Development, 93(Spring), 21–57. 14 Keller, T.E., Pryce, J.M., & Neugebauer, A. (2003) Observational methods for assessing the nature and course of mentor-child interactions. Unpublished manual, University of Chicago. 176 HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE
  • 18. EARLYADOLESCENT URBAN,MINORITYGIRLS •Lowself-esteem •Depression •Victimization •Healthriskbehaviors: Diet/nutrition,exercise, substanceuse,violence,risky sexualbehavior,self-harm •Academicunderachievement PREVENTIONPROGRAMS FORGIRLS •Lackofeffectiveness •Lackofgender-specific strategiesandcontent FINANCIAL •NIMHGrantFunding •BBBSSubcontract PERSONNEL •2.20FTEBBBSStaff •Servicesofcommunity agencies(workshop presentations) •10BBBSfemalevolunteer mentors •Consultation:UICResearch Team MATERIALS •Programmanual •Supplies(participanthandouts andnotebooks,disposable cameras,picturepuzzles, refreshments,workshop props) FACILITIES •Spaceforworkshopsandgoal- settingandprogresssessions SUPPORTS •Stafftrainingandsupervision •Mentortraining •Bi-monthlysupervisionof mentorsandparent/youth check-ins DIRECTSERVICES •Bi-monthlyworkshopseries formentors/youth:focused onrelationshipandteam building,promotion ofhealthyself-esteem, preventionofriskbehaviors/ promotionofhealthy behaviors(11workshopstotal) •Goal-settingandprogress sessionsforindividual matches •Between-sessionstructured activitiesformatchesduring workshopseries(Power Builders) •Continuedone-on-one interactionsbetweenmentors andyouthfollowing workshopseriesto1-year mark(includesPower Builders) •Groupreunionsession EVALUATION •Built-inprogramevaluation activities FIDELITY •Implementationoftraining sessionsforstaffandmentors •Implementationofworkshops &reunion •Qualityofimplementationof trainingsessions,workshops, supervision/check-ins,goal- setting,andprogresssessions •Mentor/staffsatisfactionwith training •Mentor/parent/youth satisfactionwithworkshops, supervision/check-ins,goal- settingsessions,program materials •Youth/mentor/parent satisfactionwithmentoring relationship DOSAGE •Avg.#ofworkshopsessions attendedbymentorsand youth •Parentattendanceat orientation&talent show/graduation •Avg.#ofsupervision contacts/check-insfor mentors/parents/youth •Avg.#ofgoal-settingand progresssessionscompleted •Avg.hoursofweeklyone-one- onementor/youthinteractions •Avg.#ofPowerBuilders completed •%ofevaluationmaterials completedbystaff/mentors/ youth/parents •%ofrelationshipssustained oneyear INITIAL •jjsocialsupportfrom non-parentaladult(mentor): emotional,companionship, instrumental,informational •jjhealth-related knowledge/attitudes •jjgenderandracialidentity INTERMEDIATE •jjself-esteem/self-efficacy beliefs •jjsocialcompetence •jjskillsforavoidingrisky behaviors/engaginginpositive healthbehaviors •jjqualityofrelationshipswith parents,peers,andother adults LONGTERM •llriskyhealthbehaviors: substanceuse,violence- related,unsafesexual behavior,self-harm,etc. •jjpositivehealthbehaviors: exercise,diet/nutrition,etc. •llmentalhealthproblems: internalizing(e.g.,depression) andexternalizing(e.g., conductdisorder) •jjpositivementalhealth: happiness,lifesatisfaction •jjsocial,educational, occupationalfunctioningat laterstagesofdevelopment LogicModelforGirlPOWER!* NeedsProgramProgramProgramProgram InputsActivitiesOutputsOutcomes *ThisprogramwasdevelopedthroughcollaborationbetweenBig BrothersBigSistersofMetropolitanChicago(BBBS)andthe GirlsMentoringProjectatUniversityofIllinoisatChicago(UIC), DavidDuBois,Ph.D.,Director.FTE=full-timeequivalent. Whatneedsdoesthe programaddress? Whatgoesintotheprogram?Whatgoesonintheprogram?Whathappensasaresultof theprogram? Whatarethebenefitsof participatingintheprogram? 177LOGIC MODEL FOR GIRLPOWER!
  • 19. 179 NOTES
  • 20. 180 NOTES
  • 21. SUMMARY Once again, congratulations on your commitment to quality youth mentoring. Regardless of whether you reviewed the tool kit in its entirety or reviewed only those sections that were relevant to your specific needs, we hope that you found it useful and that it included all the information and resources you need to follow the guidelines in the Elements of Effective Practice. We encourage you to refer to the tool kit often as you continue to build and strengthen your mentoring program. WE NEED YOUR FEEDBACK To ensure that the tool kit meets your needs, we are seeking your feedback on its content and suggestions for improvement. Your feedback will enable us to enhance the online version of the tool kit (www.mentoring. org/eeptoolkit) and to respond to emerging mentoring trends. Please complete the Tool Kit Evaluation Form that follows, and mail or fax it to: MENTOR/National Mentoring Partnership Attention: National Mentoring Institute 1600 Duke Street, Suite 300 Alexandria, VA 22314 Fax: 703-226-2581 Thank you in advance for your input. We invite you to visit Mentoring.org often for the latest in mentoring news, information and resources. Together, we can connect more of America’s young people with caring adult mentors. 181WRAP-UP AND FEEDBACK Section VIII. Wrap-Up and Feedback
  • 22. SECTION VIII 183EVALUATION FORM What materials in the tool kit did you find most useful for your mentoring program? _________________________________________________________________________________ _________________________________________________________________________________ What materials in the tool kit did you find least useful for your mentoring program? _________________________________________________________________________________ _________________________________________________________________________________ Please rate the following sections: Section III: Introduction to Mentoring and Program Building Narrative ❑ Excellent ❑ Good ❑ Fair Section IV: How to Design and Plan a Mentoring Program Narrative ❑ Excellent ❑ Good ❑ Fair List of Additional Resources ❑ Excellent ❑ Good ❑ Fair Checklist of Program Progress ❑ Excellent ❑ Good ❑ Fair Tools ❑ Excellent ❑ Good ❑ Fair What tools in Program Design and Planning did you find most useful? _________________________________________________________________________________ _________________________________________________________________________________ What tools in Program Design and Planning did you find least useful? _________________________________________________________________________________ _________________________________________________________________________________ How to Build a Successful Mentoring Program Using the Elements of Effective Practice EVALUATION FORM
  • 23. Section V: How to Manage a Program for Success Narrative ❑ Excellent ❑ Good ❑ Fair List of Additional Resources ❑ Excellent ❑ Good ❑ Fair Checklist of Program Progress ❑ Excellent ❑ Good ❑ Fair Tools ❑ Excellent ❑ Good ❑ Fair What tools in Program Management did you find most useful? _________________________________________________________________________________ _________________________________________________________________________________ What tools in Program Management did you find least useful? _________________________________________________________________________________ _________________________________________________________________________________ Section VI: How to Structure Effective Program Operations Narrative ❑ Excellent ❑ Good ❑ Fair List of Additional Resources ❑ Excellent ❑ Good ❑ Fair Checklist of Program Progress ❑ Excellent ❑ Good ❑ Fair Tools ❑ Excellent ❑ Good ❑ Fair What tools in Program Operations did you find most useful? _________________________________________________________________________________ _________________________________________________________________________________ What tools in Program Operations did you find least useful? _________________________________________________________________________________ _________________________________________________________________________________ 184 HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE
  • 24. SECTION VIII Section VII: How to Establish Evaluation Criteria and Methods Narrative ❑ Excellent ❑ Good ❑ Fair List of Additional Resources ❑ Excellent ❑ Good ❑ Fair Checklist of Program Progress ❑ Excellent ❑ Good ❑ Fair Tools ❑ Excellent ❑ Good ❑ Fair What tools in Evaluation Criteria and Methods did you find most useful? _________________________________________________________________________________ _________________________________________________________________________________ What tools in Evaluation Criteria and Methods did you find least useful? _________________________________________________________________________________ _________________________________________________________________________________ Feedback Please provide your comments on the overall content of the tool kit and any suggestions for additional information or improvement. Please send the completed Tool Kit Evaluation Form by mail or fax to: MENTOR/National Mentoring Partnership Attention: National Mentoring Institute 1600 Duke Street, Suite 300 Alexandria, VA 22314 Fax: 703-226-2581 185EVALUATION FORM
  • 25. 187ENDNOTES Section II 1. Elements of Effective Practice, second edition, MENTOR/National Mentoring Partnership, 2003. Section IV 1. Some information is adapted from Business Guide to Youth Mentoring, The Connecticut Mentoring Partnership. Section V 1. Document Kit, Friends for Youth, Mentoring Institute, 2003. 2. Dr. Susan G. Weinberger, president, Mentor Consulting Group, 2004. 3. Partnerships for Success: A Mentoring Program Manual, The Enterprise Foundation and the United Way of America, 1990. 4. Partnerships for Success, 1990. 5. Partnerships for Success, 1990. 6. Adapted from “Getting your Message Heard: Understanding Nonprofit Marketing,” MENTOR/National Mentoring Partnership Online Community Forum, November 2003, Mentoring.org/community. 7. The Community Collaboration Manual, The National Assembly of National Voluntary Health and Social Welfare Organizations, 1991. Section VI 1. “Mentoring: A Promising Strategy for Youth Development,” Susan M. Jekielek, M.A., Kristin A. Moore, Ph.D., Elizabeth C. Hair, Ph.D., and Harriet J. Scarupa, M.S., Child Trends Research Brief, Washington, D.C., February 2002, p. 6. 2. Allstate Insurance Co. Mentoring Program, Dr. Susan G. Weinberger, Norwalk, CT: Mentor Consulting Group, 2000. 3. “Mentoring in America 2002,” MENTOR/AOL Time Warner Foundation, Mentoring.org/poll. 4. Partnerships for Success: A Mentoring Program Manual, The Enterprise Foundation and the United Way of America, 1990. 5. “Mentoring in America 2002.” 6. Mentoring: A Guide for Local Broadcasters, National Association of Broadcasters and Harvard Mentoring Project, 2002. ENDNOTES
  • 26. 7. “Mentoring Matters: A National Survey of Adults Mentoring Young People,” The Commonwealth Fund, K.T. McLearn, D. Colasanto and C. Schoen, in J.B. Grossman (ed.), Contemporary Issues in Mentoring, Philadelphia: Public/Private Ventures, 1998, pp. 67–83. 8. Connections Newsletter, Friends for Youth Mentoring Institute, Fall 2003. www.friendsforyouth.org 9. Adapted from Vision to Reality: Mentoring Program Development Guide, The Maryland Mentoring Partnership, 2003. 10. Mentor screening information compiled from The ABCs of Mentoring, Mentoring Partnership of Long Island, 2003 and adapted from Mentoring Manual: A Guide to Program Development and Implementation, The Abell Foundation, Inc., Baltimore, MD. 11. Partnerships for Success, 1990. 12. Vision to Reality, 2003. 13. Adapted in part from Mentoring in the Faith Community: An Operations Manual for Program Coordinators, MENTOR/National Mentoring Partnership and The Mentoring Partnership of New York, 2002. Section VII 1. Dr. David DuBois, Institute for Health Research and Policy, University of Illinois at Chicago, 2004. 2. Partnerships for Success, 1990. 3. Substantial portions of these sections were adapted from Program Evaluation: Methods and Case Studies, E.J. Posavac & Carey, Englewood Cliffs, NJ: Prentice-Hall; and “Assessing the Effectiveness of Mentoring Programs,” J.B. Grossman and A. Johnson, in J.B. Grossman (ed.), Contemporary Issues in Mentoring. Philadelphia: Public/Private Ventures, 1998. 4. Grossman and Johnson, 1998. 5. “Effectiveness of Mentoring Programs for Youth: A Meta-Analytic Review,” D.L. DuBois, B.E. Holloway, J.C. Valentine, and H. Cooper, American Journal of Community Psychology, 30 (2002), pp. 157–197. 6. “Toward Predicting Successful Youth Mentoring Relationships: A Preliminary Screening Questionnaire,” J. Roffman, R. Reddy, and J. Rhodes, 2002. (Submitted for publication.) 7. Making a Difference: An Impact Study of Big Brothers/Big Sisters (reissue of 1995 study), Joseph P. Tierney and Jean Baldwin Grossman, with Nancy L. Resch, 2000. A publication of Public/Private Ventures. 188 HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE