Your SlideShare is downloading. ×
Empowering youth to be evaluators: Involving Young People in Evaluating Informal Education Programs  - Callahan Schreiber Article
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Empowering youth to be evaluators: Involving Young People in Evaluating Informal Education Programs - Callahan Schreiber Article


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Participatory evaluation: A case study of involving stakeholders in the evaluation process by Amy Grack Nelson & Robby Callahan Schreiber Citation: Grack Nelson, A. & Callahan Schreiber, R. (2009). Participatory evaluation: A case study of involving stakeholders in the evaluation process. Visitor Studies, 12(2), 199 – 213. Abstract: One way to ensure that an evaluation has “utility” is to use a participatory evaluation approach where the evaluator partners with primary users to carry out various phases of an evaluation. This article provides a case study of how participatory evaluation was used in an out-of-school youth development and employment program at the Science Museum of Minnesota's Kitty Andersen Youth Science Center. Youth staff participated in a series of evaluation workshops where they learned about evaluation, made their own meaning of evaluation data, generated recommendations to improve their museum work, and ultimately took more ownership and control of their program. As a result of the workshops, youth also carried out their own formative evaluation of four museum programs. The program manager and the youth staff reaped a number of benefits from this process. The factors leading to its success are discussed. Keywords: participatory evaluation, formative evaluation, program evaluation, youth development, evaluation capacity building, museum According to The Joint Committee on Standards for Educational Evaluation's Program Evaluation Standards (1994), one of the four essential features of all evaluations is "utility." The Joint Committee's rationale for the utility standard was, "An evaluation should not be done if there is no prospect for its being useful to some audience" (Stufflebeam, 1980, pp. 90). One way to help ensure use is to increase the primary intended users’ level of participation in the evaluation (Cousins & Earl, 1995a; Patton, 2008). A means to do this is to undertake evaluation using a participatory evaluation process, which has the core purpose of increasing evaluation use (Cousins & Earl, 1995a). Participatory evaluation is "applied social research that involves a partnership between trained evaluation personnel and practice-based decision-makers, organizational members with program responsibility, or people with a vital interest in the program" (Cousins & Earl, 1992, pg. 399-400). The process recognizes and draws upon the evaluation expertise of the evaluator and the program expertise of the primary user. The evaluator partners with primary intended users to carry out various phases of the evaluation including constructing evaluation questions, designing instruments, collecting data, analyzing results, generating recommendations, and disseminating findings (Cousins & Earl, 1995a). Involving users in the evaluation enhances the relevance of the evaluation, understanding of the data, and ownership of the findings, all important to increasing use (Cousins & Whitmore, 1998; King & Stevahn, 2002; Patton, 2008). In addition to increasing use, participating in the evaluation provides opportunities for users to develop the analytic and evaluative skills necessary for meaningful participation in the evaluation process (Cousins & Earl, 1992). This article provides a case study of how a participatory evaluation approach was used in an out- of-school youth program at the Science Museum of Minnesota's Kitty Andersen Youth Science Center (KAYSC). In the youth development field, participatory evaluation involving youth is referred to as youth participatory evaluation (Sabo Flores, 2008). Youth participatory evaluation 1
  • 2. actively involves youth in a meaningful way to improve the experience they have in a program, instead of being passive recipients of program services or subjects in an evaluation (Checkoway, Dobbie, & Richards-Schuster, 2003; Horsch, Little, Chase Smith, Goodyear, & Harris, 2002; London, Zimmerman, & Erbstein, 2003). When adults treat youth as equals and acknowledge and respect their voices in the evaluation process, it increases youth self-confidence that they have valuable assets, knowledge and insights (Checkoway et al., 2003; Sabo Flores, 2008). This article discusses the participatory process that was used with a group of youth, the factors that led to a successful process, and the benefits of the process to the program manager and youth. CASE STUDY The KAYSC was created in 1996 to encourage young people to engage in experiential learning that develops their confidence in and appreciation for STEM fields (Science, Technology, Engineering and Mathematics). The KAYSC is built on the belief that youth are active participants in their own learning where discovery, ownership, and responsibility incite a positive attitude toward the sciences. KAYSC programs annually engage approximately 100 youth 12-18 years old in STEM educational and professional opportunities. Youth work as museum volunteers or part-time, paid staff in small teams facilitated by adult program managers. They educate museum visitors, build exhibits, develop community STEM projects/presentations, and do scientific research, which make their experiences real and meaningful (Velure Roholt & Steiner, 2005). Participants are empowered to take an active role in the planning, research, design, implementation, and evaluation of their own program and projects. The Park Crew is one of the senior level teams in the KAYSC. Throughout this article, Park Crew youth staff will be referred to as “participants” while those who participated in the crew’s programs will be referred to as “visitors.” The team is comprised of eight to twelve junior and senior high school students who are led by an adult program manager, Robby Callahan Schreiber. The Park Crew began in 2003 through funding from the National Center for Earth- surface Dynamics (NCED), a National Science Foundation Science and Technology Center. The team provides public outreach activities based on the work of NCED and related topics. As a result of participating in the Park Crew, participants learn about earth-surface processes, specifically those related to water, and the role humans play in affecting the processes; personally develop and strengthen teaching skills so they can educate the public (museum visitors and school outreach groups) about earth-surface processes; and become aware of STEM careers and explore educational and professional opportunities in these fields. The primary activities of the Park Crew are developing and sharing water-based earth-surface process activities with the public. The participants visit after-school program sites during the winter to share activities with elementary students. During the summer, the Park Crew facilitates activities in the museum’s Big Back Yard (BBY), a 1.75-acre outdoor science park. Over 2,000 museum visitors interact with the Park Crew’s activities each summer. The activities include Watershed Walk, Fossil Making, Macroinvertebrate Investigation, and Restore the River. • Watershed Walk is a hands-on demonstration of non-point source pollution that includes helpful suggestions of how visitors can improve local water quality. Watershed Walk is the only activity with a predefined schedule of demonstrations every half hour. • Fossil Making is a make-and-take activity in which visitors learn about local fossils and create their own fossil to take home. 2
  • 3. • Macroinvertebrate Investigation is a free-choice activity where visitors lead their own up- close exploration of aquatic macroinvertebrates using microscopes and hand lenses. • Restore the River is an interactive demonstration of what happens to a dammed river before and after the removal of the dam. During the summer of 2007, Amy Grack Nelson, Evaluation & Research Associate at the Science Museum of Minnesota, carried out a summative evaluation of the Park Crew program1. The purpose of the evaluation was to understand how the participants implemented the activities and what they learned about earth-surface processes, teaching others, and STEM careers. A mixed-methods design was used, which included observations and pre- and post-program interviews of the participants. The evaluator observed two activities with varying presentation styles, Watershed Walk and Macroinvertebrate Investigation. Observations focused on how participants presented the activities and interacted with visitors. Pre-program interviews included questions related to what participants wanted visitors to learn from each of the four activities, while post-program interviews asked participants about their experience working in the BBY and their comfort level presenting the activities. The summative evaluation not only provided insight into the extent to which the program’s goals and outcomes had been addressed, it also brought to light many areas of potential improvement. Since the program was ongoing, the program manager wanted to find ways to address some of the program’s shortcomings and make improvements before the following summer, thus using the summative findings in a formative manner. The evaluator and program manager decided they wanted to share the evaluation data with the participants as a way to help identify areas of improvement. Since the evaluation findings were not entirely positive, it was important to share the data in a way that allowed participants to make their own conclusions instead of having an adult appear to point out what they did wrong. It was also important that the Park Crew's first experience with evaluation data was positive and did not result in participants having negative feelings about their work or evaluation. A situation had to be devised to allow participants to personally identify their successes and take ownership of what improvements needed to be made. For this reason, a participatory evaluation approach was used to share and make sense of the data with the participants. Youth participation in evaluation was not new to the KAYSC. In the past, external evaluators used a participatory process with two KAYSC program evaluations. Unfortunately, a participatory approach did not carry over into other evaluations. Staff turnover, lack of evaluation capacity of the staff, and the absence of an internal evaluator, made it difficult to establish a commitment to participatory evaluation processes across all KAYSC youth programs. In order for participatory evaluation to become integrated and a routine part of an organization's culture, repeated participatory evaluation processes are required (Cousins & Earl, 1995b). Various factors have contributed to a recent commitment to youth participation in evaluation within the KAYSC. In 2005, the Science Museum of Minnesota's Department of Evaluation & Research in Learning was established and an internal evaluator, Amy Grack Nelson, was designated to work closely with the KAYSC. The internal evaluator began to build the evaluation capacity of the KAYSC staff in order to create an organizational culture of evaluation. This work was supported by KAYSC's 2007 strategic plan that stressed a commitment to a collaborative approach to evaluation and the need for evaluation to be a component of all new 3
  • 4. programs. Participatory evaluation also aligns with KAYSC's philosophy of transforming youth from being passive consumers of knowledge to creators and connectors of knowledge, recognizing that youth can and should be in a constant state of learning and improvement, thinking critically about and evaluating their experiences in and outside of the KAYSC. Involving Participants in Interpreting Data and Generating Recommendations Participants were brought into the evaluation process during the data analysis phase of the Park Crew summative evaluation. Since they were not actively involved in the evaluation from the beginning, it was not a full participatory evaluation process, but a youth participatory evaluation approach was used to analyze data and generate recommendations. As a result of the experience, the participants requested to engage in a full participatory evaluation process to formatively evaluate their BBY activities with visitors. In April 2008, both new Park Crew youth and veteran youth from the previous summer (who were involved in the summative evaluation) participated in four evaluation workshops facilitated by their program manager and internal evaluator (See Figure 1). There were four goals of the workshops: 1) Teach participants about evaluation; 2) Give participants an opportunity to make their own meaning of evaluation data; 3) Increase participants’ understanding of BBY activities; and 4) Allow participants to take more ownership and control of their work. Over the course of four three-hour workshops, participants learned about evaluation, reflected on their activities' learning objectives, interpreted the previous summer's evaluation data and created recommendations for the upcoming summer. At the end of each workshop, the program manager led the participants through brief, facilitated discussions to provide an opportunity for them to reflect on their experience participating in the evaluation. Through focused questioning from the program manager, participants talked about parts of the participatory workshops that resonated with them, shared feelings that came up throughout the workshops, connected the process to the bigger picture of their work in the KAYSC, and identified ways to move forward. These discussions allowed participants to continually check-in with each other and the program manager and keep the workshops moving in a direction and speed with which they were comfortable. The success of the participatory workshops became evident when the Park Crew requested to lead their own formative evaluation to improve visitors’ experiences with their activities. 4
  • 5. Figure 1: Evaluation workshop agenda Workshops 1 & 2: Reflecting on Experience and Data The first workshop began with an activity to help the participants gain a deeper understanding of evaluation, why their activities were evaluated, and how the evaluation data were gathered. To explain evaluation in a way that connected to their current experiences and interests, the evaluator led the participants through an activity in which they evaluated a cell phone. The team identified what they wanted teens to be able to do with a cell phone (the objectives). They talked about what information they would need to gather and how to gather it to ensure teens were able to successfully use the phone and were satisfied with the product. The evaluator then explained how they could look at the data and create recommendations to communicate to the manufacturer how to improve the phone. Parallels were then drawn between how participants said they would evaluate a phone and how the evaluator evaluated the Park Crew program. The program manager talked about the goals and objectives of the Park Crew, identified the goals of the evaluation, explained the methods the evaluator used to gather data, and emphasized that the participants were brought into the evaluation process to interpret the data and find ways to improve the work of the Park Crew. During the first two workshops, participants reflected on learning objectives for the activities, discussed what they felt worked well the previous summer and what should be changed, and reviewed the evaluation data. The participants began the workshop by reflecting on what they did for each activity. Next they were asked to list what they wanted visitors to learn from the activity. The evaluator then introduced similar data from the previous summer’s summative evaluation, where participants were asked the same question related to learning objectives. 5
  • 6. Participants added any additional objectives that were mentioned in the summative interviews to their list. This led to a discussion about learning objectives and how activities can be taught to ensure objectives are addressed. After defining the learning objectives for an activity, participants reflected on their experience leading that activity. The participants were asked to answer two questions for each of their activities: • What would you keep and why? (e.g. things they enjoyed, visitors enjoyed, worked well, etc.) • What would you change and why? (e.g. things they did not enjoy, visitors were not interested in, did not work well, etc.) The team worked in groups of three to come up with as many ideas as they could for each activity. They were encouraged to brainstorm without limitations of cost or feasibility. Participants considered all aspects of doing a program including the training they received to learn the activities, their own process of learning the activities, setting up the activities in the BBY, leading the activities with visitors, and cleaning up after the activities. The program manager provided encouragement to participants as they worked through the process and reminded them to elaborate on ideas when needed. When the participants finished coming up with their ideas, they shared them with the larger group. Using a sticky wall, participants placed their ideas under the corresponding keep or change column for the activity (See Figure 2). As a large group, they identified any similarities among ideas and created clusters of things that worked well or things they would change. Figure 2: Photograph of sticky wall with keep and change data for Watershed Walk After participants reflected on the activity, they were presented with evaluation data for that activity. Instead of seeing the interpretations of the data from the summative evaluation report, the participants were presented with tables and graphs of the data. This allowed them to come to their own conclusions about the data without the evaluator’s interpretations influencing their 6
  • 7. thinking. The evaluator started the data discussion by leading the team through a brief overview of reading graphs, sampling, means, and medians. This introduction also provided a learning opportunity for participants to apply and improve their math skills. The participants were then shown the data for each corresponding activity. To help them think about and discuss the data, the following questions were posed: • What are your first impressions of the data? • Is there anything that surprises you? • Based on the data, what are some changes you would suggest? • What would you keep doing the same way? Participants wrote down additional things they wanted to change or keep based on their interpretation of the data and added them to the sticky wall. During the data discussions, the evaluator and program manager did not interpret the data, but allowed each participant to come up with his or her own interpretations. As needed, the evaluator asked probing questions to help participants correctly read a graph, make comparisons between some of the data, and think about why they were interpreting the data in certain ways. Participants’ interpretations of the data were often similar to those of the evaluator in the summative evaluation report. One example was the interpretation of how effectively the content was delivered for the Watershed Walk activity. While observing this activity, evaluators had recorded the total number of times participants mentioned reasons why something was considered a pollutant and what people could do to prevent that particular kind of pollution. When participants looked at graphs of this data, they were surprised at how infrequently they had talked about the reasons and actions for the various pollutants, because they felt they had covered these topics in all of their demonstrations (See Figure 3). One participant said, “We should be talking about these 100% of the time.” The evaluator pointed out that this feeling was also reflected in the interviews from the previous year’s summative evaluation, when two-thirds of the participants had said they wanted visitors to learn about these two topics. Participants had identified that they wanted visitors to learn this information, but when they actually carried out the activity, they did not follow through. This discrepancy led to a discussion about objectives and the disconnect between what they saw as objectives and what they were actually teaching visitors. Participants said in some cases they did not talk about certain pollutants because they either did not know the information or feel confident enough to talk about it. In order to attain these objectives, participants decided they needed to increase their knowledge about why certain things are considered pollutants and what people can do to prevent them. They created cards for the sticky wall that reflected their expectation of what content should be delivered during every Watershed Walk and their desire for more specific content-related training. Participants ended up creating their own expectations for their performance, expectations that the program manager always had of them but now the participants were adopting for themselves. 7
  • 8. Figure 3: How often participants talked about why something is considered a pollutant (n=27) There was one instance where participants interpreted the data quite differently than the evaluator and program manager. For the Watershed Walk activity, participants were expected to present the walk every half hour. Between walks, they cleaned up from the previous walk, set up the next walk, or waited for visitors to stop by. One of the expected outcomes of participating in the Park Crew is for participants to acquire skills and confidence interacting with museum visitors. These skills were measured by observing participant interactions with visitors passing by and/or stopping at the watershed model (the basis for the Watershed Walk activity). Observations revealed that participants were not comfortable with the most basic levels of interaction, greeting visitors and inviting them to participate in the activity. Most of the time (86%), participants said nothing to visitors as they passed by, not even “hello.” When participants were shown these data, they did not interpret it as a potential area of improvement. Instead they talked about their need to set up or clean up for the activity. It did not occur to any of the participants that they should stop for a moment to at least acknowledge the visitor, if not invite her or him back for the Watershed Walk. This was eye opening for the program manger. It became clear that training for the BBY needed to include the same customer service training that other museum staff receive, instead of solely focusing on presenting the four activities. This was one case during the workshops where the program manager added an idea to the sticky wall to make sure the necessity of customer service training did not get lost in the process. Workshop 3: Generating Recommendations Participants spent the third workshop drawing connections from the ideas generated during the first two workshops to their future BBY work. The program manager facilitated this process using a consensus-building workshop technique (Bucki, 2008). The primary goal of the consensus-building workshop was to give the participants a voice in creating recommendations for program improvement. While the evaluator and program manager could have developed recommendations on their own, it was important to include participants’ thoughts and opinions as part of the democratic process of learning and decision-making. The workshop began by 8
  • 9. posing the question, “What do we need to do in order to reach our goals working in the Big Back Yard?” Participants utilized their list of intended objectives for each activity and their keep/change reflections from the first two workshops to answer the question. The participants and program manager individually brainstormed answers to the question and shared their ideas with a partner. The partners chose 8-10 of their best ideas and wrote them down on half sheets of paper. Then they posted several of their ideas on the sticky wall and identified ones with shared meaning. They clustered these into pairs and each became a column, or theme. They placed all their remaining ideas under the column to which each best fit. By the end of the consensus- building workshop, the team had identified and named seven themes of recommendations they wanted to focus on in preparation for their upcoming BBY work: 1) things to change, 2) seeking new information, 3) hands-on field trips, 4) visitor interaction, 5) activity preparation, 6) advertising activities, and 7) space and comfort. 1) Things to change. The team identified a number of large and small changes to each activity that would improve either their ability to teach the activity or the visitor's experience. 2) Seeking new information. The team decided to seek new information on local watershed issues and local fossil history based on where they lacked or desired more content knowledge. 3) Hands-on field trips. They also wanted to increase their content knowledge by going on more field trips. 4) Visitor interaction. In order to improve visitor interactions, participants suggested creating cards for visitor questions they were unable to answer that could be sent back to the visitor after the answer was found. Participants also decided they wanted to plan multiple ways to talk to and interact with visitors, as well as survey visitors about their experience in the BBY. 5) Activity preparation. The team desired time to revise the activities’ learning objectives and practice teaching the activities to each other. 6) Advertising activities. Responding to occasional low attendance in the BBY, participants proposed new advertising techniques such as placing one Park Crew staff inside the museum to recruit visitors and creating and posting additional signs throughout the museum. 7) Space and comfort. Participants identified ways to improve their working conditions such as adding tents or awnings to their outdoor workspaces. Workshop 4: Implementing Recommendations After participants brainstormed, clustered, and named their ideas, the program manager facilitated a final workshop to incorporate the team’s recommendations into training sessions for the four BBY activities. He mapped out the remaining shifts until the start of summer and had the participants think about how to best use the shifts to prepare for the BBY season. The team decided to work in small groups to plan training sessions for the four activities. Each group chose one of the activities and was responsible for planning two days of training for the rest of the crew. Participants made sure their training days addressed each of the seven recommendation themes as they related to their activity. This final workshop allowed participants to have a direct say in how they were trained and what skills and knowledge they needed to feel comfortable teaching their activities. 9
  • 10. It was important that participants had the chance to make suggestions and offer feedback on their experience throughout the four participatory evaluation workshops. Rather than simply receiving a mixture of positive and negative feedback from the evaluation report via the program manager, participants were given the opportunity to interpret and respond critically to the evaluation data. The team worked together to make sense of the data in their own way and synthesized it into a comprehensive list of training suggestions for the upcoming season, some of which the program manager might not have considered had he not sought their input. Participants left the workshops with a stronger sense of ownership and control of their work because they were given time to express their opinions, provide their own interpretations of what was observed in the BBY, and create recommendations for improvement, many of which were implemented in the upcoming summer. Involving Participants in the Entire Evaluation Process The participatory evaluation workshops led to the team’s participation in a full participatory evaluation process. After reviewing the Park Crew summative evaluation data, the team realized there was an information gap. One participant said, "Well, we need to talk to visitors 'cuz we want to find out what they think!" Shortly after the workshops the participants came to the evaluator with their desire to formatively evaluate each of the four BBY activities with visitors that summer. The participants had already identified the goals of their evaluation, to understand what visitors learn from their activities and to improve visitors’ experiences. The program manager helped participants identify questions they wanted to ask visitors, which they then shared with the evaluator. The evaluator and participants discussed the questions and the types of information the questions would elicit to ensure they gathered their desired information. Participants also identified additional questions to include in the survey. The participants stressed that they wanted two versions of the survey, one for adults and a “kid-friendly” version. The evaluator formatted the surveys and had the participants review them. The evaluator trained the participants to collect survey data from visitors participating in BBY activities. Evaluators modeled the process and then shadowed participants during their first day of data collection to offer pointers and be available for questions. After that point, the participants managed the survey data collection. Due to scheduling constraints, the participants did not have time to enter the data so the evaluator completed that step. The survey data was then shared in a participatory manner, similar to how the participants engaged with the summative data from the previous summer. The team discussed graphs and tables of data, decided what they wanted to keep or change based on the data, and generated recommendations for their future work in the BBY. DISCUSSION Factors That Led to a Successful Participatory Evaluation Experience The participatory evaluation process was an overall success and a beneficial experience for all involved. The literature identifies a variety of factors that lead to a successful participatory evaluation process, including balanced partnerships, sufficient time allocated to the process, and the program manager's regard for evaluation and interest in including participants in the process (Cousins & Earl, 1992, 1995a; Cousins & Whitmore, 1998; King, 1995). These factors were present in carrying out the participatory evaluation workshops, leading to the success of the process.. 10
  • 11. Balanced partnerships are vital to participatory evaluation, with the evaluator and participants bringing their own knowledge and expertise to the process (Cousins & Earl, 1995a; Cousins & Whitmore, 1998). The program manager brought his expertise of working with the participants. He identified strategies to engage participants with the data, kept them on task during the workshops, and offered them encouragement when needed. The evaluator brought her evaluation expertise. She taught participants about evaluation and helped them use their analytical skills to make sense of the data. Participants brought their expertise as front-line staff working directly with visitors. They were able to provide their unique perspective to the interpretation of the data and generate recommendations based on their first-hand knowledge of the activities. Sufficient time is necessary to engage in participatory evaluation without rushing the process (King, 1995). Originally, the workshops were scheduled to cover two three-hour sessions. However, participants became deeply engaged, often spending considerable amounts of time reflecting on their prior experiences in the BBY. The participants’ positive energy and engagement made it clear by the end of the first day that the process of interpreting data and generating recommendations would have to be extended to allow enough time for a meaningful experience. Rushing participants through the workshops could have negatively affected their first participatory evaluation experience and diminished the use of the findings (King, 1998). Because of the program manager's flexibility with the team’s schedule, two additional three-hour workshop days were added. Participants must value evaluation and have an interest in engaging in the evaluation process (Cousins & Earl, 1992; King, 1995). Without the interest of those involved, the evaluator will have a difficult time successfully carrying out a participatory process (King, 1995). The interest of the program manager to engage participants in the evaluation process was vital to ensuring the successful implementation of a participatory evaluation approach. The engagement of participants during the workshops also led to its overall success. Participant engagement was evident by the lengthy discussions of the four activities. It was clear participants wanted to improve their work in the BBY on both a personal and team level. The interest and enthusiasm for the process was also evident in the participants’ desire to evaluate their own activities with visitors, as well as the program manager advocating participatory evaluation to other KAYSC staff. Benefits of the Participatory Evaluation Process The program manager and youth reaped a number of benefits from their involvement in the participatory evaluation process, as evident from their reflections and behaviors after the workshops. The youth reflected on their experience and the benefits after each workshop and while planning for the upcoming summer. The evaluator and the program manager debriefed on the evaluation process, discussing benefits of the process for them and what they had observed and heard as benefits for the youth. Benefits to the Program Manager A major outcome of participatory evaluation is evaluation capacity building of those involved in the process (King & Stevahn, 2002). The participatory evaluation process provided professional 11
  • 12. development for the program manager, who had little evaluation experience going into the workshops. The collaboration with the evaluator increased the program manager's understanding of evaluation, his ability to interpret evaluation data and generate recommendations, and his confidence in including youth as participants in future evaluations. The program manager's evaluation capacity led to his desire to engage a different youth team, the Water Crew, in a participatory evaluation process. The team was developing interactive quizzes for an upcoming exhibition and wanted to find out if the quizzes successfully communicated their desired messages. Having recently completed the participatory evaluation workshops, the program manager suggested some possible evaluation methods and the team contacted the internal evaluator to help carry out the evaluation. The participants had already identified some questions they wanted to ask visitors, so the evaluator helped them refine their interview questions and trained the participants in conducting visitor interviews. The Water Crew then carried out cued interviews with visitors. The evaluator offered participants support and guidance as needed. The evaluator and program manager then shared the data with the participants. The participants interpreted the data and identified revisions to make to their quizzes. The evaluation of the Water Crew quizzes is an example of a small-scale participatory evaluation that was prompted by the program manager's evaluation knowledge and capacity. In addition to increasing his evaluation capacity, the participatory evaluation process also provided the program manager with important feedback about the Park Crew's work. Throughout the workshops, participants reflected on their work, talking about what they personally felt worked well and what they would change. Participants also gave thoughtful suggestions on how to better prepare for their work in the BBY and keep motivated throughout the summer. They shed light on aspects of their work the program manager had not considered, such as their lack of confidence in delivering some of the information and their desire for more training opportunities. As a result of the participatory process, the program manager had a deeper understanding of the participants’ experience in the BBY and could proactively identify and respond to their needs. Benefits to the Participants The participatory evaluation process also increased participants’ understanding of evaluation. Meaningful involvement in evaluation can help youth develop higher order thinking skills, specifically analytic and evaluative skills (London et al., 2003). Participants also become more reflective about themselves, their work, and their overall program, as was evident in the Park Crew’s deep engagement in the workshops and the thoughtful recommendations they created (Sabo Flores, 2008). Participants’ understanding and comfort with evaluation was also evident by their desire to engage in a formative evaluation process to further inform and improve their BBY activities. Involving participants in the evaluation process transforms their understanding of their program (Checkoway et al., 2003). The workshops served the dual purpose of program improvement and training for the upcoming year. Some of the participants were new to the Park Crew and had not led activities in the BBY before. By taking part in such a thorough reflection of the team's previous work and planning for the upcoming summer, new participants left the workshops well poised to begin training. One participant shared his increased understanding, "I know more about 12
  • 13. the things we do in the Big Back Yard." Participant comments also reflected their increased level of knowledge about the activities and confidence in sharing that knowledge with visitors. One participant said, "I feel very confident now," while another shared, "I feel we are going to do really good this summer." The team's increased knowledge and confidence created a shared positive attitude toward their BBY work which provided a motivational springboard to the training work. Participants experience a sense of empowerment and pride when they have an influence on the way programs are run and see their ideas acted upon (Checkoway et al., 2003; Horsch et al., 2002; London et al., 2003). The participants generated thoughtful suggestions for how they could improve their work experience, but they needed outlets to implement their ideas in order to feel empowered by the process. The program manager, rather than planning and leading the BBY training himself, had the participants use their suggestions to develop their own training sessions. Participants also shared their ideas with a museum operations staff member who was responsible for BBY operations. The team suggested ways to increase the number of BBY visitors and amenities that would allow them to more effectively do their job. The museum operations staff member implemented several ideas that were realistic given time and financial constraints. Participants were able to see their suggestions put in place over the course of the summer. They learned firsthand how to make changes within a large organization and realized they shared a collective voice as valued staff members. Participants also made connections that helped them understand the evaluation work that occurs within the museum. Initially, some of the participants were a bit uncomfortable being observed by the evaluation staff as part of the summative evaluation. After meeting the evaluator and engaging with the evaluation data, they had a deeper understanding of what observations can reveal about a program. One participant expressed a renewed sense of comfort with the evaluation process sharing, "I feel being watched paid off in the long run." The team learned the importance of evaluation in a museum setting and the different ways evaluators help improve museum programs and projects. It was empowering for participants to feel a deeper tie to the museum by knowing more about the various types of work done in the institution. CONCLUSION As a result of using a participatory evaluation approach to involve youth in evaluation, the KAYSC, and more broadly the Science Museum of Minnesota, now have a format for engaging participants in evaluation practice and training them to be more self-reflective in their work. A sustainable cycle of formative evaluation (evaluation, reflection, change, and more evaluation) has been developed for a group of participants that previously had little or no knowledge of evaluation practice. The internal evaluator and program manager’s partnership increased their confidence and interest in participatory evaluation, which is evident in their use of the process with other KAYSC evaluations. Additionally, the successful Park Crew participatory evaluation experiences influenced the KAYSC's commitment to using a youth participatory evaluation approach with future evaluations. Although this article focused on youth participatory evaluation, participatory processes can be used with all types of primary users. Participatory evaluation is also not limited to program 13
  • 14. evaluation. Exhibit evaluation can also benefit from a participatory evaluation approach. Partnering with primary intended users, building their evaluation capacity, and increasing their level of participation in carrying out various phases of a formative evaluation enhances their understanding and ownership of the findings, which in turn leads to increased use of the data to improve their exhibit or program (Cousins & Earl, 1995a; Cousins & Whitmore, 1998; King & Stevahn, 2002; Patton, 2008). Users also gain a deeper understanding of evaluation practice and enhance their evaluative thinking skills through direct involvement in carrying out the evaluation, offering input, and reflecting on the process (Cousins & Earl, 1992). Primary intended users become more reflective of their work throughout the exhibit or program development process and develop the capacity to become more active participants in future evaluations (King, 1995). Note 1. To obtain a copy of the KAYSC BBY Park Crew Summative Evaluation Report, which also includes the participants’ recommendations, visit Acknowledgments The Park Crew and related evaluation work are funded by the National Center for Earth-surface Dynamics, a National Science Foundation Science and Technology Center. We thank the Science Museum of Minnesota's KAYSC and Department of Evaluation & Research in Learning staff for their feedback and support throughout the project. We also express our gratitude to the 2007 and 2008 Park Crew youth staff for their energetic participation and openness to the youth participatory evaluation process. References Bucki, J. S. (2008). Participatory leadership skills. St. Paul, MN: Checkoway, B., Dobbie, E., & Richards-Schuster, K. (2003). The Wingspread Symposium: Involving young people in community evaluation research. CYD Journal, 4(1), 7-11. Cousins, J.B., & Earl, L. (1995a). The case for participatory evaluation: Theory, research, practice. In Cousins, J.B., & Earl, L. (Eds.). Participatory evaluation in education: Studies in evaluation use and organizational learning. (pp. 3-18). London: Falmer. Cousins, J.B., & Earl, L. (1995b). Participatory evaluation in education: What do we know? Where do we go? In Cousins, J.B., & Earl, L. (Eds.). Participatory evaluation in education: Studies in evaluation use and organizational learning. (pp. 159-180). London: Falmer. Cousins, J.B., & Earl, L. (1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14, 397-418. Cousins, J.B., & Whitmore, E. (1998). Framing participatory evaluation. New Directions for Evaluation, 80, 5-23. Horsch, K., Little, P., Chase Smith, J., Goodyear, L., & Harris, E. (2002). Youth involvement in evaluation and research. (Issues and Opportunities in Out-of-School Time Evaluation No. 1). Retrieved December 14, 2008, from King, J. A. (1995). Involving practitioners in evaluation studies: How viable is collaborative evaluation in schools? In Cousins, J.B., & Earl, L. (Eds.). Participatory evaluation in education: Studies in evaluation use and organizational learning. (pp. 159-180). London: 14
  • 15. Falmer. King, J.A. & Stevahn, L. (2002). Three frameworks for considering evaluator role. In Ryan, K.E., & Schwandt, T.A. (Eds.). Exploring evaluator role and identity. (pp. 1-16). Greenwich, CT: Information Age Publishing. London, J., Zimmerman, K., & Erbstein, N. (2003). Youth-led research and evaluation: Tools for youth, organizational, and community development. New Directions for Evaluation, 98, 33-45. Patton, M.Q. (2008). Utilization-focused evaluation. (4th ed.) Thousand Oaks, CA: Sage. Sabo Flores, K. (2008). Youth participatory evaluation: Strategies for engaging young people. San Francisco, CA: Jossey-Bass. Stufflebeam, D. L. (1980). An interview with Daniel L. Stufflebeam. Educational Evaluation and Policy Analysis, 2(4): 90-92. The Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.) Thousand Oaks, CA: Sage. Velure Roholt, R., & Steiner M. (2005). "Not Your Average Workplace" - the Youth Science Center, Science Museum of Minnesota. Curator, 4(2), 141-157. About the Authors Amy Grack Nelson is an Evaluation and Research Associate in the Science Museum of Minnesota's Department of Evaluation and Research in Learning. Email: Robby Callahan Schreiber is a Youth Program Manager in the Science Museum of Minnesota's Kitty Andersen Youth Science Center. Email: 15