Good afternoon. It’s good to see all of you again. I am here again this year to present headlines from the NCSF evaluation and before I get started I wanted to give a quick “heads up” about the format of today’s presentation and the slides you have in your folder. We used a different format for this year’s presentation, with a more explicit focus on the take-home messages than on slide after slide of data. I’m excited about this shift and hope that it makes for a more engaging presentation and results in even more discussion compared to last year. That said, we wanted to make sure that you had the data slides if you wanted/needed them and so they are included in your handout for additional reference. You should have one table or figure for each data point that I talk about today. We have a longer version of the data slides as well. If you are interested in the full deck of slides then please send a quick email after the presentation today and I am happy to share those.
Okay, so here we go. One thing that continues to make the NCSF unique is the scale of our event. And I like this image for two reasons. First, when I have talked to sponsors and the NCSF leadership team in the past about quality versus quantity, the answer I usually get is a simply “Yes, please!” The second reason that I like this slide is that the evaluation data back it up – we have great data from this year to document both the scale of the NCSF and the quality of the events included and each of the next few slides will focus on quality, quantity, or both. Throughout the presentation today I’m going to compare this year’s results to those from 2012 and we have some new national data to use as a comparison point as well, so I’m going to use each as a yardstick to measure this year’s success.
So let’s start on the quantity side of things. 178K attendees took part in a total of 677 events. When it comes to scale, we were slightly down in our attendance numbers this year and higher in the amount of programming produced. The 2012 Festival was attended by approximately 233K people, including 70K who attended the opening of the Nature Research Center. One of the discussion topics for later will focus on the scale of individual events and whether/how to implement large-scale events as a strategy to continue meeting attendance goals. This year’s Festival had almost 100 additional events compared to last year, and those were hosted by a slightly smaller number of event hosts and a larger number of schools. The Festival had 19 sponsors in 2012 and 20 in 2013.
So the scale of our Festival makes it unique and another unique aspect to the NCSF is the number of Expos included. This year’s Festival included 13 Expo events. This year’s evaluation plan focused heavily on collecting data at Expo events, and so we sent field researchers to seven of the 13 Expo events to collect data from attendees using intercept surveys. Overall, our results for this year were even stronger than those from last year. We saw a nice increase in event ratings overall – last year the average rating as in the very good range (a 4 out of 5 on our scale) and this year we had a nice little pop in ratings so that the average is between very good and excellent (the highest rating). ). We also saw Expos holding steady or improving slightly in relation to the key components of the Festival mission – we have had attendees rate four components in each of the last two years from whether they had fun to different types of learning. Ratings were up for three of the four categories this year (fun, new learning, and awareness), and holding steady for the fourth category (STEM careers).
A similar pattern of results was found for this year’s ticketed event. This year the Festival featured a ticketed event with Alton Brown that was hosted in two locations. The 2012 Festival also featured two ticketed events, one with Neil deGrasse Tyson and one with Adam & Jamie from the MythBusters. If you remember from last year, the ratings for Neil deGrasse Tyson were quite high across the board, from overall ratings to those related to the Festival mission. This year’s ticketed event did quite well by comparison. Alton Brown’s overall rating were slightly higher than those for Neil deGrasse Tyson. The ratings related to the mission components were slightly lower for Alton Brown compared to Neil deGrasse Tyson, though still high overall.
Statewide Star Party was a new event this year; 53 stargazing events were hosted on April 5 to kick off this year’s Festival. We collected data from attendees at four Star Parties from around the state and from all hosts. The results for this program were strong, especially given that it was its first year. Overall ratings were high, as were ratings for fun, new learning, and feeling inspired to learn more. There was also an interesting split in attendance for the Star Parties we attended, with some having an almost entirely adult audience and others having adult audiences mixed with families with kids under 18.
TSN stands for Thorp Science Night, and if you remember, this is the NCSF’s elementary school program. It was also our top-rated program across the board last year. This year the number of TSN events was increased from 50 to 71. The evaluation collected data from parents and children at approximately half of the events (46%) and from almost all teachers who hosted a TSN. Across the board, the off-the-chart positive ratings have continued. Teacher, parents, and students all seem to love this program. Ratings are in the very good to excellent range for every item rated again this year. Given the high and consistent ratings, this program has the potential to impact the larger Festival community as well. Effectively reaching K12 audiences has been an interest for some other sites in the Festival community. Based on the success of TSN, I think this program is poised to serve as a model for other Festivals around the country if the NCSF is interested.
The Festival’s second K12 program is Invite a Scientist or IAS for short. Like TSN, the number of IAS sites was increased this year. IAS more than doubled this year, from 26 sites in 2012 to 58 events hosted this year. Evaluation data were collected from students at approximately one-third of those events (38%). Most of the teachers who hosted the events and the scientists who presented also completed an evaluation. The IAS program moved from high school to middle school this year and included a collection of new resources and materials created by the NCSF team. The changes really paid off. IAS was rated positively last year but ratings were lower when compared to the other Festival programs. This year’s ratings saw big jumps across the board, and that was true for teachers who hosted the programs, scientists who presented during the programs, and students who benefited from the program.
The IAS program is one way that the public gets to interact with scientists (and vice versa) as part of the NCSF, but it is actually the hope and expectation that most Festival programs will provide the opportunity for interaction between the public and scientists. You may remember that doing a hands-on activity with a scientist and listening to a scientist talk about his/her work are each considered best practices in Festivals because those activities are associated with higher ratings. We saw a nice jump in the number of attendees who believed they had these experiences this year – from approximately half last year to almost three-quarters this year. You may also remember that we replicated the national results from the Science Festival Alliance last year but documenting that our Festival attendees also report significantly higher ratings and outcomes if they had one of these experiences, and we have replicated that result again this year.
Another method and analysis that we have modeled after the Science Festival Alliance is the use of a Returning Attendee Survey. This survey was introduced in 2013 and used at each of the seven Expos attended by the evaluation team. Those how had also attended the 2012 Festival were asked a small number of additional questions to find out whether they had continued to think about and engage with the topics they learned about at the 2012 Festival, and the numbers were quite high. Almost everyone who attended in 2012 had talked about the Festival with others, 70% had looked for more information on something they had learned about at the Festival, and more than half had completed an activity related to a Festival topic. Compared to the national data from the SFA, these numbers are quite high and indicate that the NCSF is doing a nice job of catalyzing continued engagement with STEM.
Last year when we met we spent a lot of time talking about the demographic data collected at the 2012 Festival and so I wanted to provide some key updates on that front as well. You may remember that, compared to the 2010 census data, the 2012 Festival was attended by a disproportionate percentage of non-Hispanic White attendees while racial and ethnic groups that are traditionally underrepresented in STEM were underrepresented. The Festival team took several steps in 2013 to try to reach attendees that were a more direct match for the state population, and the results indicate some success in this regard. Expo attendees were more racially and ethnically diverse this year, with the percentage of African American and Hispanic attendees doubled compared to 2012. You may also remember that last year, the population of IAS students was a good match for the state-level race/ethnicity; the data for 2013 show that IAS programs are now serving a greater proportion of African American and Hispanic students compared to state census data. We also have new data from the SFA to consider to help think about the Festival’s success in this regard – of the four original SFA Festivals, one serves a matched percentage of non-White Hispanic attendees compared to their city population data and the other three still serve a disproportionate percentage of their non-White Hispanic population. Our data also indicate that the type of event really matters when it comes to the demographic data we collect – the ticketed events in 2013 served a predominately White group of attendees with very few African American or Hispanic attendees while IAS was tipped in the opposite direction across these different racial/ethnic groups. One of the discussion topics later will focus on the role of K12 in the Festival and demographics may be a key component of that conversation.
K12 programming also makes a difference in the demographic profile related to the educational attainment of attendees. You may remember that last year we were a bit surprised to report that almost three-quarters of Festival attendees had a college degree or higher. Data collected from the SFA Festivals and reported in October 2012 indicate that this is a common challenge. Most of the original SFA Festivals serve a disproportionate percentage of those with college degrees or higher compared to the city population. In NC, we still have work to do in this area if this is a priority. As with the race/ethnicity data, our numbers look quite different when K12 is added to the mix. If you look at the event data alone, our data are quite similar from 2012 to 2013 – there is almost an inverse relationship between the state census data and the Festival data. Adding in the TSN data, the gap is not nearly as wide, and so again we see K12 playing an interesting role in the overall profile of Festival attendees.
Most of the results discussed thus far have been focused on the attendees. I want to spend a few minutes now talking about the hosts and host institutions who create Festival events, with a focus on both the short and longer-term benefits of being part of the Festival. We have surveyed event hosts each year and one of the consistent findings has been the short-term benefit of the Festival on employees who participate – most hosts believe that the Festival not only increases the opportunity for their employees to interact with the public but that their employees also learn new strategies for working with the public and gain confidence in their ability to do outreach. Longer-term, and at an organizational rather than individual level, the Festival has created new partnerships between hosts, community groups, and other Festival organizations. Finally, about one-third of those who hosted an event in both 2012 and 2013 believed that their organization had seen an increase in follow-up phone calls and/or attendance at other events based on their participation in the Festival.
And so perhaps it isn’t surprising the folks are interested in coming back for more. A key indicator for success for volunteer programs like Festivals is whether people want to continue to play in your sandbox, and almost all of the NCSF’s hosts, teachers, and scientists plan to come back next year. An even greater percentage would recommend participation in the Festival to their colleagues.
Finally, we also have some nice convergence this year if we look across ratings. We’ve already talked about the relation between attendees’ ratings and whether they experienced a hands-on activity or heard a scientist talk, and we are starting to see some similar trends for how hosts feel about their Festival events as well. For example, hosts who included a hands-on event in their program provide higher ratings of their event overall and higher ratings in relation to outcomes. A similar pattern was found between use of the NCSF resources, ratings of the NCSF staff, and ratings of events and outcomes.
1. 2013 Evaluation Headlines
Karen Peterman, Ph.D.
Jane Robertson, Ph.D.