Download to read offline
More Related Content
Similar to CPPW and Transform Evaluation
CPPW and Transform Evaluation
- 1. Wisconsin Farm toSchool Evaluation Wisconsin Farm to School Summit 27 June 2013 Andrea Bontrager Yoder, Dale Schoeller
- 2. Outline • Context: Whytransform? – Back to our roots • Wisconsin Farm to School Evaluation – 2010-2011 Communities Putting Prevention to Work – 2013-2014 Transform Wisconsin • What else can we learn?
- 3.
- 4. Background: Energy Balance Physical Activity ThermicEffect of Foods Basal Energy Expenditure Growth Food Caloric beverages
- 5. Energy Balance • Isit really that simple? • Yes – it’s simple physics. Energy cannot be created nor destroyed. • No – biological systems are regulated. There are feedback systems
- 7. Background: Why Farm toSchool? • Fruits and Vegetables for Health! • Protective against chronic disease • Believed to be protective against overweight – displace energy-dense, nutrient-poor foods CDC strategies to increase FV consumption • Objective data? – 2004 review linking dietary energy density to body weight (Drewnowski et al, Nutrition Reviews) • Inconsistent, cross-sectional data • no longitudinal data
- 8. Early Farm toSchool • Early 1900’s Wisconsin “Pint Jar Program” – Bring a pint Mason jar from home – Soup, macaroni, leftovers – Heat in water bath on stove or room heater – Serve piping hot at noon • 2013 – No more pot bellied stoves in classroom – Really! Bring a glass jar?
- 9. Déjà vu allover again Yogi Berra Some Google Hits
- 10. Wisconsin AmeriCorps Farm toSchool Evaluation
- 11. Context • ARRA grantto promote public health initiatives in communities • Centers for Disease Control – required program evaluation: “What works?” – Community Guide, http://www.thecommunityguide.org/index.html
- 12. Context • Document activity •Assessed student outcomes relative to time (years) in F2S programming – Knowledge & Attitudes Survey (KA) – Block Kids Food Frequency Questionnaire (FFQ) – Lunch Tray Photo Observation (LTPO) • Qualitative assessment of perceived barriers, strengths – School Food Service directors; Farmers
- 13. Context • 9 schools •3rd, 4th, 5th grades • 2010-2011 school year
- 14. Farm to SchoolRelated Activity # Schools Classroom lessons 8/9 Local items on school lunch menu 8/9 School garden 7/9 Information sent home to parents 7/9 Farmers selling to district 7/9 Classroom tastings 7/9 Cafeteria tastings 6/9 Local items on school breakfast menu 6/9 Other 4/9 Classroom F/V snacks 3/9 Field trips to farms 3/9 Local items on after-school snack menu 2/9 Local foods fundraiser 2/9 Farmers visiting student classrooms 1/9
- 15. 1Activity reports from1 of these 2 schools were missing for 3 of 10 months. Prior F2S, years 0 1 2+ Number of schools 2 21 5 Classroom lessons 2/2 1/2 5/5 Local items on school lunch menu 2/2 2/2 4/5 School garden 1/2 2/2 4/5 Information sent home to parents 2/2 1/2 4/5 Farmers selling to district 1/2 2/2 4/5 Classroom tastings 2/2 1/2 4/5 Cafeteria tastings 2/2 1/2 3/5 Local items on school breakfast menu 2/2 1/2 3/5 Other 0/2 1/2 4/5 Classroom F/V snacks 1/2 1/2 1/5 Field trips to farms 2/2 0/2 1/5 Local items on after-school snack menu 0/2 0/2 2/5 Local foods fundraiser 0/2 0/2 2/5 Farmers visiting student classrooms 0/2 0/2 1/5
- 16. KA Survey • Positiveeffects on indicators of students’: – Knowledge – Attitudes toward eating fruits & vegetables – Willingness to try fruits & vegetables 0 3 6 9 12 15 0 1 ≥2 Score(range0-15) Knowledge Fall 2010 May 2011 20 30 40 50 60 70 80 0 1 ≥2 Score(range20-80) Prior years of F2S programming Attitudes 0 2 4 6 8 10 12 14 16 18 20 0 1 ≥2Score(range0-20) Willingness
- 17. Block Kids FFQ •No baseline differences or across the year • Students with lowest reported baseline fruit intake significantly ↑ • ↓ percent of students reporting very low fruit and vegetable intake 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 VeryLow Low Adequate VeryLow Low Adequate Fruit Vegetables cups/1000kcal Average Reported Intake of Fruits and Vegetables Fall 2010 May 2011 0 10 20 30 40 50 60 70 PercentofStudents Percent of Students with Varying Levels of Intake Fruit Vegetable
- 18.
- 19. LTPO • ↓ inpercent of trays with no FV items • ↓ in percent of trays with no FV consumed 0 10 20 30 40 50 0 1 ≥2 No fruit/vegetable disappearance 0 10 20 30 40 50 0 1 ≥2 %ofTrays No fruit/vegetable items Fall 2010 May 2011 Prior years of F2S programming
- 20. Qualitative: Key Stakeholder Interviews •8 sites – 6 school food service directors (SFS) – Farmers from four different sites • Feasibility of F2S – SFS: supply-related barriers – Farmers: supply/demand-balance related barriers
- 21. Qualitative: F2S Sustainability • SFSdirectors (continuing sites) reported positive interactions with the widest variety of stakeholder groups where the most F2S activities were taking place • Those interviewed appeared to be psychologically engaged in F2S motivated action sustainability
- 22. Qualitative: Business or PersonalSuccess? • SFS: – Program expansions – what will benefit the school? (e.g., salad bar, school garden) – Personal job satisfaction – student enjoyment • Farmers: – Pro-social overall slant: student interaction/positive student outcomes – Anticipated future economic benefit (2 farmers)
- 23. Summary of Findings •F2S programming… – Improves indicators of: • Knowledge of nutrition and agriculture concepts • Attitudes toward eating, trying fruits and vegetables – Increased exposure to fruits and vegetables – More favorably affects students with poor initial dietary behaviors – Feasible, but time needed to establish program and overcome barriers – Psychological involvement of key stakeholders may augment outcomes for students and overall program
- 24. Transform Wisconsin Farm toSchool Evaluation
- 25. Transform WI • Inthe context of new and continuing F2S sites and their specific activities: – Assess student health, behaviors, and attitudes – Assessed perceived challenges, strengths, and opportunities
- 26. Future Analysis • Ongoingcomparison of longitudinal exposure to F2S programming – Combined TWi and ARRA data • Differences on student outcomes according to types of activities
- 27. Acknowledgements • Partnering organizations –WI Dep’t of Health Services – WI Dep’t of Agriculture, Trade, and Consumer Protection – AmeriCorps Farm to School – UW-Madison (Nutritional Sciences, Family Medicine, WI Clearinghouse for Prevention) • Community partners, past and present
- 28. Many unanswered questions •SchoolMeals: –Caloric intakes, before vs with F2S –Change in amount of fresh FV served and consumed –How much is F2S: amount, product, calories, frequ ency served? •Programming: –School garden– how much PA? what level? (frequency, duration, intensity) –Frequency and intensity of engagement activities •Economic impact: –NSLP Participation rates –Willingness to Pay –Local purchasing data –Cost of programming activities •Long-term: –Program start time - high school vs middle vs elementary? –How long to get the full effect (i.e., carry-over into adulthood)? –Potential to link FV factors against chronic health problems
- 30. FV Waste ~Preparation 0.00 0.05 0.10 0.15 0.20 0.25 0.30 MeanCupsWaste Preparation Method Vegetables 0.00 0.05 0.10 0.15 0.20 0.25 0.30 MeanCupsWaste Preparation Method Fruit
- 31. Most Wasted FVItems 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 MeanCupsWasted
- 32. Least Wasted Items* 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 MeanCupsWasted *Amongitems appearing on at least 20 trays
- 33. FV Waste ~ MealComponent Type 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 0.20 Entrée Side Dish Topping MeanCupsWaste Meal Component
Editor's Notes
- #3 To start out with some context for why we in the academic community believe that F2S programs are important, we are going to give some background on energy balance and the prevalence of overweight and obesity among children, and show how F2S programming – a program that is considered rather novel in its approach to teaching children - gets “back to our roots” (yes, pun intended!).
- #4 [overweight and obesity prevalence trends]This is a problem because overweight and obese children are more likely to begin experiencing weight-related disease at an earlier age, and are more likely to be obese throughout their adult years.
- #5 Dale
- #6 Dale
- #7 Dale
- #9 Dale
- #10 There are many different levels of impacting a person’s energy balance (individual [knowledge, attitudes, skills], interpersonal [social network – family, peers], schools [environment, ethos], community [cultural values, norms], public policy). Farm to School programming, when comprehensive, can impact children at the individual, interpersonal, and school level; perhaps even at the community and public policy (school policy) level. Because there are multiple spheres of influence, it’s probably one of the reasons that the CDC also sees it as a viable strategy for positively influencing children’s dietary habits. The emphasis on fresh produce (particularly local produce, since that is a goal of F2S) is an important strategy for health overall in our ‘convenience-food’ culture. If we can teach (and convince) children to make healthy food choices – that is, choose fruits and vegetables, we have a chance at improving their health.
- #11 -While we’re not quite going there with our evaluation, other research groups have examined the cost of obesity to society, and it’s massive. If we can improve children’s long-term health by instilling healthy food habits in children, and those habits last a lifetime (obviously not assessed in our studies yet!), we have the chance at reducing health care costs. -There is still a lot to be done to determine if FV intake (low energy-density foods) is associated with lower body weight.
- #12 Dale
- #13 Dale
- #14 Dale
- #16 In 2010, the Wisconsin Department of Health Services (DHS) was awarded an American Reinvestment & Recovery Act (ARRA) grant. This grant aimed to implement and evaluate a variety of public health initiatives across Wisconsin. One of the initiatives that DHS selected to promote was Farm to School, which the Centers for Disease Control (CDC) identified as a promising strategy for reducing the prevalence of overweight and obesity in the United States by way of increasing access to fruits and vegetables.Because of the grant award and the decision to pursue Farm to School as a statewide strategy for obesity prevention, DHS needed to coordinate and implement programs and an evaluation. It was an opportunity to add to the knowledge base that is increasingly required in today’s society – “evidence-based practice.” Very little formal evaluation of Farm to School programs had been performed prior to this, so DHS partnered with UW-Madison, the Department of Agriculture, Trade, and Consumer Protection (DATCP; where the Wisconsin AmeriCorps Farm to School program is housed), and other local-food organizations to design and implement an evaluation that would help us learn “what works” in Farm to School programming. The CDC’s goal was to be able to synthesize the results of multiple evaluations across the nation to bolster their “Community Guide”, which is a free resource to help communities choose programs and policies that proven to be effective in improving health and preventing diseases.
- #17 The evaluation comprised a range of activities to understand the context of Wisconsin’s Farm to School programs, and their potential impacts on students and their communities. Today I will discuss some key highlights of the student outcomes that were assessed in the 2010-2011 school year.The KA survey assessed six areas: knowledge about nutrition and agriculture; attitudes towards trying and eating fruits and vegetables (in general, and in various situations); students’ own perception/self-efficacy of eating a healthy diet; student’s exposure to 20 specific fruits and vegetables; students’ liking of the fruits/veggies tried among those same 20, and their willingness to try those that they hadn’t previously tasted.The FFQ and LTPO assessed students dietary behaviors, both overall dietary patterns (FFQ) and school lunch specifically (LTPO).-The FFQ asked students to report what foods they had eaten over the past 7 days (what food, how frequently, how much).-The LTPO involved taking pictures of students’ school lunch trays before and after they’d eaten, to assess what and how much they’d eaten.
- #18 The evaluation was conducted in 9 schools during the 2010-11 school year, including students from 3rd, 4th, and 5th grades.Each school conducted Farm to School programs in a way that drew on the strengths and resources of their communities. Activities ranged from Harvest of the Month programs, to cooking competitions and demonstrations, to school gardens, to classroom lessons, to field trips to farms, and more. (Parts of this evaluation were also conducted in Wood County with similar but separate funding, but the timeline was different and we are not able to combine the results.)
- #19 Here we are looking at the different activities that we asked about on the monthly activity reports to document F2S activity. This shows the number of schools that performed a given activity at least one time (in at least one grade) during the 2010-11 academic year.The activities that occurred at the most schools were classroom lessons and local items on the school lunch menu, with that happening in 8 of 9 schools. Seven schools also included activities in a school garden, farmers selling to the district, classroom tastings, and sending information home to parents that pertained to F2S activities. Other popular activities included cafeteria tastings and local items on the school breakfast menu.As might be expected, the less-frequent activities are ones that may take more coordination with a wider variety of stakeholders: field trips to farms, a local foods fundraiser, or farmers visiting student classrooms (the farmers are busy!).
- #20 If we look at these same activities, listed in the same order as the previous slide, broken down by prior years of F2S programming. We don’t see any extremely strong trends. Again, this doesn’t detail the actual number of activities that are taking place in each school, just the number of different activities that happen at least once at each school. The most common activities are the same across all prior exposure groups (classroom lessons, local items on the school lunch menu). School gardens increase across exposure groups, as do farmers selling to the district and classroom tastings, local items on after-school snack menus, local foods fundraisers, and “other” activities. There aren’t activities that generally decline across exposure years in this group of schools, but there are different ways of looking at this information that may give us a better idea of what is happening in F2S programs. (Of course, it would be great if we had information from more schools to look at – but this is what we have for now.) It’s a complex set of data, and we are working on processing that data in the next several months. So what impact do these activities have? Like I said, we assessed student outcomes with a knowledge & attitudes survey, a food frequency questionnaire, and lunch tray photos. Let’s start with the KA survey.
- #21 Knowledge questions centered on topics covered in curricular lessons used by AmeriCorps F2S members in elementary classrooms, ranging from age-appropriate health and nutrition concepts to basic agricultural concepts (such as the benefits of composting, and whether certain FV grow in Wisconsin or not). At baseline, there were no differences in baseline scores, but students from schools in continuing programs significantly increased knowledge scores. Students from new schools did not increase their knowledge scores and were significantly lower than continuing-program students at follow-up.The Attitudes questions asked, for fruits and vegetables separately, how much a student liked eating fruits and vegetables, and how likely they were to eat a fruit or vegetable in various situations (at home, at school, at a friend’s home; if they had never tried it before, if they didn’t know what it was, if it looked strange). -At baseline, students with more prior F2S program years had higher Attitudes scores than students from schools that were new to F2S. At follow-up, students in schools that were continuing F2S programs (not new-to-program schools) significantly increased their Attitudes scores.We also asked whether students had tried 20 specific FV. While there were significant increases across the year overall and for each prior-year group individually, there were not significant differences according to the program exposure time groups. -The Liking questions are follow-up to those 20 Exposure questions, specifically the FV students said they had tried. The scores for this construct are calculated as a percent of Fv the students said they had liked of the FV they had tried (for ex, if they had only tried 4 of the 20 FV and liked only 1 of those 4, they would receive a liking score of 25). There were no changes overall in liking scores, although students from new schools decreased the percent of fruits and vegetables liked among those that they’d tasted.Willingness scores were also calculated as a number of the 20 specific FV that students would be willing to try. While there was a significant change overall in students’ willingness to try the 20 specific FV, there were not significant differences between baseline scores or in the degree of change across the school year for any of the prior-program exposure groups. -The scores of the other construct assessed (perception/self-efficacy) showed no differences at baseline or from start to end of the year.
- #22 We used the FFQ to examine children’s total FV intake. Overall, we saw no differences at baseline (according to prior years of F2S programming), nor across the year.We decided to look at the data by breaking into levels of intake: very low (the lowest 25% of students), low, and adequate (meeting dietary guidelines for their age).
- #23 Photographs of students’ school lunch trays were analyzed for the number and amounts of fruits and vegetables available (before eating) and the amounts disappeared (at the end of the lunch period). In schools with more prior F2S years:More fruit and vegetable items on traysLarger volume (cups) of fruits/vegetables available on and disappeared from traysNumber of items increased from fall to spring (continuing schools only)Continuing schools onlyNo change in amount disappeared (consumed) from fall to spring
- #24 All prior-exposure groups (no prior exposure, 1 prior year, 2+ prior years) significantly decreased the percent of trays that had no fruit/vegetable items from the start to the end of the school year. All groups also significantly decreased the percent of trays from which no fruit/vegetable volume disappeared (so, a higher percent of students’ trays showed evidence of at least some volume of fruit/vegetable disappeared – was consumed). The schools that were new to farm to school had significantly higher proportions of trays with no FV items and no FV disappearance/consumption.
- #25 Supply related barriers: -reliable access to local products (d/t production uncertainties, plus inflexible school planning processes).-seasonality-contacting/communicating with other stakeholder groups (i.e., those key to the food supply chain) this access-related barrier was not often overcome-***food cost: but most SFS and farmers reported making significant headway against this barrier (negotiation/other cooperative behavior). Farmers reported “receiving a negligible amount of revenue from their participation in F2S, suggesting that potential problems may exist with current solutions.”--New F2S sites had more persisting barriers, and more barriers than successes – reiterating the concept that comprehensive F2S programs (i.e., inclusion of local food procurement) take time to establish & settle.
- #26 -F2S programs need communication and cooperation between various stakeholder groups (school admin, teachers, parents, farmers, SFS directors, community members). This was higher among longer-running programs. Farmers increased their interactions when they sold multiple (as opposed to single) products.
- #31 The aims of the TWi F2S evaluation are similar to those that we had for the original ARRA grant. Our evaluation is important for telling a story as we educate ourselves and legislators, and help the CDC bolster the Community Guide (that I mentioned earlier). We want to help ensure that public health programs like F2S continue to receive state and/or federal funding. To do this, we want to know potential changes that occur among students (our primary target impact audience), as well as the context of what is happening in the programs.
- #32 -using essentially the same tools, with some improvements/modifications
- #33 Frame in terms of questionswe’re answering
- #34 AmyMeinen, Jan Liebhart, Amanda Knitter, Kelli Stader, Sarah Foster, Camilla Vargas and all AmeriCorps members; Tara LaRowe; many others!
- #35 Many unanswered questions in several domains – this serves to show that that there is always more to learn, and we’re hoping that we can provide at least preliminary answers that will spur additional studies that will ultimately show how much impact F2S programs can have across the board. Caloric intakes, before vs with F2SAre students eating more FV AND less high calorically-dense foods, or are they just eating more food?Change in amount of fresh FV served and consumed pre and post program implementationi.e., if they had canned fruit before, how did F2S items change the nutritional content?How much of the meal is F2S: amount, product, calories, frequency served?Tied to nutrition education?School garden specifics – how much PA and what level? (frequency, duration, intensity)Frequency and intensity of engagement activitiesStandardized test scores, AbsenteeismEconomic impact data:Participation rates (in NSLP)Willingness to Pay: will students/parents, food service directors pay more for local?Local purchasing data: how much, how often; more…Cost of programming activities: taste testing, field trips, school gardens, education curricula, food service training/kitchen equipmentImpact of starting program in high school vs middle vs elementary?How long must the student go through the program to get the full effect (i.e., carry-over into adulthood)?Potential for linking FV factors against chronic health problems (diabetes, COPD, asthma)