The document discusses using formative and summative evaluations to improve a field trip program called Think Science! Pathways. Formative evaluations in 2002 found that few teachers used the Pathways materials due to access issues, length, and complexity. It did increase time spent and engagement at exhibits. In response, materials were shortened and simplified. A 2003 summative evaluation found low Pathways use and no clear cognitive impact yet, but suggested in-class visits before field trips improved learning. Evaluations highlighted the need to better understand teacher and chaperone needs to improve program use and effectiveness.

1. Using Formative and Summative Evaluation to Improve Field Trip Programs Kimberly M. Burtnyk Amgen Center for Science Learning California Science Center

2. Pre-/Post-visit classroom activities Curriculum compatible K8 CA standards Gallery worksheets (atypical) “ Chaperone Sheets” Available free from our website Think SCIENCE ! Pathways Components

3. Think SCIENCE ! Pathways Goals of Pathways Programs Help teachers implement curriculum-compatible field trips Increase educational guidance for field trips Increase interactions with staff Help chaperones facilitate Increase concept understanding through chaperone facilitation

4. Formative Evaluation Spring 2002 Focus: Access, Use, Chaperone/student behavior Summative Evaluation Timeline Spring 2003 Focus: Cognitive gains Think SCIENCE ! Pathways Evaluation Plan

5. Recruiting Groups with existing reservations Questionnaires Pathways usage issues Field trip planning and behavior Chaperone roles Observations Stopped/didn’t stop Time spent--we want Pathways groups to stay longer Panels Read, Discussions, Gestures Evaluation Method

6. Recruiting 37 Schools recruited from March-June Questionnaires 82 Returned (37 Teacher 45 Chaperone) Observations 15 schools tagged for observation (8 PW, 7 Non-PW) 72 Chaperones observed Formative Evaluation Results Summary

7. Only 3 of 8 PW recruits used it Reasons Technical difficulties Materials looked too complicated and long Confused about where to find them Unaware of the materials Timing Formative Evaluation Questionnaires Hi-lites

8. Formative Evaluation Observations Hi-lites 72 Chaperones observed Stopped/Did not Stop at Capsules

9. Formative Evaluation Observations Hi-lites Mean t: 66 s Mean t: 138 s Time spent at Capsules

10. Formative Evaluation Observations Hi-lites Of the 35 chaperones who stopped... Reading, Discussion, Gestures

11. Formative Evaluation Conclusions Pathways usage was limited due to: Access to computers/printers Length and perceived complexity of the materials Communication gaps between school administrators and teachers BUT... it did produce some significant behavioral changes in chaperones who used it

12. How Did Formative Evaluation Impact the Program? In-class/On-site activities separated Shortened Map removed One activity removed (Gallery incomplete) More direct link created quick easy obvious

13. Summative Evaluation Method Recruiting Talked directly to lead teacher Materials sent to groups In-class visit prior to field trip Pre/Post Test Questionnaires --dropped Observations Gestures dropped

14. Summative Evaluation Results Summary 298 Students Recruited 217 had in-class visit 254 Pre- and Post-tests completed 28 PW 226 Non-PW Observations 26 chaperones observed ONLY 4 CHAPERONES USED PATHWAYS

15. Summative Evaluation Conclusions Jury still out on cognitive impact of Pathways The numbers are too small Will resume attempts next year Pre/Post test shows impact of In-class Visit Micrometeoroid question Chi Square p<0.004 for those who had visit vs those who did not

16. 1. Even with intensive intervention and support, teachers did not use PW There were special circumstances in the A&S Gallery And timing, curriculum, age groups were an issue too BUT: Are such materials worth spending $$ developing? What do teachers really want/need, if anything? We don’t yet know. 2. Whether or not PW improves learning outcomes as the lit. suggests is still unknown Warrants further testing and refinement of good and USEFUL field trip practices. What did Formative and Summative Evaluation Teach us About Pathways?

17. 3. We really need to learn more about teachers and chaperones Can’t assume that teachers have access to net sources Chaperones might be willing facilitators 4. We can apply statistics to tell us about the use and effectiveness of the program Further refining statistical techniques Learning curve Exciting prospects What did Formative and Summative Evaluation Teach us About Pathways?