Evaluation of STEM interventions such as camps and magnet schools often focuses only on the intervention. Yet evaluations focused on the "in the moment" outcomes fail to consider the program's connection to a student's life. How students arrive in optional programs, and what they do after the program, can be as or even more important to long-term program impact than the program itself. For example, while researching a middle grades engineering design program, we found several factors related to student participation, including prior achievement and having parents that work in a STEM field. Evaluators need to identify in advance which contextual factors to measure to evaluate the fit of the program into the larger context of students' lives. In this presentation, we use concepts from developmental evaluation, along with data from our studies, to situate the evaluation of STEM education activities in the broader context of student growth and development.

1. Evaluating the Role of Contextual and
Support Factors in STEM Programs for
Elementary and Middle Grades
James P. Van Haneghan, University of South Alabama
Jessica M. Harlan, University of South Alabama
(now at Johns Hopkins University School of
Medicine)

2. Taking a broader perspective
Simple to look at pre-post program outcomes
Assumptions:
Change means the program worked
No change means the program didn’t work
Getting to the “truth” might require looking at contextual factors

3. STEM Camp Attendance
Variable B SE B Exp (B)
Gender 0.32 .21 1.38
Ethnicity* -0.52 .22 0.59
4th Reading Score 0.01 .00 1.01
4th Math Score 0.00 .00 1.00
Mother in STEM field 0.21 .23 1.23
Father in STEM field* 0.43 .21 1.54
*p < .05
Χ2 (6, N = 1,437) = 25.50, p < .001

4. EYE Camps
Based on Engineering is
Elementary (Museum of
Science, Boston)
Goal: learn more about
engineering, see it in
practice
Water, Water
Everywhere
Classroom work
Math/science skills
Knowledge about
engineering
Hands-on engineering
tasks
Field Trips
Local water and sewer
system facilities
Marine research facility
Department of
Conservation and
Natural Resources

5. Math/Science/Technology Magnet School
• Accelerated school for students at or above grade level
• On average, school district scores below standardized test benchmarks
• Electives unusual for the district (robotics, pre-engineering
careers)
• Requires parents to complete lottery application

6. 6th Grade Math/Science
Magnet School Attendance
Variable B SE B Exp (B)
Gender .004 .140 1.00
Ethnicity*** .725 .149 2.07
STEM Camp Attendance** -.907 .304 0.40
5th Reading Score*** .024 .003 1.02
5th Math Score*** .014 .003 1.01
Mother in STEM field .215 .155 1.24
Father in STEM field* .278 .145 1.32
*p = .06 **p < .05 *** p < .001
Χ2 (6, N = 1,585) = 224.19, p < .001

7. 8th Grade Math/Science
Magnet School Attendance
Variable B SE B Exp (B)
Gender .077 .144 1.08
Ethnicity*** .705 .153 2.03
STEM Camp Attendance** -.861 .306 0.42
5th Reading Score*** .022 .003 1.02
5th Math Score*** .014 .003 1.01
Mother in STEM field .279 .159 1.32
Father in STEM field .213 .149 1.24
*p = .05 **p < .01 *** p < .001
Χ2 (7, N = 1,381) = 194.78, p < .001

8. EYE Middle School Program
Classroom-based integrated
STEM experiences
Implemented by math and
science teachers

9. 8th Grade Confidence in STEM Skills
1 1.5 2 2.5 3 3.5 4
Mother in STEM
Father in STEM
Participating School
No
Yes
School: F(1, 1419) = 3.44, p = .025, d = .157
Father: F(1, 1419) = 5.00, p = .007, d = .273
Mother: F(1, 1419) = 4.53, p = .010, d = .215

10. 8th Grade Math/Science Efficacy
1 1.5 2 2.5 3 3.5 4
Mother in STEM
Father in STEM
Participating School
No
Yes
School: F(1, 1454) = 5.65, p = .018, d = .134
Father: F(1, 1454) = 6.62, p = .010, d = .276
Mother: F(1, 1419) = 5.24, p = .022, d = .178

11. Importance of Knowing
Developmental Evaluation
Use information about users/the environment to inform program
adaptations
Most STEM programs only collect short term, program-specific
data
Can’t examine data you don’t have
Are your findings because of the program or an artifact of who self-selected into the
program?
Are changes needed?

12. Other Factors: Experience & Interest
Difference if this is your first versus your fifth or sixth
Multiple years of EYE elementary camp
Multiple years of EYE middle school
Different impact?
Stage of Interest (Hidi and Renninger, 2006)
What is the goal?

13. Levels of interest
Triggered
situational
interest
Maintained
situational
interest
Emerging
individual
interest
Well-
developed
individual
interest
Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational
Psychologist, 41(2), 111-127.

14. Competing Initiatives
Other programs that influence impact
Technology initiative that had to compete with High Schools that Work
Initiative
May impact both student and teacher performance
Will you know there’s another initiative?
Is there a cumulative effect of different inititives?
Positive: Accumulation of benefits
Negative: Burnout (teacher & students)

15. Interpreting context variables with EYE
Additional efforts to recruit underrepresented students to camps
& magnet school
Additional supports needed for lower performing students? EYE
middle school students without STEM parents?
Differences in how the camps/magnet school impacts these new
students?
Can you really have 1 program that serves all students in a
school?

16. Questions?
Contact: James P. Van Haneghan, jvanhane@southalabama.edu