Impact or Bias? Measuring Cause and Effect in Jewish Education Adam Gamoran University of Wisconsin-Madison
Ripped from the Headlines! <ul><li>“ Is Birthright Israel an Intermarriage Panacea? ” –  The Forward </li></ul><ul><li>“ B...
Searching for the “Cure” <ul><li>Israel trips, day schools, summer camps have often been touted as “cures” for intergenera...
Education Research Falls Short <ul><li>Enthusiasm for untested policies is common in general education as well </li></ul><...
Education Research Falls Short <ul><li>Failing schools should be “restructured” </li></ul><ul><ul><li>Education research h...
The New Education Science <ul><li>The U.S. Institute of Education Sciences (IES) is trying to change the focus of educatio...
The New Education Sciences <ul><li>This change is not really about the method, but about the QUESTION: </li></ul><ul><ul><...
The New Education Sciences <ul><li>Why do we lack evidence about cause and effect in education? </li></ul><ul><ul><li>We h...
Fundamental Problem of Causal Inference <ul><li>We cannot observe a unit in both the treated and untreated conditions simu...
Fundamental Problem of Causal Inference <ul><li>If two different units are identical, we might infer causation </li></ul><...
Examples of Selectivity in Education <ul><ul><li>Class size effects </li></ul></ul><ul><ul><ul><li>Better teachers may use...
Examples of Selectivity in Education <ul><ul><li>Jewish day school effects </li></ul></ul><ul><ul><ul><li>Students who att...
Source: Grover Whitehurst, “The Institute of Education Sciences: New wine, new bottles,” 2003.  http://www.ed.gov/rschstat...
Source: Grover Whitehurst, “The Institute of Education Sciences: New wine, new bottles,” 2003.  http://www.ed.gov/rschstat...
Addressing the Selectivity Problem <ul><li>A randomized experiment is the optimal way to rule out selectivity bias </li></...
Addressing the Selectivity Problem <ul><li>U.S. education law calls for randomized trials </li></ul><ul><li>“…using rigoro...
Addressing the Selectivity Problem <ul><li>Why is experimental research so rare in education? </li></ul><ul><ul><li>Educat...
Lessons from Experiments <ul><li>What have we learned from recent experiments? </li></ul><ul><ul><li>Barriers to cooperati...
Overcoming Barriers to Experiments <ul><li>Will school districts agree to random assignment? </li></ul><ul><ul><li>Distric...
Overcoming Barriers to Experiments <ul><li>Milwaukee, Los Angeles, San Antonio, and Phoenix are all examples of districts ...
Overcoming Barriers to Experiments <ul><li>Design options </li></ul><ul><ul><li>Treatment versus control </li></ul></ul><u...
Overcoming Barriers to Experiments <ul><li>Example: Randomized evaluation of Success for All (SFA) </li></ul><ul><ul><li>G...
Analyzing Data from Experiments <ul><li>The SFA evaluation is an example of a “cluster-randomized” design </li></ul><ul><u...
Success for All Findings (Effect sizes) Source: Adapted from Borman et al. (2007), Table 5.
Lessons from Experiments <ul><li>The SFA evaluation also illustrates the importance of patience </li></ul><ul><ul><li>A on...
Lessons from Experiments <ul><li>The three keys to a successful randomized trial </li></ul><ul><ul><li>Implementation </li...
Lessons from Experiments <ul><li>Example: Instructional Technology </li></ul><ul><ul><li>Many small-scale studies have sho...
Lessons from Experiments <ul><li>Challenges of patience, implementation, and scaling up are also salient in my work </li><...
Science PD Evaluation <ul><li>Scaling up: Would this help science learning throughout the district? </li></ul><ul><ul><li>...
Science PD Evaluation <ul><li>Implementation: Would teachers attend the summer professional development? </li></ul><ul><ul...
Science PD Evaluation <ul><li>Findings: Implementation “dip” </li></ul><ul><ul><li>No difference pre-treatment </li></ul><...
Science PD Evaluation: Grade 4
Science PD Evaluation <ul><li>Interpretation: Better implementation or abandonment of immersion? </li></ul><ul><ul><li>Obs...
Lessons From Experiments <ul><li>Include educators and schools as partners </li></ul><ul><ul><li>We cannot impose interven...
Lessons from Experiments <ul><li>Experiments are strong on internal validity, but weak on external validity </li></ul><ul>...
Generalizability in Education Experiments <ul><li>Limits of generalizability </li></ul><ul><li>Class size research </li></...
Generalizability in Education Experiments <ul><li>National survey analysis also failed to find class size effects </li></u...
Is the Birthright Study an Experiment? <ul><li>Recent study by Leonard Saxe and colleagues indicates positive effects of B...
Is the Birthright Study an Experiment? <ul><li>Natural experiment: Comparison of applicants who attended to applicants who...
Is the Birthright Study an Experiment? <ul><li>“ Observable” – a condition that has been measured </li></ul><ul><ul><li>Ag...
Is the Birthright Study an Experiment? <ul><li>No difference on observables other than age </li></ul><ul><ul><li>Jewish sc...
Is the Birthright Study an Experiment? <ul><li>Additional concerns </li></ul><ul><ul><li>Differential response rates </li>...
Is the Birthright Study an Experiment? <ul><li>Additional concerns </li></ul><ul><ul><li>Censoring on marriage </li></ul><...
Is the Birthright Study an Experiment? <ul><li>The Birthright study is closer to an experiment than most research in Jewis...
Could the Birthright Study Have Been Conducted as a True Experiment? <ul><li>Yes, if it had been set up that way </li></ul...
Advancing the New Education Science <ul><li>With all the advances in curricula and teaching methods, we should be asking s...
Further Reading on  Randomized Trials in Education <ul><li>Bloom, H. S.  (2006).  Learning more from social experiments: E...
Further Reading on  Randomized Trials in Education <ul><li>Cook, T. T.  (2003).  Why have educational evaluators chosen no...
Upcoming SlideShare
Loading in …5
×

Measuring Cause and Effect in Jewish Education - Professor Adam Gamoran

586 views
502 views

Published on

This spreadsheet accompanies Professor Gamoran's February 1 lecture/webcast for the Berman Jewish Policy Archive @ NYU Wagner:

Education researchers have become increasingly aware of the challenges of measuring the impact of educational practices, programs, and policies. Too often what appears to be cause and effect may actually reflect pre-existing differences between program participants and non-participants. A variety of strategies are available to surmount this challenge, but the strategies are often costly and difficult to implement. Examples from general and Jewish education will highlight the challenges, identify strategies that respond to the challenges, and suggest how the difficulties posed by these strategies may be addressed.

Published in: Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
586
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
8
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Measuring Cause and Effect in Jewish Education - Professor Adam Gamoran

  1. 1. Impact or Bias? Measuring Cause and Effect in Jewish Education Adam Gamoran University of Wisconsin-Madison
  2. 2. Ripped from the Headlines! <ul><li>“ Is Birthright Israel an Intermarriage Panacea? ” – The Forward </li></ul><ul><li>“ Birthright Alumni Found Less Likely to Marry Non-Jews” – Ha’Aretz </li></ul><ul><li>“ Birthright Study Offers Mixed Bag of Results on Jewish Connections” – The Jewish World </li></ul>
  3. 3. Searching for the “Cure” <ul><li>Israel trips, day schools, summer camps have often been touted as “cures” for intergenerational loss of Jewishness </li></ul><ul><li>Most programs have not been studied at all </li></ul><ul><li>Available research often uses designs that are not suited to answer questions about cause and effect </li></ul>
  4. 4. Education Research Falls Short <ul><li>Enthusiasm for untested policies is common in general education as well </li></ul><ul><li>US federal policy advocates policies that lack evidence </li></ul><ul><ul><li>Pay for performance </li></ul></ul><ul><ul><li>Charter schools </li></ul></ul><ul><li>These may or may not be good policies, but they have not been carefully tested </li></ul>
  5. 5. Education Research Falls Short <ul><li>Failing schools should be “restructured” </li></ul><ul><ul><li>Education research has provided almost no evidence about how to undertake such radical reforms successfully </li></ul></ul><ul><ul><li>All we have is practical wisdom </li></ul></ul><ul><ul><li>No studies about restructuring permit judgments of cause and effect </li></ul></ul><ul><li>Meanwhile, educators have to make decisions every day </li></ul>
  6. 6. The New Education Science <ul><li>The U.S. Institute of Education Sciences (IES) is trying to change the focus of education research </li></ul><ul><ul><li>Limit causal claims to research designs that support causal inference </li></ul></ul><ul><ul><li>Encouraged experimental and quasi-experimental research with attention to causal inference </li></ul></ul>
  7. 7. The New Education Sciences <ul><li>This change is not really about the method, but about the QUESTION: </li></ul><ul><ul><li>What education programs, practices, and policies are effective at raising scores and reducing gaps? </li></ul></ul><ul><ul><li>If the question is “what works,” it is clear we need research methods that permit judgments about cause and effect </li></ul></ul>
  8. 8. The New Education Sciences <ul><li>Why do we lack evidence about cause and effect in education? </li></ul><ul><ul><li>We have not responded to the Fundamental Problem of Causal Inference </li></ul></ul>
  9. 9. Fundamental Problem of Causal Inference <ul><li>We cannot observe a unit in both the treated and untreated conditions simultaneously </li></ul><ul><li>If a unit undergoes a treatment, and changes in some way, we may want to attribute the change to the treatment </li></ul><ul><li>But how do we know the unit would not have changed in the absence of the treatment? </li></ul><ul><ul><li>Depends on assumptions </li></ul></ul>
  10. 10. Fundamental Problem of Causal Inference <ul><li>If two different units are identical, we might infer causation </li></ul><ul><li>But this also depends on assumptions </li></ul><ul><ul><li>In particular, inferring causation from comparison of two different units assumes no selectivity bias </li></ul></ul><ul><ul><li>In education, this assumption rarely holds </li></ul></ul>
  11. 11. Examples of Selectivity in Education <ul><ul><li>Class size effects </li></ul></ul><ul><ul><ul><li>Better teachers may use clout to get smaller classes </li></ul></ul></ul><ul><ul><ul><li>Or better teachers may be more often requested and end up with larger classes! </li></ul></ul></ul><ul><ul><li>Teacher professional development impact </li></ul></ul><ul><ul><ul><li>Because participation is often voluntary, it is difficult to distinguish effects of participation from effects of who participates and who does not </li></ul></ul></ul>
  12. 12. Examples of Selectivity in Education <ul><ul><li>Jewish day school effects </li></ul></ul><ul><ul><ul><li>Students who attend day schools may come from more Jewishly active families than other students </li></ul></ul></ul><ul><ul><ul><li>Or the students themselves may be more committed to Jewish involvement than other students </li></ul></ul></ul><ul><ul><ul><li>These conditions make it difficult to distinguish the effects of day schools from the effects of the students and families who choose day schools </li></ul></ul></ul>
  13. 13. Source: Grover Whitehurst, “The Institute of Education Sciences: New wine, new bottles,” 2003. http://www.ed.gov/rschstat/research/pubs/ies.html
  14. 14. Source: Grover Whitehurst, “The Institute of Education Sciences: New wine, new bottles,” 2003. http://www.ed.gov/rschstat/research/pubs/ies.html
  15. 15. Addressing the Selectivity Problem <ul><li>A randomized experiment is the optimal way to rule out selectivity bias </li></ul><ul><ul><li>Participants are assigned to treatment and control groups at random </li></ul></ul><ul><ul><li>Self-selection does not occur </li></ul></ul><ul><ul><li>Bias is eliminated </li></ul></ul>
  16. 16. Addressing the Selectivity Problem <ul><li>U.S. education law calls for randomized trials </li></ul><ul><li>“…using rigorous methodological designs and techniques, including control groups and random assignment, to the extent feasible, to produce reliable evidence of effectiveness.” </li></ul><ul><li>Not without controversy </li></ul><ul><li>When it comes to measuring impact , no substitute for random assignment </li></ul>
  17. 17. Addressing the Selectivity Problem <ul><li>Why is experimental research so rare in education? </li></ul><ul><ul><li>Education viewed more as an art than a science </li></ul></ul><ul><ul><li>Ethical problem of “denying treatment” to those who deserve it </li></ul></ul><ul><ul><li>Practical problems of mounting an experiment </li></ul></ul><ul><ul><li>Trade-offs between internal and external validity </li></ul></ul>
  18. 18. Lessons from Experiments <ul><li>What have we learned from recent experiments? </li></ul><ul><ul><li>Barriers to cooperation can be overcome </li></ul></ul><ul><ul><li>Need for patience to let results emerge </li></ul></ul><ul><ul><li>Importance of implementation </li></ul></ul><ul><ul><li>Difficulties of scaling up </li></ul></ul><ul><ul><li>Limitations to generalizability </li></ul></ul><ul><li>Illustrations from recent work </li></ul>
  19. 19. Overcoming Barriers to Experiments <ul><li>Will school districts agree to random assignment? </li></ul><ul><ul><li>Districts are increasingly recognizing the need for unbiased assessments of new programs </li></ul></ul><ul><ul><li>If we KNOW the benefits of the program, it is unethical to conduct an experiment </li></ul></ul><ul><ul><li>If we do NOT KNOW the benefits of the program, it is unethical NOT to evaluate it </li></ul></ul>
  20. 20. Overcoming Barriers to Experiments <ul><li>Milwaukee, Los Angeles, San Antonio, and Phoenix are all examples of districts that are engaged in pioneering randomized trials </li></ul><ul><li>Is random assignment a “denial of services”? </li></ul><ul><ul><li>Not when a program is being phased in </li></ul></ul><ul><ul><li>Not if the program is only available because of the research funds </li></ul></ul>
  21. 21. Overcoming Barriers to Experiments <ul><li>Design options </li></ul><ul><ul><li>Treatment versus control </li></ul></ul><ul><ul><li>Lagged treatment </li></ul></ul><ul><ul><li>Lottery-based individual assignment </li></ul></ul><ul><ul><li>Grade-by-grade randomization </li></ul></ul>
  22. 22. Overcoming Barriers to Experiments <ul><li>Example: Randomized evaluation of Success for All (SFA) </li></ul><ul><ul><li>Geoffrey Borman, Robert Slavin, et al. </li></ul></ul><ul><ul><li>Difficulties recruiting schools </li></ul></ul><ul><ul><li>Solution </li></ul></ul><ul><ul><ul><li>Randomized assignment of grade K-2 or 3-5 to SFA </li></ul></ul></ul><ul><ul><ul><li>Grades not assigned to SFA served as controls for other schools </li></ul></ul></ul>
  23. 23. Analyzing Data from Experiments <ul><li>The SFA evaluation is an example of a “cluster-randomized” design </li></ul><ul><ul><li>Schools, not individuals, are randomly assigned </li></ul></ul><ul><ul><li>Requires a multilevel model for estimation </li></ul></ul><ul><ul><li>Treatment effect are captured at the cluster level, not the student level </li></ul></ul>
  24. 24. Success for All Findings (Effect sizes) Source: Adapted from Borman et al. (2007), Table 5.
  25. 25. Lessons from Experiments <ul><li>The SFA evaluation also illustrates the importance of patience </li></ul><ul><ul><li>A one-year study would have missed the results </li></ul></ul><ul><li>SFA has high fidelity of implementation </li></ul><ul><ul><li>It is tightly scripted </li></ul></ul><ul><ul><li>Instruction can be monitored to see whether teachers are following the script </li></ul></ul>
  26. 26. Lessons from Experiments <ul><li>The three keys to a successful randomized trial </li></ul><ul><ul><li>Implementation </li></ul></ul><ul><ul><li>Implementation </li></ul></ul><ul><ul><li>Implementation </li></ul></ul><ul><li>If the program is not implemented with fidelity, it will not achieve its desired impact </li></ul>
  27. 27. Lessons from Experiments <ul><li>Example: Instructional Technology </li></ul><ul><ul><li>Many small-scale studies have shown benefits of technology-based instruction </li></ul></ul><ul><ul><li>Federally-sponsored, large-scale study conducted by Mathematica showed zero impact </li></ul></ul><ul><ul><li>Implementation was limited </li></ul></ul><ul><ul><ul><li>E.g. in middle school math: 15 minutes per week </li></ul></ul></ul>
  28. 28. Lessons from Experiments <ul><li>Challenges of patience, implementation, and scaling up are also salient in my work </li></ul><ul><li>Study of professional development for elementary science teaching in Los Angeles </li></ul><ul><ul><li>Science “immersion” – an extended, inquiry-oriented science curriculum for grades 4-5 </li></ul></ul><ul><ul><li>Summer professional development with ongoing mentoring to help teachers implement </li></ul></ul>
  29. 29. Science PD Evaluation <ul><li>Scaling up: Would this help science learning throughout the district? </li></ul><ul><ul><li>Goal: Include all schools, do not “cherry-pick” </li></ul></ul><ul><ul><li>Were able to include about half the schools in our pool for random selection (191 schools) </li></ul></ul><ul><ul><li>80 schools randomly selected for the study </li></ul></ul><ul><ul><li>40 randomly assigned to “immersion,” 40 were comparison schools </li></ul></ul>
  30. 30. Science PD Evaluation <ul><li>Implementation: Would teachers attend the summer professional development? </li></ul><ul><ul><li>30/40 schools sent grade 4 teachers, and 22/40 schools sent grade 5 teachers </li></ul></ul><ul><ul><ul><li>36/40 schools sent at least one teacher </li></ul></ul></ul><ul><ul><li>Follow-up participation was weak </li></ul></ul><ul><li>Immersion is not tightly scripted </li></ul><ul><ul><li>Instruction varied greatly across classes </li></ul></ul>
  31. 31. Science PD Evaluation <ul><li>Findings: Implementation “dip” </li></ul><ul><ul><li>No difference pre-treatment </li></ul></ul><ul><ul><li>Year 1: Lower scores in immersion schools </li></ul></ul><ul><ul><li>Year 2: Equal scores in immersion and comparison schools </li></ul></ul>
  32. 32. Science PD Evaluation: Grade 4
  33. 33. Science PD Evaluation <ul><li>Interpretation: Better implementation or abandonment of immersion? </li></ul><ul><ul><li>Observations: Use of immersion went from 80% in Year 1 to 30% in Year 2 </li></ul></ul><ul><ul><li>Surveys: 26% of teachers in immersion schools reported using immersion “a lot” in Year 2 </li></ul></ul><ul><li>Probably some of both is taking place </li></ul><ul><li>We are awaiting results for Year 3 </li></ul>
  34. 34. Lessons From Experiments <ul><li>Include educators and schools as partners </li></ul><ul><ul><li>We cannot impose interventions on educators </li></ul></ul><ul><ul><li>Implementation will succeed only with buy-in from school staff </li></ul></ul><ul><ul><li>Study recruitment often requires school district partnership </li></ul></ul>
  35. 35. Lessons from Experiments <ul><li>Experiments are strong on internal validity, but weak on external validity </li></ul><ul><li>We can have confidence in our judgment about cause and effect, but not in whether the effect would generalize to other places </li></ul>
  36. 36. Generalizability in Education Experiments <ul><li>Limits of generalizability </li></ul><ul><li>Class size research </li></ul><ul><ul><li>Tennessee STAR experiment showed that smaller classes boost achievement in grades K-1 </li></ul></ul><ul><ul><li>Effects are sustained but do not increase </li></ul></ul><ul><li>Similar efforts in California and Florida have failed </li></ul><ul><ul><li>FL: Lack of funding </li></ul></ul><ul><ul><li>CA: Lack of trained teachers and space </li></ul></ul>
  37. 37. Generalizability in Education Experiments <ul><li>National survey analysis also failed to find class size effects </li></ul><ul><ul><li>Unobserved selectivity or lack of generalizability? </li></ul></ul><ul><li>Early Childhood Longitudinal Study </li></ul><ul><ul><li>Survey of the Kindergarten class of 1988 </li></ul></ul><ul><ul><li>Comparisons of teachers with two classes confirms the finding of no effect </li></ul></ul>
  38. 38. Is the Birthright Study an Experiment? <ul><li>Recent study by Leonard Saxe and colleagues indicates positive effects of Birthright Israel </li></ul><ul><ul><li>23% greater likelihood of sense of connection to Israel </li></ul></ul><ul><ul><li>50% more likely to feel “very confident” of ability to explain current situation in Israel </li></ul></ul><ul><ul><li>22% more likely to belong to a congregation </li></ul></ul><ul><ul><li>57% more likely to have a Jewish spouse (married non-Orthodox respondents) </li></ul></ul>
  39. 39. Is the Birthright Study an Experiment? <ul><li>Natural experiment: Comparison of applicants who attended to applicants who did not </li></ul><ul><li>Generalizable only to applicants </li></ul><ul><li>Unbiased comparison? </li></ul><ul><ul><li>Main reason for not attending: timing of dates offered for trip was inconvenient </li></ul></ul><ul><ul><li>“ The selection process was more or less random” (p.10) – more details would help!! </li></ul></ul><ul><ul><li>No difference on observables other than age </li></ul></ul>
  40. 40. Is the Birthright Study an Experiment? <ul><li>“ Observable” – a condition that has been measured </li></ul><ul><ul><li>Age, gender, denomination, etc. </li></ul></ul><ul><li>Contrasted with “unobservable” – a condition that has not been measured </li></ul><ul><ul><li>Motivation, commitment </li></ul></ul><ul><li>Observables can be addressed with statistical methods, unobservables are harder to control </li></ul>
  41. 41. Is the Birthright Study an Experiment? <ul><li>No difference on observables other than age </li></ul><ul><ul><li>Jewish schooling, gender, ritual practices, etc. </li></ul></ul><ul><li>What about unobservables? </li></ul><ul><ul><li>Motivation, commitment, interest in being Jewish, exploring Israel </li></ul></ul><ul><ul><li>Involvement in non-Jewish activities </li></ul></ul><ul><li>If inconvenient timing of trips was the main reason for non-participation, who got preferred dates, why? </li></ul><ul><li>Differences in such unobserved characteristics may bias the results </li></ul>
  42. 42. Is the Birthright Study an Experiment? <ul><li>Additional concerns </li></ul><ul><ul><li>Differential response rates </li></ul></ul><ul><ul><ul><li>61.8% participants vs 42.3% non-participants </li></ul></ul></ul><ul><ul><ul><li>Addressed with weighting, but that assumes respondents and non-respondents are similar </li></ul></ul></ul><ul><ul><ul><li>Reasons for non-response </li></ul></ul></ul><ul><ul><ul><ul><li>No contact (26% participants, 30.5% non-participants) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Lack of cooperation (6.4% participants, 19.6% non-participants) </li></ul></ul></ul></ul><ul><ul><ul><li>Differential response could bias the results in either direction </li></ul></ul></ul>
  43. 43. Is the Birthright Study an Experiment? <ul><li>Additional concerns </li></ul><ul><ul><li>Censoring on marriage </li></ul></ul><ul><ul><ul><li>This study captures people who marry young </li></ul></ul></ul><ul><ul><ul><li>In-marriage rates may differ for those who marry older </li></ul></ul></ul><ul><ul><ul><li>For this reason, the intermarriage finding should be treated with particular caution </li></ul></ul></ul>
  44. 44. Is the Birthright Study an Experiment? <ul><li>The Birthright study is closer to an experiment than most research in Jewish education </li></ul><ul><li>Deserves special attention </li></ul><ul><li>Yet recognize limited generalizability and potential problems due to non-random selection and differential response rates </li></ul>
  45. 45. Could the Birthright Study Have Been Conducted as a True Experiment? <ul><li>Yes, if it had been set up that way </li></ul><ul><li>For many years, Birthright has been oversubscribed </li></ul><ul><ul><li>Instead of first-come, first-served, establish a deadline and then select participants by lottery </li></ul></ul><ul><ul><li>That would have ruled out bias due to unobserved characteristics </li></ul></ul><ul><li>Also get better contact information to reduce non-response rate among non-participants </li></ul>
  46. 46. Advancing the New Education Science <ul><li>With all the advances in curricula and teaching methods, we should be asking some “what works” questions </li></ul><ul><li>Though not always feasible, randomized trials are the optimal method for answering these questions </li></ul><ul><li>There are many pitfalls in executing randomized trials, but the potential benefits make the effort worthwhile </li></ul>
  47. 47. Further Reading on Randomized Trials in Education <ul><li>Bloom, H. S. (2006). Learning more from social experiments: Evolving analytic approaches. New York: Russell Sage Foundation. </li></ul><ul><li>Bloom, H. S., Bos, J. M., & Lee, S. W. (1999). Using cluster random assignment to measure program impacts: Statistical implications for the evaluation of education programs. Evaluation Review, 23 , 445–469. </li></ul><ul><li>Boruch, R., May, H., Turner, H., Lavenberg, J., Petrosino, A., & de Moya, D. (2004). Estimating the effects of interventions that are deployed in many places: Place-randomized trials. American Behavioral Scientist, 47 , 608–633. </li></ul><ul><li>Borman, G. D. (2002). Experiments for educational evaluation and improvement. Peabody Journal of Education , 77 , 7-27. </li></ul><ul><li>Borman, G. D., Gamoran, A., & Bowdon, J. (2008). A randomized trial of teacher development in elementary science: First-year effects. Journal of Research on Educational Effectiveness, 1, 237-264. </li></ul><ul><li>Borman, G. D., Slavin, R. E., Cheung, A. C. K., Chamberlin, A. M., Madden, N. A., & Chambers, B. (2007). Final reading outcomes of the national randomized field trial of Success for All. American Educational Research Journal , 44 , 701-731. </li></ul>
  48. 48. Further Reading on Randomized Trials in Education <ul><li>Cook, T. T. (2003). Why have educational evaluators chosen not to do experiments? Annals, APASS, 589, 114-149. </li></ul><ul><li>Dynarski, M., et al. (2007). Effectiveness of reading and mathematics software products: Findings from the first student cohort. Washington, DC: U.S. Department of Education. </li></ul><ul><li>Milesi, C., & Gamoran, A. (2006). Effects of class size and instruction on kindergarten achievement.” Educational Evaluation and Policy Analysis, 28 , 287-313. </li></ul><ul><li>Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2 , 173–185. </li></ul><ul><li>Schneider, B., Carnoy, M., Kilpatrick, J., Schmidt, W. H., & Shavelson, R. J. (2007). Estimating causal effects using experimental and observational designs . Washington, DC: American Educational Research Association. </li></ul><ul><li>Whitehurst, G. (2003). The Institute of Education Sciences: New wine, new bottles . http://www.ed.gov/rschstat/research/pubs/ies.html </li></ul>

×