Your SlideShare is downloading. ×
What We Know
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

What We Know


Published on

Published in: Education

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. What Do We Know? SEEKING EFFECTIVE MATH AND SCIENCE INSTRUCTION THE URBAN INSTITUTE February 2005 As requested by the GE Foundation, the with relatively credible evaluations to pro- main goal of this review was to identify vide some choice to districts and schools mathematics and science curricula as well that wanted guidance in selecting a cur- as professional development models at riculum and that wished to use effective- the middle and high school levels that ness as a selection criterion. had been deemed effective based on their success in increasing student achievement. Because of its emphasis on student achieve- Evaluation Study Selection ment, this approach differs from that Criteria Used typically used at the local level and that adopted by other recent studies conducted To be included, studies had to have to find “effective” curricula and profes- 1. Rigorous methodological design sional development models. including comparison groups; Historically, curriculum choice at the 2. Measures of impact on student local level has often been made by a com- outcomes that included, but were mittee that decides which curriculum to not limited to, test scores; and adopt based on considerations only periph- 3. Comparative data with experi- erally related to student achievement—such mental or quasi-experimental as state-imposed standards, recommenda- design preferred. tions of others, cost, and presentations by publishers’ representatives. Choice of pro- fessional development models follows a As can be seen from the criteria above, similar pattern. Indeed, there has been very we chose increases in student achievement little else available to guide school districts as the measure of effectiveness.2 We limited in their curriculum selection process, since our review to evaluations that compared for most curricula and textbooks the only student achievement outcomes elicited by data at hand are publishers’ figures on the curriculum being studied with student the number of adoptions. That has been achievement outcomes of another curricu- changing. There is a growing movement lum. Where data were available, we also to assess the effectiveness of math and took into account the content validity of the science curricula through various meth- assessments used as well as the statistical The Urban Institute odologies, including content analyses, significance and the effect size of the differ- Beatriz Chu Clewell comparative studies, case studies, and syn- ences reported. Our review made a special Clemencia Cosentino de thesis studies.1 And while there have been effort to identify eligible studies that dis- Cohen several studies of the effectiveness of pro- aggregated results by performance of stu- Nicole Deterding Sarah Manes fessional development practices, very few dent subgroups. Our review of evaluation Lisa Tsui have measured the effects of these practices studies of professional development was on student achievement. similarly limited to studies that used Campbell-Kibler The focus of this review, reflecting the student achievement as the measure of Associates, Inc. GE Foundation’s interests, was to identify effectiveness. Patricia B. Campbell current math and science curricula and Lesley Perlman professional development at the middle Professional Development: Shay N.S. Rao and high school levels that showed evi- Becky Branting dence of positive impact on student What We Found Lesli Hoey achievement. Our goal was to come up We identified 18 evaluation studies of pro- Rosa Carson with enough math and science curricula fessional development models that used 1
  • 2. What Do We Know? Seeking Effective Math and Science Instruction student achievement outcomes as both change in the teaching mathematics curricula that were measures of effectiveness. Features of behavior of teachers and change developed as part of whole school these studies can be found in a full in the classroom environment. reform efforts. A total of 156 studies report of this review.3 We found that One study found that behavioral of student mathematics achievement change was only evident after with comparison group data were Providing professional develop- teachers had received a minimum found for 18 of the curricula (20 per- ment for teachers of standards- of 80 hours of intensive profes- cent of the total number of curricula based science curricula is sional development; change in originally identified). A list of the effective in increasing student the classroom environment was 18 mathematics curricula, together achievement. There is evidence achieved after 160 hours.4 It is with overviews of the curricula and from the research that standards- important to note that at the indi- their impact studies, appears in based science programs that pro- vidual level, teacher preparation in Appendix A. vide professional development to the content area was an important The following findings relate to teachers produce higher student factor in influencing effectiveness. methodology: achievement outcomes than those Some widely held beliefs about that do not. what constitutes effective profes- Most middle and high school Professional development that is sional development are not sup- mathematics curricula do not tied to curriculum, to knowledge ported by research linked to have studies of student achieve- of subject matter, and/or to how student achievement. ment with comparison groups students learn the subject is more that can be found through litera- ● Myth: Professional development effective in terms of improving ture or web searches. Only three should be distributed over the student achievement than is of the studies found specified the school year. While professional professional development that curriculum to which the target cur- development in science seems focuses only on teaching behav- riculum was being compared. The to benefit from distributed time, iors. A number of studies have rest compared their curriculum to the same effect has not been concluded that the content of pro- some unnamed curriculum, mak- found for math. fessional development—what is ● Myth: More is better. There is no ing comparisons across curricula taught—is more important than its impossible. evidence to support the belief format—how it is taught—and that that more contact time over and If students are going to be judged content should be linked to subject above 80 hours and 160 hours on the results of an external test, matter knowledge, a specific cur- results in changes in teaching the mathematics curriculum riculum, or the process of student behavior or classroom environ- selected should cover the areas learning. For example, some pro- ment, respectively. and skills that are included on fessional development focuses on ● Myth: Classroom visits are neces- that test (i.e., the test and curricu- training teachers to model specific sary. Visits to classrooms for lum should be aligned). As would pedagogical approaches and skills consultation or coaching as part be expected, students tend to score using generic lesson formats. The of professional development do higher on tests focusing on the goal of these efforts is to get teach- not necessarily result in greater content and skills covered in their ers to change the way they teach, student achievement. curriculum. The content and skills expecting that this will increase ● Myth: Schoolwide programs are covered in traditional math cur- student achievement. More effec- more effective. Providing services ricula tend to reflect the content tive professional development to whole schools rather than and skills covered in most stan- focuses on increasing teacher individuals may not be the most dardized and statewide tests. knowledge of the subject matter, important element of profes- Alternatively, the content and on a specific curriculum, or on sional development that leads to skills covered in standards-based the student learning process. The increased student achievement. curricula tend to better reflect that goal of these types of efforts is to change teacher knowledge, ex- which is covered in standards- pecting that this will, in turn, Mathematics and Science based tests. (For more on this both change teacher behavior and Curricula: What We Found issue, see the text box, “Inter- increase student achievement. preting the Results.”) The amount of professional Mathematics Curricula development provided is an Our review netted 89 middle and The following findings relate to important factor in influencing high school curricula, including eight outcomes and impacts: 2
  • 3. What Do We Know? Seeking Effective Math and Science Instruction Studies of six of the curricula ricula and the results of impact stud- more effective than traditional found that students who use the ies, is provided in Appendix B. science curricula as measured by curriculum being tested scored The following findings relate to student achievement. The prepon- higher than comparison students methodology: derance of evidence provided by on both a majority of standard- meta-analyses and evaluations of ized and/or state tests used and As with mathematics curricula, individual curricula seems to con- on a majority of the curriculum- most middle and high school cur- firm that inquiry-based science based tests used. These curricula ricula do not have evaluation curricula produce larger effects were Cognitive Tutor, Connected studies of student achievement on student achievement than do Mathematics (CMP), Interactive with comparison groups that can the more “traditional” science Mathematics Program (IMP), be found through published liter- curricula.6 Prentice Hall: Tools for Success, ature or web searches. However, It is difficult to determine the Saxon Math: An Incremental in contrast to the mathematics cur- effect of these science curricula Development, and University of ricula in Table 1 in Appendix A on different subgroups of stu- Chicago School Mathematics (many of which had been the sub- dents—such as girls, minority Project. Because students using jects of multiple studies), most of group members, and urban stu- these curricula scored higher on the science curricula in Table 2 in dents. Very few of the curricula both types of tests, it is reasonable Appendix B had only one eval- had studies that met our criteria to assume that the effectiveness of uation or study, usually unpub- and disaggregated their findings each curriculum is not an effect of lished works available through the by sex, language minority status, developer. or urban location. Surprisingly, the type of test used. none of the studies reported data It is difficult to determine any The following findings relate to disaggregated by race/ethnicity. differential effect of these math outcomes and impacts: Most science curricula include a curricula on girls and boys. Very professional development compo- few of the curricula had studies Science curricula based on the nent. Inclusion of a professional that met our criteria and disaggre- inquiry approach are consistently development component as a part gated their findings by sex. Among those that did, the results were inconsistent. With the exception of Connected Mathematics, too few results per Interpreting the Results: Validity of Tests Used curriculum were broken out by To Measure Student Achievement race/ethnicity to allow us to draw The evaluation studies for mathematics and science curricula on our lists general conclusions regarding used a variety of assessments—some standardized, some state-mandated, racial/ethnic effects. Connected some self-developed, and some developed by others specifically for stan- Mathematics, however, appeared dards-based curricula. In the case of mathematics curriculum studies, the to reduce racial/ethnic gaps. content and skills covered by most standardized achievement tests tend to reflect more closely the content and skills covered by traditional mathe- Science Curricula matics curriculum than those covered by standards-based curriculum, causing them to have better “content validity” for traditional curriculum. We identified 80 science curricula at Many researchers of standards-based curriculum are aware of this and the middle and high school levels.5 develop their own student achievement tests that more accurately test the Similar to the mathematics cur- skills and content of standards-based curriculum. riculum, we found seven science If students taking one curriculum score higher than others on both curricula that had been developed types of test, there is no question of interpretation. Beyond that, however, specifically as a part of whole school judgments of efficacy must take into account the content validity of the reform. A total of 45 studies of stu- tests in order to determine which type of curriculum is more effective. dent achievement in science were Although science curriculum studies face a similar dilemma, several stud- found that met our criteria, covering ies comparing the results of standardized tests with those of other forms of 21 (or 26 percent) of the total science assessment have found very small differences. The issue of content valid- curricula identified. The 21 curricula ity, therefore, seems to be less of a concern in judging the effectiveness of are listed in Table 2, which together science curricula. with brief descriptions of these cur- 3
  • 4. What Do We Know? Seeking Effective Math and Science Instruction of the curriculum is far more com- mines what students will learn. Once an enormous task and an enormous mon in science than in mathemat- a decision is made, selection should responsibility. We hope that our ics because science curricula tend be guided by how effective a par- review can provide some guidance to be more discretionary and vari- ticular curriculum is for the student and assistance in the process. able than mathematics curricula. population to be taught. The National Science curriculum developers, Research Council report, On Eval- therefore, may feel the need to pro- Endnotes uating Curricular Effectiveness: Judging vide more guidance to teachers the Quality of K–12 Mathematics 1. The majority of these efforts have been undertaken by AAAS, the National Research about the appropriate instructional Evaluations (2004) describes the value Council (NRC), and the U.S. Department of approaches to be used for specific of this knowledge to the decision- Education. The AAAS study used content curricula. making process: analysis, the NRC study did not rate specific curricular math programs, and the U.S. Department of Education study reviewed Clearly, knowing how effective a middle school math programs only. What Can We Conclude particular curriculum is, and for 2. As determined by quantifiably measurable whom and under what conditions it from Our Review? outcomes, such as test scores. is effective, represents a valuable The major conclusions of our review and irreplaceable source of informa- 3. The full report, on which this summary tion to decisionmakers, whether report is based and which contains refer- that should be most useful to those they are classroom teachers, par- ences and a bibliography of all studies wishing to invest in sustainable described in this report, can be found on ents, district curriculum specialists, school reform in science and mathe- school boards, state adoption the Urban Institute web site, http://www. matics can be summarized as follows: boards, curriculum writers and evaluators, or national policymak- 4. “Teacher behavior” in this context refers Effective mathematics curricula in ers. Evaluation studies can provide to teachers’ use of specific pedagogical middle and high school may be that information but only if those approaches in instruction, such as inquiry- either traditional or integrative evaluations meet standards of qual- based teaching practices. “Change in the ity (p.1).7 classroom environment” refers to teacher (standards-based). facilitation of an investigative classroom cul- Effective science curricula in ture through seating arrangements to stimu- middle and high school should This review of math and science late discussion, use of cooperative learning be inquiry-based rather than curricula has tried to simplify for groups, encouraging students to explain schools and districts the complex, concepts to one another, and other such traditional. practices. Effective professional develop- time-consuming process of determin- ment programs are those that ing curriculum effectiveness by iden- 5. Of the 80 science curricula identified, 59 had tifying programs that have what we no evaluations or had evaluations that did focus on content rather than for- not meet our criteria, or evaluations were mat and that have the following consider to be credible evaluations. out of print, or we did not receive a re- features: We have also distilled the findings of sponse from the developers. achievement-based research on pro- 6. As measured by both standardized tests and ● Content tied to curriculum, fessional development to a handful of tests developed for standards-based curric- knowledge of subject matter, principles that reflect effective prac- ula in both meta-analyses and evaluations of and/or how students learn a tice. Once schools and districts have individual curricula. subject; decided on a curriculum and an 7. Jere Confrey and Vicki Stohl, eds. 2004. On ● A minimum of 80 contact hours appropriate assessment tool, they Evaluating Curricular Effectiveness: Judging the to effect changes in teachers’ might wish to collect their own Quality of K–12 Mathematics Evaluations. impact data to evaluate how well the Committee for a Review of the Evaluation instructional behaviors; and Data on the Effectiveness of NSF-Supported ● A minimum of 160 contact curriculum they choose is working and Commercially Generated Mathematics hours to effect changes in the with their own students.8 It is the Curriculum Materials. National Research classroom environment. responsibility of the school and dis- Council. Washington, DC: National Acad- trict community to ensure that the emy Press. content that they want students to 8. For those schools and districts that already How to Use This Review learn is embodied in the curriculum, have programs that they feel are effective Choice of a curriculum in mathe- that the curriculum is effective for but are not included in Tables 1 and 2, we matics and science involves deciding suggest that they collect their own effective- this purpose, and that appropriate ness data via evaluation studies that adhere what aspects of these subjects are measures are used to assess whether to the criteria established in this review. important to address and emphasize students are indeed learning what the They may wish to contract with an evalua- in schools—this choice thus deter- community wants them to learn. It is tor for this purpose. 4
  • 5. What Do We Know? Seeking Effective Math and Science Instruction Appendix A DETAILED FINDINGS OF THE REVIEW OF MATH CURRICULA This appendix provides a list of the mathematics curricula included in our review, together with a descrip- tion of each curriculum and its related evaluation studies. Table 1 lists the mathematics curricula identified as having evaluation studies that met our criteria. TABLE 1. Math Curricula with Studies Grades Covered Curriculum Name Subject Matter K 12 Edison Schools Mathematics K 8 Direct Instruction Mathematics middle school Cognitive Tutor *[sex, race/ethnicity] Mathematics middle school Connected Mathematics (CMP) *[sex, race/ethnicity] Mathematics middle school Integrated Mathematics, Science, and Technology (IMaST) Mathematics middle school Mathematics in Context (MiC) *[race/ethnicity] Mathematics 6 8 MATHThematics Mathematics middle school Prentice Hall: Tools for Success Mathematics middle school Saxon Math: An Incremental Development Mathematics 7 12 Mathematics with Meaning *[sex, race/ethnicity] Mathematics 7 12 University of Chicago School Mathematics Project (UCSMP) Mathematics 9 12 College Preparatory Mathematics (CPM) Mathematics 9 12 Contemporary Mathematics in Context: A Unified Approach (Core-Plus Mathematics Project CPMP) Mathematics 9 12 Interactive Mathematics Program (IMP) *[sex] Mathematics 9 12 Math Connections *[sex, race/ethnicity] Mathematics 9 12 Mathematics: Modeling Our World (MMOW/ARISE) Mathematics 9 12 Systemic Initiative for Montana Mathematics and Science (SIMMS) Mathematics 11-12 Advanced Placement (AP) Calculus Calculus Note: Shaded curricula are those for which we have found the strongest evidence of effectiveness, that is, quantitative evidence that their use in instruction elicits higher achieve- ment/performance in students than other curricula to which they are compared on both standardized and/or state tests and on curriculum developed tests. There are several curricula for which this evidence of effectiveness has not been collected but which might also qualify as effective should appropriate studies be conducted. An omission from this list of many curricula signifies merely that these curricula have not yet provided quantitative evidence of effectiveness that meets our criteria. *An asterisk marks curricula for which effectiveness data are provided for subgroups of students, indicated in brackets. (See full report on the web for discussion of research find- ings by subgroup.) 5
  • 6. What Do We Know? Seeking Effective Math and Science Instruction A Description of the Mathematics Curricula ity, and differential and integral calculus, with Calculus BC and Evaluation Studies Found covering these topics more extensively than Calculus AB. Prerequisites include knowledge of analytic geometry and In addition to a description of each curriculum, these elementary functions in addition to college preparatory overviews (presented in alphabetical order) include the algebra, geometry, and trigonometry. Enrolled students are type of student achievement measure used, the number expected to take the Advanced Placement examination in and direction of the results, and, if available, the size of Calculus AB or BC. any differences between groups. If the results were broken out by sex and/or race and ethnicity, this too is indicated. Contact The types of measures used in the impact studies are College Board Headquarters broken into five categories: 45 Columbus Avenue New York, NY 10023-6992 Standardized achievement tests (i.e., PSAT, SAT, SAT-9, Tel: 212-713-8066 and Iowa Test of Basic Skills) Web site:,3045, State-mandated achievement tests (i.e., FCAS, MEAP, 151-165-0-2178,00.html and MCAS) Standards-based, curriculum-driven measures (i.e., Results Balanced Assessment, Problem Situation Test) Teacher-based measures (i.e., GPA, school tests, and Five results were found from studies on the effect of AP mathematics courses taken) Calculus courses. These studies compared college mathe- matics performance of students who took AP Calculus Percent passing different mathematics courses with other students who did not. In the charts that follow within this appendix, results were tracked rather than the number of studies. Multiple Number grades and multiple measures were counted as multiple Type of measure of results Results Effect size results and broken out as such. For example, a study that Teacher-based 3 AP students scored higher No effect looked at the impact of different subsets of a curriculum measures/GPA in two results; there were size reported for sixth, seventh, and eighth grades was counted as three no differences in one results. Cohort studies that tracked students across multi- result. ple grades were counted as one result. Also counted as one Pass rates/ 2 AP students scored higher No effect result were findings from different sites within the same courses taken in one result, and there size reported study using the same measure. If a majority of sites had were no differences in changes favoring the tested curriculum, the result was one result. indicated as positive. If the majority of sites did not differ from the comparison, the result was indicated as no Results were not reported by sex, race, or ethnicity. change. If a majority of sites had changes favoring the comparison curriculum, the result was indicated as negative. Cognitive Tutor Note: Most impact studies for math curricula reported the statistical sig- The Cognitive Tutor (CT), from Carnegie Learning, nificance of their results. Only differences that have reached the conserva- includes full curricula in Algebra I, Geometry, Algebra II, tive minimum acceptable statistical significance level of .05 were included in the results reported for each study. If differences are statistically signifi- an Integrated Math Series, and Quantitative Literacy cant, then there is another measure, called an effect size, which shows Through Algebra. Each curriculum combines software- how big the difference is. In our description of the study results, we pro- based, individualized computer lessons with collaborative, vide effect sizes where available, although very few studies reported real-world problem-solving activities. Students spend these results. Effect sizes greater than .4 are considered large; between .2 about 40 percent of their class time using the software and and .4 are considered moderate; and less than .2 are considered small. the remainder of their time engaged in classroom problem- solving activities. Contact Advanced Placement (AP) Calculus Carnegie Learning, Inc. The AP Calculus curriculum includes two courses: AP 1200 Penn Avenue Calculus AB, which is comparable to one semester of col- Suite 150 lege-level calculus, and AP Calculus BC, which is com- Pittsburgh, PA 15222 parable to two semesters of college-level calculus. Both Tel: 888-851-7094 courses include elementary functions, limits and continu- E-mail: 6
  • 7. What Do We Know? Seeking Effective Math and Science Instruction Results Results Twenty-one results from studies on the effect of CT were Six results from studies on the effect of CPM were found, found, all of which focused on Algebra I at the middle and one of which focused on algebra and five of which focused high school level. on grade 9–11 math. Number Number Type of measure of results Results Effect size Type of measure of results Results Effect size Standardized 8 CT students scored higher Moderate Standardized 5 There were no differences No effect achievement in all eight results. effect size achievement in all five results. size reported tests in one result tests Statewide tests 5 CT students scored higher No effect Teacher- 1 There were no differences. No effect in three results; there size reported developed size reported were no differences in measure two results. Results were not reported by sex, race, or ethnicity. Curriculum-driven, 7 CT students scored higher Large effect skill-specific tests in seven results. sizes in two results Passing rates 1 CT students had higher No effect Connected Mathematics (CMP) passing rates. size reported The Connected Mathematics (CMP) curriculum is com- posed of eight models, each focusing on one important Two results looked at sex differences; in one result, CT area of mathematics and emphasizing previously learned boys’ scores were higher than comparison boys. There content. Connected Mathematics is designed to develop were no differences between boys in one result, and there students’ knowledge and understanding of mathematics were no differences for the girls’ scores in either result. through attention to connections between mathematical Two results reported differences by race and ethnicity. In ideas and their applications in the world outside school; one result, African-American CT students had higher among the core ideas in mathematics; among the strands scores than comparison African-American students, and in a modern mathematics curriculum; and between the there were no differences for Hispanic students. In the planned teaching-learning activities and the special apti- second result, Hispanic CT students scored higher than tudes and interests of middle school students. comparison Hispanic students. Contact Pearson Education College Preparatory Mathematics (CPM) P.O. Box 2500 The CPM series offers a four-year integrated curriculum Lebanon, IN 46052-3009 where mathematics topics are revisited and built upon Tel: 800-848-9500 through the years. Problem-solving strategies are empha- Web site: sized as a vehicle for learning mathematics, and student study teams are an integral part of the learning process. Results Based on the belief that concept mastery requires time, the curriculum spirals through practice of the main course Thirty-four results from comparison studies looking at concepts throughout each year and emphasizes students’ the effect of CMP were found focusing on middle grade supportive group work. The program sees the teacher’s mathematics. role as a guide. Number Contact Type of measure of results Results Effect size CPM Business Office Standardized 11 CMP students scored No effect 1233 Noonan Drive achievement higher for six results; they size reported Sacramento, CA 95822 tests scored lower for one result; Tel: 916-681-3611 and there were no differ- Web site: ences for four results. (continued) 7
  • 8. What Do We Know? Seeking Effective Math and Science Instruction Number Number Type of measure of results Results Effect size Type of measure of results Results Effect size Statewide 16 CMP students scored No effect Standardized 43 In 19 of the results, CPMP No effect tests higher for 14 results; there size reported achievement students scored higher; size reported were no differences for tests in 24 results, there were two results. no differences. Curriculum-driven, 6 CMP students scored Large effect Statewide 5 Students scored higher in No effect skill-specific tests higher for all six results. sizes were tests all five results. size reported found for Curriculum-driven, 2 Students scored higher in No effect one result. skill-specific tests both results. size reported Teacher-based 1 CMP students scored No effect Teacher-based 6 CPMP students scored No effect measure higher in one result. size reported measure/GPA higher in one result, no size reported differently in another, and lower in four results. Seven results were broken out by race/ethnicity and two by sex. No significant sex differences were found. African-American CMP students were found to score Results were not reported by sex, race, or ethnicity. higher than African-American comparison students in six results, while Hispanic CMP students scored higher than Hispanic comparison students in four results; in a fifth result, there were no differences. African-American and Hispanic CMP students showed greater gains than others Direct Instruction in two results, while African Americans alone showed greater gains in one result. In one result, Native-American In Direct Instruction (DI), each program is fully scripted, CMP student performance decreased. from what the teacher says and anticipated student responses, to correctional procedures. Each skill is broken down into its component parts, and then each component of the skill is taught to mastery. Afterward, the skills are Contemporary Mathematics in Context: combined within a larger context where they may be uti- lized across settings, resulting in generalized fluency. The A Unified Approach DI mathematics curriculum covers grades K–8, but focuses (Core PLUS Mathematics CPMP) primarily on grades K–6. The mathematics curriculum Core-Plus Mathematics consists of a single core sequence covers 19 different topics ranging from addition and sub- for both college-bound and employment-bound students traction to money, mathematics study skills (graphs, during the first three years of high school. A flexible fourth charts, maps, and statistics), and geometry. year course can be used to prepare students for college mathematics. Contact Contact The McGraw-Hill Companies P.O. Box 182604 Core-Plus Mathematics Project Columbus, OH 43272 Department of Mathematics Tel: 888-772-4543 Western Michigan University Web site: Kalamazoo, MI 49008-5248 curriculumsolutions/di/connectingmath/114 Tel: 866-407-CPMP (2767) E-mail: Results Results Nine results from comparison studies on the effect of DI Fifty-six results from comparison studies on the effect of were found which focus on seventh and eighth grade CPMP were found focusing on high school math. math. 8
  • 9. What Do We Know? Seeking Effective Math and Science Instruction Number Integrated Mathematics, Science, Type of measure of results Results Effect size and Technology (IMaST) Standardized 6 DI students scored higher No effect The Integrated Mathematics, Science, and Technology pro- achievement for four results; they size reported tests scored lower for two gram provides integrated sixth, seventh, and eighth grade results. curricula that promote hands-on learning for students and teamwork among teachers from different disciplines. State test 3 In all three results, DI No effect students scored higher. size reported IMaST emphasizes learning based on constructivist theory and active student participation involving a hands-on Results were not reported by sex, race, or ethnicity. approach comprising a wide variety of activities. Edison Schools Contact Founded in 1992, Edison Schools is focused on raising stu- Ronjon Publishing, Inc. dent achievement through research-based school design, 1001 S. Mayhill Rd. aligned assessment systems, interactive professional devel- Denton, TX 76208 opment, integrated use of technology, and other program Tel: 800-262-3060 features, including a longer school day and school year. The Web site: math curriculum for grades 6–8 includes applied arithmetic, prealgebra, and pregeometry, using a spiral curriculum approach to teach concepts and ideas. The mathematics in grades 9–10 provides three years of high school math in two Results years’ time, using an integrated application-based approach to algebra, geometry, and trigonometry with additional One study was found looking at the effect of IMaST focus- emphasis on probability, statistics, and discrete mathemat- ing on seventh and eighth grade students. ics. Grades 11 and 12 incorporate advanced mathematical modeling, Calculus and Statistics, Algebra 2, and Number Precalculus, as well as AP Statistics and Calculus. Type of measure of results Results Effect size Contact Standardized 1 IMaST students scored No effect 521 Fifth Avenue, 11th Floor achievement higher for the one result. size reported New York, NY 10175 tests Tel: 212-419-1600 Web site: Results were not reported by sex, race, or ethnicity. Results Seventeen results from comparison studies on the effect of Edison Schools were found; fifteen focus on middle school math and two focus on high school math. Interactive Mathematics Program (IMP) Number Type of measure of results Results Effect size This four year, problem-based curriculum incorporates tra- Standardized 8 For four results, Edison No effect ditional branches of mathematics (algebra, geometry, and achievement students scored higher; size reported trigonometry) with additional topics recommended by the tests in the remaining four, there NCTM Standards, such as statistics, probability, curve fit- were no differences. ting, and matrix algebra. Students are encouraged to State test 9 For two results, Edison No effect experiment, investigate, ask questions, make and test con- students scored higher; size reported jectures, reflect, and accurately communicate their ideas for four results, Edison students scored lower; and conclusions. Although each unit has a specific mathe- and for three results, there matical focus, other topics are brought in as needed to were no differences. solve the central problem. Ideas that are developed in one unit are usually revisited and deepened in one or more Results were not reported by sex, race, or ethnicity. later units. Algebra and geometry are distributed through- out the four years. 9
  • 10. What Do We Know? Seeking Effective Math and Science Instruction Contact Results Key Curriculum Press Fifteen results from comparison studies were found on the 1150 65th Street effect of MC, all of which focused on high school students. Emeryville, CA 94608 Tel: 800-995-MATH (6824) Number Web site: Type of measure of results Results Effect size textbooks/Prod_IMP.html Standardized 6 MC students scored No effect Results achievement higher for three results; for size reported tests three results, there were Twenty-two results from comparison studies were found were no differences. on the effect of IMP, all focusing on high school. State test 8 MC students scored There was a Number higher for all eight results. large effect Type of measure of results Results Effect size size for three results. Standardized 12 IMP students scored No effect achievement higher for eight results; size reported Curriculum-driven, 1 No difference was found. N/A tests there were no differences skill-specific test for four results. Pass rates or 5 In all five results, IMP No effect One result looked at sex, race, and ethnic differences courses taken students had higher pass- size reported and found no differences within MC students. ing rates and/or were more likely to take more mathe- matics courses. Curriculum-driven, 3 IMP students scored Large effect skill-specific tests higher for all three results. sizes were Mathematics in Context (MiC) found in the three results Mathematics in Context is a four-year middle school cur- Teacher-based 2 In both results, IMP No effect riculum (grades 5–8) that encourages students to discover measure students scored higher. size reported mathematical concepts and skills through engaging prob- lems and meaningful contexts. Each year includes lessons Results were not reported by race or ethnicity. The one in the four strands (numbers, algebra, geometry and statis- result reported by sex found IMP girls were slightly more tics, and probability) that are interwoven through 10 units. apt to continue three or more years in math than IMP boys, For example, sample algebra units include Patterns and while the reverse was the case for comparison students. Symbols (grade 5), Expressions and Formulas (grade 6), Ups and Downs (grade 7), and Graphing Equations (grade 8). MATH Connections Contact MATH Connections (MC) is an integrated curriculum that blends ideas from traditionally separate mathematical Holt, Rinehart and Winston fields (e.g., algebra, geometry, statistics, and discrete math- Attn: Ms. Web1 ematics) in ways that blur the lines between them. This 10801 N. MoPac Expressway three year curriculum replaces the traditional Algebra I, Building 3 Geometry, Algebra II sequence and is designed for all stu- Austin, TX 78759 dents in grades 9, 10, and 11, with honors students begin- Tel: 800-HRW-9799 (800-479-9799) ning the curriculum in grade 8. Web sites: Contact 750 Old Main Street Suite 303 Results Rocky Hill, CT 06067-1567 Tel: 860-721-7010 Twenty results from comparison studies were found on the Web site: effect of MiC, which focused on middle school students. 10
  • 11. What Do We Know? Seeking Effective Math and Science Instruction Number Number Type of measure of results Results Effect size Type of measure of results Results Effect size Standardized 17 In 15 results, MiC In two Standardized 4 For three results, MMOW No effect achievement students scored higher; in results, a achievement students scored higher; size reported tests two results, there were no majority of tests for one result, there were differences. classes no differences. showed at least moder- Results were not reported by sex, race, or ethnicity. ate effect sizes. State test 3 MiC students scored No effect Mathematics with Meaning higher for all three results. size reported Mathematics with Meaning (MwM) is not a complete cur- ricular program; rather, it is a combination of professional One result was broken out by race. No statistically sig- development, instructional strategies, and carefully nificant difference was found between African-American planned materials designed to alter the pedagogy and students using MiC and comparison students. content of middle school and high school mathematics courses in order to improve student achievement. The pro- gram consists of instructional units that teachers may use on a supplementary basis or as their entire instructional program. MwM takes a student-centered approach based Mathematics: Modeling Our World on exploratory learning and problem solving, focusing on (MMOW/ARISE) developing conceptual understanding, connections, and communication with mathematical concepts through fre- In Mathematics: Modeling Our World (MMOW), students quent group work and hands-on activities. are taught to use a variety of resources to solve problems and to choose resources that meet the needs of a particular Contact situation. As in real life, MMOW’s problems do not neces- College Board sarily have perfect solutions. MMOW works to strengthen Dept CBO the students’ ability to solve problems by setting goals and P.O. Box 869010 thinking strategically about how to achieve these goals, Plano, TX 75074 solving problems through trial and error and/or process Tel: 212-713-8260 of elimination, using technology like calculators and com- Tel: 800-323-7155 puters, and working together to solve semi-structured E-mail: problems and communicating the solutions. Web site: Contact Results W.H. Freeman and Company Eleven results from comparison studies on the effect of 41 Madison Avenue MwM were found; three focused on middle school math New York, NY 10010 achievement and eight on high school math achievement. Tel: 800-446-8923 Web sites: Number Type of measure of results Results Effect size contact_hs_rep.asp Statewide tests 11 In six results, there were No effect introduction.htm no differences; in five reported results, MwM students scored higher. Results Six results were broken out by sex, four by race. In the Four results related to MMOW were found, one focusing six results where sex differences were given, boys slightly on middle school students and three focusing on high outperformed girls. In four results where race/ethnic dif- school students. ferences were given, achievement scores were significantly 11
  • 12. What Do We Know? Seeking Effective Math and Science Instruction lower for African-American students than other students school level. Each lesson includes a Think and Discuss sec- in both MwM and comparison groups. tion that presents the new material along with questions to get students actively thinking about and discussing impor- tant concepts. Textbooks in this curriculum include Work Together activities that allow students to work in groups, Middle Grades MATHThematics often doing hands-on activities to reinforce math topics. Middle Grades MATHThematics (STEM) is a three-year curriculum designed for use in grades 6–8. Four unifying Contact concepts—Proportional Reasoning, Multiple Represen- Pearson Education tations, Patterns and Generalizations, and Modeling—are P.O. Box 2500 used across the three years with seven content strands: Lebanon, IN 46052-3009 Number, Measurement, Geometry, Statistics, Probability, Tel: 800-848-9500 Algebra, and Discrete Mathematics. Web site: Contact Results McDougal Littell Customer Service Center Eleven results from comparison studies on the effect of A Houghton Mifflin Company PHM were found. 1900 S. Batavia Geneva IL 60134 Number Tel: 617-351-5326 Type of measure of results Results Effect size Tel: 800-462-6595 Web sites: Standardized 8 PHM students scored Moderate achievement higher in five results; in effect sizes tests three results, there in one result were no differences. Results Statewide tests 2 PHM students scored No effect Six results from comparison studies were found on the higher in both results. size reported effect of STEM. Curriculum-driven, 1 PHM students scored No effect skill-specific tests higher in this result. size reported Number Type of measure of results Results Effect size Results were not reported by sex, race, or ethnicity. Standardized 2 In one result, STEM stu- No effect achievement dents scored higher; in size reported tests the second, there were no differences. Saxon Math: An Incremental Development Statewide tests 2 In one result, STEM stu- No effect Saxon Math (SM) is a K–12 curriculum that systematically dents scored higher; in size reported distributes instruction, practice, and assessment through- the second, there were no out the academic year rather than concentrating concepts differences. in a single unit or chapter. Each increment builds upon the Curriculum-driven, 2 In both results, STEM stu- No effect foundation of earlier increments, to lead students toward skill-specific tests dents scored higher. size reported a deeper understanding of mathematical concepts. Instruction of related concepts is spread throughout the Results were not reported by sex, race, or ethnicity. grade level, ensuring that students have an opportunity to master each concept before they are introduced to the next one. Prentice Hall: Tools for Success Contact In Prentice Hall Math (PHM), various mathematical Saxon Publishers strands (such as number sense, algebra, geometry, mea- 2600 John Saxon Blvd. surement, data analysis, and problem solving) are inte- Norman, OK 73071 grated throughout the series to ensure that students are Tel: 800-284-7019 prepared for subsequent mathematics courses at the high E-mail: 12
  • 13. What Do We Know? Seeking Effective Math and Science Instruction Results Number Type of measure of results Results Effect size A total of eight results from comparison studies were found on the impact of SM, focusing on both middle and Standardized 6 No differences were found No effect high school. achievement in the six results. size reported tests Curriculum-driven, 6 In one result, SIMMS stu- No effect Number skill-specific tests dents scored higher; no size reported Type of measure of results Results Effect size differences were found in Standardized 3 SM students scored No effect the other five. achievement higher in all three results. size reported tests Results were not reported by sex, race, or ethnicity. Statewide tests 3 SM students scored No effect higher in all three results. size reported Curriculum-driven, 1 SM students scored No effect University of Chicago School Mathematics skill-specific tests higher in this result. size reported Project (UCSMP) Teacher-based 1 SM students scored No effect The UCSMP secondary curriculum consists of six courses: measure/GPA higher in this result. size reported Transitional Mathematics; Algebra; Geometry; Advanced Algebra; Functions, Statistics, and Trigonometry; and Pre- Results were not reported by sex, race, or ethnicity. Calculus and Discrete Mathematics. Transitional Mathe- matics, which was originally designed for average to above average seventh graders (but can be started earlier or later), weaves together three more or less equal strands of major content: applied arithmetic, prealgebra, and elementary geometry. Algebra, geometry, and some discrete mathematics Systemic Initiative for Montana are integrated into all courses, as are statistics and probability. Mathematics and Science (SIMMS) Contact The SIMMS curriculum is divided into six levels, each UCSMP consisting of one year of work. Level 1 is typically offered 5835 South Kimbark Avenue to ninth graders, followed by level 2 in grade 10. After Chicago, IL 60637 completing level 2, students may choose between levels 3 Tel: 773-702-1130 and 4 and then proceed to either level 5 or 6 in the subse- Web site: quent year. The sequence for potential math and science majors is 1-2-4-6. Levels 1 and 2 offer basic mathematical Results literacy. Fourteen results were found from comparison studies on the effect of UCSMP. The following UCSMP courses Contact were covered: Transitional Mathematics (2), Algebra (2), Kendall/Hunt Publishing Geometry (5), Advanced Algebra (4), and Pre-Calculus and 4050 Westmark Drive Discrete Mathematics (1). P.O. Box 1840 Number Dubuque, IA 52004-1840 Type of measure of results Results Effect size Tel: 800-542-6657 Standardized 5 In all five results, UCSMP No effect Web sites: achievement students scored higher. size reported tests Curriculum-driven, 8 In all eight results, UCSMP No effect Results skill-specific tests students scored higher. size reported Teacher-based 1 No differences were found. No effect Twelve results from comparison studies were found on the measure/GPA size reported effect of SIMMS. Four results each focused on levels 1 and 2, and two each focused on levels 4 and 6. Results were not reported by sex, race, or ethnicity. 13
  • 14. What Do We Know? Seeking Effective Math and Science Instruction Appendix B DETAILED FINDINGS OF THE REVIEW OF SCIENCE CURRICULA This appendix provides a list of the science curricula included in our review, together with a description of each curriculum and its related evaluation studies. Table 2 below lists the science curricula that we identified as having evaluation studies that met our criteria. TABLE 2. Science Curricula with Studies Grades Covered Curriculum Name Subject Matter K 12 Expeditionary Learning Outward Bound (ELOB) *[LEP] Whole School Reform K 8 Full Option Science System (FOSS) *[LEP] Multi-Science PK 8 Great Explorations in Math and Science (GEMS) Math & Multi-Science K 6 The Science Curriculum Improvement Study (SCIS) Multi-Science 2 8 National Science Curriculum for High Ability Learners Multi-Science 6 8 DESIGNS/DESIGNS II *[sex] Physical Science 6 8 Integrated Math, Science and Technology (IMaST) Math, Science & Technology 6 8 Center for Learning Technologies in Urban Schools (LeTUS) *[sex] Multi-Science 6 8 Learning By Design (LBD) Multi-Science 6 8 Science 2000/Science 2000+ Multi-Science 6 8 Science and Technology Concepts for Middle School (STC/MS) Multi-Science 6 9 Event-Based Science (EBS) Earth Science 78 Constructing Ideas in Physical Science (CIPS) *[sex] Physical Science 7 9 Foundational Approaches in Science Teaching (FAST) Multi-Science 7 9 Global Lab Curriculum (GLC) Multi-Science, Environmental Focus 7 9 Issues, Evidence and You (IEY)/SEPUP Multi-Science 9 11 BSCS: An Inquiry Approach Multi-Science 9 12 High Schools that Work (HSTW) Whole School Reform 9 12 Modeling Instruction in High School Physics *[sex] Physics 9 12 World Watcher/Learning About the Environment (LATE) *[urban] Environmental Science 10 12 Physics Resources and Instructional Strategies for Motivating Students (PRISMS) Physics Note: Shaded curricula are those for which we have found the strongest evidence of effectiveness, that is, quantitative evidence that (1) their use in instruction elicits higher achieve- ment/performance in students than other curricula to which they are compared on both standardized and/or state tests and on curriculum developed tests, or (2) they showed large effect sizes in terms of increasing student achievement. All science curricula listed in Table 2, however, have credible evaluations that show evidence of effectiveness. There are sev- eral curricula for which this evidence of effectiveness has not been collected but which might also qualify as effective should appropriate studies be conducted. An omission from this list of many curricula signifies merely that these curricula have not yet provided quantitative evidence of effectiveness that meets our criteria. *An asterisk marks curricula for which effectiveness data are provided for subgroups of students, indicated in brackets. (See full report on the web for discussion of research find- ings by subgroup.) 14
  • 15. What Do We Know? Seeking Effective Math and Science Instruction A Description of the Science Curricula Number and Evaluation Studies Found Type of measure of results Results Effect size The attached descriptions of science curricula, which Content test, 1 Average student gains at No effect appear in alphabetical order, include the type of student unknown if self- both 9th and 10th grade size reported achievement measure used, the number and direction of designed or levels were between 20 state-issued and 25 percent. the results, and, if available, the size of any differences between groups. When known, effect sizes have been Note: Statistical significance levels were not reported. listed. Where results have been provided by race/ethnic- The field test was conducted across 10 states and ity, sex, or other demographic characteristics, we have re- included students in urban, suburban, and rural schools. ported these. The absence of such notation means that no However, data regarding specific demographic groups data were reported by subgroup. In the case of three cur- was not provided. ricula, descriptions of studies do not follow the normal format because results reported are not best described in that particular format. Center for Learning Technologies Note: Almost all impact studies for science curricula reported the statisti- in Urban Schools (LeTUS) cal significance of their results. Only differences that have reached the conservative minimum acceptable statistical significance level of .05 were LeTUS, developed by researchers at the University of included in the results reported for each study. If differences are statisti- Michigan and Detroit public school teachers, includes pro- cally significant, then there is another measure, called an effect size, that ject-based curriculum materials that build from district, shows how big the difference is. In our description of the study results, we provide effect sizes where available, although few studies reported state, and national standards to support the development these results. Effect sizes greater than .4 are considered large, between .2 of integrated science understanding for middle school stu- and .4 are considered moderate, and less than .2 are considered small. dents. The materials support students’ science learning through engaging them in inquiry about real world prob- lems, providing them with multiple opportunities to work with concepts, and integrating the use of learning tech- BSCS: An Inquiry Approach nologies in instruction. LeTUS is focused on learning This program introduces 9th, 10th, and 11th grade stu- about and developing a new machine to construct large dents to the core concepts in inquiry, the physical sciences, buildings and bridges, an area that has been identified as the life sciences, and the earth–space sciences as articu- of interest to young urban students. lated in the National Science Education Standards. In addi- Contact tion, the curriculum engages students in integration across Joseph Krajcik the disciplines in relevant contexts that explore the stan- School of Education dards related to science in a personal and social perspec- University of Michigan tive. This program provides high school students with an 610 East University Avenue, Rm. 4109 alternative to the traditional sequence of biology, chem- Ann Arbor, MI 48109-1259 istry, and physics. Included with this program is a profes- Tel: 734-647-0597 sional development component designed to help teachers Fax: 734-615-5245 and school districts implement the materials. E-mail: Contact Results Pamela Van Scotter Two evaluations of the LeTUS program were found, one Director that utilized skill-specific instruments and one that used The BSCS Center for Curriculum Development statewide achievement test scores to measure effectiveness. BSCS 5415, Mark Dabling Blvd. Colorado Springs, Colorado 80918 Type of Number Web site: measure of results Results Effect size development|high_school_9-12|An_Inquiry_Approach Curriculum- 1 Significant content and Large effect size for driven,skill- process gains that in- content achieve- Results specific creased with program ment and more Most of the evaluative work on this was of the materials, tests revision and scale-up moderate effects for process skills. not student achievement; however, the results of a nation- wide field test are summarized below. (continued) 15
  • 16. What Do We Know? Seeking Effective Math and Science Instruction CIPS participation did not appear to have closed either Type of Number measure of results Results Effect size sex (male/female) or racial (white/Asian versus non-Asian minority) achievement gaps on either multiple-choice con- Statewide 1 Students who completed Moderate effect tent or process questions or open-ended content items. tests at least one LeTUS unit size for all three in seventh or eighth content areas and grade outperformed both process peers in all content and areas. Moderate DESIGNS/DESIGNS II process categories effects in increas- measured by the test. test passing rates. Project DESIGNS (Doable Engineering Science Inves- tigations Geared for Non-science Students), developed Program- 1 Sixth graders showed Large effect size by the Harvard-Smithsonian Center for Astrophysics, designed significant overall im- for achievement includes design-based activity modules for use in physical measures provement on pre- and gains pre- and science and technology education courses in grades 5–9. post-test measures over post-test The project’s six topics cover chemistry, static forces, elec- a four-year period. tricity and magnetism, potential and kinetic energy, energy transfer, and force, work, power, and torque. Designs II The study looking at differences by sex indicated that resulted from a follow-up effort to develop a full-year participation in at least one LeTUS unit is associated with middle school (grades 7–9) physical science course based an apparent reduction in boy-girl achievement differences on the modules. Project pedagogy was derived from the on statewide examinations. constructivist model of learning and takes into account stu- dents’ personal theories. The project’s goal was to open sci- ence concepts to students through activities involving the Constructing Ideas in Physical Science (CIPS) design, construction, and optimization of simple devices. Constructing Ideas in Physical Science (CIPS) is an Contact inquiry-based, yearlong physical science course that Program overview: attempts to engage seventh or eighth grade middle school Web site: http://cfa- students in constructing meaningful understanding of physical science concepts. The CIPS course is based on the Publisher: themes of interactions and energy transfers between Kendall/Hunt objects. CIPS has five units. Each unit consists of two or Tel: 800-542-6657 three cycles of activities designed to help students develop Web site: physics and chemistry concepts. &PGI=152 Contact Results CIPS Project Number 6475 Alvarado Road, Suite 206 Type of measure of results Results Effect size San Diego, CA 92120 Self-designed 1 Designs II students gained Large Fax: 619-594-1581 test to measure over control students. E-mail: Web site: process skills Publisher: Self-designed 1 Students with initially lower It’s About Time, Inc. tests to measure (same scores gain more. Large Tel: 888-698-TIME (8463) conceptual as knowledge above) Results Analysis by sex for conceptual knowledge shows no Number significant difference in gains. Type of measure of results Results Effect size Multiple-choice 1 CIPS students scored Small content items higher than non-CIPS com- parisons after controlling Event-Based Science (EBS) Multiple-choice for prior knowledge, stu- Small The Event-Based Science (EBS) series is a module-based process items dent demographics, and program designed for students in grades 6–9, with a focus weeks of instruction in on current events. The series has 18 modules designed to Open-ended Physical Science. Moderate last four to six weeks, each focusing on different themes content items and concepts across the domains of earth, life, and physical 16
  • 17. What Do We Know? Seeking Effective Math and Science Instruction sciences. The modules may be sequenced over all middle through projects, fieldwork, and service. Learning expedi- school grade levels and combined with other instructional tions are designed with clear learning goals aligned with materials in order to build a comprehensive middle school district and state standards. Ongoing assessment is woven science program. One or two modules typically are used in throughout each learning expedition. a year, with teachers selecting particular units based on the district’s science standards, the local curriculum program, Contact the interests of the student population, and their own back- Expeditionary Learning ground knowledge in specific topics. EBS is not a “stand 100 Mystery Point Road alone” curriculum. By design, teachers and students sup- Garrison, NY 10524 plement each module with additional data about a specific Tel: 845-424-4000 event from various resources included as part of the mate- E-mail: rials, or suggested by the program. Web site: Contact Results Russ Wright, Project Director Montgomery County Public Schools Only one study of many found for ELOB looked at science 850 Hungerford Drive achievement. Two schools are included in this study, but Rockville, MD 20850 only one includes data for science. Tel: 301-806-7252 Fax: 301-279-3153 Number Type of measure of results Results Effect size E-mail: Web site: Standardized 1 ELOB students showed No effect eventscience/ tests steady gains in science. size reported Results One of the schools in the study experienced an in- One interim study of the impact of EBS was found. This crease of about 22 percent in immigrant students (who study measures the first three years of the EBS project. The were limited English proficient) over the five years report of the final findings from the six-year project is not accounted for in the evaluation. The school showed consis- yet available. tent gains in all subject areas. This same school also has a high number of students qualifying for free and reduced Number Type of measure of results Results Effect size price lunch. The evaluation concludes that ELOB, in this instance, is “particularly successful with a challenging, Curriculum- 1 In two of the three tested Small normally low-achieving population.” driven, years, EBS students out- skills-specific, performed the control group, multiple-choice after controlling for prior test science performance. Foundational Approaches Science attitudes EBS students displayed Small in Science Teaching (FAST) survey more positive attitudes about science than the Foundational Approaches in Science Teaching (FAST) is a control group. three-volume, comprehensive curriculum program based Task-based EBS students outperformed Moderate on a constructivist philosophy of learning in which stu- performance the control group, controlling dents construct their own knowledge through experiential, assessment rubric for prior science performance. hands-on learning. Investigations help students build on existing knowledge and reinforce conceptual understand- ing throughout their work. The continual reinforcement and return to concepts allows students to achieve a deep understanding of the material and to arrive at that under- Expeditionary Learning Outward Bound (ELOB) standing at different points in the curriculum. FAST puts As a whole school reform effort, Expeditionary Learning the teacher in the role of “director of research.” The pro- Outward Bound (ELOB), K–12, organizes curriculum, gram emphasizes an instructional strategy that is based on instruction, assessment, school culture, and school struc- the teacher’s question development practices and other tures around producing high quality student work in techniques that encourage students to think critically. It is learning expeditions. These expeditions are long-term, heavily laboratory-based, with most concepts taught in-depth investigations of themes or topics designed through laboratory experiences where students develop to engage students both in and beyond the classroom skills in measurement and lab procedures. 17
  • 18. What Do We Know? Seeking Effective Math and Science Instruction Contact Phone: 510-642-8941 Fax: 510 642-7387 Donald Young, CRDG Director Email: The University of Hawaii Web sites: The Curriculum Research & Development Group, shtml Science Section 1776 University Avenue, UHS Building 2, Room 202 Results Honolulu, HI 96822-2463 Tel: 808-956-4969 Two studies of FOSS materials met our criteria, though in Fax: 808-956-6730 each of these, use of FOSS was included as only one part E-mail: of a larger curriculum or professional development Web site: intervention. A 2003 evaluation of the NSF-sponsored Project In- Results quiry, a broad professional development intervention Number including the adoption and use of FOSS materials, revealed Type of measure of results Results Effect size that fifth grade students of teachers trained in kit use per- formed significantly higher on both multiple-choice and Laboratory skills 2 FAST students outper- No effect constructed/open-ended response assessments than did the Process skills 1 formed non-FAST compar- size reported control group. Self-reported hours of teacher professional Knowledge skills 1 ison group. development were also positively associated with science Statewide test 1 In one of two years tested, No effect achievement. The authors conclude that professional devel- scores FAST students scored sig- size reported opment is associated with better implementation of inquiry- nificantly higher than the based instruction and greater science topic coverage, which state average, after control- is in turn associated with science achievement. ling for student background. A 2002 study of the use of multiple kit- and inquiry- Standardized 2 FAST students outper- No effect based science materials (including FOSS) in fourth and test scores formed non-FAST students size reported sixth grade classes for English language learners revealed in one study. In the second, a positive relationship between years in the science pro- both FAST and non-FAST gram and standardized science test scores. This relation- groups scored above na- ship remained after controlling for the students’ increasing tional norms. English language proficiency. Global Lab Curriculum (GLC) Full Option Science System for Middle School (FOSS) The Global Lab Curriculum (GLC) was a four-year project to create a science course emphasizing student collabora- The Full Option Science System (FOSS) was developed by tive inquiry. Organized around six units, GLC culminates the Lawrence Hall of Science, University of California– in the design and conduct of original student investiga- Berkeley. Funded by the National Science Foundation, FOSS tions. The structure purposefully provides substantial combines science content and process with a goal of increas- guidance to students in initial investigations and gradu- ing scientific literacy for students and instructional effi- ally peels away the support to allow students to exercise ciency for teachers. The curriculum is organized into topical acquired skills in designing and conducting their own units, called courses, under each of three strands: Earth and collaborative experiments. GLC is also designed to cap- Space Science, Life Science, and Physical Science and italize on the Internet as both a communication and moti- Technology. Each course is an in-depth unit requiring 9 to 12 vational tool that helps establish a cross-cultural science weeks of instruction. The units have approximately community. 10 investigations, each with three to seven parts. The system advocates that students should learn important scientific Contact concepts and develop the ability to think well if they are Harold McWilliams engaged in situations where they actively construct ideas TERC Center for Earth and Space Science Education through their own explorations, applications, and analyses. 2067 Massachusetts Ave. Cambridge, MA 02140 Contact Tel: 617-547-0430 FOSS Project E-mail: Lawrence Hall of Science, University of California Web sites: Berkeley, CA 94720 18
  • 19. What Do We Know? Seeking Effective Math and Science Instruction Results High Schools that Work (HSTW) Number High Schools that Work (HSTW) is a whole-school, Type of measure of results Results Effect size research- and assessment-based reform that offers a frame- Self-designed 1 Students with low levels No effect work of goals and key practices for improving the acade- tests of prior knowledge benefit size reported mic, technical, and intellectual achievement of high school more from GLC than those students. HSTW blends traditional college-preparatory with medium or high levels content with quality technical and vocational studies. of prior knowledge. HSTW provides technical assistance and staff develop- ment focused on techniques and strategies such as team- work, applied learning, and project-based instruction. The HSTW assessment is based on the National Assessment of Great Explorations in Mathematics Educational Progress (NAEP). The developer does not and Science (GEMS) offer specific subject area programs, but consultants pro- Great Explorations in Math and Science (GEMS) is a vide workshops customized to fit an individual school’s supplemental enrichment program for students from needs. preschool through eighth grade. GEMS provides teachers with more than 70 teacher’s guides, support documents, Contact and pedagogical handbooks; professional development Gene Bottoms, Senior Vice President opportunities; an active web site; and a national support net- Southern Regional Education Board work of GEMS leaders and associates and over 45 regional 592 10th St., NW sites. GEMS uses generally accessible materials that inte- Atlanta, GA 30318 grate science and mathematics. The program’s units, pre- Tel: 404-875-9211, ext. 249 sented as flexible enhancements or in curriculum sequence, E-mail: are designed to help all teachers reach all students and fea- Web site: ture clear, step-by-step teacher instructions. In addition to the specific standards-based learning goals and content, the Results program emphasizes cooperative learning and problem- Two studies were conducted using the same measure. solving, literature and language arts connections, and real world relevance. GEMS units feature an inquiry-based, Number guided-discovery, student-centered approach to learning. Type of measure of results Results Effect size An assessment component is in place for the entire series. HSTW 1 Gains in achievement No effect Contact assessment increase with the number size reported Jacqueline Barber of students in a school Lawrence Hall of Science #5200 completing the HSTW University of California, Berkeley curriculum. 1 Centennial Drive, Berkeley, CA 94720-5200 Tel: 510-642-7771 HSTW 1 Over time, more students No effect Fax: 510-643-0309 assessment met achievement goals size reported Contact e-mail: and completed program. Program e-mail: Web site: Results Several studies of the GEMS units, which serve grades Integrated Mathematics, Science, K–8, have been produced. Only one of these met our crite- and Technology Curriculum (IMaST) ria and focused on the fourth through eighth grade astron- Integrated Mathematics, Science, and Technology Cur- omy unit Earth, Moon, and Stars. riculum (IMaST) was developed by the Center for Math- Number ematics, Science, and Technology at Illinois State University. Type of measure of results Results Effect size IMaST is a standards-based, integrated curriculum for Content-specific, 3 Students receiving GEMS No effect grades 6–8. IMaST integrates technology, science, and math- skills-driven test instruction had signifi- size reported ematics and includes connections to the language arts and cantly higher pre–post social sciences, as well as readings that profile typical gains than control groups careers related to the curriculum content. The curriculum is without GEMS instruction. based on the constructivist learning theory that allows con- cept development to take place in a structured venue. 19
  • 20. What Do We Know? Seeking Effective Math and Science Instruction Contact 17 Colt Court Ronkonkoma, NY 11779 Dr. Karen Lind, Director Tel: 800-381-8003 Center for Mathematics, Science, and Technology Fax: 631-737-1286 Illinois State University E-mail: Campus Box 5960 Normal, IL 61790-5960 Tel: 309-438-3089 Results Fax: 309-438-3592 An assessment of the pilot implementation of IEY was con- E-mail: ducted as an experimental study using pre- and post-tests Web site: of student knowledge for two groups: one taught using imast.shtml IEY and another using the traditional science curriculum. The study spanned 15 multi-school sites in 12 states, cov- Results ering 26 classrooms and a total of 830 students. Student achievement was measured by the assessments from the Number Type of measure of results Results Effect size IEY program, which consisted of multiple choice and short answer items. TIMMS sub-test 1 IMaST students outper- No effect form traditional peers, size reported especially regarding Number science processes. Type of measure of results Results Effect size Curriculum-driven, 1 IEY students experienced Moderate skill-specific tests an increase in ability to present an evidence-based Issues, Evidence, and You (IEY)/SEPUP line of reasoning, while comparison classes did The University of California–Berkeley Lawrence Hall of not experience significant Science developed the SEPUP IEY curriculum for middle improvement in this skill. school and junior high use with support from the National Science Foundation. Issues, Evidence, and You (IEY) focuses on environmental issues in a social context. The program builds upon earlier SEPUP modules and is designed to address students’ increasing ability to think Learning by Design (LBD) abstractly and builds upon students’ need for peer inter- action and support. The developer intended the curricu- Developed by Georgia Tech’s EduTech Institute, Learning lum to serve as the physical science component of an by Design (LBD) is an approach to science learning in integrated science program (physical, life, and earth sci- which middle school students learn as a result of collab- ence) or as a year-long physical science program. The oratively engaging in design activities and reflecting course consists of 65 activities or investigations presented appropriately on their experiences. Scaffolding is a key in a conceptual sequence. The instructional times of the component of LBD; interventions combine teacher facili- activities vary from one to three class periods. The cur- tation, paper-and-pencil design diaries and other paper riculum can accompany Science and Life Issues (SALI), prompts, and software tools and prompts. LBD has units another SEPUP program now in the piloting phase. The in both physical and earth science. program also integrates student assessment into the curriculum, providing teachers with a basis for under- Contact standing gaps in student knowledge of core scientific concepts and a plan for addressing such gaps as they are The EduTech Institute identified. Georgia Institute of Technology 801 Atlantic Drive Contact Atlanta, Georgia, 30332-0280 Web site: Tel: 404-894-3807 Publisher: Web site: LAB-AIDS lbdview.html 20
  • 21. What Do We Know? Seeking Effective Math and Science Instruction Results Results Number Four evaluations of the impact of Modeling Instruction on Type of measure of results Results Effect size student achievement were found. Self-designed 1 LBD gains in content learn- No effect Number test incorporat- ing higher than comparison size reported Type of measure of results Results Effect size ing some students. Typical LBD stu- standardized test dents do as well or better Tests of alternate 2 Average pre–post test gains No effect items (NAEP, than honors-level non-LBD methods of phys- in Modeling Instruction size TIMSS) students. ics instruction classrooms were double reported (Comparison of average those in traditional class- Performance 2 Typical LBD students score No effect classroom scores be- tween teachers using rooms, and 10 percentage assessments higher than typical com- size reported Modeling Instruction, points higher than those in panion group students in traditional methods, and reform methods) reform-method classrooms. applying collaborative science skills and prac- Tests of alternate 2 Post-test scores after a No effect tices. Typical LBD student’s methods of phys- teacher is trained in Modeling size performance was on par ics instruction Instruction were between 6 reported with non-LBD honors stu- (Comparison of average and 10 percentage points classroom scores dents, and LBD honors stu- between teachers pre– higher than their classroom dents outperformed non- Modeling Instruction averages before training. training and the same LBD honors students. teachers post-training) Data from three studies show that male students con- sistently outperform female students. Modeling Instruction in High School Physics Modeling Instruction in High School Physics is grounded in the thesis that scientific activity centers on modeling: the National Science Curriculum construction, validation, and application of conceptual models to understand and organize the physical world. for High Ability Learners The program uses computer models and modeling to The National Science Curriculum for High Ability develop the content and pedagogical knowledge of high Learners Project is a supplemental program that has been school physics teachers and train them to be leaders in implemented across grades 2–8 with a broad group of stu- science teaching reform and technology infusion. The pro- dents within the average-to-gifted range of ability. The gram relies heavily on professional development work- curriculum units employ problem-based learning for shops to equip teachers with a teaching methodology. engaging students in the study of the concept of systems, Teachers are trained to develop student abilities to make specific science content, and the scientific research process. sense of physical experience, understand scientific claims, Students engage in a scientific research process that leads articulate coherent opinions of their own, and evaluate evi- them to create their own experiments and design their dence in support of justified belief. For example, students own solutions to each unit’s central problem. The units analyze systems using graphical models, mathematical encourage in-depth study, and content areas cover a models, and pictorial diagrams called system schema. breadth of scientific subject matter drawn from the physi- cal, life, and earth sciences. Each unit constitutes approxi- Contact mately 30 hours of instruction, with students typically David Hestenes receiving two units within an academic year. Major com- Director, Modeling Instruction Program ponents of the program include a curriculum framework Department of Physics and Astronomy that contains goals and learning outcomes linked to indi- Arizona State University vidual lesson plans; embedded and post assessments that P.O. Box 871504 focus on science content, concept, and process learning; Tempe, AZ 85287-1504 25 lesson plans that address these goals and outcomes Tel: (480) 965-6277 with relatively equal emphasis on each of the goals; and a E-mail: real world problem that serves as the catalyst for subse- Web Site: quent learning in the unit. 21
  • 22. What Do We Know? Seeking Effective Math and Science Instruction Contact University of Northern Iowa Cedar Falls, IA 50614-0150 Joyce VanTassel-Baska Tel: 319- 273-2380 Center for Gifted Education Fax: 319-273-7136 College of William and Mary E-mail: 427 Scotland St. Web Site: Williamsburg, VA 23185 Tel: 757-221-2362 Fax: 757-221-2184 Results E-mail: Web Site: Number Type of measure of results Results Effect size Results State 2 PRISMS students have No effect assessment higher achievement gains size reported Number Effect than comparison students. Type of measure of results Results size Program- 1 PRISMS students’ gains in No effect Open-ended 1 Students in National Science High designed reasoning and problem- size reported assessment to Curriculum (NSC) classrooms measures solving skills were greater test gifted scored better on one unit than those of comparison science students tested when compared to stu- students. dents in non-NSC classrooms. Open-ended 4 Same as above High assessment to Science 2000/Science 2000+ test gifted Science 2000+ (previously known as Science 2000) is a mul- science students timedia, multiyear science curriculum for high schools that takes an integrated, thematic approach to the earth, life, and physical sciences. At each grade level, the yearlong course includes four nine-week units, connected by central Physics Resources and Instructional Strategies themes and a storyline—a narrative that sets a real-world for Motivating Students (PRISMS) context for the science content. Each unit poses problems related to real-life scientific and social issues. Students The goal of Physics Resources and Instructional Strategies address these problems by drawing information from CD- for Motivating Students (PRISMS) is to provide learning ROM-based resources (text, images, video, simulations) activities to promote understanding of physics principles supplemented by laser disc, web references, and manipu- in the context of experiences relating to the daily lives of lative kits. Multimedia resources (text, images, and video) secondary school students. PRISMS includes a guide with are cross-referenced and linked within the CD. over 130 activities in the form of student instructions and teacher notes with background information on the activi- Contact ties. The program’s resources include several videotapes from which students make observations and take data, Ellen M. Nelson and recommended software and interfacing for schools Decision Development Corp. that have access to microcomputers. A complete student 2303 Camino Ramon, Suite 220 evaluation and testing program is included in a three- San Ramon, CA 94583-1389 to four-diskette set. The instructional strategies blend Tel: 800-835-4332 or 925-830-8896 exploratory activities, concept development, and applica- Fax: 925-830-0830 tion activities to stimulate problem-solving skills and the E-mail: understanding of major physics concepts. The guide can Web site: be integrated with the use of any physics textbook and is designed to be individualized to meet the needs of each Results teacher. The guide also contains activities dealing with sci- Number entific, technological, and social issues as well as career Type of measure of results Results Effect size information. The new version is called PRISMS PLUS. Self-designed 1 All Science 2000 students No effect Contact test instruments showed gains in content size reported Roy D. Unruh, Director knowledge from pre- to post-test. Physics Department, Room 303 22
  • 23. What Do We Know? Seeking Effective Math and Science Instruction Study showed the curriculum to be effective in strated significantly higher performance than control increasing the content knowledge of all students, regard- groups and national/international comparison groups. less of gender, ethnicity, or language classification. The quasi-experimental evaluation design, however, makes it difficult to control for prior knowledge or instruc- tion of students in comparison groups. Science and Technology Concepts for Middle Schools (STC/MS) The Science Curriculum Science and Technology Concepts for Middle Schools Improvement Study (SCIS) (STC/MS) is a modular program composed of 24 units. The Science Curriculum Improvement Study (SCIS) There are four units for each grade level, one each in the was developed at the Lawrence Hall of Science at the following strands: life science, earth science, physical University of California between 1962 and 1974 for use in science, and technology. Each STC unit generally has grades K–6. The goal of the program is the development of 16 lessons with hands-on investigations. Teachers can use scientific literacy, defined as a combination of basic knowl- the four modules to comprise the science curriculum for edge concerning the natural environment, investigating abil- the entire school year or use one or two individual units as ity, and curiosity. The program consists of 12 units, one life supplements to other curriculum pieces. Eight modules for and one physical science unit at each elementary grade level. grades 7 and 8 are currently under development. When About 10 major concepts are developed each year. The con- completed, STC/MS will include eight units for science in cepts are interrelated and are intended to provide a concep- grades 7 and 8. The instructional units will be balanced tual framework for the child’s thinking. Opportunities are among life, earth, physical sciences, and technological provided for developing science processes as well. The gen- design. The components are designed to be offered as two eral instructional pattern is free exploration of new materials, one-year courses (one unit from each of the scientific the introduction of a new concept, and the application of the strands) or as four single semester courses. new concept in a range of new situations. Contact Contact National Science Resources Center Note: Science Curriculum Improvement Study (SCIS) is now called Arts and Industries Building, Room 1201 SCIS 3+ 900 Jefferson Drive, SW Publisher: Washington, DC 20560-0403 Delta Education Tel: 202-357-3364 Tel: 800-258-1302 Fax: 202-633-9136 Web Site: Web site: collection=N&menuID=11 middle_school.html Publisher: Results Carolina Biological A 1983 meta-analysis of 57 controlled studies of SCIS and Tel: 800-334-5551 two other activity-based science programs (ESS and SAPA) Web site: draws conclusions about the programs across process, con- tent, and affective outcomes. Seventy percent of the stud- Results ies were dissertations, and a conservative estimate of the combined students tested is 13,000 in over 900 classrooms. In a 2001 study of four of the eight STC/MS modules, a Seventy-nine percent of the studies had a quasi-experi- post-test only design was used to compare the perfor- mental design. Forty-eight percent of the studies tested mance between groups receiving STC/MS instruction and effects after more than one year of program use. those who received traditional instruction. The impact of The overall effect of these three programs on all outcome the curriculum on student achievement in each particular areas was positive, though not dramatically so; thirty-two content area was measured by multiple-choice and short- percent of comparison studies had statistically significant answer tests developed to measure concepts specific to results favoring the treatment group, while each unit, including TIMMS and NAEP. When possible, 6 percent favored the nontreatment group. These results sup- test items were taken from previously existing assessments port an overall conclusion of a positive program effect. The so that national and international comparisons were possi- mean effect size for all studies, with all outcomes weighted ble. In all four STC/MS curriculum units, students demon- equally, was .35, or a 14 percentile point improvement over 23
  • 24. What Do We Know? Seeking Effective Math and Science Instruction non-activity-based instruction. The effects on measures of Contact science process, intelligence, and creativity were positive. Daniel C. Edelson Small positive effects were observed in attitudes towards Northwestern University science. Contrary to a common worry about activity-based Learning Sciences Program programs, content achievement was not negatively affected. Annenberg Hall A second meta-analysis conducted in 1986 largely con- Evanston, Ill 60208-2610 firmed these results, emphasizing in particular the differ- E-mail: ences in attitude toward science and process skills (17 and Web site: 19 percentile point gains, respectively) among students curriculum.htm who were taught with activity-based modules. Results Number World Watcher/Learning about the Type of measure of results Results Effect size Environment Curriculum (LATE) Self-designed 1 Gains at all grade levels Large test and all populations (urban, (urban), The World Watcher/Learning about the Environment Cur- suburban). Biggest differ- moderate riculum (LATE) is a yearlong, inquiry-based, technology- ences in 10th grade. (suburban) supported environmental science curriculum for high school developed at Northwestern University. It is based on the Urban students showed higher gains (5 points) than Learning-for-Use model of learning that conceptualizes con- suburban students (3.5 points). Researchers caution not to tent and process learning as complementary and mutually interpret this result as greater effectiveness of the curricu- facilitating, rather than at odds. LATE incorporates the use of lum for the urban group. It is possible that a subgroup of a scientific visualizations and is centered on three key issues: sample that scores below average on a test tends to do bet- population growth and resource availability, electricity gen- ter on retests. eration and energy demand, and managing water resources The views expressed here are those of the authors and do not necessarily reflect for agriculture and human consumption. those of the Urban Institute, its trustees, or its funders. THE URBAN INSTITUTE Nonprofit Org. 2100 M Street, NW U.S. Postage Washington, DC 20037 PAID Permit No. 8098 Ridgely, MD Address Service Requested