What Do We Know?
SEEKING EFFECTIVE MATH AND SCIENCE INSTRUCTION


     THE URBAN INSTITUTE


February 2005            As r...
What Do We Know? Seeking Effective Math and Science Instruction


    student achievement outcomes as                 both...
What Do We Know? Seeking Effective Math and Science Instruction


  Studies of six of the curricula         ricula and the...
What Do We Know? Seeking Effective Math and Science Instruction


      of the curriculum is far more com-       mines wha...
What Do We Know? Seeking Effective Math and Science Instruction


                                                 Appendi...
What Do We Know? Seeking Effective Math and Science Instruction


    A Description of the Mathematics Curricula          ...
What Do We Know? Seeking Effective Math and Science Instruction


Results                                                 ...
What Do We Know? Seeking Effective Math and Science Instruction


                     Number                             ...
What Do We Know? Seeking Effective Math and Science Instruction


                 Number                                 ...
What Do We Know? Seeking Effective Math and Science Instruction


     Contact                                            ...
What Do We Know? Seeking Effective Math and Science Instruction


                 Number                                 ...
What Do We Know? Seeking Effective Math and Science Instruction


     lower for African-American students than other stud...
What Do We Know? Seeking Effective Math and Science Instruction


Results                                                 ...
What Do We Know? Seeking Effective Math and Science Instruction


                                                   Appen...
What Do We Know? Seeking Effective Math and Science Instruction


A Description of the Science Curricula                  ...
What We Know
What We Know
What We Know
What We Know
What We Know
What We Know
What We Know
What We Know
What We Know
Upcoming SlideShare
Loading in …5
×

What We Know

1,461 views

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,461
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

What We Know

  1. 1. What Do We Know? SEEKING EFFECTIVE MATH AND SCIENCE INSTRUCTION THE URBAN INSTITUTE February 2005 As requested by the GE Foundation, the with relatively credible evaluations to pro- main goal of this review was to identify vide some choice to districts and schools mathematics and science curricula as well that wanted guidance in selecting a cur- as professional development models at riculum and that wished to use effective- the middle and high school levels that ness as a selection criterion. had been deemed effective based on their success in increasing student achievement. Because of its emphasis on student achieve- Evaluation Study Selection ment, this approach differs from that Criteria Used typically used at the local level and that adopted by other recent studies conducted To be included, studies had to have to find “effective” curricula and profes- 1. Rigorous methodological design sional development models. including comparison groups; Historically, curriculum choice at the 2. Measures of impact on student local level has often been made by a com- outcomes that included, but were mittee that decides which curriculum to not limited to, test scores; and adopt based on considerations only periph- 3. Comparative data with experi- erally related to student achievement—such mental or quasi-experimental as state-imposed standards, recommenda- design preferred. tions of others, cost, and presentations by publishers’ representatives. Choice of pro- fessional development models follows a As can be seen from the criteria above, similar pattern. Indeed, there has been very we chose increases in student achievement little else available to guide school districts as the measure of effectiveness.2 We limited in their curriculum selection process, since our review to evaluations that compared for most curricula and textbooks the only student achievement outcomes elicited by data at hand are publishers’ figures on the curriculum being studied with student the number of adoptions. That has been achievement outcomes of another curricu- changing. There is a growing movement lum. Where data were available, we also to assess the effectiveness of math and took into account the content validity of the science curricula through various meth- assessments used as well as the statistical The Urban Institute odologies, including content analyses, significance and the effect size of the differ- Beatriz Chu Clewell comparative studies, case studies, and syn- ences reported. Our review made a special Clemencia Cosentino de thesis studies.1 And while there have been effort to identify eligible studies that dis- Cohen several studies of the effectiveness of pro- aggregated results by performance of stu- Nicole Deterding Sarah Manes fessional development practices, very few dent subgroups. Our review of evaluation Lisa Tsui have measured the effects of these practices studies of professional development was on student achievement. similarly limited to studies that used Campbell-Kibler The focus of this review, reflecting the student achievement as the measure of Associates, Inc. GE Foundation’s interests, was to identify effectiveness. Patricia B. Campbell current math and science curricula and Lesley Perlman professional development at the middle Professional Development: Shay N.S. Rao and high school levels that showed evi- Becky Branting dence of positive impact on student What We Found Lesli Hoey achievement. Our goal was to come up We identified 18 evaluation studies of pro- Rosa Carson with enough math and science curricula fessional development models that used 1
  2. 2. What Do We Know? Seeking Effective Math and Science Instruction student achievement outcomes as both change in the teaching mathematics curricula that were measures of effectiveness. Features of behavior of teachers and change developed as part of whole school these studies can be found in a full in the classroom environment. reform efforts. A total of 156 studies report of this review.3 We found that One study found that behavioral of student mathematics achievement change was only evident after with comparison group data were Providing professional develop- teachers had received a minimum found for 18 of the curricula (20 per- ment for teachers of standards- of 80 hours of intensive profes- cent of the total number of curricula based science curricula is sional development; change in originally identified). A list of the effective in increasing student the classroom environment was 18 mathematics curricula, together achievement. There is evidence achieved after 160 hours.4 It is with overviews of the curricula and from the research that standards- important to note that at the indi- their impact studies, appears in based science programs that pro- vidual level, teacher preparation in Appendix A. vide professional development to the content area was an important The following findings relate to teachers produce higher student factor in influencing effectiveness. methodology: achievement outcomes than those Some widely held beliefs about that do not. what constitutes effective profes- Most middle and high school Professional development that is sional development are not sup- mathematics curricula do not tied to curriculum, to knowledge ported by research linked to have studies of student achieve- of subject matter, and/or to how student achievement. ment with comparison groups students learn the subject is more that can be found through litera- ● Myth: Professional development effective in terms of improving ture or web searches. Only three should be distributed over the student achievement than is of the studies found specified the school year. While professional professional development that curriculum to which the target cur- development in science seems focuses only on teaching behav- riculum was being compared. The to benefit from distributed time, iors. A number of studies have rest compared their curriculum to the same effect has not been concluded that the content of pro- some unnamed curriculum, mak- found for math. fessional development—what is ● Myth: More is better. There is no ing comparisons across curricula taught—is more important than its impossible. evidence to support the belief format—how it is taught—and that that more contact time over and If students are going to be judged content should be linked to subject above 80 hours and 160 hours on the results of an external test, matter knowledge, a specific cur- results in changes in teaching the mathematics curriculum riculum, or the process of student behavior or classroom environ- selected should cover the areas learning. For example, some pro- ment, respectively. and skills that are included on fessional development focuses on ● Myth: Classroom visits are neces- that test (i.e., the test and curricu- training teachers to model specific sary. Visits to classrooms for lum should be aligned). As would pedagogical approaches and skills consultation or coaching as part be expected, students tend to score using generic lesson formats. The of professional development do higher on tests focusing on the goal of these efforts is to get teach- not necessarily result in greater content and skills covered in their ers to change the way they teach, student achievement. curriculum. The content and skills expecting that this will increase ● Myth: Schoolwide programs are covered in traditional math cur- student achievement. More effec- more effective. Providing services ricula tend to reflect the content tive professional development to whole schools rather than and skills covered in most stan- focuses on increasing teacher individuals may not be the most dardized and statewide tests. knowledge of the subject matter, important element of profes- Alternatively, the content and on a specific curriculum, or on sional development that leads to skills covered in standards-based the student learning process. The increased student achievement. curricula tend to better reflect that goal of these types of efforts is to change teacher knowledge, ex- which is covered in standards- pecting that this will, in turn, Mathematics and Science based tests. (For more on this both change teacher behavior and Curricula: What We Found issue, see the text box, “Inter- increase student achievement. preting the Results.”) The amount of professional Mathematics Curricula development provided is an Our review netted 89 middle and The following findings relate to important factor in influencing high school curricula, including eight outcomes and impacts: 2
  3. 3. What Do We Know? Seeking Effective Math and Science Instruction Studies of six of the curricula ricula and the results of impact stud- more effective than traditional found that students who use the ies, is provided in Appendix B. science curricula as measured by curriculum being tested scored The following findings relate to student achievement. The prepon- higher than comparison students methodology: derance of evidence provided by on both a majority of standard- meta-analyses and evaluations of ized and/or state tests used and As with mathematics curricula, individual curricula seems to con- on a majority of the curriculum- most middle and high school cur- firm that inquiry-based science based tests used. These curricula ricula do not have evaluation curricula produce larger effects were Cognitive Tutor, Connected studies of student achievement on student achievement than do Mathematics (CMP), Interactive with comparison groups that can the more “traditional” science Mathematics Program (IMP), be found through published liter- curricula.6 Prentice Hall: Tools for Success, ature or web searches. However, It is difficult to determine the Saxon Math: An Incremental in contrast to the mathematics cur- effect of these science curricula Development, and University of ricula in Table 1 in Appendix A on different subgroups of stu- Chicago School Mathematics (many of which had been the sub- dents—such as girls, minority Project. Because students using jects of multiple studies), most of group members, and urban stu- these curricula scored higher on the science curricula in Table 2 in dents. Very few of the curricula both types of tests, it is reasonable Appendix B had only one eval- had studies that met our criteria to assume that the effectiveness of uation or study, usually unpub- and disaggregated their findings each curriculum is not an effect of lished works available through the by sex, language minority status, developer. or urban location. Surprisingly, the type of test used. none of the studies reported data It is difficult to determine any The following findings relate to disaggregated by race/ethnicity. differential effect of these math outcomes and impacts: Most science curricula include a curricula on girls and boys. Very professional development compo- few of the curricula had studies Science curricula based on the nent. Inclusion of a professional that met our criteria and disaggre- inquiry approach are consistently development component as a part gated their findings by sex. Among those that did, the results were inconsistent. With the exception of Connected Mathematics, too few results per Interpreting the Results: Validity of Tests Used curriculum were broken out by To Measure Student Achievement race/ethnicity to allow us to draw The evaluation studies for mathematics and science curricula on our lists general conclusions regarding used a variety of assessments—some standardized, some state-mandated, racial/ethnic effects. Connected some self-developed, and some developed by others specifically for stan- Mathematics, however, appeared dards-based curricula. In the case of mathematics curriculum studies, the to reduce racial/ethnic gaps. content and skills covered by most standardized achievement tests tend to reflect more closely the content and skills covered by traditional mathe- Science Curricula matics curriculum than those covered by standards-based curriculum, causing them to have better “content validity” for traditional curriculum. We identified 80 science curricula at Many researchers of standards-based curriculum are aware of this and the middle and high school levels.5 develop their own student achievement tests that more accurately test the Similar to the mathematics cur- skills and content of standards-based curriculum. riculum, we found seven science If students taking one curriculum score higher than others on both curricula that had been developed types of test, there is no question of interpretation. Beyond that, however, specifically as a part of whole school judgments of efficacy must take into account the content validity of the reform. A total of 45 studies of stu- tests in order to determine which type of curriculum is more effective. dent achievement in science were Although science curriculum studies face a similar dilemma, several stud- found that met our criteria, covering ies comparing the results of standardized tests with those of other forms of 21 (or 26 percent) of the total science assessment have found very small differences. The issue of content valid- curricula identified. The 21 curricula ity, therefore, seems to be less of a concern in judging the effectiveness of are listed in Table 2, which together science curricula. with brief descriptions of these cur- 3
  4. 4. What Do We Know? Seeking Effective Math and Science Instruction of the curriculum is far more com- mines what students will learn. Once an enormous task and an enormous mon in science than in mathemat- a decision is made, selection should responsibility. We hope that our ics because science curricula tend be guided by how effective a par- review can provide some guidance to be more discretionary and vari- ticular curriculum is for the student and assistance in the process. able than mathematics curricula. population to be taught. The National Science curriculum developers, Research Council report, On Eval- therefore, may feel the need to pro- Endnotes uating Curricular Effectiveness: Judging vide more guidance to teachers the Quality of K–12 Mathematics 1. The majority of these efforts have been undertaken by AAAS, the National Research about the appropriate instructional Evaluations (2004) describes the value Council (NRC), and the U.S. Department of approaches to be used for specific of this knowledge to the decision- Education. The AAAS study used content curricula. making process: analysis, the NRC study did not rate specific curricular math programs, and the U.S. Department of Education study reviewed Clearly, knowing how effective a middle school math programs only. What Can We Conclude particular curriculum is, and for 2. As determined by quantifiably measurable whom and under what conditions it from Our Review? outcomes, such as test scores. is effective, represents a valuable The major conclusions of our review and irreplaceable source of informa- 3. The full report, on which this summary tion to decisionmakers, whether report is based and which contains refer- that should be most useful to those they are classroom teachers, par- ences and a bibliography of all studies wishing to invest in sustainable described in this report, can be found on ents, district curriculum specialists, school reform in science and mathe- school boards, state adoption the Urban Institute web site, http://www. matics can be summarized as follows: boards, curriculum writers and urban.org. evaluators, or national policymak- 4. “Teacher behavior” in this context refers Effective mathematics curricula in ers. Evaluation studies can provide to teachers’ use of specific pedagogical middle and high school may be that information but only if those approaches in instruction, such as inquiry- either traditional or integrative evaluations meet standards of qual- based teaching practices. “Change in the ity (p.1).7 classroom environment” refers to teacher (standards-based). facilitation of an investigative classroom cul- Effective science curricula in ture through seating arrangements to stimu- middle and high school should This review of math and science late discussion, use of cooperative learning be inquiry-based rather than curricula has tried to simplify for groups, encouraging students to explain schools and districts the complex, concepts to one another, and other such traditional. practices. Effective professional develop- time-consuming process of determin- ment programs are those that ing curriculum effectiveness by iden- 5. Of the 80 science curricula identified, 59 had tifying programs that have what we no evaluations or had evaluations that did focus on content rather than for- not meet our criteria, or evaluations were mat and that have the following consider to be credible evaluations. out of print, or we did not receive a re- features: We have also distilled the findings of sponse from the developers. achievement-based research on pro- 6. As measured by both standardized tests and ● Content tied to curriculum, fessional development to a handful of tests developed for standards-based curric- knowledge of subject matter, principles that reflect effective prac- ula in both meta-analyses and evaluations of and/or how students learn a tice. Once schools and districts have individual curricula. subject; decided on a curriculum and an 7. Jere Confrey and Vicki Stohl, eds. 2004. On ● A minimum of 80 contact hours appropriate assessment tool, they Evaluating Curricular Effectiveness: Judging the to effect changes in teachers’ might wish to collect their own Quality of K–12 Mathematics Evaluations. impact data to evaluate how well the Committee for a Review of the Evaluation instructional behaviors; and Data on the Effectiveness of NSF-Supported ● A minimum of 160 contact curriculum they choose is working and Commercially Generated Mathematics hours to effect changes in the with their own students.8 It is the Curriculum Materials. National Research classroom environment. responsibility of the school and dis- Council. Washington, DC: National Acad- trict community to ensure that the emy Press. content that they want students to 8. For those schools and districts that already How to Use This Review learn is embodied in the curriculum, have programs that they feel are effective Choice of a curriculum in mathe- that the curriculum is effective for but are not included in Tables 1 and 2, we matics and science involves deciding suggest that they collect their own effective- this purpose, and that appropriate ness data via evaluation studies that adhere what aspects of these subjects are measures are used to assess whether to the criteria established in this review. important to address and emphasize students are indeed learning what the They may wish to contract with an evalua- in schools—this choice thus deter- community wants them to learn. It is tor for this purpose. 4
  5. 5. What Do We Know? Seeking Effective Math and Science Instruction Appendix A DETAILED FINDINGS OF THE REVIEW OF MATH CURRICULA This appendix provides a list of the mathematics curricula included in our review, together with a descrip- tion of each curriculum and its related evaluation studies. Table 1 lists the mathematics curricula identified as having evaluation studies that met our criteria. TABLE 1. Math Curricula with Studies Grades Covered Curriculum Name Subject Matter K 12 Edison Schools Mathematics K 8 Direct Instruction Mathematics middle school Cognitive Tutor *[sex, race/ethnicity] Mathematics middle school Connected Mathematics (CMP) *[sex, race/ethnicity] Mathematics middle school Integrated Mathematics, Science, and Technology (IMaST) Mathematics middle school Mathematics in Context (MiC) *[race/ethnicity] Mathematics 6 8 MATHThematics Mathematics middle school Prentice Hall: Tools for Success Mathematics middle school Saxon Math: An Incremental Development Mathematics 7 12 Mathematics with Meaning *[sex, race/ethnicity] Mathematics 7 12 University of Chicago School Mathematics Project (UCSMP) Mathematics 9 12 College Preparatory Mathematics (CPM) Mathematics 9 12 Contemporary Mathematics in Context: A Unified Approach (Core-Plus Mathematics Project CPMP) Mathematics 9 12 Interactive Mathematics Program (IMP) *[sex] Mathematics 9 12 Math Connections *[sex, race/ethnicity] Mathematics 9 12 Mathematics: Modeling Our World (MMOW/ARISE) Mathematics 9 12 Systemic Initiative for Montana Mathematics and Science (SIMMS) Mathematics 11-12 Advanced Placement (AP) Calculus Calculus Note: Shaded curricula are those for which we have found the strongest evidence of effectiveness, that is, quantitative evidence that their use in instruction elicits higher achieve- ment/performance in students than other curricula to which they are compared on both standardized and/or state tests and on curriculum developed tests. There are several curricula for which this evidence of effectiveness has not been collected but which might also qualify as effective should appropriate studies be conducted. An omission from this list of many curricula signifies merely that these curricula have not yet provided quantitative evidence of effectiveness that meets our criteria. *An asterisk marks curricula for which effectiveness data are provided for subgroups of students, indicated in brackets. (See full report on the web for discussion of research find- ings by subgroup.) 5
  6. 6. What Do We Know? Seeking Effective Math and Science Instruction A Description of the Mathematics Curricula ity, and differential and integral calculus, with Calculus BC and Evaluation Studies Found covering these topics more extensively than Calculus AB. Prerequisites include knowledge of analytic geometry and In addition to a description of each curriculum, these elementary functions in addition to college preparatory overviews (presented in alphabetical order) include the algebra, geometry, and trigonometry. Enrolled students are type of student achievement measure used, the number expected to take the Advanced Placement examination in and direction of the results, and, if available, the size of Calculus AB or BC. any differences between groups. If the results were broken out by sex and/or race and ethnicity, this too is indicated. Contact The types of measures used in the impact studies are College Board Headquarters broken into five categories: 45 Columbus Avenue New York, NY 10023-6992 Standardized achievement tests (i.e., PSAT, SAT, SAT-9, Tel: 212-713-8066 and Iowa Test of Basic Skills) Web site: http://apcentral.collegeboard.com/article/0,3045, State-mandated achievement tests (i.e., FCAS, MEAP, 151-165-0-2178,00.html and MCAS) Standards-based, curriculum-driven measures (i.e., Results Balanced Assessment, Problem Situation Test) Teacher-based measures (i.e., GPA, school tests, and Five results were found from studies on the effect of AP mathematics courses taken) Calculus courses. These studies compared college mathe- matics performance of students who took AP Calculus Percent passing different mathematics courses with other students who did not. In the charts that follow within this appendix, results were tracked rather than the number of studies. Multiple Number grades and multiple measures were counted as multiple Type of measure of results Results Effect size results and broken out as such. For example, a study that Teacher-based 3 AP students scored higher No effect looked at the impact of different subsets of a curriculum measures/GPA in two results; there were size reported for sixth, seventh, and eighth grades was counted as three no differences in one results. Cohort studies that tracked students across multi- result. ple grades were counted as one result. Also counted as one Pass rates/ 2 AP students scored higher No effect result were findings from different sites within the same courses taken in one result, and there size reported study using the same measure. If a majority of sites had were no differences in changes favoring the tested curriculum, the result was one result. indicated as positive. If the majority of sites did not differ from the comparison, the result was indicated as no Results were not reported by sex, race, or ethnicity. change. If a majority of sites had changes favoring the comparison curriculum, the result was indicated as negative. Cognitive Tutor Note: Most impact studies for math curricula reported the statistical sig- The Cognitive Tutor (CT), from Carnegie Learning, nificance of their results. Only differences that have reached the conserva- includes full curricula in Algebra I, Geometry, Algebra II, tive minimum acceptable statistical significance level of .05 were included in the results reported for each study. If differences are statistically signifi- an Integrated Math Series, and Quantitative Literacy cant, then there is another measure, called an effect size, which shows Through Algebra. Each curriculum combines software- how big the difference is. In our description of the study results, we pro- based, individualized computer lessons with collaborative, vide effect sizes where available, although very few studies reported real-world problem-solving activities. Students spend these results. Effect sizes greater than .4 are considered large; between .2 about 40 percent of their class time using the software and and .4 are considered moderate; and less than .2 are considered small. the remainder of their time engaged in classroom problem- solving activities. Contact Advanced Placement (AP) Calculus Carnegie Learning, Inc. The AP Calculus curriculum includes two courses: AP 1200 Penn Avenue Calculus AB, which is comparable to one semester of col- Suite 150 lege-level calculus, and AP Calculus BC, which is com- Pittsburgh, PA 15222 parable to two semesters of college-level calculus. Both Tel: 888-851-7094 courses include elementary functions, limits and continu- E-mail: info@carnegielearning.com 6
  7. 7. What Do We Know? Seeking Effective Math and Science Instruction Results Results Twenty-one results from studies on the effect of CT were Six results from studies on the effect of CPM were found, found, all of which focused on Algebra I at the middle and one of which focused on algebra and five of which focused high school level. on grade 9–11 math. Number Number Type of measure of results Results Effect size Type of measure of results Results Effect size Standardized 8 CT students scored higher Moderate Standardized 5 There were no differences No effect achievement in all eight results. effect size achievement in all five results. size reported tests in one result tests Statewide tests 5 CT students scored higher No effect Teacher- 1 There were no differences. No effect in three results; there size reported developed size reported were no differences in measure two results. Results were not reported by sex, race, or ethnicity. Curriculum-driven, 7 CT students scored higher Large effect skill-specific tests in seven results. sizes in two results Passing rates 1 CT students had higher No effect Connected Mathematics (CMP) passing rates. size reported The Connected Mathematics (CMP) curriculum is com- posed of eight models, each focusing on one important Two results looked at sex differences; in one result, CT area of mathematics and emphasizing previously learned boys’ scores were higher than comparison boys. There content. Connected Mathematics is designed to develop were no differences between boys in one result, and there students’ knowledge and understanding of mathematics were no differences for the girls’ scores in either result. through attention to connections between mathematical Two results reported differences by race and ethnicity. In ideas and their applications in the world outside school; one result, African-American CT students had higher among the core ideas in mathematics; among the strands scores than comparison African-American students, and in a modern mathematics curriculum; and between the there were no differences for Hispanic students. In the planned teaching-learning activities and the special apti- second result, Hispanic CT students scored higher than tudes and interests of middle school students. comparison Hispanic students. Contact Pearson Education College Preparatory Mathematics (CPM) P.O. Box 2500 The CPM series offers a four-year integrated curriculum Lebanon, IN 46052-3009 where mathematics topics are revisited and built upon Tel: 800-848-9500 through the years. Problem-solving strategies are empha- Web site: http://www.phschool.com/math/ sized as a vehicle for learning mathematics, and student study teams are an integral part of the learning process. Results Based on the belief that concept mastery requires time, the curriculum spirals through practice of the main course Thirty-four results from comparison studies looking at concepts throughout each year and emphasizes students’ the effect of CMP were found focusing on middle grade supportive group work. The program sees the teacher’s mathematics. role as a guide. Number Contact Type of measure of results Results Effect size CPM Business Office Standardized 11 CMP students scored No effect 1233 Noonan Drive achievement higher for six results; they size reported Sacramento, CA 95822 tests scored lower for one result; Tel: 916-681-3611 and there were no differ- Web site: http://www.cpm.org ences for four results. (continued) 7
  8. 8. What Do We Know? Seeking Effective Math and Science Instruction Number Number Type of measure of results Results Effect size Type of measure of results Results Effect size Statewide 16 CMP students scored No effect Standardized 43 In 19 of the results, CPMP No effect tests higher for 14 results; there size reported achievement students scored higher; size reported were no differences for tests in 24 results, there were two results. no differences. Curriculum-driven, 6 CMP students scored Large effect Statewide 5 Students scored higher in No effect skill-specific tests higher for all six results. sizes were tests all five results. size reported found for Curriculum-driven, 2 Students scored higher in No effect one result. skill-specific tests both results. size reported Teacher-based 1 CMP students scored No effect Teacher-based 6 CPMP students scored No effect measure higher in one result. size reported measure/GPA higher in one result, no size reported differently in another, and lower in four results. Seven results were broken out by race/ethnicity and two by sex. No significant sex differences were found. African-American CMP students were found to score Results were not reported by sex, race, or ethnicity. higher than African-American comparison students in six results, while Hispanic CMP students scored higher than Hispanic comparison students in four results; in a fifth result, there were no differences. African-American and Hispanic CMP students showed greater gains than others Direct Instruction in two results, while African Americans alone showed greater gains in one result. In one result, Native-American In Direct Instruction (DI), each program is fully scripted, CMP student performance decreased. from what the teacher says and anticipated student responses, to correctional procedures. Each skill is broken down into its component parts, and then each component of the skill is taught to mastery. Afterward, the skills are Contemporary Mathematics in Context: combined within a larger context where they may be uti- lized across settings, resulting in generalized fluency. The A Unified Approach DI mathematics curriculum covers grades K–8, but focuses (Core PLUS Mathematics CPMP) primarily on grades K–6. The mathematics curriculum Core-Plus Mathematics consists of a single core sequence covers 19 different topics ranging from addition and sub- for both college-bound and employment-bound students traction to money, mathematics study skills (graphs, during the first three years of high school. A flexible fourth charts, maps, and statistics), and geometry. year course can be used to prepare students for college mathematics. Contact Contact The McGraw-Hill Companies P.O. Box 182604 Core-Plus Mathematics Project Columbus, OH 43272 Department of Mathematics Tel: 888-772-4543 Western Michigan University Web site: http://www.sraonline.com/index.php/home/ Kalamazoo, MI 49008-5248 curriculumsolutions/di/connectingmath/114 Tel: 866-407-CPMP (2767) E-mail: cpmp@wmich.edu Results Results Nine results from comparison studies on the effect of DI Fifty-six results from comparison studies on the effect of were found which focus on seventh and eighth grade CPMP were found focusing on high school math. math. 8
  9. 9. What Do We Know? Seeking Effective Math and Science Instruction Number Integrated Mathematics, Science, Type of measure of results Results Effect size and Technology (IMaST) Standardized 6 DI students scored higher No effect The Integrated Mathematics, Science, and Technology pro- achievement for four results; they size reported tests scored lower for two gram provides integrated sixth, seventh, and eighth grade results. curricula that promote hands-on learning for students and teamwork among teachers from different disciplines. State test 3 In all three results, DI No effect students scored higher. size reported IMaST emphasizes learning based on constructivist theory and active student participation involving a hands-on Results were not reported by sex, race, or ethnicity. approach comprising a wide variety of activities. Edison Schools Contact Founded in 1992, Edison Schools is focused on raising stu- Ronjon Publishing, Inc. dent achievement through research-based school design, 1001 S. Mayhill Rd. aligned assessment systems, interactive professional devel- Denton, TX 76208 opment, integrated use of technology, and other program Tel: 800-262-3060 features, including a longer school day and school year. The Web site: math curriculum for grades 6–8 includes applied arithmetic, http://www.ilstu.edu/depts/cemast/programs/imast.shtml prealgebra, and pregeometry, using a spiral curriculum approach to teach concepts and ideas. The mathematics in grades 9–10 provides three years of high school math in two Results years’ time, using an integrated application-based approach to algebra, geometry, and trigonometry with additional One study was found looking at the effect of IMaST focus- emphasis on probability, statistics, and discrete mathemat- ing on seventh and eighth grade students. ics. Grades 11 and 12 incorporate advanced mathematical modeling, Calculus and Statistics, Algebra 2, and Number Precalculus, as well as AP Statistics and Calculus. Type of measure of results Results Effect size Contact Standardized 1 IMaST students scored No effect 521 Fifth Avenue, 11th Floor achievement higher for the one result. size reported New York, NY 10175 tests Tel: 212-419-1600 Web site: http://www.edisonschools.com/ Results were not reported by sex, race, or ethnicity. Results Seventeen results from comparison studies on the effect of Edison Schools were found; fifteen focus on middle school math and two focus on high school math. Interactive Mathematics Program (IMP) Number Type of measure of results Results Effect size This four year, problem-based curriculum incorporates tra- Standardized 8 For four results, Edison No effect ditional branches of mathematics (algebra, geometry, and achievement students scored higher; size reported trigonometry) with additional topics recommended by the tests in the remaining four, there NCTM Standards, such as statistics, probability, curve fit- were no differences. ting, and matrix algebra. Students are encouraged to State test 9 For two results, Edison No effect experiment, investigate, ask questions, make and test con- students scored higher; size reported jectures, reflect, and accurately communicate their ideas for four results, Edison students scored lower; and conclusions. Although each unit has a specific mathe- and for three results, there matical focus, other topics are brought in as needed to were no differences. solve the central problem. Ideas that are developed in one unit are usually revisited and deepened in one or more Results were not reported by sex, race, or ethnicity. later units. Algebra and geometry are distributed through- out the four years. 9
  10. 10. What Do We Know? Seeking Effective Math and Science Instruction Contact Results Key Curriculum Press Fifteen results from comparison studies were found on the 1150 65th Street effect of MC, all of which focused on high school students. Emeryville, CA 94608 Tel: 800-995-MATH (6824) Number Web site: http://www.keypress.com/catalog/products/ Type of measure of results Results Effect size textbooks/Prod_IMP.html Standardized 6 MC students scored No effect Results achievement higher for three results; for size reported tests three results, there were Twenty-two results from comparison studies were found were no differences. on the effect of IMP, all focusing on high school. State test 8 MC students scored There was a Number higher for all eight results. large effect Type of measure of results Results Effect size size for three results. Standardized 12 IMP students scored No effect achievement higher for eight results; size reported Curriculum-driven, 1 No difference was found. N/A tests there were no differences skill-specific test for four results. Pass rates or 5 In all five results, IMP No effect One result looked at sex, race, and ethnic differences courses taken students had higher pass- size reported and found no differences within MC students. ing rates and/or were more likely to take more mathe- matics courses. Curriculum-driven, 3 IMP students scored Large effect skill-specific tests higher for all three results. sizes were Mathematics in Context (MiC) found in the three results Mathematics in Context is a four-year middle school cur- Teacher-based 2 In both results, IMP No effect riculum (grades 5–8) that encourages students to discover measure students scored higher. size reported mathematical concepts and skills through engaging prob- lems and meaningful contexts. Each year includes lessons Results were not reported by race or ethnicity. The one in the four strands (numbers, algebra, geometry and statis- result reported by sex found IMP girls were slightly more tics, and probability) that are interwoven through 10 units. apt to continue three or more years in math than IMP boys, For example, sample algebra units include Patterns and while the reverse was the case for comparison students. Symbols (grade 5), Expressions and Formulas (grade 6), Ups and Downs (grade 7), and Graphing Equations (grade 8). MATH Connections Contact MATH Connections (MC) is an integrated curriculum that blends ideas from traditionally separate mathematical Holt, Rinehart and Winston fields (e.g., algebra, geometry, statistics, and discrete math- Attn: Ms. Web1 ematics) in ways that blur the lines between them. This 10801 N. MoPac Expressway three year curriculum replaces the traditional Algebra I, Building 3 Geometry, Algebra II sequence and is designed for all stu- Austin, TX 78759 dents in grades 9, 10, and 11, with honors students begin- Tel: 800-HRW-9799 (800-479-9799) ning the curriculum in grade 8. Web sites: http://www.hrw.com http://mic.britannica.com/mic/common/home.asp Contact 750 Old Main Street Suite 303 Results Rocky Hill, CT 06067-1567 Tel: 860-721-7010 Twenty results from comparison studies were found on the Web site: http://www.mathconnections.com effect of MiC, which focused on middle school students. 10
  11. 11. What Do We Know? Seeking Effective Math and Science Instruction Number Number Type of measure of results Results Effect size Type of measure of results Results Effect size Standardized 17 In 15 results, MiC In two Standardized 4 For three results, MMOW No effect achievement students scored higher; in results, a achievement students scored higher; size reported tests two results, there were no majority of tests for one result, there were differences. classes no differences. showed at least moder- Results were not reported by sex, race, or ethnicity. ate effect sizes. State test 3 MiC students scored No effect Mathematics with Meaning higher for all three results. size reported Mathematics with Meaning (MwM) is not a complete cur- ricular program; rather, it is a combination of professional One result was broken out by race. No statistically sig- development, instructional strategies, and carefully nificant difference was found between African-American planned materials designed to alter the pedagogy and students using MiC and comparison students. content of middle school and high school mathematics courses in order to improve student achievement. The pro- gram consists of instructional units that teachers may use on a supplementary basis or as their entire instructional program. MwM takes a student-centered approach based Mathematics: Modeling Our World on exploratory learning and problem solving, focusing on (MMOW/ARISE) developing conceptual understanding, connections, and communication with mathematical concepts through fre- In Mathematics: Modeling Our World (MMOW), students quent group work and hands-on activities. are taught to use a variety of resources to solve problems and to choose resources that meet the needs of a particular Contact situation. As in real life, MMOW’s problems do not neces- College Board sarily have perfect solutions. MMOW works to strengthen Dept CBO the students’ ability to solve problems by setting goals and P.O. Box 869010 thinking strategically about how to achieve these goals, Plano, TX 75074 solving problems through trial and error and/or process Tel: 212-713-8260 of elimination, using technology like calculators and com- Tel: 800-323-7155 puters, and working together to solve semi-structured E-mail: Collegeboardcustomerservice@pfsweb.com problems and communicating the solutions. Web site: http://www.collegeboard.com Contact Results W.H. Freeman and Company Eleven results from comparison studies on the effect of 41 Madison Avenue MwM were found; three focused on middle school math New York, NY 10010 achievement and eight on high school math achievement. Tel: 800-446-8923 Web sites: http://www.whfreeman.com/highschool/ Number Type of measure of results Results Effect size contact_hs_rep.asp http://www.comap.com/highschool/projects/mmow/ Statewide tests 11 In six results, there were No effect introduction.htm no differences; in five reported results, MwM students scored higher. Results Six results were broken out by sex, four by race. In the Four results related to MMOW were found, one focusing six results where sex differences were given, boys slightly on middle school students and three focusing on high outperformed girls. In four results where race/ethnic dif- school students. ferences were given, achievement scores were significantly 11
  12. 12. What Do We Know? Seeking Effective Math and Science Instruction lower for African-American students than other students school level. Each lesson includes a Think and Discuss sec- in both MwM and comparison groups. tion that presents the new material along with questions to get students actively thinking about and discussing impor- tant concepts. Textbooks in this curriculum include Work Together activities that allow students to work in groups, Middle Grades MATHThematics often doing hands-on activities to reinforce math topics. Middle Grades MATHThematics (STEM) is a three-year curriculum designed for use in grades 6–8. Four unifying Contact concepts—Proportional Reasoning, Multiple Represen- Pearson Education tations, Patterns and Generalizations, and Modeling—are P.O. Box 2500 used across the three years with seven content strands: Lebanon, IN 46052-3009 Number, Measurement, Geometry, Statistics, Probability, Tel: 800-848-9500 Algebra, and Discrete Mathematics. Web site: http://www.phschool.com/math/ Contact Results McDougal Littell Customer Service Center Eleven results from comparison studies on the effect of A Houghton Mifflin Company PHM were found. 1900 S. Batavia Geneva IL 60134 Number Tel: 617-351-5326 Type of measure of results Results Effect size Tel: 800-462-6595 Web sites: http://www.mcdougallittell.com/ Standardized 8 PHM students scored Moderate achievement higher in five results; in effect sizes http://www.classzone.com/math_middle.cfm tests three results, there in one result were no differences. Results Statewide tests 2 PHM students scored No effect Six results from comparison studies were found on the higher in both results. size reported effect of STEM. Curriculum-driven, 1 PHM students scored No effect skill-specific tests higher in this result. size reported Number Type of measure of results Results Effect size Results were not reported by sex, race, or ethnicity. Standardized 2 In one result, STEM stu- No effect achievement dents scored higher; in size reported tests the second, there were no differences. Saxon Math: An Incremental Development Statewide tests 2 In one result, STEM stu- No effect Saxon Math (SM) is a K–12 curriculum that systematically dents scored higher; in size reported distributes instruction, practice, and assessment through- the second, there were no out the academic year rather than concentrating concepts differences. in a single unit or chapter. Each increment builds upon the Curriculum-driven, 2 In both results, STEM stu- No effect foundation of earlier increments, to lead students toward skill-specific tests dents scored higher. size reported a deeper understanding of mathematical concepts. Instruction of related concepts is spread throughout the Results were not reported by sex, race, or ethnicity. grade level, ensuring that students have an opportunity to master each concept before they are introduced to the next one. Prentice Hall: Tools for Success Contact In Prentice Hall Math (PHM), various mathematical Saxon Publishers strands (such as number sense, algebra, geometry, mea- 2600 John Saxon Blvd. surement, data analysis, and problem solving) are inte- Norman, OK 73071 grated throughout the series to ensure that students are Tel: 800-284-7019 prepared for subsequent mathematics courses at the high E-mail: info@saxonpublishers.com 12
  13. 13. What Do We Know? Seeking Effective Math and Science Instruction Results Number Type of measure of results Results Effect size A total of eight results from comparison studies were found on the impact of SM, focusing on both middle and Standardized 6 No differences were found No effect high school. achievement in the six results. size reported tests Curriculum-driven, 6 In one result, SIMMS stu- No effect Number skill-specific tests dents scored higher; no size reported Type of measure of results Results Effect size differences were found in Standardized 3 SM students scored No effect the other five. achievement higher in all three results. size reported tests Results were not reported by sex, race, or ethnicity. Statewide tests 3 SM students scored No effect higher in all three results. size reported Curriculum-driven, 1 SM students scored No effect University of Chicago School Mathematics skill-specific tests higher in this result. size reported Project (UCSMP) Teacher-based 1 SM students scored No effect The UCSMP secondary curriculum consists of six courses: measure/GPA higher in this result. size reported Transitional Mathematics; Algebra; Geometry; Advanced Algebra; Functions, Statistics, and Trigonometry; and Pre- Results were not reported by sex, race, or ethnicity. Calculus and Discrete Mathematics. Transitional Mathe- matics, which was originally designed for average to above average seventh graders (but can be started earlier or later), weaves together three more or less equal strands of major content: applied arithmetic, prealgebra, and elementary geometry. Algebra, geometry, and some discrete mathematics Systemic Initiative for Montana are integrated into all courses, as are statistics and probability. Mathematics and Science (SIMMS) Contact The SIMMS curriculum is divided into six levels, each UCSMP consisting of one year of work. Level 1 is typically offered 5835 South Kimbark Avenue to ninth graders, followed by level 2 in grade 10. After Chicago, IL 60637 completing level 2, students may choose between levels 3 Tel: 773-702-1130 and 4 and then proceed to either level 5 or 6 in the subse- Web site: http://socialsciences.uchicago.edu/ucsmp/ quent year. The sequence for potential math and science majors is 1-2-4-6. Levels 1 and 2 offer basic mathematical Results literacy. Fourteen results were found from comparison studies on the effect of UCSMP. The following UCSMP courses Contact were covered: Transitional Mathematics (2), Algebra (2), Kendall/Hunt Publishing Geometry (5), Advanced Algebra (4), and Pre-Calculus and 4050 Westmark Drive Discrete Mathematics (1). P.O. Box 1840 Number Dubuque, IA 52004-1840 Type of measure of results Results Effect size Tel: 800-542-6657 Standardized 5 In all five results, UCSMP No effect Web sites: http://www.simms-im.com achievement students scored higher. size reported http://www.montana.edu/~wwwsimms/ tests Curriculum-driven, 8 In all eight results, UCSMP No effect Results skill-specific tests students scored higher. size reported Teacher-based 1 No differences were found. No effect Twelve results from comparison studies were found on the measure/GPA size reported effect of SIMMS. Four results each focused on levels 1 and 2, and two each focused on levels 4 and 6. Results were not reported by sex, race, or ethnicity. 13
  14. 14. What Do We Know? Seeking Effective Math and Science Instruction Appendix B DETAILED FINDINGS OF THE REVIEW OF SCIENCE CURRICULA This appendix provides a list of the science curricula included in our review, together with a description of each curriculum and its related evaluation studies. Table 2 below lists the science curricula that we identified as having evaluation studies that met our criteria. TABLE 2. Science Curricula with Studies Grades Covered Curriculum Name Subject Matter K 12 Expeditionary Learning Outward Bound (ELOB) *[LEP] Whole School Reform K 8 Full Option Science System (FOSS) *[LEP] Multi-Science PK 8 Great Explorations in Math and Science (GEMS) Math & Multi-Science K 6 The Science Curriculum Improvement Study (SCIS) Multi-Science 2 8 National Science Curriculum for High Ability Learners Multi-Science 6 8 DESIGNS/DESIGNS II *[sex] Physical Science 6 8 Integrated Math, Science and Technology (IMaST) Math, Science & Technology 6 8 Center for Learning Technologies in Urban Schools (LeTUS) *[sex] Multi-Science 6 8 Learning By Design (LBD) Multi-Science 6 8 Science 2000/Science 2000+ Multi-Science 6 8 Science and Technology Concepts for Middle School (STC/MS) Multi-Science 6 9 Event-Based Science (EBS) Earth Science 78 Constructing Ideas in Physical Science (CIPS) *[sex] Physical Science 7 9 Foundational Approaches in Science Teaching (FAST) Multi-Science 7 9 Global Lab Curriculum (GLC) Multi-Science, Environmental Focus 7 9 Issues, Evidence and You (IEY)/SEPUP Multi-Science 9 11 BSCS: An Inquiry Approach Multi-Science 9 12 High Schools that Work (HSTW) Whole School Reform 9 12 Modeling Instruction in High School Physics *[sex] Physics 9 12 World Watcher/Learning About the Environment (LATE) *[urban] Environmental Science 10 12 Physics Resources and Instructional Strategies for Motivating Students (PRISMS) Physics Note: Shaded curricula are those for which we have found the strongest evidence of effectiveness, that is, quantitative evidence that (1) their use in instruction elicits higher achieve- ment/performance in students than other curricula to which they are compared on both standardized and/or state tests and on curriculum developed tests, or (2) they showed large effect sizes in terms of increasing student achievement. All science curricula listed in Table 2, however, have credible evaluations that show evidence of effectiveness. There are sev- eral curricula for which this evidence of effectiveness has not been collected but which might also qualify as effective should appropriate studies be conducted. An omission from this list of many curricula signifies merely that these curricula have not yet provided quantitative evidence of effectiveness that meets our criteria. *An asterisk marks curricula for which effectiveness data are provided for subgroups of students, indicated in brackets. (See full report on the web for discussion of research find- ings by subgroup.) 14
  15. 15. What Do We Know? Seeking Effective Math and Science Instruction A Description of the Science Curricula Number and Evaluation Studies Found Type of measure of results Results Effect size The attached descriptions of science curricula, which Content test, 1 Average student gains at No effect appear in alphabetical order, include the type of student unknown if self- both 9th and 10th grade size reported achievement measure used, the number and direction of designed or levels were between 20 state-issued and 25 percent. the results, and, if available, the size of any differences between groups. When known, effect sizes have been Note: Statistical significance levels were not reported. listed. Where results have been provided by race/ethnic- The field test was conducted across 10 states and ity, sex, or other demographic characteristics, we have re- included students in urban, suburban, and rural schools. ported these. The absence of such notation means that no However, data regarding specific demographic groups data were reported by subgroup. In the case of three cur- was not provided. ricula, descriptions of studies do not follow the normal format because results reported are not best described in that particular format. Center for Learning Technologies Note: Almost all impact studies for science curricula reported the statisti- in Urban Schools (LeTUS) cal significance of their results. Only differences that have reached the conservative minimum acceptable statistical significance level of .05 were LeTUS, developed by researchers at the University of included in the results reported for each study. If differences are statisti- Michigan and Detroit public school teachers, includes pro- cally significant, then there is another measure, called an effect size, that ject-based curriculum materials that build from district, shows how big the difference is. In our description of the study results, we provide effect sizes where available, although few studies reported state, and national standards to support the development these results. Effect sizes greater than .4 are considered large, between .2 of integrated science understanding for middle school stu- and .4 are considered moderate, and less than .2 are considered small. dents. The materials support students’ science learning through engaging them in inquiry about real world prob- lems, providing them with multiple opportunities to work with concepts, and integrating the use of learning tech- BSCS: An Inquiry Approach nologies in instruction. LeTUS is focused on learning This program introduces 9th, 10th, and 11th grade stu- about and developing a new machine to construct large dents to the core concepts in inquiry, the physical sciences, buildings and bridges, an area that has been identified as the life sciences, and the earth–space sciences as articu- of interest to young urban students. lated in the National Science Education Standards. In addi- Contact tion, the curriculum engages students in integration across Joseph Krajcik the disciplines in relevant contexts that explore the stan- School of Education dards related to science in a personal and social perspec- University of Michigan tive. This program provides high school students with an 610 East University Avenue, Rm. 4109 alternative to the traditional sequence of biology, chem- Ann Arbor, MI 48109-1259 istry, and physics. Included with this program is a profes- Tel: 734-647-0597 sional development component designed to help teachers Fax: 734-615-5245 and school districts implement the materials. E-mail: krajcik@umich.edu Contact Results Pamela Van Scotter Two evaluations of the LeTUS program were found, one Director that utilized skill-specific instruments and one that used The BSCS Center for Curriculum Development statewide achievement test scores to measure effectiveness. BSCS 5415, Mark Dabling Blvd. Colorado Springs, Colorado 80918 Type of Number Web site: http://www.bscs.org/page.asp?id=curriculum_ measure of results Results Effect size development|high_school_9-12|An_Inquiry_Approach Curriculum- 1 Significant content and Large effect size for driven,skill- process gains that in- content achieve- Results specific creased with program ment and more Most of the evaluative work on this was of the materials, tests revision and scale-up moderate effects for process skills. not student achievement; however, the results of a nation- wide field test are summarized below. (continued) 15

×