Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

RELNW 1.2.40 OSPI-Final Report-e RS OTA Report 12-01-2009

  • Login to see the comments

  • Be the first to like this

RELNW 1.2.40 OSPI-Final Report-e RS OTA Report 12-01-2009

  1. 1. REL Technical Assistance Report A Linking Study of the Grade 3 DIBELS ORF Test and the WASL Reading Test Prepared by: Richard H. Smiley, Ph.D. Education Northwest November 30, 2009
  2. 2. Summary At the request of the Washington Office of the Superintendent of Public Instruction (OSPI), Education Northwest conducted a study of the relationship between two tests used in Washington State. The project consisted of a linking study of the Oral Reading Fluency test of the Dynamic Indicators of Basic Early Literacy Skills (DIBELS ORF) and the reading comprehension test of the Washington Assessment of Student Learning (WASL). Subjects for the study were grade 3 students in all 94 Reading First schools in Washington during the 2007/08 school year. The research methodology employed a single group design that linked both fall and winter 2007 DIBELS ORF scores with spring 2008 WASL reading comprehension scores for grade 3 students. A correlation coefficient was computed for the winter DIBELS ORF and spring WASL reading test. This was followed with a regression analysis of the fall DIBELS and spring WASL scores and the computation of descriptive statistics for various student subpopulations. Finally, a table of conditional probabilities at selected levels was computed for the corresponding DIBELS ORF scores. Results of the study found a correlation of r = .62 and r = .64 between the fall and winter DIBELS ORF and the spring WASL, respectively. The regression analysis showed that the fall DIBELS ORF was a reasonable predictor of student performance on the subsequent WASL reading test. On the fall DIBELS, 70 percent of the Benchmark, 23 percent of the Strategic, and 7 percent of the Intensive students scored proficient or higher on the WASL reading comprehension test the following spring. Seventy-five percent of students who obtained a cutscore of 70 on the fall DIBELS ORF scored at or above the proficient level on the spring WASL. Ninety-five percent of students who obtained a cutscore of 135 on the fall DIBELS ORF scored at or above the proficient level on the spring WASL reading test.
  3. 3. Introduction The No Child Left Behind Act of 2001 (NCLB) requires states to establish accountability systems to ensure that all students meet academic proficiency by 2014. Under NCLB, states must develop academic standards for students and examinations to measure their progress in attaining these standards. In Washington, the performance standards are known as the Essential Academic Learning Requirements and the exam developed to measure them is the Washington Assessment of Student Learning (WASL). At grade 3, the WASL, administered each spring, contains a reading test that primarily focuses on reading comprehension. In 2003, Washington implemented a Reading First program funded under Title I of NCLB. The Washington Reading First program uses the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) for progress monitoring and screening as well as for measuring school-level outcomes. Reading First schools in Washington are required to administer the DIBELS in fall, winter, and spring in grades K–3. In grade 3, the Oral Reading Fluency score (ORF) is the primary measure of student progress in reading and scores consist of the number of words in a reading passage correctly read in one minute. The publisher of the DIBELS classifies student scores on the basis of three categories of risk for reading failure: low risk (Benchmark), some risk (Strategic), and high risk (Intensive). Since 2003, Education Northwest has been the evaluator of Washington’s statewide Reading First program. On several occasions, staff at the Washington Office of Superintendent of Public Instruction (OSPI) expressed an interest in a linking study of the DIBELS ORF and the WASL grade 3 reading test. The authors of the DIBELS Administrative and Scoring Guide recommend such studies (Good and Kamanski 2003) and OSPI staff members pointed to similar studies that have been conducted in Arizona, Florida, Idaho, Colorado, Ohio, and Oregon. In 2007, Montana joined the ranks of states with linking studies between the DIBELS and statewide standards-based examinations. OSPI staff wanted to know if fall DIBELS ORF scores could be used to predict which students were at risk of scoring below the proficient level on the spring WASL grade 3 reading test. To address this issue, OSPI requested that REL Northwest, operated by Education Northwest, investigate two questions: 1. What is the correlation between the grade 3 winter DIBELS ORF test and the grade 3 WASL reading test?
  4. 4. 2. What threshold fluency scores on the fall grade 3 DIBELS are associated with a high probability of scoring at the proficient level on the grade 3 WASL reading test? Methodology The study employed a single-group research design. The form of linking used was projection (Kolen and Brennan 2004) involving the DIBELS ORF test and the WASL reading comprehension test administered to the same students. The target population was all grade 3 students in Washington Reading First schools. Since the 94 Reading First schools in Washington are in only 35 school districts, data were collected from all Reading First students rather than a sample. Scaled scores on the WASL (Pearson Educational Measurement 2008) and “number correct” scores for the DIBELS ORF test (Good and Kaminski 2003) were the variables used in the study. Data Collection Data for the study came from two online databases: one operated by OSPI, which contained WASL test scores and demographic information, and the other operated by the University of Oregon and containing DIBELS test information and demographic information. Student records from these two sources were matched to create two datasets for analysis. The first dataset resulted from matching demographic information, fall 2007 DIBELS ORF scores, and spring WASL reading scores. A total of 3,506 records containing complete information were obtained from this matching. The second dataset resulted from matching demographic information, winter 2007 DIBELS ORF scores, and spring WASL reading scores. A total of 3,957 records with complete information were obtained for the second dataset. Data analysis consisted of a Pearson correlation of winter DIBELS and spring grade 3 WASL reading scores. Additionally, a regression analysis was conducted using fall 2007 DIBELS ORF scores as a predictor of spring 2008 WASL reading scores at grade 3. Also, conditional probabililities for passing the WASL, based on DIBELS scores, were calculated. Finally, test statistics for the major student subgroups were calculated. Results There was a significant correlation between winter DIBELS ORF scores and spring WASL reading scores (r=.64, p. ≤ .01). For fall 2007 DIBELS ORF, the correlation was slightly lower but still significant (r=.62, p. ≤ .01). These correlations are similar to those for the grade 3 DIBELS ORF and the Oregon Statewide Assessment (r=.67, p. ≤ .001) as reported by Good, Simmons, and Kame’enui (2001). 2
  5. 5. Figure 1 shows the relationship between the fall 2007 DIBELS ORF and the spring 2008 WASL reading score at the grade 3 level. Figure 1. Relationship between fall 2007 DIBELS ORF scores and spring 2008 WASL scores Source: Author’s analysis of data. Students scoring at or above the horizontal line scored at the Proficient or Advanced level on the WASL reading test while those beneath scored Below Proficient or Basic. The vertical lines represent demarcations between the three Instructional Support Recommendations (ISR) for the fall administration of the DIBELS ORF test as reported in the administration manual for the DIBELS (Good and Kamanski 2003). The diagonal regression line shows that the DIBELS is a reasonably good predictor of the WASL reading test, although predictions seem better at the lower end of the DIBELS ORF score continuum than at the upper end. For example, 33 students obtained a spring WASL score of 479, the second highest score possible. Although their previous DIBELS ORF ISR was at the Benchmark level, these scores ranged from 89 to 225. However, since the primary concern is identifying students who may not obtain a proficient score on the WASL, scores at the lower end of the fall DIBELS continuum appear to be reasonably good predictors for at-risk students. 3
  6. 6. To further illustrate the relationship between the two tests, students were divided into two groups on the basis of whether they scored at or above the cutscore of 400 for proficiency on the spring 2007 WASL reading test (Yes) or whether they scored below it (No). In figure 2, the percentage of students in each of the fall 2007 DIBELS ISRs is shown by the Yes or No WASL classification. As expected the largest group of students in the Yes group had ISR scores at the Benchmark level (70 percent) while the smallest was Intensive (7 percent). Conversely, the largest group in the No category was at the Intensive ISR (42 percent) while the smallest was at the Benchmark ISR (26 percent). The Strategic ISR group was relatively evenly represented between the Yes and No categories (23 percent and 31 percent, respectively). Figure 2. Percent of students above and below proficiency on the spring 2008 WASL reading test by fall 2007 DIBELS ORF ISR. Source: Author’s analysis of data. For a breakdown of performance by student subpopulations, see appendix A. To show selected threshold scores on the DIBELS ORF associated with proficient scores on the WASL reading test, a table of conditional probabilities was calculated. In table 1, three cutscores on the fall DIBELS ORF are shown with the associated percent of students obtaining scores at the proficient or above level on the spring WASL reading 4
  7. 7. test. These cutscores range from a fall DIBELS score of 108 (75% score proficient or above) to 135 (95% score proficient or above). Table 1: Percent of Students Scoring Proficient or Above on the Spring WASL Reading Test Based on Fall DIBELS ORF Cutscores. Fall DIBELS ORF Cut Score Percent scoring proficient or above on spring WASL reading test 1 70 75% 2 105 90% 3 135 95% Source: Author’s analysis of data. Conclusions  Pearson correlation coefficients between the fall and spring DIBELS ORF and the spring WASL reading test are r=.62 and r=.64, respectively. These correlations are significant at the p.≤.01 level of confidence and are similar to those between the grade 3 Oregon Statewide Assessment and the DIBELS ORF as reported by Good, Simmons, and Kame’enui (2001).  The grade 3 fall DIBELS ORF is a reasonable predictor of the spring WASL reading test. The prediction is stronger at the lower end of the DIBELS score continuum than at the upper end.  The percentages of students in the three fall DIBELS ORF ISR categories who did not pass the spring WASL reading test were 42 percent at Intensive, 31 percent at Strategic, and 26 percent at Benchmark. For those students who did pass the WASL, the percentages were 7 percent at Intensive, 23 percent at Strategic, and 70 percent at Benchmark.  The percentage of students obtaining a proficient or better score on the spring WASL reading test based on fall DIBELS ORF cutscores ranged from 75 percent for a DIBELS cutscore of 70 to 95 percent for a DIBELS ORF cutscore of 135. 5
  8. 8. References Good, R.H., and Kaminski, R.A. (2003). DIBELS Dynamic Indicators of Basic Early Literacy Skills, 6th Ed: Administration and scoring guide. Frederick, CO: Sopris West Educational Services. Good, R.H., Simmons, D.C., and Kame’enui, E. (2001). The importance and decision- making utility of a continuum of fluency-based indicators of foundational reading skills for third-grade high-stakes outcomes. Scientific Studies of Reading, 5, 257–288. Kolen, M.J., and Brennan, R.L. (2004). Test equating, scaling, and linking: Methods and practices (2nd ed.). New York, NY: Springer. Pearson Educational Measurement. (2008). Washington Assessment of Student Learning grade 3 2007 technical report. Olympia, WA: Office of the Superintendent of Public Instruction. 6
  9. 9. Appendix A Subgroup Breakouts by WASL Proficiency and DIBELS ISR Student Sub- Groups1 WASL- Proficient2 N Fall 2007 ISR r3 Intensive Strategic Benchmark All Students Yes 2032 7% 23% 70% .62* N=3506 No 1474 43% 31% 26% Male Yes 959 7% 24% 69% .62* N=1817 No 858 41% 33% 27% Female Yes 1073 6% 23% 71% .62* N=1689 No 616 45% 30% 25% American Indian Yes 68 2% 19% 79% .70* N=123 No 55 35% 38% 27% Asian Yes 122 2% 9% 89% .71* N=195 No 73 29% 33% 38% Black Yes 168 5% 23% 71% .60* N=317 No 149 36% 31% 33% Hispanic Yes 898 8% 26% 66% .61* N=1775 No 877 44% 32% 24% White Yes 754 7% 23% 70% .61* N=1060 No 306 46% 27% 27% Title I Reading Yes 1233 6% 22% 72% .62* N=2139 No 906 41% 32% 27% Non Title I Reading Yes 799 8% 26% 67% .63* N=1367 No 568 45% 30% 25% Bilingual/ESL Yes 471 10% 28% 62% .60* N=1225 No 754 48% 31% 21% Non Bilingual/ESL Yes 1561 6% 22% 72% .59* N=2281 No 720 37% 31% 32% Special Ed. Yes 95 12% 39% 49% .61* N=377 No 282 60% 22% 19% Non Special Ed. Yes 1937 7% 23% 71% .60* N=3129 No 1192 38% 34% 28% 1 Because of low frequencies, data for the Hawaiian or Other Pacific Islander, Multiracial, and Not Provided race/ethnic classifications are not provided. 2 Students coded “Yes” scored Proficient or Advanced and students coded “No” scored Below Proficient or Basic on the spring 2008 WASL reading test. 3 Correlation between fall DIBELS ORF and spring WASL reading test *p≤.01. 7
  10. 10. Appendix A Subgroup Breakouts by WASL Proficiency and DIBELS ISR Student Sub- Groups1 WASL- Proficient2 N Fall 2007 ISR r3 Intensive Strategic Benchmark All Students Yes 2032 7% 23% 70% .62* N=3506 No 1474 43% 31% 26% Male Yes 959 7% 24% 69% .62* N=1817 No 858 41% 33% 27% Female Yes 1073 6% 23% 71% .62* N=1689 No 616 45% 30% 25% American Indian Yes 68 2% 19% 79% .70* N=123 No 55 35% 38% 27% Asian Yes 122 2% 9% 89% .71* N=195 No 73 29% 33% 38% Black Yes 168 5% 23% 71% .60* N=317 No 149 36% 31% 33% Hispanic Yes 898 8% 26% 66% .61* N=1775 No 877 44% 32% 24% White Yes 754 7% 23% 70% .61* N=1060 No 306 46% 27% 27% Title I Reading Yes 1233 6% 22% 72% .62* N=2139 No 906 41% 32% 27% Non Title I Reading Yes 799 8% 26% 67% .63* N=1367 No 568 45% 30% 25% Bilingual/ESL Yes 471 10% 28% 62% .60* N=1225 No 754 48% 31% 21% Non Bilingual/ESL Yes 1561 6% 22% 72% .59* N=2281 No 720 37% 31% 32% Special Ed. Yes 95 12% 39% 49% .61* N=377 No 282 60% 22% 19% Non Special Ed. Yes 1937 7% 23% 71% .60* N=3129 No 1192 38% 34% 28% 1 Because of low frequencies, data for the Hawaiian or Other Pacific Islander, Multiracial, and Not Provided race/ethnic classifications are not provided. 2 Students coded “Yes” scored Proficient or Advanced and students coded “No” scored Below Proficient or Basic on the spring 2008 WASL reading test. 3 Correlation between fall DIBELS ORF and spring WASL reading test *p≤.01. 7

×