The document summarizes New York State's annual assessment and accountability discussion. It provides data on student performance on English and math assessments from 2006 to 2008. Achievement increased in most grades and subgroups in both subjects. The performance of English language learners and students with disabilities also improved overall. The state is proposing a growth model for accountability that measures student progress toward proficiency from year to year.

1. DATAG: Annual State Assessment/Accountability Discussion David Abrams Assistant Commissioner for Standards, Assessment, and Reporting DATAG Fall Conference October 3, 2008

2. Achievement is up in English statewide, except in Grade 8. Percentage of Students Scoring at Levels 3 and 4 Grade 3 185,603 198,457 195,777 Grade 4 190,951 197,499 197,016 Grade 5 201,262 202,133 198,022 Grade 6 204,249 204,463 200,505 Grade 7 210,735 211,839 207,278 Grade 8 212,320 213,971 209,180 Grades 3-8 1,205,120 1,228,362 1,207,778 Number Tested 2006 2007 2008 2008 3-8 ELA results

3. At all grade levels, the mean scale score was above 650, the score that denotes meeting the ELA standards. 2008 3-8 ELA results 650

4. More Students with Disabilities are meeting the English standards at every grade level, except Grade 8. Percentage of Students with Disabilities Scoring at Levels 3 and 4 Grade 3 23,811 26,692 27,285 Grade 4 26,474 28,281 29,983 Grade 5 28,987 29,985 30,661 Grade 6 28,883 29,055 31,195 Grade 7 29,237 29,842 31,180 Grade 8 29,119 29,514 31,077 Grades 3-8 166,511 173,369 181,381 Number Tested 2006 2007 2008 2008 3-8 ELA results

5. The achievement gap in English is narrowing for Black and Hispanic students. Percentage of Students Scoring at Levels 3 and 4 2008 3-8 ELA results

6. The performance of ELL students on the English test improved overall. Percentage of ELL Students Scoring at Levels 3 and 4 2006 2007 2008 Grade 3 3,684 17,093 17,433 Grade 4 4,379 14,200 14,683 Grade 5 6,686 11,480 11,916 Grade 6 5,585 9,934 10,323 Grade 7 6,234 9,299 9,798 Grade 8 5,852 10,076 9,046 Grades 3-8 32,420 72,082 73,199 Number of ELL Students Tested 2008 3-8 ELA results

7. Achievement in math is up statewide. Percentage of Students Scoring at Levels 3 and 4 Number of Students Tested 2006 2007 2008 Grade 3 201,956 200,217 197,500 Grade 4 202,791 199,391 198,730 Grade 5 209,242 203,956 199,746 Grade 6 211,428 206,220 202,058 Grade 7 217,308 213,436 209,039 Grade 8 219,414 215,415 210,716 Grades 3-8 1,262,139 1,238,635 1,217,789 2008 3-8 math results

8. At all grade levels, the mean scale score was above 650, the score that denotes meeting the math standard. 650 2008 3-8 math results

9. More Students with Disabilities met the math standards at every grade level. Percentage of Students with Disabilities Scoring at Levels 3 and 4 Grade 3 27,045 26,780 27,325 Grade 4 29,043 28,327 30,072 Grade 5 30,290 29,960 30,662 Grade 6 30,077 29,040 31,119 Grade 7 29,791 29,659 31,037 Grade 8 29,539 29,305 30,899 Grades 3-8 175,785 173,071 181,114 Number Tested 2006 2007 2008 2008 3-8 math results

10. The achievement gap in math is narrowing for Black and Hispanic students. Percentage of Students Scoring at Levels 3 and 4 2008 3-8 math results

11. Regents English Students Tested and Scores Received: 1996-2007 All Students

12. Regents Mathematics Students Tested and Scores Received: 1996-2007 Data for 1999–2002 include both Mathematics A and Sequential Mathematics, Course I. Data for 2003 forward are for Mathematics A only. All Students

13. Regents Global History and Geography Students Tested and Scores Received: 1996-2007 The data for 2001 forward are for the Regents Global History and Geography examination only. The data for 2000 are for both the Regents Global History and Geography and Global Studies examinations. The data for previous years are for Regents Global Studies only. All Students

14. Regents U.S. History & Government Students Tested and Scores Received: 1996-2007 All Students

15. Regents Living Environment/Biology Students Tested and Scores Received: 1996-2007 All Students Data for 1996 through 2000 are for the Regents Biology examination. Data for 2001 are for both the Regents Biology and the Regents Living Environment examinations. Data for 2002 through 2005 are for the Regents Living Environment examination.

16. Percentage of Students Graduating with Regents or Local Diploma After 4 Years Results Through June, All Students By Need/Resource Capacity Category Total Cohort Graduation Rate Public Schools 2001, 2002, 2003 Cohorts

17. Percentage of Students Graduating with Regents or Local Diploma After 4 Years Results Through June 2007 and Through August 2007 By Need/Resource Capacity Category 2003 Total Cohort Graduation Rate Public Schools, All Students

18. Students Who Started 9 th Grade in 2003, Results After Four Years 69% of students statewide in the 2003 Total Cohort graduated by June 2007; 17% were still enrolled. 2003 Total Cohort Students = 220,332 All Students in Public Schools

19. Students Who Started 9 th Grade in 2003, Results After Four Years 71% of students statewide in the 2003 Total Cohort graduated by August 2007; 15% were still enrolled. 2003 Total Cohort Students = 220,332 All Students in Public Schools

20. Total Cohort Graduation Rate All Students Percentage of Students Graduating with Regents or Local Diploma After 4, 5 and 6 Years Results Through June Cohort Membership 212,272 214,729 220,332

21. 2003 Total Cohort Graduation Rate By Racial/Ethnic Group, By Gender Percentage of Students Graduating with Regents or Local Diploma After 4 Years Through June 2007 Asian American Indian / Alaska Native Hispanic Black White Solid colors: Females Stripes: Males

22. Total Cohort Graduation Rate for By Racial/Ethnic Group 2001, 2002, 2003 Total Cohorts Percentage of Students Graduating with Regents or Local Diploma After 4 Years Through June

23. Total Cohort Graduation Rate for By Racial/Ethnic Group, By Gender 2001, 2002, 2003 Total Cohorts Percentage of Students Graduating with Regents or Local Diploma After 4 Years Through June Solid colors: Females Stripes: Males

24. Total Public High School Graduates Each Year 164,790 2006-07 161,732 2005-06 153,202 2004-05 153,202 2003-04 143,818 2002-03 143,070 2001-02 141,634 2000-01 141,510 1999-00 140,365 1998-99 139,531 1997-98 138,990 1996-97 136,754 1995-96 Number of Students Earning Regents or Local Diplomas Year

25. Integrated Algebra Pre-standard setting measurement review committee Standard setting committee Post-standard setting measurement review committee

26. Standard Setting Use operational sample: N=142,286 Calibration using the operational sample Rasch and Partial Credit Model Produce ordered item book, RP 0.67: 39 operational items 22 anchor items

27. Standard Setting Two committees, 30 and 31 people, completely independent meetings Item mapping methodology Process: Achievement Level Descriptors Total of three rounds Use of p value and impact data

28. Integrated Algebra: Standard-Setting

29. Integrated Algebra Year II Identify representative sample Use enrollment data Use Need/Resource Categories Validity study: Reliability analysis Classification accuracy Operational and field test results contrast IRT model fit and dimensionality analysis Summary statistics by NCLB subgroups

30. Public Forum on New York State’s NCLB Growth Model Proposal David Abrams Assistant Commissioner for Standards, Assessment, and Reporting

31. “… to ensure that all children have a fair, equal, and significant opportunity to obtain a high-quality education and reach, at a minimum, proficiency on challenging state academic achievement standards and state academic assessments.” Purpose of No Child Left Behind

32. Chapter 57 of the Laws of 2007: Growth Model “ By the start of the 2008-2009 school year, the Regents shall establish, using existing state assessments, an interim, modified accountability system for schools and districts that is based on a growth model, subject to approval of the United States department of education where required under federal law.”

33. Key Questions: How do we design accountability models that move students from low performance to proficient as well as from proficient to distinction? How can we ensure that improved scores represent improved learning? How do we take data and turn it into actionable information that improves teaching and learning? How do we move from beating the odds to changing the odds? Next Generation Accountability System Design

34. Accountability: Status vs. Growth Status Models: take a snapshot of a subgroup’s or school’s level of student proficiency at one point in time and often compare that proficiency level with an established target. Growth Models: measure progress by tracking the achievement scores of the same students from one year to the next to determine student progress.

35. Why Growth? Types of Performance Improve-ment Status Low Growth High Growth Low Status High Status Growth Acceleration Effective-ness Achieve-ment Status Change Low/Low Low/High High/Low High/High Status/Growth Combinations

36. Betebenner, Jan. 2008, for RI project

37. Two Types of Growth Targets Policy driven targets start with a policy goal (what should be) and then establish the targets for performance that are necessary to achieve this goal. Data driven targets start with historical performance (what has been) and use that as a basis to project what should be expected of the units to be measured.

38. New York State’s Proposal Use growth in two different ways: To make more refined AYP determinations (must be approved by USED) To supplement AYP and make a more comprehensive system, attending in particular to growth of students who are proficient or higher (not necessary to be approved by USED)

39. Constraints Using growth in AYP is highly constrained: Focused on reaching proficiency and reducing the achievement gap Specifics dictated by USED Proposal must be submitted by Oct. 15 to USED; if approved, would apply to 2008-09 data Using growth outside of AYP is less constrained, but there is less agreement on approaches: NYSED will work with partners to create this growth proposal Regents would like proposal by end of this school year

40. New York State: Local Initiatives A number of NYS districts have developed local growth and value-added models. The two most prominent are: NYC’s Progress Report initiative. Capital Region BOCES initiative. These initiatives are neither endorsed by SED nor require SED’s endorsement. These initiatives are not constrained by USED’s growth model guidelines and were not designed for use in making AYP decisions. These models can inform the development of State growth and/or value-added models for which USED’s permission is not required.

41. SED’s Interim Growth Model Design Principles Interim Growth Model shall be implemented in 08-09 school year (with Regents and USED approval). Model shall meet core principles of Spellings 11/21/05 correspondence. Model shall be based upon NY’s current State assessment program & shall not require the implementation of new assessments. Model shall utilize such data as is currently collected through State data collection processes and shall not require the collection of new data elements. Model’s purpose shall be to make more refined determinations of student progress, identify with greater precision high performing schools and districts, and support greater differentiation in support and services to schools and districts in need of improvement.

42. SED’s Interim Growth Model Design Principles The model shall be based upon measuring whether students are proficient or on track towards proficiency within a prescribed time period. Model shall use an “open architecture.” All calculations should be transparent. The interim growth model shall be a stage in a process leading, by 2010-11, to the development of an enhanced system that includes a value-added model. The NCLB model should be combined with a State model that includes consequences and/or incentives for promoting growth for all students, while placing no school or district at risk of failing to make AYP, if it would make AYP under the current status model.

43. USED Seven Core Principles All students proficient or on track to proficiency by 2014; set annual goals to close subgroup gaps. Expectations for annual achievement based on meeting grade level proficiency, not based on demographic characteristics or school characteristics. Produce separate decisions for math and ELA/reading. Include students, subgroups, schools, and districts in accountability.

44. USED Core Principles – cont. Assessment data: annual, 3-8 & high school, operational for more than one year (i.e., at least two years’ of data), produce comparable results grade-to-grade and year-to-year; approved in Peer Review. Track student progress (longitudinal). Include participation rates and additional academic indicator for accountability.

45. USED Peer Review Additional Specifications Fully approved assessment system Uniform minimum-n for all groups No confidence intervals for growth Very limited recalculation of student growth target Cannot use with multiple other non-Status approaches, such as Safe Harbor and Index Apply growth to all students for reporting and accountability (preference) Report student growth results (preference) State has vertical scale (preference?)

46. SED Draft Proposal For grades 3-8, utilize a “proficiency plus” growth model for grades 3-8 similar to North Carolina’s approved model. For high school, utilize a “value tables” model similar to Delaware’s approved grades 3-8 model. Include an enhanced middle level and high school component in the proposal. Build a “growth for all” State component that sets growth targets for all students, including those who are already proficient.

47. NCLB Growth Model: General Approach If a student scores proficient or above (Level 3/4) in the current year, include that student’s results in the Performance Index as is done under the present status model. Use growth to check whether students who did not yet score proficient (Level 2) have grown enough that it is likely they will become proficient within a designated amount of time. For purposes of calculating the Performance Index, give schools and districts “full credit” for any student who either scores proficient or above or who is deemed to be on track for proficiency.

48. “ On-Track” Growth to Proficiency Example 3 4 5 6 7 8 Prof. 6 Prof. 5 Prof. 4 Prof. 3 Observed growth Gr. 3-4 projected to Gr. 8 Proficient On track to be proficient Proficient or Prof. 8 Prof. 7

49. 3-8 Growth Model: Simplified Example Level 3 Scale Score = 650. Billy scores a 614 in Grade 3 ELA. Billy is 36 points below proficiency (650- 614). Billy has four years to become proficient. Billy must close the gap by ¼ (9 points) in Grade 4. Billy’s proficiency target in Grade 4 is 623 (614 + 9). Billy scores 635 in Grade 4. Billy now has three years to become proficient. Billy must close the gap by 1/3 (5 points) in Grade 5. Billy’s proficiency target in Grade 5 is 640 (635 + 5).

50. Growth Model: Middle School Extension Students in middle school would be evaluated on whether they made sufficient growth to become proficient by the designated high school Regents examination. The designated high school target is proficient on the Regents Examination in Integrated Algebra and proficient on the Regents Comprehensive Examination in English. Students in middle school would have until the target assessment to be projected proficient; the number of years permitted would be based upon the grade the student entered middle school. This middle school extension will only apply to schools (and their subgroups), not to district AYP decisions in instances where students transfer among schools within a district.

51. Middle Level Extension: Simplified Example Level 3 Math Regents Exam is equated to scale score of 663*. Billy scores a 623 in Grade 5 Math. Billy enrolls in a new middle school in Grade 6. Billy is 40 points below proficiency (663-623). Billy has four years to become proficient. Billy must close the gap by 1/4 (10 points) in Grade 6. Billy’s proficiency target in Grade 6 is 633 (623 + 10). If Billy remained in his original school in Grade 6, then his proficiency target would have been 637 . (650**-623= 27/2 years = 13.5 + 623 = 637.) *Actual equated score not yet determined **Represents Grade 7 Level 3 Scale Score

52. 3-8 Growth Model: Implications for Schools 136 5 5 35 46 9 Totals NA NA 4 1 Less Number of Students On Track Towards Proficiency 130 NA 5 35 50 10 Number of Students (2 Index Points per Student) (2 Index Points per Student) (2 Index Points per Student) (1 Index Point per Student) (0 index points) Performance Index On Track Towards Proficiency Level 4 Level 3 Level 2 Level 1

53. Growth Model: High School Extension Students who enter high school having scored Level 1 or at low-Level 2 on the Grade 8 ELA or Mathematics tests are considered on track towards proficiency if they score between 55-64 on the designated Regents examinations prior to Grade 12. Schools have five years for certain English Language Learners, certain students with disabilities, and students who enter high school far below standards to demonstrate proficiency in English language arts and mathematics. Value table is interim model to be used for cohorts prior to the 2008-2009 school year cohort (i.e. next three years).

54. Growth Model: High School Values Table 200 100 0 4 200 100 0 3 200 100 0 2-plus 200 200 0 2-minus 200 200 0 Level 1 65 and higher 55-64 Less than 55 Initial Grade 8 level Score on High School Regents Exam

55. Timeline for NCLB Growth Models July/September 2008 : Proposed model submitted to Board of Regents for Review. September/October, 2008: Discussion with the Field of the Model. October, 2008: Submission to USED. Fall 2008: Approval of model by USED. September 2009: Use of model to make AYP decisions based on 2008-09 school year data, subject to availability of resources.

56. Building a “Growth for All” Model Regents have directed SED to provide recommendations for how to hold schools and districts accountable for growth of students beyond proficiency as part of the process of moving towards creation of a value-added accountability model. This “growth for all” model can be separate from NY’s NCLB accountability system and need not be constrained by NCLB growth model rules. The Regents will need to decide what rewards and/or consequences should be based upon a “growth for all” model.

57. Building a “Growth for All” Model One possibility would be to modify the current process for designation of High Performing and Rapidly Improving schools to include a “growth for all” component: other possibilities include rewards, regulatory relief, and differentiated consequences.

58. Approaches to “Growth for All” Models Student growth in terms of what other reference schools or reference groups have achieved (e.g. “peer schools,” “low-income Hispanic students”). Growth of students compared to other students who started with similar growth histories. Student growth in relation to statistical expectations for what the student would have learned with a typical teacher/school.

59. Define “Value-added” Increase over previous score or performance Increase over what was expected Attribution of performance changes (increases/decreases) to agents/conditions

60. What do we value? “A Year’s Worth of Growth” More than “A Year’s Worth of Growth” Not Going Backwards Relative Growth Absolute Growth Growth to Proficiency Growth to a Point Beyond Proficiency

61. What happens next? Questions and Answers Small group discussions and completion of surveys Summary and next steps

62. More Information To submit questions or requests for more information, please e-mail: [email_address]