Value-added assessment

1. Value-Added Carol Thompson July 2010

2. Where we are: Adequate Yearly Progress (AYP) NCLB measures school performance based on AYP, which calculates the percentage of various student populations that annually met or exceed the state’s academic standards, toward a universal fixed point.

3. No Child Left Behind Benefits Set measureable goals Reduce achievement gap among subpopulations Federal funding provided All students should reach proficiency standards in reading and math within 12 years

4. With NCLB States can choose own testing. Standardized tests measure achievement levels, but not change. Systematic evaluation is necessary to determine which educational approaches are working and which are not.

5. Assumptions IF . . . good schools are those that have high test scores, . . . then . . .

6. Assumptions Bad schools are those that have low test scores.

7. Assumptions Students with higher scores are learning more. HOWEVER . . . Schools that admit students with low scores and raise them to average are better schools than those that admit high scoring students and graduate them at the same level.

8. comparison Under NCLB we make comparisons across cohorts. Example: This year’s reading scores in 5th grade are higher than last year’s reading scores. Does that show improvement or were this year’s students ahead of last year’s students? Value-added measures the progress of each child where they are as they progress through school. Value-added measures change, not just achievement.

9. Purpose of Value-Added Data The value-added approach focuses on changes in test scores over time, rather than on a single test score at a given moment. Inequity of standardized-results: Some schools doing a good job of teaching severely disadvantaged students could be sanctioned Some poor schools ‘shepherding’ top students could be imitated, Some excellent schools using effective strategies to help high-risk students may NOT be imitated due to below average test scores. (Crane, 2002)

10. Value added Value-added Assessment Analysis of student achievement that uses longitudinal student achievement data to obtain estimates of the impact schools or teachers have on student learning as measured by test scores. It measures individual student progress from a relative starting point.

11. Value-added Levels the playing field,. Calculates a projected test score for a student in a given grade and subject, and Bases the projected score on the student’s prior academic achievement.

12. Value–Added Is a way of analyzing test data that measures both teaching and learning, Allows us to see how teachers add to a student’s knowledge over and above what their families and communities do, and Separates student effects from school effects.

13. Value-added IF Achievement test data is available for each student each year, Highly correlated scales of curricular objectives are used, The scales have sufficient stretch to measure progress of previously low and high scoring students, and Scales have appropriate repeatabiities, THEN All data that meet these conditions can be used in a value-added assessment system, regardless of test source. (Crane, 2002)

14. School effectivenes The “differences in schooling effectiveness is the dominant factor affecting the speed that students move up the ‘ramp’ of curriculum.” (Sanders, 2000).

15. Teacher quality “An accountability system based on the academic progress of student populations is one that will hold people accountable for things over which they have control, rather than for things over which they do not. Teachers do have primary control of the rate of academic progress of their students. (Sanders, 2000)

16. Achievement vs. growth Achievement is best predicted by family income. Growth is best predicted by the quality of instruction. Teachers have control over the quality of their instruction.

17. Classroom patterns Disproportionate attention is given in classrooms to ‘bubble kids’—those who score just below the state standard for NCLB; more difficult kids get help as a result. Research from Tennessee revealed three classroom patterns, shed, reverse shed, and tepee. The names reflect the shape of the slope. Even though these patterns can be found in any classroom, they occur disproportionately in some circumstances. Data is based on three year averages.

18. Shed pattern Typically found in low-income communities. Elementary teachers focus on the low achievers in order to get high gains while previous high-achievers get low gains.

19. Shed pattern If this pattern continues through elementary, there will be few high achieving children by middle school.

20. Reverse shed pattern Opposite focus found in high income communities. Teachers concentrate on their highest performers.

21. Reverse shed pattern Low achievers get low gains while previous high achievers continue to get high gains.

22. Tepee pattern Prevalent in most classrooms in most communities. Teachers focus on the average student; also called teaching to the middle.

23. Tepee pattern This results in previously average achievers getting high gains and both previously low and high achievers getting low gains.

24. Teaching to the Middle Biggest impediment to higher achievement are the years where individual students aren’t making realistic growth. A pattern can exist whereby lower scoring students are given opportunities to make progress but earlier high achieving students are held to the same pace and place as their lower achieving peers. As this pattern repeats over the years it becomes a self-fulfilling prophecy that these students lose ground (Sanders, 2002).

25. Teaching to the Middle Value-added provides valuable data for teachers to use in focusing their instructional practices with planned intentionality. As educators we need to remember that high achieving students come from housing projects, remote rural areas, and million dollar homes.

26. Teacher variability Once again, data suggests that teacher quality may be the single most important in-school factor determining how much students learn. (Crane, 2002, Sanders & Rivers, 1996; Jordan, Mendro, & Weerasinghe, 1997; Haycock, 1998)

27. Teacher Quality The Sanders and Rivers study found that good teachers raised students math scores at least 2-3 years into the future (regardless of class size) The Sanders study (1991-1995) found the top 1/5 of teachers raised students’ achievement test scores 39 percentile points more than teachers of bottom 1/5. (These effects were consistent whether heterogeneous or homogeneous grouping or excelling vs. struggling students.)

28. Teacher Quality Layered model—all student achievement data for 5 years is used simultaneously over all subjects tested, linking each year’s data to the current and previous teachers.: teachers given credit for teaching beyond standards assessed as well as not held accountable for one year’s poor performance due to student illness, drug involvement, etc. Accommodates ‘real world’ situations; different modes of instruction such as self-contained, team teaching, fractured student records, and data from non-vertically scaled tests.

29. Teacher Variability Struggling students aren’t randomly distributed into classrooms. They are found disproportionately in classrooms where they receive poor instruction.

30. Teacher variability The best way to improve test scores is to improve teacher quality. This can be done by applying lessons learned from value-added analysis to teacher education and professional development. Crane, 2002

31. Teacher variability Personnel decisions can also be made given value-added data. Incentives could be offered. Help offered to teachers who raised test scores the least. Teachers promoted to instructional leadership positions who raised scores the most. Over time, lowest performing teachers would leave the profession.

32. Teacher effectiveness “Of all the factors we study—class size, ethnicity, location, poverty—these pale to triviality in the face of teacher effectiveness.” Sanders address to Metropolitan School Board in Nashville, 2010.

33. Sustained growth over time By having sustained growth over time, we will begin to close the gap and all students will still be making gains.

34. Sustained Academic Growth Data is not the enemy but a valuable tool in an educator’s toolbox they can use to fine tune instruction to provide the best opportunity for every child. (Sanders, 2000)

35. Advantages of value-added Tests don’t have to be nearly so closely aligned as they do when judgments are made from a single year of test results each year. Reporting of simple test averages is singularly inaccurate. Disaggregation of socioeconomic strata is an improvement on raw score reporting. Use of regression models are sometimes labeled as ‘value-added’. (Sanders, 2000)

36. Additional information A value-added analysis is statistical and there is always a margin of error: they are better able to make distinctions at either end with more accuracy than the middle. Distinguishing between true learning and teaching to the test: can limit effects by changing the tests frequently.

37. Local example of value-added Dallas measures individual student progress from a relative starting point. It measures only that knowledge that the school/teacher is responsible for imparting, calculating the value-added by each school. (Toch, 2005)

38. Benefits of Value-Added Provide a more accurate picture of school, districts, and states success/failure. Generate objective measures of teacher performance that could be used to raise teacher quality. Provide useful data for evaluating school reform programs (Crane, 2002). Use of up to five years data for each student with different subjects tested and item analysis (40+ each subtest) could result in over 1,000 items collectively contributing to information array for that student; not a snapshot in time (Sanders, 2000).

39. Provides data To improve data-driven decision making To build professional learning communities To differentiate instruction To measure through growth, not just achievement

40. Resources Crane, J. (2002). The promise of value-added testing. Policy Report from http://www.ppionline.org Doran H. & Fleischman, S. (2005). Challenges of value-addedassessment. Educational Leadership. 63:3 http://www.ascd.org Hershberg, T. (2004). Value-added assessment. The Center for Greater Philadelphia. http://www.cgp.upenn.edu Holland, R. (2010). How to build a better teacher. Hoover Institution Stanford University. http://www.hoover.org/publications/policy-review/article/7841 Sanders, W. (2000). Value-added assessment from student achievement data: opportunities and hurdles. Journal of Personnel Evaluation in Education 14:4. http://www.sas.com/govedu/edu/research.html Schneider, C. (2002). Can value added assessment raise the level of student accomplishment? Peer Review. http://www.aacu.org Toch, T. (2005). Measure for measure. Washington Monthly. http://www.washingtonmonthly.com/featues/2005/0510.toch.html Zurawsky, C. (2004). Teachers matter: Evidence from value-added assessments. Research Points. 2:2 http://www.aera.researchpoints.com (2007). Roundtable discussion on value-added analysis of student achievement: a summary of findings. Working Group on Teacher Quality http://www.tapsystem.org/pubs/value_added_roundtable_08.pdf