1. Data Quality Challenges When
Implementing Classroom Value-
Added Models
Chris Thorn
Sara Kraemer
Jeff Watson
Center for Data Quality and Systems Innovation
Value-Added Research Center
UW Madison

2. Outline
• Overview VARC/CDQSI
• Overview Value-Added methods
• Student Teacher Linkage
• Threats to validity for Classroom Value-Added
– Student teacher linkage data quality
– Assignment and matching of students and
teachers

3. VARC and CDQSI Overview
• Value-Added Research Center
Director – Rob Meyer
• Center for Data Quality and Systems Innovation
Director – Chris Thorn
• Areas of work
– Data systems / Data quality / Decision support
– Professional development
– Reporting systems
– Evaluation
– Mixed methods
• Places where we work

4. Districts and States working with VARC
Minneapolis Milwaukee
Madison Racine
Chicago New York City
US Dept. of Ed.
Los Angeles Tulsa
Atlanta
Hillsborough
County
Broward
County

5. What is Value-Added Anyways?

6. How do we measure
student performance?
• How do we do this?
(example: the current NCLB method… percent proficient)
(example: Colorado Growth Model)
(example: VARC’s Value-Added Analysis)
• The following non-education example tries to
illustrate the difference between these measures.

7. The Oak Tree Analogy

8. Explaining the concept of Value-Added
by evaluating the performance of two gardeners
• For the past year, these gardeners have been tending to their oak trees trying to maximize
the height of the trees.
• Each gardener used a variety of strategies to help their own tree grow… which of these
two gardeners was more successful with their strategies?

9. To measure the performance of the gardeners, we will measure
the height of the trees today (1 year after they began tending to the trees).
• Using this method, Gardener B is the superior gardener.
This method is analogous to using an Attainment Model.

10. … but this attainment result does not tell the whole story.
• These trees are 4 years old.
• We need to find the starting height for each tree in order to more fairly evaluate each
gardener’s performance during the past year.
• The trees were much shorter last year.
Oak A Oak B
Age 4
3 Age 4
3
(1 (Today)
year ago) (1 (Today)
year ago)

11. We can compare the height of the trees one year ago to the height today.
• By finding the difference between these heights, we can determine how many inches the
trees grew during the year of gardener’s care.
• Oak B had more growth this year, so Gardener B is the superior gardener.
This is analogous to a Simple Growth Model, also called Gain.

12. … but this simple growth result does not tell the whole story either.
• We do not yet know how much of this growth was due to the strategies used by the
gardeners themselves.
• This is an “apples to oranges” comparison.
• For our oak tree example, three environmental factors we will examine are:
Rainfall, Soil Richness, and Temperature.

13. External condition Oak Tree A Oak Tree B
Rainfall amount High Low
Soil richness Low High
Temperature High Low

14. How much the gardeners’ own strategies contributed to the growth of the trees…
• We can take out each environmental factor’s contribution to growth.
• After these external factors are accounted for, we will be left with the effect of just the
gardeners.
• To find the correct adjustments, we will analyze data from all oaks in the region.

15. In order to find the impact of rainfall, soil richness, and temperature, we will plot the
growth of each individual oak in the region compared to its environmental conditions.

16. Now that we have identified growth trends for each of these environmental
factors, we need to convert them into a form usable for our calculations.
Rainfall Low Medium High
Growth in inches
relative to the -5 -2 +3
average
Soil Richness Low Medium High
Growth in inches
relative to the -3 -1 +2
average
Temperature Low Medium High
Growth in inches
relative to the +5 -3 -8
average
Now we can go back to Oak A and Oak B to adjust for their growing conditions.

17. To calculate our new adjusted growth, we start with simple growth.
• Next, we will use our numerical adjustments to account for the effect of each tree’s
environmental conditions.
• When we are done, we will have an “apples to apples” comparison of the gardeners’
influence on growth.
+14 Simple +20 Simple
Growth Growth

18. Based on data for all oak trees in the region, we found that high rainfall resulted in
3 inches of extra growth on average.
For having high rainfall, Oak A’s growth is adjusted by -3 to compensate.
Similarly, for having low rainfall, Oak B’s growth is adjusted by +5 to compensate.
+14 Simple +20 Simple
- 3 for Rainfall + 5 for Rainfall

19. For having poor soil, Oak A’s growth is adjusted by +3 to compensate.
For having rich soil, Oak B’s growth is adjusted by -2 to compensate.
+14 Simple +20 Simple
- 3 for Rainfall + 5 for Rainfall
+ 3 for Soil - 2 for Soil

20. For having high temperature, Oak A’s growth is adjusted by +8 to compensate.
For having low temperature, Oak B’s growth is adjusted by -5 to compensate.
+14 Simple +20 Simple
- 3 for Rainfall + 5 for Rainfall
+ 3 for Soil - 2 for Soil
+ 8 for Temp - 5 for Temp

21. Now that we have removed the effect of environmental conditions, our adjusted
growth result puts the gardeners on a level playing field.
We calculate that Gardener A’s effect on Oak A is +22 inches
We calculate that Gardener B’s effect on Oak B is +18 inches
+14 Simple +20 Simple
- 3 for Rainfall + 5 for Rainfall
+ 3 for Soil - 2 for Soil
+ 8 for Temp - 5 for Temp
_________ _________
+22 inches +18 inches
Adjusted Growth Adjusted Growth

22. Using this method, Gardener A is the superior gardener.
By accounting for last year’s height and environmental conditions of the trees during
this year, we found the “value” each gardener “added” to the growth of the tree.
This is analogous to a Value-Added Model.
+14 Simple +20 Simple
- 3 for Rainfall + 5 for Rainfall
+ 3 for Soil - 2 for Soil
+ 8 for Temp - 5 for Temp
_________ _________
+22 inches +18 inches
Adjusted Growth Adjusted Growth

23. How does this analogy relate to Value-Added calculations in the education context?
Oak Tree Analogy Value-Added in Education
Level of analysis? • Gardeners • IHE
• Districts/Schools
• Grades
• Classrooms
• Programs and Interventions
What are we using to • Growth in Inches • Relative Growth in Scale Score
measure success? Points
Sample • Single Oak Tree • Groups of Students
Control Factors • Rainfall • Students’ Prior Performance
• Soil Richness (most significant predictor)
• Temperature
Potential other variables collected
for ALL students
• Free/Reduced Lunch Status
• English Language Learner Status
• IEP / Special Education Status
• Race
• Gender

24. Where is Classroom-level
Value-Added Used?
• Recognizing Effective Practice
• Providing Support
• Performance Pay
• Tenure Decisions
• Higher Education Evaluation of Teacher
Preparation
• Performance Management

25. Overview of Data Requirements for
Classroom VAA
• Student demographics
• Teacher roster with background and accreditation
• School list and information
• Student and teacher attendance
• Course enrollment with assigned teachers
• Achievement data
• Mobility data

26. Simple is good….
….unless its wrong.

27. Defining Student Teacher Linkages:
Who is Teaching Who?
School
Student Teacher
Course
Outcomes: Practice:
Formative assessment, Pedagogy,
Achievement, Curriculum,
Graduation, Tutoring,
Behavior, Coaching, PD,
Post-graduation Scheduling,
success Class size

28. Warning: Reality Approaching
School
Student Teacher
Course

29. … movement between schools
School
Student School Teacher

30. … and movement within schools
School
Student Teacher
Course
Course
Course

31. … and complicated workflows
School enrollment
Schedule may
is estimated
change based
on completion
School
Student Staffing needs
are estimated
Determined
sometime between
Course Feb. and July
Maybe
hired in Sept.
Teacher

32. … and absence rates
School
Student Teacher
Attendance rate, Extended leave,
discipline moved to cover,
infractions Course absences?

33. … Instructional strategies like grouping,
pull-outs, room aides, team teaching
School
Student Teacher
Instructional
Course Support staff
Specialists

34. … non-traditional schools
School
Non-district
Student employed
Teachers
Course
Unique
instructional
approaches

35. … data errors, integration issues, SIS work
arounds, multiple districts
School SIS
Student
Student Teacher
Teacher
Matching
Records
Course HR
Data entry Creative
problems scheduling

36. Ouch
Course
Scheduling
Attendance rate,
School SIS
discipline
infractions Student Teacher Extended leave,
Student Student Teacher
moved to cover,
Matching Enrollment absences?
Records Instructional
Course HR
support staff
Data entry Creative HIRING
problems scheduling Specialists workflow
Unique School staffing
instructional adjustments
approaches
Mobility
School

37. Challenges for Classroom Level Value-
Added
• Threats to Validity
– Student teacher linkages
– Assignment of students and teachers
• Mobility
• Assessment Quality
• Assessment Coverage
• Integration

38. ST Linkages: Harvesting and Validating
Jeff Watson
Chris Thorn
Peter Witham
Timothy St. Louis
Center for Data Quality and Systems Innovation
Value-Added Research Center
UW Madison

39. 2 ST Linkage Data Quality Studies
• On-Site validation of ST Linkage data in a large
urban district
• Review of TIF districts processes for managing
and verifying ST Linkage data

40. Validation Study
Goals:
1. Determine how good / bad ST linkage data is
2. Correct bad data
3. Collect new data (e.g., team teaching)

41. Selecting a Sample
• Sample method goal is to be confident that estimates
of error rates are relatively accurate
– Sample first to assess DQ levels
• Target a small number of schools so that DQ error rates can be
estimated more accurately
• Census method goal is to confirm and adjust SIS ST
linkages to improve the validity of classroom VA
– Expanded sample as needed
• Include all schools within program
– Staged verification

42. Determine Sample Size
How to estimate sigma?
Estimate of σ (in
percent points) ±2.5% ±5.0%
10 64 16
15 144 36
20 256 64
25 400 100
30 576 144
Based on approach for determining samples size for a targeted confidence interval from
Wackerly, Mendenhall, Scheaffer (1996)

43. Target = ?? to ensure 100 Classrooms
• N=100
• Target more classrooms so that the resulting
sample in at least 100
• 75% participation rate
• 50% content rate
• Starting Sample = 100 / .75 /.5
= 267 classrooms
resulted in us recruiting about 9-10 urban
schools, mixed size and grade levels.

44. Constraints
• Control for mobility elsewhere
– Student
– Teacher
• Focus on grades and courses that are aligned
with the assessments and VA analysis
– 3-8, Math/Reading
– Formative assessments
– End of course assessments
– College readiness assessments

45. Process for Validation
• Collect data from district administration:
– Initial data files (enrollment, entity data)
– Course content case logic
• Collect data from principals:
– School – teacher assignments
– Teacher – course linkages
• Collect data from teachers:
– Teacher – course assignments
– Student-course assignments
– Team teaching

46. So…. Who is Teaching Who?
Accuracy Rate 74.2%
Team Taught Students 15.7%
Incorrect Students 2.5%
Incorrect Courses 7.6%
TOTAL 100%

47. Data from 9 schools in 1 urban district
100.0%
90.0%
80.0%
70.0%
60.0%
50.0% Accuracy Rate
40.0% Team Taught Students
30.0% Incorrect Students
20.0% Incorrect Courses
10.0%
0.0%
A B C D E F G H I

48. Counts of Student – Teacher Errors
No Add. Pull-out Push-in Targeted
Services Services Services Services NULL Grand Total
Traditional 7952 57 0 26 0 8035
Type of Classroom
Team Taught 1150 126 25 297 0 1598
“Was not in
class” 10 92 2 0 163 267
“Did not teach
class” 0 0 0 0 817 817
Grand Total 9112 275 27 323 980 10717

49. Summary of Findings
• Validated 74% of the time
• Adjusted 16% of the time
• Wrong 10% of the time
• Schools’ ST Linkage data varied between 55% and 93%
accurate
• Inaccurate records were the result of:
– not knowing about team teaching (4-30%)
– inaccurate teacher – course linkages (0-31%)
– inaccurate student – course linkages (0-15%)

50. So How Do Performance Pay Projects
Manage ST Linkage Data Quality?
1. How do TIF grantees obtain student-teacher
linkage data?
2. How do TIF grantees validate student-teacher
linkage data?
3. Are there any lessons to be learned?

51. Process Flow
Roster
Roster
Roster
Census Performance
SIS Rosters Verify Estimation
$$Payout$$

52. Process Flow
Roster
Roster
Roster
Census Performance
SIS Rosters Verify Estimation
Data
Warehouse $$Payout$$

53. Managing Complexity
 Courses: Multiple subject areas, Project-based
learning
 Teachers: Team teaching, Cross assignments
 Students: Mobility, Absenteeism

54. Enhancing Performance Pay Programs
• Verification: Critical and Strategic
– Accuracy vetted
– Enhanced stakeholder buy-in
– Opportunity to capture nuance
• Changes in time
• Team teaching

55. Leverage Personnel for Different Purposes.
– When / where to involve district staff
– When/where to involve principals
– When/where to involve teachers

56. Organization Design
• TIF as a Business Driver (not compliance
focused; high stakes DSS)
• Multi-level connections
• Technical staff involvement
• Focus on managing teacher data
• Implementation → Challenge → Innovation

57. External partners can increase both
capacity and overhead.
• SIS vendors, Database mgmt., Verification
processes and tools
• Positive consequences:
– Increased capacity, specialized and expert
knowledge, external agency
• Negative consequences:
– Vendor silos, more project mgmt. and
communication overhead for the grantee

58. Summary
• Classroom Value-Added requires high-quality
student – teacher linkage data
• SIS Student – teacher linkage data may not
capture certain things well, like team teaching
• Verification can be staged
• Verification improves stakeholder trust and
support as well as improving the validity of VA
estimates and the decisions those estimates
inform

59. Extending the Validity and
Interpretation of
Value-Added Data
Classroom Assignment and Matching
Sara Kraemer, Robert H. Meyer, and
Robin Worth

60. Assignment and Value-Added
• To what extent are teachers systematically
assigned to certain kinds of students?
– Rothstein Critique
• What are the unique attributes of students
and teachers?
– Can we make them “observable” in the data?
• What is the process of assignment in schools?

61. Student Assignment & Classroom Value-
Added
• Classroom assignment and student-teacher
matching (validity and accuracy of VA)
– Implications for student-teacher linkages
– Sociotechnical analysis of school processes
• Sources to inform our analysis:
– Principals and former principals from 3 large
urban school districts
• 2 districts with performance compensation systems.

62. Classrooms Design and Attributes
• Purposive classroom assignment
– Heterogeneous (mix ability levels, demographics, etc…)
– Homogeneous (sort on ability levels)
– Vertical and horizontal alignment
– Ability grouping
– Split classrooms, looping, team teaching
– Other cases: Mobility, special education students
• Matching: Attributes
– Teachers
• Classroom management skills, personality, etc…
– Student
• Behavior, special cases, etc…
• Social and technical considerations
– Value-added
– Linking students to teachers
– Organizational design and process of assignment and matching

63. Value added and assignment/matching
• Extent of unbalanced classrooms?
– Ex: Teachers w/ strong classroom management
skills
• Heterogeneous assignment perceived as more
“fair”
• Implications for measuring “true” teacher
effectiveness
• Some characteristics not measured in VA
model

64. Social and organizational factors in
assignment/matching - 1
• Process of assignment and matching in schools
– Principal led
– Teams of teachers (within and across grades)
– School leader/teacher/principal hybrid team
– Collaborative versus non-collaborative model
– Timing: End of academic year
• Input factors
– Test scores, input from teachers/school leaders, principal
observation, parent input
• Environmental issues
– Teacher labor issues (turnover/retention, shortages,
teacher allocation)

65. Social and organizational factors in
assignment/matching - 2
• Contextual factors
– Levels of complexity
– Size of school, number of grades served
– Breadth and depths of courses offered
– School types
• Elementary versus high school, charters
– Demographics of student, teacher population
– Degrees of community and parental involvement
– Mobility of students across classrooms or/and schools

66. Assignment and Matching:
Linking Students to Teachers in IS
• Verification of student-teacher linkages in data
• Information systems need capture reality of
classrooms
• Assignment and matching models for
explanatory power in linkage data

67. Next steps
• How can data systems support classroom
assignment and value-added validity?
– Assessment of school assignment practices within
a district (survey development)
– System requirements for data quality validation of
student-teacher linkages
– Data systems or other tools for school-level
decision support
– Reporting to schools, includes assignment
assessment

68. Questions?