This document discusses approaches to developing comprehensive teacher evaluation systems using student achievement data. It outlines five propositions for validating such systems and the design claims and evidence needed to support each proposition. It then describes different approaches used across the US, including student learning objectives, subject- and grade-alike measures, universal pre-/post-tests, and value-added composites. Both advantages and disadvantages are provided for each approach. Finally, it discusses examples from Minneapolis and Edina that use value-added measures and standardized tests to evaluate teacher effectiveness.

1. Creating a Comprehensive
Teacher Effectiveness System
Christopher A. Thorn
University of Wisconsin-Madison
Center for Data Quality and Systems Innovation
Value-Added Research Center

3. Propositions and Claims to the
Validity Evaluation
Proposition 1: The standards clearly define learning
expectations for the subject area and grade level
Design Claims: Evidence:
• Clarity • Expert reviews
• Feasibility • Research studies validating
• Explicit progressions progressions

4. Proposition 2a: The assessment instruments have
been designed to yield scores that can accurately and
fairly reflect student achievement of standards.
Design Claims: Evidence:
• Alignment with standards • Expert reviews of alignment
(specs and items) • Small scale studies
• Fair and accessible • Sensitivity reviews
• Reliable/Replicable • Measurement reviews of
procedures design, administration, and
scoring procedures

5. Proposition 2b: The assessment instruments have
been designed to yield scores that accurately and
fairly reflect student growth over the course of the
year
Design Claims: Evidence:
• Sample the range of where • Expert review
students may start and end • Teacher review
the school year
• Designed to be sensitive to
instruction

6. Proposition 3: Assessment scores accurately and fairly
reflect the status of students’ knowledge and skills
relative to learning expectations
Design Claims: Evidence:
• Psychometric analyses • Psychometric analysis
confirm the assessment’s • Bias analysis
blueprint
• Scores are sufficiently
precise and reliable
• Scores are fair/unbiased

7. Proposition 4: Student growth scores accurately and
fairly measures student progress over the course of
the year
Design Claims: Evidence:
• Score scale reflects the full • Psychometric modeling and
distribution of where fit statistics
students may start and end • Sensitivity/bias analysis
the year
• Growth scores are
sufficiently precise and
reliable for all students
• Growth scores are
fair/relatively free of bias

8. Proposition 5: Value-added scores represent teachers’
contribution to student growth
Design Claims: Evidence:
• Scores are instructionally • Assumption checking
sensitive • Advanced statistical
• Sores representing teacher modeling
contribution are sufficiently • Research on instructional
precise and reliable sensitivity
• Scores representing teachers
contributions are relatively
free of bias

9. What do we see across the US?
• Student Learning Objectives
– Austin, Texas; Charlotte- Mecklenburg, North Carolina;
and Washington, DC
• Subject- and grade-alike measures
– Delaware and Tennessee
• Universal pre-/post-tests
– Hillsborough County, Florida, and Washington, DC
• Value-added composite (school/cohort)
– Charlotte- Mecklenburg, North Carolina; Delaware;
and Tennessee

10. SLOs
Teacher (overseen by administrator) selects
appropriate measure(s); assesses students at
beginning of year; sets specific objectives for
performance; and assesses students at end of
year. Administrator ultimately determines the
teacher’s success in achieving SLOs
(Goe 2011).

11. Pro/Con for SLOs
• Advantages: Adaptable; permits
specialization; and tends to be credible among
educators.
• Disadvantages: Requires significant attention
from administrators; difficult to create
comparability and rigor across classrooms;
and does not account for differences across
teachers in the students served
(RttT Assistance Network 2011).

12. Subject- and grade-alike measures
Teachers meet in grade- and/ or subject-specific
teams to consider growth measures (existing,
adapted, or new assessments; portfolios of
work; performances, and the like), then agree
on measure(s) all will use to determine
individual contributions to student growth.
District/state then reviews and approves the list
for each grade and subject. (May include SLOs,
pre-/ post-tests, or other measures.)

13. Subject- and grade-alike measures
• Advantages: Designed to yield comparable
measures across classrooms/districts.
• Disadvantages: Requires significant attention
to ensuring comparability and much dedicated
time for teachers to work together to develop
consistent scoring patterns
(RttT Assistance Network 2011).

14. Universal pre-/post-tests
• Written pre-/post-tests developed for every
grade and every subject.
• Advantages: Enables annual growth
calculations for all students.
• Disadvantages: Requires extensive test
development and analysis capacity; and may
be difficult to link previous years of test data,
given course variety at secondary level.

15. Value-added composite
(school/ cohort)
Nontested teachers are assigned a composite
value- added score based on average test
performance of either the school as a whole or
the particular cohort of students “claimed” by
the teacher
(Goe 2011).

16. Value-added composite
(school/ cohort)
• Advantages: Can contribute to collective
efforts around student achievement.
• Disadvantages: May mask high and low
performers and hold individual teachers
accountable for measures over which they
have limited influence.

17. A Tale of Two Cities – K-2
• Value-Added Measures of Math and Reading
• Assessments administered 2-4 times a year
• Administered 1-1 by retired teachers
• Minneapolis
– Off the shelf public domain assessments
– Used to study Beat the Odds Kindergarten Teachers
• Edina
– Purchased NWEA early grade assessments
– Used whole grade and school to study overall
effectiveness

18. Build Decision Points
Test date
Test Demographic Grade
timing and
Properties Controls Retention
frequency
Classroom/ Aggregation
Student School-year
Teacher Units (grades,
Mobility Mobility
Indicators teams, schools)
Special
Aggregation Multiple
Education Multi-year data
over time components
detail

19. References
Webinar on Evaluating Teacher Quality
http://www.aacompcenter.org/cs/aacc/view/rs/26579
A Practical Guide to Designing Comprehensive
Teacher Evaluation Systems
http://tqsource.org/publications/practicalGuideEvalSyst
ems.pdf
Measuring Teachers’ Contributions to Student
Learning Growth for Nontested Grades and
Subjects
http://www.lauragoe.com/LauraGoe/MeasuringTeacher
sContributions.pdf