A review of the latest literature on the usefulness of Student Evaluations of Teaching (SETs)? Do they measure teaching quality? Or Something Else?

1. Student Evaluations of
Teaching
(SETs)
What do they REALLY tell us?
Denise Wilson -- University of Washington, Department of Electrical and Computer Engineering
The truth will set you free, but first it will piss you off.
Gloria Steinem

2. An Overview of SETs
• SETs have been around for a hundred years and have
evolved from their original intent as formative instruments
(to help instructors improve their teaching) to summative
tools (to judge teaching quality).
• A large body of research has argued that the use of SETs as
a summative tool to measure teaching quality for personnel
decisions is at best misguided and at worst unethical or
illegal.
1920
First SETs are
completed at the
University of
Washington
1960’s
SETs are adopted
nationwide.
1970’s
SETs transition from
formative
instruments to
summative tools
used in firing, hiring,
and merit
decisions.
2009
France mandates
that SETs can only
be used to help
instructors improve
teaching and not for
merit, hiring, or
firing decisions.
2014
Stark and Freishtat
publish “An
Evaluation of
Course Evaluations”
which demonstrates
statistically that
SETs are rarely a
good tool to
measure teaching
effectiveness.
2018
Formal arbitration
mandates Ryerson
University to ensure
that SETs, “are not
used to measure
teaching
effectiveness for
promotion or
tenure.”

3. Student Evaluations of
Teaching
(SETs)
The State of SET Research

4. Do SETs measure Student Learning?
Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty's teaching effectiveness: Student
evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22-42.
A meta-analysis analyzes data from several independent studies
to determine an overall trend (or lack thereof) on the subject
that is common to all of the studies.
The Uttl meta-analysis re-analyzed 51 multi-section studies from
seven previous meta-analyses dating from 2009 back to 1981.
A multi-section study strives to randomly assign students to
multiple sections of a course that are essentially the same
except for the instructor teaching the section. The students
from all of these sections then take the same final exam.
After accounting for small sample sizes (small numbers of
students in each section) and publication bias using updated
statistical methods, the authors found no or minimal
correlations between SET ratings and student learning

5. Do SETs measure Student Learning?
Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty's teaching effectiveness: Student
evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22-42.
Conclusion:
SETs fail to measure
student learning

6. Do SETs measure Teaching Quality?
Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of
the art. Review of Educational Research, 83(4), 598-642.
A review analyzes a large number of articles within a
predefined timeframe that meet certain inclusion criteria. The
goal of the review can vary depending on the authors’ purpose
and motivation in pursuing the review.
The Spooren et al. (2013) literature review analyzed 160
articles published since 2000 with a ”meta-validation” model in
order to make conclusions as to how valid (i.e. sound and
logical) SETs are as a measure of teaching quality.
This review evaluated many different forms of validity but here,
we will emphasize only two that are most relevant to whether
or not SETs are useful to teachers:
• Construct Validity: Do SETs really evaluate teaching quality? (construct
validity requires both convergent and discriminant validity)
• Outcome validity: Are SETs used in appropriate ways? (is the outcome
valid?)

7. Do SETs measure Teaching Quality?
Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of
the art. Review of Educational Research, 83(4), 598-642.
Construct Validity answer the question: Do SETs really evaluate
teaching? but requires evidence of both convergent and
discriminant validity.
Convergent Validity:
Are measures that should be related really related?
• SETs should be related to objective measures of learning.
Are they? YES then NO. Multi-section studies show positive and moderate
correlations between SET ratings and grades – but the more recent Uttl et al.
(2017) study refutes these results.
• SETs should be related to subjective measures of students’
perceived learning.
Are they? YES. Multiple studies show strong correlations between students’
perception of learning and SET ratings.

8. Do SETs measure Teaching Quality?
Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of
the art. Review of Educational Research, 83(4), 598-642.
Discriminant Validity:
Are measures that should not be related really not related?
• The literature on bias on SETs has shown many measures that
should not be related to SETs and are. Although Spooren et al
(2013) do not go into depth on these biases, discriminant
validity has been very mixed in the literature.
• For example, recent studies have shown that being a female results in
low SETs even when a male and female teacher exhibit identical
teaching behaviors. This result implies low discriminant validity
because gender should have nothing to do with SETs.
Construct Validity answer the question: Do SETs really evaluate
teaching? but requires evidence of both convergent and
discriminant validity.

9. Do SETs measure Teaching Quality?
Conclusions:
SETs have questionable
convergent validity – meaning that
they may or may not measure
teaching quality.
Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of
the art. Review of Educational Research, 83(4), 598-642.

10. Are SETs useful?
Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of
the art. Review of Educational Research, 83(4), 598-642.
Outcome Validity answers the question: Do SETs lead to the
desired outcomes of administering them?
• Do students take them seriously and offer feedback focused on
teaching quality?
• Do teachers believe SETs to reflect teaching quality and act on SET ratings
accordingly?
• Do administrators use SETs appropriately to infer teaching quality?
Outcome Validity: from the perspective of students
If students don’t see that SETs are resulting in impacts on
teaching (changes in instruction, teaching awards, etc.), they
may engage in mindless evaluation behaviors. In this review,
the authors found that students don’t fear negative
repercussions from SETs, are ambivalent about their value, and
find them only somewhat useful in course selection.

11. Are SETs useful?
Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of
the art. Review of Educational Research, 83(4), 598-642.
Outcome Validity: from the perspective of teachers
While teachers recognized the importance of SETs, they
generally viewed SETs as having limited value in improving
teaching. They tended to see SETs as “hints” regarding how to
improve a course, but didn’t see the ratings and comments as
a valid measure of teaching quality.
Outcome Validity answers the question: Do SETs lead to the
desired outcomes of administering them?
• Do students take them seriously and offer feedback focused on teaching
quality?
• Do teachers believe SETs to reflect teaching quality and act on SET
ratings accordingly?
• Do administrators use SETs appropriately to infer teaching quality?

12. Are SETs useful?
Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of
the art. Review of Educational Research, 83(4), 598-642.
Outcome Validity: from the perspective of administrators
Administrators tend to value SETs as a quick and simple
indication of teaching performance. They prefer to aggregate
measures and often make inferences that “jump the gun” and
violate statistical and methodological rules regarding the
proper use of SET data.
Outcome Validity answers the question: Do SETs lead to the
desired outcomes of administering them?
• Do students take them seriously and offer feedback focused on teaching
quality?
• Do teachers believe SETs to reflect teaching quality and act on SET ratings
accordingly?
• Do administrators use SETs appropriately to infer teaching quality?

13. Are SETs useful?
Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of
the art. Review of Educational Research, 83(4), 598-642.
Conclusions:
SETs have mixed outcome
validity for students, minimal
outcome validity for teachers,
and dangerous outcome validity
for administrators.

14. What do SET numbers mean?
Stark, P., & Freishtat, R. (2014). An evaluation of course evaluations. ScienceOpen. Center for Teaching and
Learning, University of California, Berkley. Retrieved from https://www. scienceopen. com/document.
An evaluation looks at a particular topic from a single lens or a
particular analytical perspective for the purposes of judging its
usefulness or applicability to a problem.
The Stark & Freishtat (2014) evaluation article looks at SETs
from a statistical perspective to understand whether the way
SET numbers are used is appropriate based on the nature and
type of questions contained on the SETs.
This evaluation looked at:
• How items on SETs are assigned numbers and combined to
represent teaching quality or effectiveness.
• What it means when students do not complete SETs.

15. What do SET numbers mean?
Stark, P., & Freishtat, R. (2014). An evaluation of course evaluations. ScienceOpen. Center for Teaching and
Learning, University of California, Berkley. Retrieved from https://www. scienceopen. com/document.
On Averages:
• Does taking the average of SET scores lead to an aggregate,
“big picture” measure of how good a teacher is compared to
other teachers at the same institution?
SET averages:
• SET scores are ordinal, categorical values – meaning a 5 (excellent) is
better than a 4 (very good) but the difference between a 5 and a 4 may
or may not be the same as the difference between a 4 and a 3, a 3 and a
2 and so on.
• The difference between a 3 and a 4 doesn’t mean the same thing to
different students.
• A 5 can mean different things in different courses.

16. What do SET numbers mean?
Stark, P., & Freishtat, R. (2014). An evaluation of course evaluations. ScienceOpen. Center for Teaching and
Learning, University of California, Berkley. Retrieved from https://www. scienceopen. com/document.
Conclusions:
Taking averages of numbers on
SETs that basically represent labels
is nonsensical and logically, leads
to nonsensical conclusions about
teaching quality.

17. What do SET numbers mean?
Stark, P., & Freishtat, R. (2014). An evaluation of course evaluations. ScienceOpen. Center for Teaching and
Learning, University of California, Berkley. Retrieved from https://www. scienceopen. com/document.
On Response Rates:
• Do low response rates lead to different information? How?
On response rates:
• Some students do not fill out surveys
• The lower the response rate, the less representative the results are of
the class
• There is no reason to assume that non-responders are just like
responders. In fact, responders in a low response rate situation tend to
be the malcontents and may lead to biased, misleading teaching scores.
• Some will be tempted to blame the instructor for low response rates.

18. What do SET numbers mean?
Stark, P., & Freishtat, R. (2014). An evaluation of course evaluations. ScienceOpen. Center for Teaching and
Learning, University of California, Berkley. Retrieved from https://www. scienceopen. com/document.
Conclusions:
Low response rates are not a
reflection of poor teaching quality
but they should and do bring into
question the usefulness of SET
data.

19. In summary
What DO SETs Measure?
a. Grade anticipation (what kind of grade does the student expect?)
b. Teaching quality (how good is the teacher?)
c. Entertainment value (how well does the teacher keep students engaged?)
d. Grades (how easy is the grading?)
e. Difficulty (how hard is the course?)
f. Mood (how does the student feel on course evaluation day?)
g. Student Satisfaction (did the course meet or exceed the student’s expectations?)
h. It’s anybody’s guess (who knows?)

20. In summary,
the general consensus is that
SETs Measure:
a. Grade anticipation (what kind of grade does the student expect?)
b. Teaching quality (how good is the teacher?)
c. Entertainment value (how well does the teacher keep students engaged?)
d. Grades (how easy is the grading?)
e. Difficulty (how hard is the course?)
f. Mood (how does the student feel on course evaluation day?)
g. Student Satisfaction (did the course meet or exceed the student’s expectations?)
h. It’s anybody’s guess (who knows?)
A large number of research studies have shown that SETs measure student satisfaction which in
turn, is strongly correlated to the grade that a student anticipates receiving in a course.

21. In Summary,
Do SETs Measure Student Learning?
a. Yes, of course
b. No, not really
c. Yes, but not in the expected way.
d. Who knows? (It’s anybody’s
guess)
A recent meta-analysis (Uttl,
White, and Gonzalez 2017)
showed no significant
correlations between SET ratings
and student learning.
One research study has shown
that learning measured at the
end of a term is highly correlated
to SETs, but when learning is
measured in subsequent courses
(for which the original course
was a pre-requisite), learning is
negatively correlated with SETs.
(Kornell and Hausman 2016).