The document compares course evaluation ratings between online and traditional courses. Contrary to expectations, the study found no significant differences in ratings for most items, including those referring to in-class procedures. The only significant difference was that students in online courses perceived a higher workload than those in traditional courses, possibly because online students consider any work for the class as "outside of class." Overall, the results suggest that instructors are viewed similarly in online and traditional courses.

1. Do Online Courses Yield Lower Student Ratings
on Selected Course Evaluation Items?
Bridget G. Hanley, Eugeniu Grigorescu, and Thomas P. Hogan
University of Scranton
Introduction
Having comparable course evaluations for online courses and tradi-
tional courses is important for valid measurement. A literature search
consisting of 17 studies showed that most had weak research designs
and conflicting results (see References). Several factors determine
whether or not results of course evaluations in online courses differ
significantly from traditional courses.
We compared course evaluation ratings between online courses and
traditional courses to determine if there were any significant differ-
ences between student ratings. The ratings of seven instructional
methods and five “other” items (response rate, initial interest in the
course, overall ratings of the course and instructor, and workload)
were analyzed for comparison.
Abstract
Typical course evaluation forms are developed in the context of tradi-
tional, in-class formats but may also be used for online courses. We
hypothesized that online courses would yield lower ratings for items
referring to in-class procedures. The study compared 9 in-class sec-
tions with 9 online sections matched for instructor, field, and level
given in adjacent semesters. Contrary to expectations, no differences
were found in ratings for items referring to in-class procedures nor
for other items.
Results
No significant differences were found between student ratings on
Method Items. The one significant difference found for “Other”
items, workload, was perceived higher in online courses. This could
be due to the wording of the item: “Average number of hours per
week I spent outside of class on work for this course.” Students in
online courses may perceive any work they perform for that one class
as “outside of class” and therefore gave higher ratings. Tables 1 and 2
present the results.
Course Evaluation Items
The course evaluation form used in this study consisted of 7
“instructional methods” items, 12 “course objectives” items, and 4
“other” items. The course objectives items were not analyzed because re-
sponses to them varied from course to course depending on which objec-
tives an instructor selected. Most items used a 5-point Likert response
scale from “strongly disagree” to “strongly agree.” A complete copy of
the form appears at:
http://www.scranton.edu/academics/ctle/oce/
GUIDE_TO_COURSE_SURVEY.PDF
Instructional Methods Items
1. Was enthusiastic about teaching the class (for example, was dynamic
and energetic, enhanced presentations with humor, style of presentation
held your interest).
2. Made students feel welcome in seeking help/advice (for example, was
friendly towards individual students, had a genuine interest in individual
students, was accessible to students).
3. Used evaluation methods which were fair and appropriate (for exam-
ple, examinations/graded materials tested class content as emphasized by
the instructor, feedback on examinations/graded material was valuable).
4. Provided clear and well organized class materials/presentations (for
example, explanations were clear, class materials were well prepared and
carefully explained).
5. Provided context for course material (for example, contrasted the im-
plications of various concepts, presented the background of concepts).
6. Encouraged students to participate in class (for example, encouraged
students to ask and answer questions, gave meaningful answers to ques-
tions).
7. Assigned readings/texts that were appropriate (for example, readings/
texts were valuable, readings contributed to appreciation and understand-
ing of subject).
Other Items Analyzed
Overall, I rate this instructor an excellent teacher.
Overall, I rate this course as excellent.
Average number of hours per week I spent outside of class on work for
this course.
Before enrolling, I really wanted to take this course REGARDLESS of
who taught it.
(The “response rate” is the percentage of students who completed the
form out of all students registered for the course.)
Discussion
Analysis of comparisons between online course evaluations and
comparable traditional course evaluations yielded no significant
differences. Our results suggest that instructors are viewed similar-
ly in the online and traditional courses they teach.
Our conclusion coincides with the technical report completed by
the IDEA at Kansas State University. That report analyzed thou-
sands of samples of classes from over one hundred institutions
from 2002—2008. Researchers concluded “the current findings in-
dicate the IDEA Student Rating System is appropriate for both
online and traditional courses.” IDEA determined “The results of
this study reveal more similarities than meaningful differences be-
tween IDEA student ratings in traditional and online cours-
es” (Benton et al., 2010, p. 28).
Our study had two principle limitations. First, the sample of data
analyzed is quite small, partly due to very few professors teaching
courses both online and traditionally in the given semesters. The
majority of online courses at the University are taught during spe-
cial terms rather than in traditional Fall and Spring terms. Second,
the matches between traditional and online courses were less than
perfect. The matches were exact for instructor and for academic
field but not for exact course.
References
Benton, S. L., Webster, R., Gross, A. B., & Pallett, W. H. (2010). An analysis of IDEA student ratings of instruc-
tion in traditional versus online courses, 2002-2008 data. The IDEA Center (Technical Report No. 15). Man-
hattan, KS: The IDEA Center.
Campbell, M. C., Floyd, J., & Sheridan, J. B. (2002). Assessment of student performance and attitudes for cours-
es taught online versus onsite. The Journal of Applied Business Research, 18(2), 45-51.
Driscoll, A., Jicha, K., Hunt, A. N., Tichavsky, L., & Thompson, G. (2012). Can online courses deliver in-class
results?: A comparison of student performance and satisfaction in an online versus a face-to-face introducto-
ry sociology course. Teaching Sociology, 40(4), 312-331.
Hale, L. S., Mirakian, E. A., & Day, D. B. (2009). Online vs. classroom instruction: Student satisfaction and
learning outcomes in an undergraduate allied health pharmacology course. Journal of Allied Health, 38(2),
e36-42.
Horspool, A., & Lange, C. (2012). Applying the scholarship of teaching and learning: Student perceptions, be-
haviours, and success online and face-to-face. Assessment & Evaluation in Higher Education, 37(1), 73-88.
Johnson, S. D., Aragon, S. R., Shaik, N., & Palma-Rivas, N. (2000). Comparative analysis of learner satisfaction
and learner outcomes in online and face to face learning environments. Journal of Interactive Learning Re-
search, 11(1), 29-49.
Lim, J., Kim, M., Chen, S. S., & Ryder, C. E. (2008). An empirical investigation of student achievement and sat-
isfaction in different learning environments. Journal of Instructional Psychology, 35(2), 113-119.
McGhee, D. E. and Lowell, N. (2003), Psychometric properties of student ratings of instruction in online and on-
campus courses. New Directions for Teaching and Learning, 2003(96), 39-48.
Mentzer, G. A., Cryan, J., & Teclehaimanot, B. (2007). Two peas in a pod? A comparison of face-to-face and web
-based classrooms. Journal of Technology & Teacher, 15(2), 233-246.
Mintu-Wimsatt, A. (2001). Traditional vs. technology-mediated learning: A comparison of students’ course eval-
uations. Marketing Education Review, 11(2), 63-73.
Mintu-Wimsatt, A., Ingram, K., Milward, M. A., & Russ, C. (2006). On different teaching delivery methods:
What happens to instructor course evaluations? Marketing Education Review, 16(3), 49-57.
Paulsen, K. J., Higgins, K., & Miller, S. P. (1998). Delivering instruction via interactive television and videotape:
student achievement and satisfaction. Journal of Special Education Technology, 13(4), 59-77.
Rabe-Hemp, C., Woollen, S., & Humiston, G. S. (2009). A comparative analysis of student engagement, learning,
and satisfaction in lecture hall and online learning settings. Quarterly Review of Distance Education, 10(2),
207-218.
Spooner, F., Jordan, L., Algozzine, B., & Spooner, M. (1999). Student ratings of instruction in distance learning
and on-campus classes. The Journal of Educational Research, 92(3), 132-140.
Summers, J. J., Waigandt, A., & Whittaker, T. A. (2005). A comparison of student achievement and satisfaction in
an online versus a traditional face-to-face statistics class. Innovative Higher Education, 29(3), 233-250.
Urtel, M. G. (2008). Assessing academic performance between traditional and distance education course for-
mats. Journal of Educational Technology & Society, 11(1), 322-330.
Warren, L. L., & Holloman Jr., H. L. (2005). On-line instruction: Are the outcomes the same? Journal of Instruc-
tional Psychology, 32(2), 148-151.
Method
We obtained data files of ratings for traditional and online courses
taught in the Spring 2013 semester and the Fall 2013 semester. We se-
lected instructors who taught both traditional and online courses in
these semesters.
Courses were matched according to instructor, subject, and level.
They were matched across terms; courses taught online in Spring or
Fall 2013 were matched with traditional courses taught in Spring or
Fall 2013.
After organizing the data, nine matches were suitable to be ana-
lyzed. These matches included four upper level marketing courses,
four upper level nursing courses, one advanced education course and
one lower level education course, and eight advanced physical thera-
py courses. Total student respondents for the online courses was 62
(median response rate, 71%). Total student respondents for the tradi-
tional courses was 182 (median response rate, 78%).
Table 1 Comparison of online and traditional
student ratings of Method Items
Online Traditional
Method
Item
Mean
(SD)
Mean
(SD) t(8) p
1 3.90
(0.99)
4.01
(0.63)
-0.28 0.79
2 4.03
(1.05)
4.01
(0.67)
0.07 0.95
3 4.12
(1.07)
4.29
(0.43)
-0.52 0.62
4 3.99
(1.12)
4.16
(0.52)
-0.53 0.61
5 4.16
(0.89)
4.26
(0.39)
-0.37 0.72
6 4.35
(0.86)
4.29
(0.41)
0.19 0.85
7 4.27
(0.70)
4.28
(0.42)
-0.04 0.97
Our Hypotheses
Out of the seven “instructional methods” items on the evaluation
form, items 2 and 6 were of specific interest to us.
Method 2 determines how welcoming students perceive their in-
structor as. We hypothesized that students in an online course feel
that their instructors are less welcoming because they do not meet
in person and must communicate online.
Method 6 determines students’ perceptions of how well their in-
structor encouraged them to participate in class. Because the meth-
od has the words “in class” built into it, we predicted that online
students would give lower ratings.
Table 2 Comparison of online and traditional
student ratings of “Other” Items
Online Traditional
Item Mean
(SD)
Mean
(SD)
t(8) p
Response rate 0.73
(0.17)
0.75
(0.07)
-0.32 0.76
Initial
interest
3.90
(0.67)
3.89
(0.37)
0.19 0.85
Overall
instructor
4.12
(0.80)
4.03
(0.64)
0.29 0.78
Overall
course
4.02
(0.92)
4.14
(0.32)
-0.39 0.71
Workload 3.61
(0.93)
2.30
(0.46)
2.02 0.003