1. Peer feedback on writing: Is more better?
A pilot study in progress
Christina Hendricks & Jeremy Biesanz, University of British Columbia
Figure 1: Instructor comments: number of
comments rated as “2” in each category
Figure 3: Student peer comments: number of
comments rated as “2” in each category
1st
Method
Course: A year-long, writing-intensive course for
first-year students at a large university. Students
write 10-12 essays over the year, and give/receive
peer feedback on each one.
Participants: 13 students agreed to participate in
this pilot study. We plan to do a larger study in 2015-
2016.
Measures: 10 essays from each of 13 participants;
all the student peer comments (n=1218) and all the
instructor comments on those essays (n=3263).
Procedures: All comments were coded according to
a rubric with four categories (and subcategories
within each): Strength of Argument, Insight,
Organization, and Style & Mechanics. Fleiss’ Kappa
for coders: 0.61; for the most frequently used
categories: 0.8. Essays will be coded with the same
rubric. Comments were rated as 1 (very negative), 2
(somewhat negative), or 3 (positive). We will analyze
all data according to the design in Figure 5.
Introduction
There is ample evidence that both giving and
receiving peer feedback improves student writing
(e.g., Topping, 1998; Cho & MacArthur, 2010; Cho
& MacArthur, 2011; Cho & Cho, 2011; Li, Liu &
Steckelberg, 2010, Crossman & Kite, 2012). This
pilot study addresses two questions for which
there isn’t yet a great deal of research evidence:
1. How do students use peer comments given
and received for improving later essays rather
than on drafts of the same essay?
2. Are students more likely to use peer comments
given and received for improving their writing
after more than one peer feedback session?
How many sessions are optimal?
We are also asking:
3. Do students use peer comments on later
essays even if the instructor did not make
similar comments?
4. Does the quality of student comments improve
over time?
Contact: c.hendricks@ubc.ca
Essay 1
quality
Essay 2
quality
Essay 3 … n=10
quality
Essay 1
comments
Essay 2
comments
Essay 3 … n=10
comments
Figure 5: Cross-lagged panel design with auto-regressive structure. After we code the essays
in Summer 2015 we will analyze these relationships for all 10 essays from each student
Figure 2: Instructor comments: number of
comments rated as “3” in each category
Figure 4: Student peer comments: number
of comments rated as “3” in each category
0 2 4 6 8 10 12
024681012
Essay Number
InstructorNumberofComments
Argument Strength
Style
Insight
Organization
0 2 4 6 8 10 12
012345
Essay Number
InstructorNumberofComments
Argument Strength
Style
Insight
Organization
0 2 4 6 8 10 12
01234
Essay Number
StudentNumberofComments
Argument Strength
Style
Insight
Organization
0 2 4 6 8 10 12
0.00.51.01.52.02.53.0 Essay Number
StudentNumberofComments
Argument Strength
Style
Insight
Organization
.31***
.11**
.08**
-.28**
-.04*
-.16**
.19**
Results so far
We have only analyzed the student and
instructor comments so far.
Trends over time: There is a strong upward
trend in instructor comments rated “3” in
each category over time. There is also a
significant downward trend in the number of
instructor comments rated “2” in Strength
and Insight. The student peer comments are
more erratic, with only a significant
downward trend in comments rated “2” in
Style.
Student/instructor agreement: The
average comment ratings agree strongly
between student and instructor: b = .48,
t(297) = 5.08, p < .00001. However, this
agreement goes down across essays:
b = -.04, t(296) = -3.02, p = .003. This is
because instructor ratings go up over time,
b = .033, t(107) = 5.84, p < .00001, while
student ratings increase at only half that
rate, b = .016, t(86) = 1.89, p = .06.
Auto-regressive analysis of comments
from essay 1 to essay 2, essay 2 to essay 3,
etc. There are significant effects for
instructor comments of “3” in Strength
(.33***) and Style (.34****). For student
comments, there are significant effects in
ratings of “3” in Strength (.20*), Style (.22**),
Organization (.20*), and for ratings of “2” in
Strength (.23*) and Style (.27**).
*p < .05, **p< .01, ***p< .001, ****p < .0001
References
Cho, Y. H., & Cho, K. (2011). Peer reviewers learn from giving comments. Instructional
Science, 39, 629-643.
Cho, K., & MacArthur, C. (2010). Student revision with peer and expert reviewing, Learning
and Instruction, 20, 328-338.
Cho, K. & MacArthur, C. (2011). Learning by reviewing, Journal of Educational Psychology,
103(1), 73-84.
Crossman, J. M., & Kite, S. L. (2012). Facilitating improved writing among students through
directed peer review, Active Learning in Higher Education, 13, 219-229.
Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or assessee: How student learning
improves by giving and receiving peer feedback. British Journal of Educational
Technology, 41(3), 525–536.
Topping, K. (1998). Peer assessment between students in colleges and universities.
Review of Educational Research, 68, 249- 276.