The document investigates the impact of the order of feedback in the writing process on student writing outcomes in two essay writing classes at a Japanese public university. It compares two groups receiving local feedback from teachers first followed by global feedback from peers, against the reverse order, analyzing various essays to assess improvements in writing quality. The study findings indicate that while both classes showed significant improvement, the class receiving global feedback first performed better in global criteria during peer-editing, while the opposite was true for local criteria.

1. PEER AND TEACHER FEEDBACK
ON STUDENT WRITING:
WHAT, HOW AND WHEN?
Richard Hodson
Joel Hensley
University of Nagasaki, Siebold
Nagasaki JALT
21 January 2012

2. Background
 Two essay writing classes in a Japanese public
university
 Fifteen 90-minute classes per semester
 Elective, TOEIC requirement: 500
 Textbook used:
Ready To Write More
by Blanchard and Root
(Longman)
 Difference between classes: order in which
feedback on writing was given

3. Class schedule
“Diagnostic” essay (no local or global check)
Paragraph practice
First essay (“Pressures”)
Classification essay Peer-editing
Cause/effect essay stage
Comparison essay
Problem/solution essay
Final essay (no local or global check)

4. Definitions
 Local
 Grammar
 Vocabulary and word choice
 Spelling, capitalization, punctuation
 Global
 Essay structure
 Style

5. Research question
 Does the order of feedback in the writing process have
an effect on student writing?
 Order of feedback:
Local first (teacher edit), followed by global (peer edit)
vs.
Global first (peer edit), followed by local (teacher edit)
 Local editing: done manually by teachers
 Global peer-editing: students used checklist of criteria
for each essay

6. Editing process
Class J Class R
First draft First draft
Student rewrites to Student rewrites to
Local check by
Global check by peer include peer incorporate teacher
teacher
suggestions corrections
Second draft Second draft
Student rewrites to Student rewrites to
Local check by
incorporate teacher Global check by peer include peer
teacher
corrections suggestions
Finished essay Finished essay

7. Study participants
 Original class size:
 J = 13 students
 R = 27 students
 Selection criteria:
 No more than one absence during the semester
 Submitted all assignments
Class Grade Female Male Total TOEIC TOEIC
average sd
1 2 3
J 1 2 3 5 1 6 625 134
R -- 10 3 11 -- 11 557 83

8. Texts for analysis
“Diagnostic” essay (no local or global check)
Paragraph practice
First essay (“Pressures”)
Classification essay Peer-editing
Cause/effect essay stage
Comparison essay
Problem/solution essay
Final essay (no local or global check)

9. Final essay topics (1)
 Classification
 Types of vacation activities
 Types of colleges/universities
 Types of websites
 Cause/Effect
 The reasons you decided to major in English/come to this
university
 The effects that the extension of the Shinkansen to Nagasaki
might have
 The effects that the Internet has had

10. Final essay topics (2)
 Compare/Contrast
 An aspect of Japanese culture and of the culture of another
country
 Japan now and Japan in another time period
 Two different products on sale in Japan
 Problem/Solution
 The falling birth rate in Japan
 People getting music, movies and TV shows illegally from the
Internet
 Bullying in schools

11. Analytical measures
 Local
 Average sentence length
 Gunning-Fog readability
 “a weighted average of the number of words per sentence,
and the number of long words per word”
% of error-free sentences (average of J & R scores)
 Global
 12 criteria
 J & R agreed score

12. Global criteria (1)
Introduction
 Is there enough background information?
 Has the author used at least one intro
technique (anecdote, etc.)?
 Thesis Statement:
 Has the subject been given?
 Has the focus (main ideas) been provided?
 Is there a clear purpose (not necessarily in
the TS itself)?

13. Global criteria (2)
Body
 Does every body paragraph contain a Topic
Sentence with topic and focus?
 Does every body paragraph provide adequate
support to develop and prove the thesis?
 Do the body paragraphs achieve unity (no
irrelevant, vague, or repetitive sentences)?
 Are the body paragraphs arranged in a logical
order?

14. Global criteria (3)
Conclusion
 Has the Thesis Statement been summarized or
restated in different words?
 Has the author used at least one conclusion
technique (question, etc.)?
Other
 Has the essay been formatted correctly?

15. Raw comparison
local
percentage of sentences
correct average sentence length Gunning-Fog readability
Class R Class J Class R Class J Class R Class J
diagnostic 29.61% 29.52% 11.72 12.23 7.72 8.19
pressures 59.86% 68.90% 13.19 15.24 7.68 8.55
causes 54.67% 60.19% 13.82 13.24 8.37 7.69
compare 63.02% 58.49% 13.05 13.41 7.79 8.21
final 25.72% 29.73% 13.35 14.32 8.21 8.44
global
Class R Class J
diagnostic 6.09 7.00
pressures 9.27 9.83
causes 9.64 9.83
compare 10.73 10.50
final 10.27 9.50

16. Results: local (by class) 1
1.5
Percentage of sentences correct (Z-score)
1
0.5
Class R (local first)
0
Class J (global first)
diagnostic pressures causes compare final
-0.5
-1
-1.5

17. Results: local (by class) 2
2
Average sentence length (Z-score)
1.5
1
0.5
Class R (local first)
0
Class J (global first)
diagnostic pressures causes compare final
-0.5
-1
-1.5
-2

18. Results: local (by class) 3
1.5
Gunning-Fog readability (Z-score)
1
0.5
0
diagnostic pressures causes compare final Class R (local first)
Class J (global first)
-0.5
-1
-1.5
-2

19. Results: local (by class) 4
1.5
Collective local variable (Z-score)
1
0.5
Class R (local first)
0
Class J (global first)
diagnostic pressures causes compare final
-0.5
-1
•Impact of starting proficiency?
•Final essay switch
-1.5

20. Results: global (by class)
1 Global criteria (Z-score)
0.5
0
diagnostic pressures causes compare final
Class R (local first)
-0.5
Class J (global first)
-1
-1.5
•Substantial improvement in both classes
•Final essay switch
-2

21. Results: local (by TOEIC)
1.5
Collective local variable (Z-score)
1
0.5
High TOEIC score
0
Low TOEIC score
diagnostic pressures causes compare final
-0.5
-1
-1.5

22. Results: global (by TOEIC)
1
Global criteria (Z-score)
0.5
0
diagnostic pressures causes compare final
High TOEIC score
-0.5
Low TOEIC score
-1
-1.5
-2

23. Results: local (by TOEIC)
80
Collective local variable (raw score)
70
60
50
40 High TOEIC score
Low TOEIC score
30
20
10
0
diagnostic pressures causes compare final

24. Results: global (by TOEIC)
Global criteria (raw score)
12
10
8
6 High TOEIC score
Low TOEIC score
4
2
0
diagnostic pressures causes compare final

25. Discussion (1)
Does the order of feedback in the writing process
have an effect on student writing?
 Both classes made substantial 1
0.5
diagnostic-final global 0
-0.5
Class R (local
first)
improvement -1
Class J (global
first)
-1.5
-2
 Local improvement?
1.5
 Less
substantial diagnostic-final 1
 More variation in peer-editing stage 0.5 Class R (local
first)
0
 Between classes -0.5
Class J (global
first)
 Between different essay types -1
 Amongst local variables -1.5

26. Discussion (2)
 Class J outperformed Class R in global criteria
during peer-editing stage
 Class R outperformed Class J in local criteria during
peer-editing stage
Hypothesis
During the peer-editing stage, classes showed
greater improvement in the area focused on first
[global/local].

27. Discussion(3)
 Class R outperformed Class J globally in final essay
 Class J outperformed Class R locally in final essay on
2/3 of criteria analyzed
Hypothesis
During the peer-editing stage, focusing on global
feedback first allowed students to make more immediate
improvement on global criteria. However, in post-peer-
editing [final essay], students focused more on the
last/most recent feedback stage [local/global] when
completing their essay independently.

28. Limitations of study
 Sample size
 17 students
 85 essays (36,636 words)
 Analytical measures
 Three local variables only
 Inter-rater reliability issues
 Nature and efficiency of feedback
 Teacher (local)
 Peers (global)
 Extent of student adoption/use of feedback
 During the semester
 When writing final essay

29. Future research possibilities
 Further analysis of existing sample
 Include more
texts
 Improve number & accuracy of local variables
 Continue existing study with larger sample
 Analyze nature, quality, and extent of adoption and
efficiency of feedback

30. Ongoing research
 Continue existing study with larger sample
 Class J: 21 students
 Class R: 12 students
 Possibly data from a further 14 students?
 Standardized local feedback:
 ETS Criterion® Online Writing Evaluation Service

31. Ongoing research issues (1)
Technical issues with Criterion

32. Ongoing research issues (2)
 Data-gathering v. pedagogy
 Difficultto replicate experimental conditions in
dynamic classrooms
 Possible need to change final essay may invalidate
comparison between groups?
 Broader question:
 Does our working hypothesis suggest a change to
pedagogical practice?
 If so, should classroom practice be changed, at the risk
of invalidating further research?

33. Any suggestions or
questions?
hodson@sun.ac.jp
jhensley@sun.ac.jp