Peer Feedback on Writing: A SoTL work in progress
Report
•4 likes•20,232 views
Peer Feedback on Writing: A SoTL work in progress
•4 likes•20,232 views
Download to read offline
Report
These slides are for a symposium on the Scholarship of Teaching and Learning, sponsored by the SoTL Institute at Mt. Royal University in Calgary, Alberta, Canada. They are about a project studying peer feedback on writing, in a course at the University of British Columbia in Vancouver (Canada) in which students write 10-12 essays over a full academic year, and engage in peer feedback sessions every week for that year as well.
Recommended
More Related Content
What's hot
What's hot(20)
Viewers also liked
Viewers also liked(20)
Similar to Peer Feedback on Writing: A SoTL work in progress
Similar to Peer Feedback on Writing: A SoTL work in progress(20)
More from Christina Hendricks
More from Christina Hendricks(20)
Recently uploaded
Recently uploaded(20)
Peer Feedback on Writing: A SoTL work in progress
- 1. Tracking a Dose-Response Curve in Peer Feedback on Writing A Work in Progress Christina Hendricks Co-Investigator: Jeremy Biesanz University of British Columbia-Vancouver SoTL Symposium, November 2015 Slides available here: http://is.gd/PFBwriting2015 Slides licensed CC-BY 4.0
- 2. Literature on peer feedback Receiving peer feedback improves writing (Paulus, 1999; Cho & Schunn, 2007; Cho & MacArthur, 2010; Crossman & Kite, 2012) Giving peer feedback improves writing (Cho & Cho, 2011; Li, Liu & Steckelberg, 2010)
- 3. GAPS: Most studies look at revisions to a single essay, not changes across different essays Draft 1 Draft 2 Draft 3 Essay 1 Essay 2 Essay 3 Essay 4 Essay …n PFB PF B PF B PFB PF B PFB Few studies look at “dose-response curve”
- 4. Pilot study research questions 1. How do students use peer comments given and received for improving different essays rather than drafts of the same essay? 1. Are students more likely to use peer comments given and received for improving their writing after more than one or two peer feedback sessions? How many sessions are optimal? 2. Does the quality of peer comments improve over time?
- 5. • Interdisciplinary, full year course for first-years • 18 credits (6 each in 1st year English, History, Philosophy) • Students write 10-12 essays (1500-2000 words) • Peer feedback tutorials every week (4 stdnts) http://artsone.arts.ubc.ca Toni Morrison, Wikimedia Commons, licensed CC BY-SA 2.0 Osamu Tezuka, public domain on Wikimedia Commons Jane Austen, public domain on Wikimedia Commons Friedrich Nietzsche, public domain, Wikimedia Commons
- 6. Data for pilot study 2013-2014 • 10 essays by 13 participants (130 essays) • Comments by students in tutorial group (4 in group) on all essays (n=1219) • Comments by instructor on all essays (n=3331) • All essays and comments coded according to a common rubric
- 7. Coding Rubric Categories (plus subcategories, for 11 options) • Strength of argument • Organization • Insight • Style & Mechanics Numerical value 1: Significant problem 2: Moderate problem 3: Positive comment/praise E.g., STREV 2: could use more textual evidence to support your claims
- 8. Progress so far All student and instructor comments coded 60 of 130 essays coded -- (10 essays each by 6 of the 13 participants)
- 9. Inter-coder reliability Fleiss’ Kappa Intra-class correlation Student comments (n=141) All categories: 0.61 (moderate) Most used categories: 0.8 (excellent) 0.96 (excellent) Essays (n=60) 0.68 (adequate) 3 coders: • Daniel Munro & Kosta Prodanovich (undergrads, former Arts One) • Jessica Stewart (author, editor)
- 10. LOOKING AT TRENDS IN COMMENTS OVER TIME
- 11. 0 2 4 6 8 10 12 024681012 Essay Number InstructorNumberofComments Argument Strength Style Insight Organization INSTRUCTOR Comments - .28**Strength Style Organiz. Insight -.04* Number of 2 comments in each categ.
- 12. 0 2 4 6 8 10 12 01234 Essay Number StudentNumberofComments Argument Strength Style Insight Organization STUDENT comments Strength Style Organiz. Insight -.16** Number of 2 comments in each categ.
- 13. 0 2 4 6 8 10 12 012345 Essay Number InstructorNumberofComments Argument Strength Style Insight Organization INSTRUCTOR Comments .31*** Strength Style Organiz. Insight .08** .19** .11** Number of 3 comments in each category
- 14. 0 2 4 6 8 10 12 0.00.51.01.52.02.53.0 Essay Number StudentNumberofComments Argument Strength Style Insight Organization STUDENT Comments Strength Style Organiz. Insight Number of 3 comments in each categ.
- 15. Student/instructor agreement on comments • Average numerical ratings for comments across all categories agree strongly between student and instructor (.48****) • However, this agreement goes down across essays (-.04**) o This is because student ratings increase over time at only half the rate that instructor ratings do *p < .05, **p< .01, ***p< .001, ****p < .0001
- 16. HOW DOES ESSAY QUALITY CHANGE OVER TIME? From this slide onwards, we are looking only at 60 essays coded so far, out of the set of 130 (essays from 6 of the 13 participants)
- 17. Essay quality improves linearly in 60 essays 0 2 4 6 8 10 3.03.54.04.55.05.5 Essay EssayQualityRating Mean essay quality rating 4.12 out of 7, SD = .62
- 19. Cross-lagged panel design with auto-regressive structure Essay Quality Time 1 Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N
- 20. Path A: Student comments Essay Quality Time 1 Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N Significant effects: • Ratings of 2 in Insight (-.53*) • Ratings of 3 in Strength (.25*)
- 21. Path C: instructor comments Essay Quality Time 1 Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N Significant effects • Ratings of “2” in Strength (-.31**) • Ratings of “3” in Strength (.51***) and Style/Mechanics (.34**)
- 22. Path C: student comments Essay Quality Time 1 Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N Significant effects • Ratings of “2” in Style (.28*) and Insight (.15*) • Ratings of “3” in Strength (.30*)
- 23. Path D: Student comments Essay Quality Time 1 Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N Significant effects: • Ratings of 2 in Organization (.16*) • Ratings of 3 in Style (.16*)
- 24. Research question 2 How do students use peer comments given and received for improving different essays rather than drafts of the same essay? o Not enough evidence yet to say much about path D o Haven’t yet looked at differences in comments given vs. received
- 25. Research question 2 Are students more likely to use peer comments given and received for improving their writing after more than one or two peer feedback sessions? How many sessions are optimal? o No evidence yet that there is any change over time in path D
- 26. Research question 3 Does the quality of peer comments improve over time? o No evidence yet that there is any change over time in path A Essay Quality Time 1 Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N
- 27. Some conclusions so far • Pilot study: is this sort of study feasible for larger sample? o Yes, but probably more so if instructors code essay quality rather than coders • Facilitating easy collection of student & instructor comments is difficult
- 28. References • Cho, K., & MacArthur, C. (2010). Student revision with peer and expert reviewing, Learning and Instruction. 20, 328-338. • Cho, Y. H., & Cho, K. (2011). Peer reviewers learn from giving comments. Instructional Science, 39, 629-643. • Cho, K. & Schunn, C. D. (2007). Scaffolded writing and rewriting in the discipline: A web-based reciprocal peer review system. Computers & Education, 48, 409–426 • Crossman, J. M., & Kite, S. L. (2012). Facilitating improved writing among students through directed peer review, Active Learning in Higher Education, 13, 219-229. • Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational Technology, 41(3), 525–536. • Paulus, T. M. (1999). The effect of peer and teacher feedback on student writing. Journal of Second Language Writing, 8, 265-289.
- 29. Thank you! Christina Hendricks University of British Columbia-Vancouver Website: http://blogs.ubc.ca/christinahendricks Blog: http://blogs.ubc.ca/chendricks Twitter: @clhendricksbc Slides available: http://is.gd/PFBwriting2015 Slides licensed CC-BY 4.0
Editor's Notes
- How much agreement do we observe relative to how much we would expect to see by chance? -- takes into account the frequency of the type of code occurring in the data -- some codes are more frequent, so you’d expect those to have more apparent agreement -1 to +1 0 = amount of agreement we’d expect to see by chance -1 is complete disagreement 0.6 is moderate agreement; 0.8 is substantial -- Kappa includes just the category Many of the mostly used categories have agreement in 0.8 range Reliability on degree: intra class correlation (ICC) of 0.96 -- to what extent is the average across the three raters reliable: average of all the numbers each gave—how does this correlate with the average of everyone who could possibly do this—get no benefit for adding more people -- average is 2.5 -- 1’s are pretty infrequent -- people agree on whether a 2 or a 3 (40% are 2s, 60% are 3s)
- Refined coding rubric Added, subtracted, condensed dimensions acc. to student comments Added examples of each sub-dimension Out of 242 peer comments: All 3 coders agree on value (1-3), regardless of dimension: 90% 2 or 3 coders agree on dimension and final decision (after meeting) is same as that: 82% 2 coders agree on dimension & final is different: 12% 3 coders agree on dimension & final is same: 56% Just last set of 70 comments Single meaning units Had several comments that could be given more than one code; needed to split them up so each comment had one code so as to better do analysis