Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Peer Feedback on Writing: A SoTL work in progress

12,861 views

Published on

These slides are for a symposium on the Scholarship of Teaching and Learning, sponsored by the SoTL Institute at Mt. Royal University in Calgary, Alberta, Canada. They are about a project studying peer feedback on writing, in a course at the University of British Columbia in Vancouver (Canada) in which students write 10-12 essays over a full academic year, and engage in peer feedback sessions every week for that year as well.

Published in: Education
  • Be the first to comment

Peer Feedback on Writing: A SoTL work in progress

  1. 1. Tracking a Dose-Response Curve in Peer Feedback on Writing A Work in Progress Christina Hendricks Co-Investigator: Jeremy Biesanz University of British Columbia-Vancouver SoTL Symposium, November 2015 Slides available here: http://is.gd/PFBwriting2015 Slides licensed CC-BY 4.0
  2. 2. Literature on peer feedback Receiving peer feedback improves writing (Paulus, 1999; Cho & Schunn, 2007; Cho & MacArthur, 2010; Crossman & Kite, 2012) Giving peer feedback improves writing (Cho & Cho, 2011; Li, Liu & Steckelberg, 2010)
  3. 3. GAPS: Most studies look at revisions to a single essay, not changes across different essays Draft 1 Draft 2 Draft 3 Essay 1 Essay 2 Essay 3 Essay 4 Essay …n PFB PF B PF B PFB PF B PFB Few studies look at “dose-response curve”
  4. 4. Pilot study research questions 1. How do students use peer comments given and received for improving different essays rather than drafts of the same essay? 1. Are students more likely to use peer comments given and received for improving their writing after more than one or two peer feedback sessions? How many sessions are optimal? 2. Does the quality of peer comments improve over time?
  5. 5. • Interdisciplinary, full year course for first-years • 18 credits (6 each in 1st year English, History, Philosophy) • Students write 10-12 essays (1500-2000 words) • Peer feedback tutorials every week (4 stdnts) http://artsone.arts.ubc.ca Toni Morrison, Wikimedia Commons, licensed CC BY-SA 2.0 Osamu Tezuka, public domain on Wikimedia Commons Jane Austen, public domain on Wikimedia Commons Friedrich Nietzsche, public domain, Wikimedia Commons
  6. 6. Data for pilot study 2013-2014 • 10 essays by 13 participants (130 essays) • Comments by students in tutorial group (4 in group) on all essays (n=1219) • Comments by instructor on all essays (n=3331) • All essays and comments coded according to a common rubric
  7. 7. Coding Rubric Categories (plus subcategories, for 11 options) • Strength of argument • Organization • Insight • Style & Mechanics Numerical value 1: Significant problem 2: Moderate problem 3: Positive comment/praise E.g., STREV 2: could use more textual evidence to support your claims
  8. 8. Progress so far All student and instructor comments coded 60 of 130 essays coded -- (10 essays each by 6 of the 13 participants)
  9. 9. Inter-coder reliability Fleiss’ Kappa Intra-class correlation Student comments (n=141) All categories: 0.61 (moderate) Most used categories: 0.8 (excellent) 0.96 (excellent) Essays (n=60) 0.68 (adequate) 3 coders: • Daniel Munro & Kosta Prodanovich (undergrads, former Arts One) • Jessica Stewart (author, editor)
  10. 10. LOOKING AT TRENDS IN COMMENTS OVER TIME
  11. 11. 0 2 4 6 8 10 12 024681012 Essay Number InstructorNumberofComments Argument Strength Style Insight Organization INSTRUCTOR Comments - .28**Strength Style Organiz. Insight -.04* Number of 2 comments in each categ.
  12. 12. 0 2 4 6 8 10 12 01234 Essay Number StudentNumberofComments Argument Strength Style Insight Organization STUDENT comments Strength Style Organiz. Insight -.16** Number of 2 comments in each categ.
  13. 13. 0 2 4 6 8 10 12 012345 Essay Number InstructorNumberofComments Argument Strength Style Insight Organization INSTRUCTOR Comments .31*** Strength Style Organiz. Insight .08** .19** .11** Number of 3 comments in each category
  14. 14. 0 2 4 6 8 10 12 0.00.51.01.52.02.53.0 Essay Number StudentNumberofComments Argument Strength Style Insight Organization STUDENT Comments Strength Style Organiz. Insight Number of 3 comments in each categ.
  15. 15. Student/instructor agreement on comments • Average numerical ratings for comments across all categories agree strongly between student and instructor (.48****) • However, this agreement goes down across essays (-.04**) o This is because student ratings increase over time at only half the rate that instructor ratings do *p < .05, **p< .01, ***p< .001, ****p < .0001
  16. 16. HOW DOES ESSAY QUALITY CHANGE OVER TIME? From this slide onwards, we are looking only at 60 essays coded so far, out of the set of 130 (essays from 6 of the 13 participants)
  17. 17. Essay quality improves linearly in 60 essays 0 2 4 6 8 10 3.03.54.04.55.05.5 Essay EssayQualityRating Mean essay quality rating 4.12 out of 7, SD = .62
  18. 18. MORE COMPLEX ANALYSES
  19. 19. Cross-lagged panel design with auto-regressive structure Essay Quality Time 1 Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N
  20. 20. Path A: Student comments Essay Quality Time 1 Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N Significant effects: • Ratings of 2 in Insight (-.53*) • Ratings of 3 in Strength (.25*)
  21. 21. Path C: instructor comments Essay Quality Time 1 Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N Significant effects • Ratings of “2” in Strength (-.31**) • Ratings of “3” in Strength (.51***) and Style/Mechanics (.34**)
  22. 22. Path C: student comments Essay Quality Time 1 Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N Significant effects • Ratings of “2” in Style (.28*) and Insight (.15*) • Ratings of “3” in Strength (.30*)
  23. 23. Path D: Student comments Essay Quality Time 1 Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N Significant effects: • Ratings of 2 in Organization (.16*) • Ratings of 3 in Style (.16*)
  24. 24. Research question 2 How do students use peer comments given and received for improving different essays rather than drafts of the same essay? o Not enough evidence yet to say much about path D o Haven’t yet looked at differences in comments given vs. received
  25. 25. Research question 2 Are students more likely to use peer comments given and received for improving their writing after more than one or two peer feedback sessions? How many sessions are optimal? o No evidence yet that there is any change over time in path D
  26. 26. Research question 3 Does the quality of peer comments improve over time? o No evidence yet that there is any change over time in path A Essay Quality Time 1 Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N
  27. 27. Some conclusions so far • Pilot study: is this sort of study feasible for larger sample? o Yes, but probably more so if instructors code essay quality rather than coders • Facilitating easy collection of student & instructor comments is difficult
  28. 28. References • Cho, K., & MacArthur, C. (2010). Student revision with peer and expert reviewing, Learning and Instruction. 20, 328-338. • Cho, Y. H., & Cho, K. (2011). Peer reviewers learn from giving comments. Instructional Science, 39, 629-643. • Cho, K. & Schunn, C. D. (2007). Scaffolded writing and rewriting in the discipline: A web-based reciprocal peer review system. Computers & Education, 48, 409–426 • Crossman, J. M., & Kite, S. L. (2012). Facilitating improved writing among students through directed peer review, Active Learning in Higher Education, 13, 219-229. • Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational Technology, 41(3), 525–536. • Paulus, T. M. (1999). The effect of peer and teacher feedback on student writing. Journal of Second Language Writing, 8, 265-289.
  29. 29. Thank you! Christina Hendricks University of British Columbia-Vancouver Website: http://blogs.ubc.ca/christinahendricks Blog: http://blogs.ubc.ca/chendricks Twitter: @clhendricksbc Slides available: http://is.gd/PFBwriting2015 Slides licensed CC-BY 4.0

×