Peer review in an economics writing course: the Pen is still mightier than the SWoRD,

941 views
826 views

Published on

Jennifer Imazeki, Economics
Scaffolded Writing and Reviewing in the Disciplines(SWoRD) is a web-based peer-review system. One of the primary innovations of SWoRD, relative to other peer review tools, is the scoring algorithm through which peer review scores are converted into student grades for both writing and reviewing. In this session, I will discuss my experience with SWoRD, which I used in Spring 2011 for an upper-division writing course for economics majors, replacing my previous system of ‘manual’ peer review (i.e., students swapping papers)

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
941
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • http://www.lrdc.pitt.edu/schunn/sword/index.html
  • Review process itself similar to other options (e.g., PeerMark, doing it manually)
  • Key point here is that the focus is not JUST writing but incorporating economic ideas into writing (so they have to know some econ)
  • - Explain process (half class writing/half reviewing)- I already had rubrics and peer review instructions for each assignment, so fixed costs low
  • This was the main reason I wanted to try SWoRD – already did some peer review but SWoRD a) allows for more feedback overall and b) potentially less grading for me
  • - the biggest change in the structurewas that because you need at least 4 or more reviews in SWoRD to get meaningful composite scores, I reduced the number of writing assignments and stretched out the timeline for each paperI had hoped to give less feedback on the first draft but after first assignment, realized that wasn’t going to happen – partly a problem with my comment and rating prompts, partly students don’t know how to give good feedback, partly that students don’t feel ‘qualified’ to give feedback (since emphasis in this course is as much on content as writing) These are challenges for peer review in general (not necessarily SWoRD): students need training in how to give good comments; prompts and ratings need to be well-designed, not vague, get students to recognize their own ability to be ‘experts’But the problems students have with giving good reviewing feedback is even more important with if student grades are going to be based on those reviews and that feedbackBack evaluations did not provide good assessment or sufficient feedback for improving comments
  • One thing I want to point out is that my class met on Mondays and Wednesdays – issues with SWoRD deadlines
  • Interface overall is not very flexible (deadlines biggest problem but also admin)Explain about how SWoRD requires curve but not transparent (show screenshots)Can see raw scores, final scores, but can’t see scores without curve applied
  • Stats for each draft
  • Draft grades (cannot change order of presentation in SWoRD)
  • So can SWoRD replace instructor grading? No
  • Peer review in an economics writing course: the Pen is still mightier than the SWoRD,

    1. 1. Peer review in an economics writing course: the Pen is still mightier than the SWoRD<br />Jennifer Imazeki<br />Department of Economics<br />jimazeki@gmail.com<br />http://economicsforteachers.blogspot.com<br />
    2. 2. SWoRD<br />Scaffolded Writing and Reviewing in the Disciplines<br />Students submit papers<br />Students review papers of 4-6 peers<br />Comments and 7-point numeric rating<br />Students ‘back evaluate’ reviews they receive<br />5-point numeric rating<br />
    3. 3. What makes SWoRD different?<br />Student review and backevaluation scores  grades for both writing and reviewing<br />Reviewing grade: “Accuracy” + “Helpfulness”<br />Accuracy: Consistency compared to other reviewers<br />Helpfulness: Back evaluation ratings<br />Writing grade: adjusted average of numeric ratings from reviewers<br />Scoring algorithm to remove bias<br />
    4. 4. Econ 449W: Economic Literacy<br />“The goal of this course is to make you better economists by making you better writers… This course will focus on how to apply and use [economic] knowledge, by incorporating economic material and the economic way of thinking into writing for a more general audience.”<br />
    5. 5. Econ 449W: Structure<br />2010:<br />6 peer-reviewed papers, exchanged in class<br />1 reviewer for each paper, plus instructor feedback<br />No feedback for reviewers, participation points only<br />
    6. 6. Potential benefits of SWoRD(vs. other peer review tools)<br />Students read multiple papers, receive feedback from multiple reviewers<br />Back evaluations provide incentive to take reviews seriously, give students feedback for improving reviews (without requiring instructor intervention) <br />Rubric scores converted to grades with adjustment for bias<br />
    7. 7. Econ 449W: 2010 vs. 2011<br />2010:<br />6 peer-reviewed papers, exchanged in class<br />1 reviewer for each paper, plus instructor feedback<br />No feedback for reviewers, participation points only<br />2011:<br />4 peer reviewed papers, all online submission<br />4-5 reviewers for each paper, no/less instructor feedback on draft 1<br />Writers back-evaluate reviewers, reviewing score based 50% on those back evaluations<br />
    8. 8. Assignment timeline<br />5 days<br />5 days<br />5 days<br />9pm Monday<br />9pm Sunday<br />9pm Friday<br />9pm Wednesday<br />First Draft<br />Final draft<br />Final reviews (5)<br />Reviews (5)<br />Back reviews by Thurs<br />Back reviews by Wednesday<br />
    9. 9. Issues<br />Lack of flexibility<br />Lack of transparency<br />Scoring algorithm questionable<br />
    10. 10.
    11. 11.
    12. 12. Peer review in an economics writing course: the Pen is still mightier than the SWoRD<br />Jennifer Imazeki<br />Department of Economics<br />jimazeki@gmail.com<br />http://economicsforteachers.blogspot.com<br />

    ×