• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Peer review in an economics writing course: the Pen is still mightier than the SWoRD,
 

Peer review in an economics writing course: the Pen is still mightier than the SWoRD,

on

  • 787 views

Jennifer Imazeki, Economics ...

Jennifer Imazeki, Economics
Scaffolded Writing and Reviewing in the Disciplines(SWoRD) is a web-based peer-review system. One of the primary innovations of SWoRD, relative to other peer review tools, is the scoring algorithm through which peer review scores are converted into student grades for both writing and reviewing. In this session, I will discuss my experience with SWoRD, which I used in Spring 2011 for an upper-division writing course for economics majors, replacing my previous system of ‘manual’ peer review (i.e., students swapping papers)

Statistics

Views

Total Views
787
Views on SlideShare
787
Embed Views
0

Actions

Likes
0
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • http://www.lrdc.pitt.edu/schunn/sword/index.html
  • Review process itself similar to other options (e.g., PeerMark, doing it manually)
  • Key point here is that the focus is not JUST writing but incorporating economic ideas into writing (so they have to know some econ)
  • - Explain process (half class writing/half reviewing)- I already had rubrics and peer review instructions for each assignment, so fixed costs low
  • This was the main reason I wanted to try SWoRD – already did some peer review but SWoRD a) allows for more feedback overall and b) potentially less grading for me
  • - the biggest change in the structurewas that because you need at least 4 or more reviews in SWoRD to get meaningful composite scores, I reduced the number of writing assignments and stretched out the timeline for each paperI had hoped to give less feedback on the first draft but after first assignment, realized that wasn’t going to happen – partly a problem with my comment and rating prompts, partly students don’t know how to give good feedback, partly that students don’t feel ‘qualified’ to give feedback (since emphasis in this course is as much on content as writing) These are challenges for peer review in general (not necessarily SWoRD): students need training in how to give good comments; prompts and ratings need to be well-designed, not vague, get students to recognize their own ability to be ‘experts’But the problems students have with giving good reviewing feedback is even more important with if student grades are going to be based on those reviews and that feedbackBack evaluations did not provide good assessment or sufficient feedback for improving comments
  • One thing I want to point out is that my class met on Mondays and Wednesdays – issues with SWoRD deadlines
  • Interface overall is not very flexible (deadlines biggest problem but also admin)Explain about how SWoRD requires curve but not transparent (show screenshots)Can see raw scores, final scores, but can’t see scores without curve applied
  • Stats for each draft
  • Draft grades (cannot change order of presentation in SWoRD)
  • So can SWoRD replace instructor grading? No

Peer review in an economics writing course: the Pen is still mightier than the SWoRD, Peer review in an economics writing course: the Pen is still mightier than the SWoRD, Presentation Transcript

  • Peer review in an economics writing course: the Pen is still mightier than the SWoRD
    Jennifer Imazeki
    Department of Economics
    jimazeki@gmail.com
    http://economicsforteachers.blogspot.com
  • SWoRD
    Scaffolded Writing and Reviewing in the Disciplines
    Students submit papers
    Students review papers of 4-6 peers
    Comments and 7-point numeric rating
    Students ‘back evaluate’ reviews they receive
    5-point numeric rating
  • What makes SWoRD different?
    Student review and backevaluation scores  grades for both writing and reviewing
    Reviewing grade: “Accuracy” + “Helpfulness”
    Accuracy: Consistency compared to other reviewers
    Helpfulness: Back evaluation ratings
    Writing grade: adjusted average of numeric ratings from reviewers
    Scoring algorithm to remove bias
  • Econ 449W: Economic Literacy
    “The goal of this course is to make you better economists by making you better writers… This course will focus on how to apply and use [economic] knowledge, by incorporating economic material and the economic way of thinking into writing for a more general audience.”
  • Econ 449W: Structure
    2010:
    6 peer-reviewed papers, exchanged in class
    1 reviewer for each paper, plus instructor feedback
    No feedback for reviewers, participation points only
  • Potential benefits of SWoRD(vs. other peer review tools)
    Students read multiple papers, receive feedback from multiple reviewers
    Back evaluations provide incentive to take reviews seriously, give students feedback for improving reviews (without requiring instructor intervention)
    Rubric scores converted to grades with adjustment for bias
  • Econ 449W: 2010 vs. 2011
    2010:
    6 peer-reviewed papers, exchanged in class
    1 reviewer for each paper, plus instructor feedback
    No feedback for reviewers, participation points only
    2011:
    4 peer reviewed papers, all online submission
    4-5 reviewers for each paper, no/less instructor feedback on draft 1
    Writers back-evaluate reviewers, reviewing score based 50% on those back evaluations
  • Assignment timeline
    5 days
    5 days
    5 days
    9pm Monday
    9pm Sunday
    9pm Friday
    9pm Wednesday
    First Draft
    Final draft
    Final reviews (5)
    Reviews (5)
    Back reviews by Thurs
    Back reviews by Wednesday
  • Issues
    Lack of flexibility
    Lack of transparency
    Scoring algorithm questionable
  • Peer review in an economics writing course: the Pen is still mightier than the SWoRD
    Jennifer Imazeki
    Department of Economics
    jimazeki@gmail.com
    http://economicsforteachers.blogspot.com