Peer review and the development of evaluative skills
Peer Review and the development of evaluative skills David Nicol Emeritus Professor of Higher Education University of Strathclyde, Scotland Visiting Professor, University of Ulster Adjunct Professor, University of Swinburne, AustraliaConsultant to JISC: Assessment and Feedback Programme JISC Webinar: 18 February 2013
Plan for Session Brief introduction to peer review You experience of peer review – produce assignment (5 mins) and review two peer assignments (10 mins) Reflection and discussion Drawing on case examples – engineering, sociology chemistry
Engineering DesignPeer Project case study DM 100 Design 1: first-year class Dr Avril Thomson, Course Leader, Design Manufacturing and Engineering Management (DMEM), University of Strathclyde email@example.com Caroline Breslin, Learning Technology Adviser, University of Srathclyde firstname.lastname@example.orgFunded by JISC: see www.reap.ac.uk/PEER.aspx
Peer reviewDefinition of peer review Peer review is an arrangement whereby students make evaluative judgements about the work of peers and provide a written feedback commentary. In peer review, students produce as well as receive feedback.
Introduction Research on peer review has been confounded by three factors (i) an over-focus on peer assessment rather than peer review (ii) a bias towards examining the benefits of receipt of feedback reviews rather than the production of feedback reviews and (iii) studies examining general benefits of peer review without distinguishing its component parts. Today the focus is primarily on producing feedback reviews
Your assignment Write a convincing argument for having students review the work of peers (the reviewing component only). Provide evidence for your argument (from literature, logical, from you own experience, convincingly anecdotal) and identify and respond to any obvious counter-arguments. Criteria for good argument are: convincingness of argument (ii) evidence in support of argument (iii) identification and responses to obvious counter-arguments. Five minutes for this task Normally students can produce about 10-14 lines of text Access to the task and instructions is here in google docs: http://bit.ly/peerevalform Keep your audio on and please keep to time.
The peer review task Review and provide feedback comments on the work of two peers using the given criteria.
Argument from peer 1I think that students would gain understanding of their own work inthe process of reviewing the work of peers. When reading anotherstudent’s work, the reviewer would more likely be able to see areaswhere improvements could be made. It is often the case that it iseasier to identify others’ weaknesses than one’s own. Whenreviewing the work of others, the student would engage in a processof comparison with their own work. This leads to a form of reflectionotherwise not available. However, it could be argued that studentsare not well-qualified to comment on the work of others. They donot have the knowledge of the subject or the pedagogical training tomake valuable comments. This I do not agree with. Students areoften close to each other in their level of knowledge and writing andwould therefore be able to give constructive criticism. At the sametime, giving criticism would heighten awareness of their ownperformance.
Argument from peer 2Having students review the work of peers should be a regularactivity in higher education because if students do this they will seethe way others tackle the same assignment and they will learn andget ideas from this. Also, when they review they will have to applysome criteria and this will help them to understand these criteriabetter. In my experience students often produce poor assignmentsbecause they do not understand what is expected, not because theycannot do the work. Indeed, when I organise peer discussion ofcriteria before a task this results in better quality work. However, itis clear that there might be problems of plagiarism as in readingpeers assignments it is likely that students will copy without owningthe ideas themselves. This could be tackled, however, by havingstudents review and just say what they would do to improve theirown assignment (if they had the opportunity) without actuallygetting them to do it. In this way, they would provide evidence ofinterpretation rather than copying.
Criteria/questions for the peer review (i) Summarise the core of the argument written by each peer in one sentence.(ii) Identify and list what evidence is actually used to support the argument?(ii) Make one suggestion that would strengthen the argument. Give a reason for your suggestion in a sentence or two.Tackle one review, then the next and submitPeer assignment 1 is here: http://bit.ly/peerarg1Peer assignment 2 is here: http://bit.ly/peerarg2Time = 10 minutes = 5 minutes each review
Discussion and reflection You: Reflection on your experience – the learning from peer review Me: presentation of some recent research findings using ‘student quotes’ Facilitators – manage chat discussion and highlight questions and ideas
Reflection (1) What did you learn from this peer review exercise?
What was the most valuable aspect of the reviewing process?Rate each of the following on the following scale (where 0 is not valuable,1 is of some value and 7 is very valuable). 0 1 3 5 71.Seeing how peers had approached this task2.Engagement with the criteria/questions3.Comparing the peer assignments with your own4.Making evaluative judgements about others’ work5.Writing the feedback commentaries6.Comparing one peer’s work with the other7.Thinking about changes to your own assignment
Results: learning from RECEIVING reviewsPlease give examples of what you learned from RECEIVING peer reviewsfrom other students (n=54)Specific content mentioned: Depth of analysis needed, more numericaldata and figures, stronger rationale, how to structure it better etc.Receiving peer reviews gave me insight into what others thought of mywork and gave me a direction to improve (reader response)Where the PDS was confusing to understand (reader response)Parts that I had previously missed were brought to eye such as marketcompetition (noticing)The person who peer reviewed my PDS gave me positive feedbackwhich helped me a lot (motivational)Not much, they...[the peer reviews]...weren’t very good (no value)
Results: learning from PROVIDING reviewsPlease give examples of what you learned from PROVIDING peer reviewsof other’s work (n=47)How to look at work critically that isn’t your own [critical judgement]Thinking from a critical point of view [critical judgement]I was given a greater understanding of the level of the work thecourse may be demanding [attention to expectations/criteria]Allowed me to see from an assessor’s perspective[expectations/criteria]When giving advice to people on theirs, it gave me greater perceptionwhen reviewing my own work by listening to my own advice forexample [reflection/transfer]I had a chance to see other peoples work and aspects of their workthat I felt were lacking in my work, this helped me to improve my work[reflection/transfer]
Results: How you carried out peer reviewCould you make any comments about how you carried out thepeer review? How did you evaluate the quality of the work toprovide a response to the peer review questions? (n=37)I compared it to mine and ...and said how I would improve itPartly by comparing my work to theirsI tried to think about what I wrote and whether this productdesign specification was better or worse
Focus groups How did you go about reviewing? ‘I read it through and compared it with what I had done to see if they had put something I had not done and then I added it in if they hadn’t. The four questions...[provided by the teacher]...were useful as they provided a framework for the review. If we hadn’t had the questions it would have been difficult. I did the reviews separately and then answered one then the other. The first was a better standard than the other – so I used the ideas from the better one to comment on the weaker one. I also read the guidelines in class when I did the peer review. There were ideas from the good one that I hadn’t even thought of in mine’
Results: reviewingIn the focus groups the effects of the review questions (criteria)was probed further. Typical comments were:You compare it (the other student’s work) to the criteria butthen in the back of your mind you’re comparing it to our own atthe same time.I went down the questions and compared it to my own..I wastrying to think what has this person done. Have they put in moreeffort or knowledge than me.I went through the questions keeping my own in mindYou’ve got what you’ve done at the back of your mind whilegoing over theirs so you see where you’ve gone wrong withoutanyone pointing it out so you learn it yourselfReviewing is grounded in comparisons with students’ own work(Nicol, Thomson and Breslin, 2013)
Summary Reviewing elicits multiple acts of evaluative judgement 1. Evaluate peer’s work against own 2. Evaluate one peer’s work against another (and own) 3. Evaluate work against given criteria to produce response The pre-condition for these effects 1 Students must first have produced an assignment in the ‘same domain’ as those that they are asked to review To what extent does your experience resonate with this finding?
Some thoughts about criteria/standards Students both create and apply evaluative criteria 1 Create criteria as they compare work with own (holistic judgements) 2 Apply explicit criteria (analytic) to instances of practice when the produce a written response (analytic judgements) ‘Through reviewing students generate richer criteria than those provided by the teacher but sounder criteria than those they might be able to formulate on their own’ (Nicol, Thomson and Breslin, 2013)Nicol, D., Thomson, A and Breslin, C. 2013. Rethinking feedback in higher education: a peer review perspective. Submitted to Assessment and Evaluation in Higher Education
Reflection Why are these findings important in relation to current debates about feedback in higher education?
Purpose of feedback Feedback should develop the students’ capacity to make evaluative judgements about their own and others work (Boud and Associates, 2010: Cowan, 2010; Sadler, 2010) Feedback should serve the function of progressively enabling students to better monitor, evaluate and regulate their own learning, independently of the teacher (Nicol and Macfarlane-Dick, 2006: Nicol, 2009)
Limitations with received feedback1. Can promote learning of scripted responses – students dependent on teacher (Orsmond and Merry, 2011)2. Assumes that ‘others are required to identify and provide the information students need to learn and that learning is driven by how others go about this’ (Boud and Malloy, 2012)3. Any use of teacher feedback involves students in acts of self-assessment (Nicol, 2012: Black and William, 1997)4. Even if you require students to act on teacher feedback it is still a transmission model5. Practicality – teacher workload
Focus groups What do you think is best for learning – giving or receiving feedback?I think when you are reviewing...[the work of peers]...it’s more a self-learning process, you’re teaching yourself; well, I can see somebody’s done that and that’s a strength, and I should maybe try and incorporate that somehow into my work. Whereas getting...[teacher]... feedback you’re kind of getting told what to do; you’re getting told this is the way you should be doing it, and this is the right way to do it. You’re not really thinking for yourself.... I think...[reviewing]... would help you not need so much of teacher feedback, if there was more of this. Whereas, I think if you’re not being able to do... [reviewing]... then you will always be needing more...[teacher feedback]...
Focus groups What do you think is best for learning – giving or receiving feedback? ‘For me it would probably be to give feedback because I think seeing what other people have done is more helpful than getting other people’s comments on what you have already done. By looking at other people’s work you can see for yourself what you have forgotten or not even thought about. When people give feedback on yours they generally just talk about what is there. They don’t say, well I did this on mine and you could put that in yours.’
Peer review: a new perspective on feedback Students construct feedback ‘meanings’ for themselves while they produce it for others Puts feedback processes in the hands of the student Reduces their need for teacher feedback Suggests another focus for teacher feedback – helping students calibrate the quality of their own judgements (reviews)Nicol, D., Thomson, A and Breslin, C. 2013. Rethinking feedback in higher education: a peerreview perspective. Submitted to Assessment and Evaluation in Higher Education
Principles of good peer review practice1. Ensure an atmosphere of trust and respect2. Encourage critical engagement with criteria and standards3. Ensure an assignment has been produced in the same domain as those to be reviewed4. Require well-reasoned explanations for reviews (not just marks)5. Give practice in making holistic as well as analytic judgements6. Facilitate dialogue around the object and quality of the review7. Integrate self-reviews into peer review designs8. Provide signposts that help students calibrate the quality of their reviews9. Encourage critical reflection on received reviewsNicol (2013) available at http://www.reap.ac.uk/PEERToolkit/Design.aspxwww.reap.ac.uk/PEER.aspx
Feedback in professional and workplace settings1. In the professions, feedback never comes from a single source: task is usually to evaluate, weigh up and reconcile and respond to different and sometimes contradictory feedback perspectives.2. Professionals are not just ‘consumers’ of feedback but also ‘producers’ Nicol , D. Thomson, A and Breslin, C. 2013. Rethinking feedback practices in higher education: a peer review perspective. Submitted to Assessment and Evaluation in Higher Education
Design decisions1. Target task – factual or open-ended (design, computer programme, essay, report etc)2. Unit for task: individual, pair, group3. Unit for review: individual, pair, group work4. Matching reviewers: random, by ability, by topic5. Number of reviews – more is better6. Privacy: anonymous or known reviewer and/or author7. Peer review rubric – not-given: guidelines: fixed format8. Review focus: holistic v analytic, content or process9. Use of received reviews: drafts, self-review, new task10. Requesting and responding to feedback11. Grading: no marks, marks for participation, for reviews, marks for self-review after peer review
ReferencesNicol, D., Thomson, A & Breslin, C. (2013). Rethinking feedback in higher education: a peer reviewpersepective, Submitted to Assessment and Evaluation in Higher Education.Nicol, D (2013), Resituating feedback from the reactive to the proactive. In D. Boud and L. Malloy (Eds)Effective Feedback in Higher and Professional Education: understanding it and doing it well, Routledge UKBoud, D. and Associates (2010) Assessment 2020: Seven propositions for assessment reform in highereducation. Sydney: Australian Learning and Teaching Council (ALTC) Available fromwww.assessmentfutures.comCowan, J. (2010) Developing the ability for making evaluative judgements, Teaching in Higher Education,15(3), 323-334.Nicol, D. and Macfarlane-Dick, D. (2006), Formative assessment and self-regulated learning: A model andseven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218Nicol, D (2009) Assessment for learner self-regulation: enhancing achievement in the first year using learningtechnologies, Assessment and Evaluation in Higher Education, 34(3), 335-352.Nicol, D (2010) From monologue to dialogue: improving written feedback in mass higher education,Assessment and Evaluation in Higher Education, 35:5, 501-517Nicol, D (2010) The foundation for graduate attributes: developing self-regulation through self and peerassessment, QAA Scotland, Enhancement Themes. Available at:http://www.enhancementthemes.ac.uk/resources/publications/graduates-for-the-21st-centuryNicol, D (2011) Developing students’ ability to construct feedback, QAA Scotland, Enhancement Themes.Available at http://www.enhancementthemes.ac.uk/resources/publications/graduates-for-the-21st-centuryRoscoe, R. & Chi, M. (2008) Tutor learning: the role of explaining and responding to questions, InstructionalScience, 36, 321-350.Sadler, D.R (2010) Beyond feedback: Developing student capability in complex appraisal, Assessment andEvaluation in Higher Education, 35:5, 535-550