Analysing feedback:opening the black boxDr Gwyneth Hughes, Institute of   Education, London
The purpose of assessment is to select learners for  further study and/or careers.Agree/disagreeThe purpose of assessment ...
OutlineWhy analyse feedback-some current issuesUsing a feedback analysis tool (Assessment Careers JISC funded   project at...
Some current issues with feedback inHE•   What is feedback?-received not just given out (Nicol &    Macfarlane-Dick, 2005)...
• Praise is not helpful to students unless accompanied by  detail on why the praise is deserved. Encourages  dependency on...
Assessment Careers Project: taking action on feedback• Year 1: Baseline and feedback analysis tool• Year 2 (2012/13): Five...
Tool to identify feedback categories P1 Giving praise P2 Recognising progress (ipsative feedback) Critique      • C1 Corre...
Analysing feedback categories • The score is the number of times a classification   appears in the feedback • The default ...
You have clearly developed your thinking on how you might usetechnologies in your teaching but as I mentioned in the previ...
You have clearly developed your thinking on how you might usetechnologies in your teaching but as I mentioned in the previ...
The structure of the essay was very strong, with a clear introduction   which nicely contextualised the general problem of...
Data• Analysed formative and summative assessment  feedback for modules on 5 postgraduate programmes  at the IOE (total 22...
Baseline feedback profile                                     Total comments for 5 programmes                             ...
Baseline feedback profile                                 Total comments for 5 programmes (N= 165) Category of feedback   ...
Overall you have written a really comprehensive and detailed review of     the document. Your strength lies in your fantas...
What do students do with feedback?                                 Formative n=50   Summative n=35didnt respond to the que...
Overview of interventionsPilot 1 MA Education, Health Promotion and International DevelopmentPilot 2 MRes in Educational a...
Assignment cover sheets     Thinking about the feedback on your draft of this essay, please     indicate what the key poin...
Is ipsative feedback appreciated bystudents?“I   made an effort to act on it (formative feedback). This     was acknowledg...
Principles for Action on Feedback1. Feedback is directed towards students concerns2. Students are encouraged to see the va...
ConclusionFindings so far:• Feedback profiling is useful for getting programme teams   talking about feedback for both sho...
ReferencesAssessment Careers project Baseline Report see:   www.ioe.ac.uk/assessmentcareersCarless, D. Slater, D.Yang, M. ...
Lizzio, A. & Wilson, K. 2008. Feedback on assessment: student’s   perceptions of quality and effectiveness. Assessment & E...
Upcoming SlideShare
Loading in...5
×

Analysing Feedback

726

Published on

CDE seminar conducted by Dr Gwyneth Hughes, Senior Lecturer in HE, Institute of Education.

In this session Dr Gwyneth Hughes, a CDE Fellow, drew on her CDE research on ipsative assessment and a JISC funded project that she is leading at the IOE to explore why it is useful to analyse feedback for distance learners. It demonstrated a feedback analysis tool that has been developed as part of the project.

Gwyneth, a CDE Fellow, teaches on Higher Education programmes within the Lifelong and Comparative Education department including the MBA in Higher Education Management. She also supervises doctoral students. She has undertaken research and published on a range of topics including: ipsative assessment, formative feedback, identity, blended learning, e‐learning, gender inclusivity, widening participation, online collaborative work, web 2.0, learning technologies and reflective practice.

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
726
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • These are not easy to reconcile and the marking and grading tends to win out.
  • 1.AC analogy for modules and programmes jobs and careers.2.Term in literature for students bringing previous experiences of assesment. An assessment career is to develop ability to self assess and manage feedback –assessment literacy.- over timeWhen I taught in universities in the 1990s feedback was a way of showing off one's knowledge as superior to the students.Occurred behind closed doors only seen be assessor and student and perhaps an external examiner. With increasing concern about student complaints about assessment and feedback and takinig care not to harm students with too much critique there is a shift towards giving encouragement to students. But we don't really know what feedback practice is amongst staff and students. Need to open the 'black box' of feedback practice.
  • Feedback accroding to hattie and tijmperlay is any information on performance but it may be given and not received and acted upon. So a broader definition encompasses a dynamic viewie.e some impact on the system or it is not feedback.Feedback rarely scrutinised, unlike marks and grades-behind closed doors. When it is practice is inconsistent Lack of learner engagement with and understanding of feedback (Lizzio & Wilson, 2008)Inefficiency in terms of effort and impactTransmitted feedback creates dependency on teacher (Carless et al. 2011) e.g. Telling students what to do correcting them or telling them what do do. Better to develop student ability to think for themselves constructivist learning.
  • Students need to know what a good performance is but praise is not helpful to students unless accompanied by detail on why the praise is deserved. Encourages dependency on others (Hattie & Timperley, 2007).Praise-critique does not encourage dialogue and self-reliance (Nicol & Macfarlane-Dick, 2005)Lack of information about progress but this could be motivating (Hughes, 2011) Read in literature about closing performance gap –a deficit model but focus can be on progress towards goals.Critique and advice are for the current assignment. Encourages focus on short-term grade improvement rather than longer-term learning. Last 3 are made worse by modularisation
  • Looked at existing practice at the IOE –not surpsinly variable. Not much info on feedback so developed tool.Pilots aim to enocurage both assessors and student to take the longituinalprogarmme level approahc to feedback.
  • Ipsative assessment is assessment which compares current work to previous work so ipsative feedback is feedback that identifies progress or lack since previous work. It is self-referential rather than praise whcih refers to external standards. Our study showed that students often have little idea whether or not they are making progress apart from relying on grades. Those will lower grades may be progressing an d also grades may be too late to help studetns map their progress.Look at Hand out and go through key points. A3 is feed forward
  • IpsativePraise plus crit plus questionfeedforward
  • Discuss profile in pairs or your own samples
  • All students entitled to formal feedback on a draft assignment so formative feedback –also other iinformal formative. Summative assessment is grade plus written comments.
  • Formative assessment was not surprisingly dominated by advice for current assignment but summative was more unexpected.Praise/critique
  • One team discussed their results and now aim to provide more feed forward and possibly reduce the amount of praise.
  • Discuss in pairs. Example of peer feedback 4 questions in that paragraph. Different approach to feedback rather than telling what is wrong and what to do it is prompting the student to think and perhaps self-critique.
  • Think about the last piece of written feedback that you received on an academic course (not necessarily this course). Was your work formative (a draft you could revise) or summative (final assessed work that contributed a grade)?  It was formative / summative (please circle) What did you do with the feedback?The ones who didn’t respond just stated what was good or bad about feedback.Only 1/4 systematically took feedback forward.
  • Plus related programmes that share modules
  • Students both reflect on feedback and invite dialogue with tutors (or peers) on their actions. This is longitudinal feedback across modules. Tutor provide feedback on the feedback encourages dialogue, feed forward and feedback on progress. Make this easier by using Moodle–or any vle that has assignment submission so students fill this in directly online and a record is kept.Wanted to make previous feedback visible for students across a programme but not currently possible in our Moodle.We are developing VLE as a repository for past feedback and assignments so both students and staff can easily look at previous feedback not just past grades as in the exam board.
  • Some early indications from the project that the assessment careers approach is appreciated.
  • David Nicol has developed assessment principles at Strathclyde university as a discourse about assessment institution wide and this is more important than the actual principles.Premis is that feedback must be acted upon3.dialogue can be electronic and peers4. project has implications for prog design5. More formative e.g. Task s peer assessed nb peer feedback need guidanceDiscuss in pairs which you agree with or are they clear?
  • Modularisation tends to limit the potential for students to reflect on feedback and take suggestions forward.
  • Analysing Feedback

    1. 1. Analysing feedback:opening the black boxDr Gwyneth Hughes, Institute of Education, London
    2. 2. The purpose of assessment is to select learners for further study and/or careers.Agree/disagreeThe purpose of assessment is to facilitate learning.Agree/disagree2
    3. 3. OutlineWhy analyse feedback-some current issuesUsing a feedback analysis tool (Assessment Careers JISC funded project at the IOE)Using assignment cover sheets for dialogue on feedback3
    4. 4. Some current issues with feedback inHE• What is feedback?-received not just given out (Nicol & Macfarlane-Dick, 2005)• Feedback rarely scrutinised, when it is practice is inconsistent• Lack of distance learner engagement with and understanding of feedback (Lizzio & Wilson, 2008)• Inefficiency in terms of effort and impact• Transmitted feedback creates dependency on teacher (Carless et al. 2011)
    5. 5. • Praise is not helpful to students unless accompanied by detail on why the praise is deserved. Encourages dependency on praise (Hattie & Timperley, 2007).• Lack of information about progress but this could be motivating –especially for distance learners (Hughes, 2011)• Formative feedback often encourages a focus on short- term grade improvement rather than longer-term learning.5
    6. 6. Assessment Careers Project: taking action on feedback• Year 1: Baseline and feedback analysis tool• Year 2 (2012/13): Five pilot programmes including 2 distance programmes• Year 3 Institutional implementationAssessment Careers: Institute of Education website:www.ioe.ac.uk/assessmentcareers and video clip:http://youtu.be/VSaGbPoXPh0
    7. 7. Tool to identify feedback categories P1 Giving praise P2 Recognising progress (ipsative feedback) Critique • C1 Correction of errors • C2 Factual criticisms • C3 Criticism of approach Giving advice • A1 Specific to content current assignment • A2 General skills in current assignment • A 3 For future assignments Q Clarifications and questions O Other unclassified statements Adapted from Orsmond & Merry, 2011 including Hughes, 2011.
    8. 8. Analysing feedback categories • The score is the number of times a classification appears in the feedback • The default unit for analysis was the sentence • Where a sentence contains clauses that make distinct points, it was split into separate clauses, each of which was classified separately. • Neutral comments that for example describe the piece of work, but do not make any judgement are unclassified.
    9. 9. You have clearly developed your thinking on how you might usetechnologies in your teaching but as I mentioned in the previousfeedback, you have not reflected on your experience of usingtechnology to identify the potential benefits and challenges.You make your rationale for innovation clear, but less clear is thelearning theory that underpins your pedagogic design. You havenow explained X’s framework, but how does your design draw onthis?When you do mention Y’s model you do so uncritically, but you haveresponded to the feedback on your draft that you should discuss thelimitations of applying model Z to your design.Try to critique all models and frameworks in future.9
    10. 10. You have clearly developed your thinking on how you might usetechnologies in your teaching but as I mentioned in the previousfeedback, you have not reflected on your experience of usingtechnology to identify the potential benefits and challenges.You make your rationale for innovation clear, but less clear is thelearning theory that underpins your pedagogic design. You have nowexplained X’s framework, but how does your design draw on this?When you do mention Y’s model you do so uncritically, but you haveresponded to the feedback on your draft that you should discuss thelimitations of applying model Z to your design.Try to critique all models and frameworks in future.10
    11. 11. The structure of the essay was very strong, with a clear introduction which nicely contextualised the general problem of extraction industries and the interest in social environments vs the needs for quite „quick and dirty‟ (my phrase) data collection. The general aims of REA was well described, although I did have questions while reading this about how representative cultural samples could be drawn quickly, and what „representative‟ meant in this context. In terms of limitations I felt that the essay had maybe missed a trick – if a closer comparison with the aims and debates in conventional ethnography had been conducted, then this may have opened a space for thinking about the extent to which REA can achieve those aims or not. The issues raised in the conclusion about ethics and trustworthiness for instance could have been discussed in some detail ... Overall, the essay was very clear and well written, but I think that the absence of relating this very particular method to the broader issues in ethnography perhaps made it limited in the level of analysis that it provided of REAs. An extremely interesting and competent11 essay all the same though.
    12. 12. Data• Analysed formative and summative assessment feedback for modules on 5 postgraduate programmes at the IOE (total 228 pieces)• Recorded the total number of comments in each category and the average per script• Ranked the categories to obtain a feedback profile at programme level as well as an aggregate profile of the 5 programmes.
    13. 13. Baseline feedback profile Total comments for 5 programmes Summative assessment (N= 165) Category of feedback Average per script Rank P1 Praise 4.4 1 P2 Ipsative (progress) 0 (negligible) 5 C1-C3 Critique 2.7 2 A1-A3 Advice for current or 1.9 (mostly for current 3 future assignments assignment) Q Questions and clarification 0.1 4 requests13
    14. 14. Baseline feedback profile Total comments for 5 programmes (N= 165) Category of feedback Average per script Rank P1 Praise 4.4 1 P2 Ipsative (progress) 0 (negligible) 5 C1-C3 Critique 2.7 2 A1-A3 Advice for current or 1.9 (mostly for current 3 future assignments assignment) Q Questions and clarification 0.1 4 requests14
    15. 15. Overall you have written a really comprehensive and detailed review of the document. Your strength lies in your fantastic ability to contextualise the document with your political and pedigogical references. […] What do you understand by the term disruptive in this context? […] You seem to suggest ( or I am interpreting wrongly) that disruptive in this context is bad? […] Finally, you quote and cite almost in every sentence- but what is your personal view of the rise of K-12 blended learning‘ in primary and secondary education? Where is your personal voice? It scares me - or at least having a purely distance online learning component for such students worries me. What about you?15
    16. 16. What do students do with feedback? Formative n=50 Summative n=35didnt respond to the question 4 3no feedback received 0 1nothing 4 5read it 3 10revised assignment 32 3systematically took forward 7 1316
    17. 17. Overview of interventionsPilot 1 MA Education, Health Promotion and International DevelopmentPilot 2 MRes in Educational and Social ResearchPilot 3 MA Clinical EducationPilot 4 MA/MSc Psychology of EducationPilot 5 PGCE PrimaryApprox. 400 students and 30 staff in total.
    18. 18. Assignment cover sheets Thinking about the feedback on your draft of this essay, please indicate what the key points were and what action you took to respond to this feedback to help you prepare for and write this essay I would like tutor feedback on: 1. 2. 3.18
    19. 19. Is ipsative feedback appreciated bystudents?“I made an effort to act on it (formative feedback). This was acknowledged in the feedback on the final assignment.”From another student:“I like the fact that the feedback demonstrates that the tutor is aware not only of my current level of achievement, but my progress.”19
    20. 20. Principles for Action on Feedback1. Feedback is directed towards students concerns2. Students are encouraged to see the value of feedback for both short- term and longer-term learning3. Assessors initiate dialogue with students about feedback their work4. Curriculum designers and assessors make it clear to students how learning accumulates over a programme5. Programme teams are aware that feedback has different purposes and agree which categories of feedback are appropriate for their students6. Students have experience early and frequent formative assessment opportunities which are designed into their programme20
    21. 21. ConclusionFindings so far:• Feedback profiling is useful for getting programme teams talking about feedback for both short and longer term learning.• Assignment cover sheets are a useful way of engaging student in discussion of feedback.• Feedback principles are useful for opening up discussion of feedback practice at the institution level.• www.ioe.ac.uk/assessmentcareers
    22. 22. ReferencesAssessment Careers project Baseline Report see: www.ioe.ac.uk/assessmentcareersCarless, D. Slater, D.Yang, M. and Lam, J. 2011. Developing sustainable feedback practices. Studies in Higher Education 36, no.4: 395-407.Gibbs, G. & Simpson, C. (2004) Conditions Under Which Assessment Supports Students’ Learning. Learning and Teaching in Higher Education, 1: 3-31Hattie, J. & Timperley, H. 2007. The Power of Feedback. Review of Educational Research 77 no. 1: 81-112.Hughes, G. (2011) Aiming for Personal Best: a Case for Introducing Ipsative Assessment in Higher Education Studies in Higher Education22 36 (3): 353 – 367
    23. 23. Lizzio, A. & Wilson, K. 2008. Feedback on assessment: student’s perceptions of quality and effectiveness. Assessment & Evaluation in Higher Education 33 no.3:263-275.Nicol, D. & Macfarlane,-Dick, D. 2006. Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education 31 no. 2: 199-218.Orsmond, P. & Merry, S. 2011. Feedback alignment: effective and ineffective links between tutors’ and students’ understanding of coursework feedback. Assessment & Evaluation in Higher Education. 36(2): 125-126.23
    1. ¿Le ha llamado la atención una diapositiva en particular?

      Recortar diapositivas es una manera útil de recopilar información importante para consultarla más tarde.

    ×