1. The document discusses analyzing feedback provided to students to identify patterns and areas for improvement. Two tools were presented for categorizing feedback from the Institute of Education and University of Dundee. 2. An analysis of 171 assignments from 4 postgraduate programs at the IOE found that praise was the most common category of feedback. 3. A feedback audit of 140 assignments at Dundee found most feedback focused on content and tasks, and that positive and negative feedback were equal. This confirmed variability in feedback between instructors.

1. Rola Ajjawi & Karen Barton
Centre for Medical Education
Gwyneth Hughes, Institute of Education, London.

2. If you wanted to make sure your students did NOT receive and act on your feedback what would you do?

3. Overview of session
•JISC project introductions
•Why analyse feedback?
•2 Tools for analysing feedback from the IOE and Dundee
•Applying each tool at the IOE and Dundee
•Resulting feedback profiles and audits
•Benefits of feedback profiling and some challenges

4. Project Introductions
•Assessment Careers-Institute of Education http://www.ioe.ac.uk/assessmentcareers
•http://youtu.be/VSaGbPoXPh0
•Interactive Assessment and Collaboration via Technology (InterACT) http://blog.dundee.ac.uk/interact/
•http://www.youtube.com/watch?v=S5bBFEbXDD0

5. Why analyse feedback?
Which is the main reason why you might analyse feedback?
A Widely inconsistent practice in feedback
B Lack of learner engagement with and understanding of feedback
C High teacher effort - low efficiency
D Transmitted feedback creates dependency on teacher
E NSS scores Result of the poll:

6. Feedback categories
•Giving praise (P1)
•Recognising progress or ipsative feedback (P2)
•Criticisms
•Correction of errors (C1)
•Factual criticisms (C2)
•Criticism of approach (C3)
•Giving advice
•Specific (current assignment)(A1)
•General (current assignment) (A2)
•For future assignments (A3)
•Clarifications (Q)
•Unclassified statements (O)
http://assessmentcareers.jiscinvolve.org/wp/about/
1. Focus of feedback?
•Content
•Style, format and language
•Sources and references 2. Level of Feedback (Hattie & Timperley 2007)
•Task
•Process
•Self-regulation
•Self 3. Depth of feedback (Brown & Glover 2006)
•Identified
•Categorised/Described
•Corrected/Exemplified
•Explained
•Future oriented http://blog.dundee.ac.uk/interact/
IOE
Dundee

7. •Which categories are most helpful for student learning?
•Giving praise (P1)
•Recognising progress or ipsative feedback (P2)
•Criticisms
•Correction of errors (C1)
•Factual criticisms (C2)
•Criticism of approach (C3)
•Giving advice
•Specific (A1)
•General (A2)
•For future assignments (A3)
•Clarifications (Q)
•Unclassified statements (O)
http://assessmentcareers.jiscinvolve.org/wp/about/
IOE
Dundee
1. Focus of feedback?
•Content
•Style, format and language
•Sources and references 2. Level of Feedback (Hattie & Timperley 2007)
•Task
•Process
•Self-regulation
•Self 3. Depth of feedback (Brown & Glover 2006)
•Identified
•Categorised/Described
•Corrected/Exemplified
•Explained
•Future oriented http://blog.dundee.ac.uk/interact/

8. •Giving praise (P1)
•Recognising progress or ipsative feedback (P2)
•Criticisms
•Correction of errors (C1)
•Factual criticisms (C2)
•Criticism of approach (C3)
•Giving advice
•Specific (A1)
•General (A2)
•For future assignments (A3)
•Clarifications (Q)
•Unclassified statements (O)
http://assessmentcareers.jiscinvolve.org/wp/about/
IOE
Dundee
•Does the context matter?
1. Focus of feedback?
•Content
•Style, format and language
•Sources and references 2. Level of Feedback (Hattie & Timperley 2007)
•Task
•Process
•Self-regulation
•Self 3. Depth of feedback (Brown & Glover 2006)
•Identified
•Categorised/Described
•Corrected/Exemplified
•Explained
•Future oriented http://blog.dundee.ac.uk/interact/

9. Coding framework
•The score is the number of times a classification appears in the feedback
•The default unit for analysis was the sentence
•Where a sentence contains clauses that make distinct points, it was split into separate clauses, each of which was classified separately.
• Neutral comments that for example describe the piece of work, but do not make any judgement are unclassified
IOE & Dundee

10. Dundee: Coding Examples
-ve
+ve
Identified
Poor
Good
Categorised / Described
In which country?
This is a good way of structuring a session
Corrected / Exemplified
This is not accurate – you may choose to read this article rather than rely on personal experience alone to support your teaching …
A safe, supportive and challenging environment is important to get open and active dialogue and discussion
Explained
One of the problems with this type of format is that the respondent can easily look up the correct answers
The pilot will allow you to see if the items are understood and the data is what you are looking for
Future-orientated
Do you encourage self and peer assessment? What would be the advantages and potential dangers of peer assessment in inter- professional learning?
Agree – how easy is this in a self- orientated patient-centred health provider?

11. Samples tested
•Formative and summative feedback for modules on 4 postgraduate programmes in Education and Psychology (171 samples)
•Recorded the total number in each category and the average per script
•Ranked the categories to obtain a feedback profile at programme level as well as an aggregate profile of the 4 programmes.
Dundee
IOE
•Access database built to provide a simple interface for data entry which can be easily exported for further data manipulation
•Random 5% sample of all summative assignments marked in 2011 (by assignment and tutor) was selected (140 samples)
•Assignments retrieved from Blackboard
•Recorded the total number in each category and average per tutor

12. Feedback analysis All assessments (n=171)
No of occurrences
Per script
Rank
P1
Praise
725
4.2
1
P2
Progress
4
0.0
C1
Criticism
382
2.2
3
C2
263
1.5
4
C3
215
1.3
5
A1
Advice
474
2.8
2
A2
182
1.1
7
A3
28
0.2
Q
Clarification
215
1.3
5
O
66
0.4
Results from the IOE
Results from the poll:

13. Vote on whether or not this is expected
No of occurrences
Per script
Rank
P1
Praise
725
4.2
1
P2
Progress
4
0.0
C1
Criticism
382
2.2
3
C2
263
1.5
4
C3
215
1.3
5
A1
Advice
474
2.8
2
A2
182
1.1
7
A3
28
0.2
Q
Clarification
215
1.3
5
O
66
0.4
Results from the IOE

14. Feedback analysis All assessments (n=171)
No of occurrences
Per script
Rank
P1
Praise
725
4.2
1
P2
Progress
4
0.0
C1
Criticism
382
2.2
3
C2
263
1.5
4
C3
215
1.3
5
A1
Advice
474
2.8
2
A2
182
1.1
7
A3
28
0.2
Q
Clarification
215
1.3
5
O
66
0.4

15. Feedback analysis All assessments (n=171)
No of occurrences
Per script
Rank
P1
Praise
725
4.2
1
P2
Progress
4
0.0
C1
Criticism
382
2.2
3
C2
263
1.5
4
C3
215
1.3
5
A1
Advice
474
2.8
2
A2
182
1.1
7
A3
For future assignments
28
0.2
Q
Clarification
215
1.3
5
O
66
0.4

16. Vote: Is this the profile you would want?
No of occurrences
Per script
Rank
P1
Praise
725
4.2
1
P2
Progress
4
0.0
C1
Criticism
382
2.2
3
C2
263
1.5
4
C3
215
1.3
5
A1
Advice
474
2.8
2
A2
182
1.1
7
A3
For future assignments
28
0.2
Q
Clarification
215
1.3
5
O
66
0.4

17. Dundee: Feedback audit results
•140 assignments assessed
–Main focus of feedback was content related (95%)
–Main level of feedback was on the task (72%)
–Positive and negative feedback was equal (51% vs. 49%)
•Confirmed the wide variety in quantity and nature of feedback provided by tutors

18. Focus of feedback

19. Level of feedback

22. •Is this the profile you would want?
•As programme leader or director would you expect this diversity?

23. Benefits of doing the audit
•Raises awareness of quantity and quality of feedback provided individually and across the centre
•Opportunity for faculty development
•Objective measure of changes in feedback culture

24. Enablers and Challenges
•Success factors
–Workshop
–Easy to use
•Challenges
–Categories overlap
–Shift from grading to feedback
Success factors
•Discussion of codes
•Double coding and negotiation of codes – establish inter-rater reliability
•Having examples of codes Challenges
•Difficulty in interpretation of codes and feedback
•Workload
What are other factors in other institutions?
How could feedback profiling be implemented without funding?
IOE
Dundee

25. Why analyse feedback?
Which is the main reason why you might analyse feedback?
A Widely inconsistent practice in feedback
B Lack of learner engagement with and understanding of feedback
C High teacher effort - low efficiency
D Transmitted feedback creates dependency on teacher
E NSS scores

26. Any other questions?

27. References
1.Brown, E. & Glover, C. (2006) Evaluating written feedback. in: B. C. & K. Klegg (Eds) Innovative assessment in higher education. London, Routledge), 81-91.
2.Hattie, J. & Timperley, H. (2007) The Power of Feedback. Review of Educational Research, 77, 81-112.
3.Hughes, G. (2011) Aiming for Personal Best: a Case for Introducing Ipsative Assessment in Higher Education Studies in Higher Education 36 (3): 353 – 367.
4.Orsmond, P. & Merry, S. (2011) Feedback alignment: effective and ineffective links between tutors’ and students’ understanding of coursework feedback. Assessment & Evaluation in Higher Education. 36(2): 125-126.