The second keynote I delivered at #eas14 continued the theme of Brookfield's lenses of reflection, but this time focussed on Assessment Analytics. The two examples identified were from the LIFTUPP project at Liverpool's Dental School, as well as a pilot from Liverpool's Medical School in using data from Turnitin and Grademark to analyse staff and student performance in written assignments.
16. TURNITIN / GRADEMARK
Pilot in School of Medicine SSMs
• Tutor name included in submission title;
• SSMs marked against 7 objectives (intro, methodology, etc);
• Objectives marked as Unsatisfactory (1), Fair (2), Good (3), Excellent (4);
• Total possible marks was 7 * 4 = 28;
• 6 Markers & 36 Student assignments;
17. TURNITIN / GRADEMARK
120
100
80
60
40
20
0
Unsatisfactory Fair Good Excellent
Count
Count of marks awarded
18. TURNITIN / GRADEMARK
25
20
15
10
5
0
Frequency of Grades Awarded for Each Objective
Ob 1 Ob 2 Ob 3 Ob 4 Ob 5 Ob 6 Ob 7
Unsatisfactory
Fair
Good
Excellent
19. TURNITIN / GRADEMARK
140
120
100
80
60
40
20
0
Number of QM Comments Made by Objective
Objective 1 Objective 2 Objective 3 Objective 4 Objective 5 Objective 6 Objective 7
22. QUESTIONS
Is anybody doing anything with this information?
Should we?
Share with Staff? With Students?
Ethical considerations?
Hawthorne effect?
Predictive analytics?
Email Interviews - Do you think the data available from the
system is useful and why?
23. EMAIL INTERVIEWS
“Yes, no Hawthorne for me, I didn't know marks were being
assessed in that way. Helps reassure students and externals that we
are monitoring for consistency.”
“I think it very useful as it will make it easy to identify conveners who
mark outside normal parameters and identify who is giving feedback
to the students and who isn't.”
24. EMAIL INTERVIEWS
“I’m not quite sure what I think about this. Obviously we’re all
interested in what our own performance is (although I’m not sure we
should be told!). Major inconsistencies between markers or between
objectives would be worrying, but I don’t think we have a solid basis
for expecting any particular result – to put it crudely, would ‘all
excellent’ be a good thing (great students and course-management)
or bad (slack marking)?”