Assessment in challenging times Mantz Yorke email@example.com LJMU 7 November 2012
ContextIn the new fee andfunding regime ...1. Increased pressure on staff2. Raised student expectationsKey questionWhat’s ‘the deal’(going to be)?
A ‘sustainability’ consideration for staffAssessors have to be able to sustain the expectations laidupon them.What volume of good assessment practice can they realisticallyundertake as the economic screw tightens?(And does ‘the management’ know?)Some serious curriculum design and quality assurance issues:they go beyond the individual teacher/assessor
Agenda (rather loosely organised)1. Curricula for autonomy2. Formative assessment, feedback and feedforward3. Summative assessment (especially grading)
Becoming a graduateOught to involve young people (and perhaps some older people)in a significant transformationAcquiescence AutonomyKohlberg 1964Perry 1970 (reprinted 1998)King and Kitchener 1994Kuhn and Weinstock 2002Baxter Magolda 2009Guiding learners through the transformation from authoritydependence to self-authorship is a primary challenge fortwenty-first century higher educationBaxter Magolda (2009, p.144)
Students often take time to ‘get it’[School study] habits can continue well into the first year ofuniversity.(Leckey and Cook, 1999)[N]ew undergraduates often see the tutor as the expertwho can (and perhaps should) give them the information.By contrast university history teachers emphasise the needfor student autonomy and independent judgement.(Booth, 2005)There may not be a ‘truth’ that can be handed downMany Art & Design students expect to create their own ‘truths’
Students often take time to ‘get it’Despite help, students did not grasp what was requiredof them as regards essay-writing.(McCune, 2004)Students need help in order to develop as autonomouslearners.(Fazey and Fazey, 2001)At the beginning you have no idea what constitutes a passas you have no frame of reference. Need feedback on earlierwork before progressing to next assessment.(Student, in Johnston and Kochanowska, 2009)But what if the next assessment task is very different?
A pedagogy for autonomyStudents should be encouraged to develop their capacityto self-assess over a lifetime (Boud, 2000).Royce Sadler is insistent that higher education should developin students an appreciation of standards and how their workmeasures up (e.g. Sadler, 2009).A ‘patient curriculum’? In which ...Assessment and feedback are of critical importance
What surveys sayFeedback has relatively low ratings from ‘graduates’ (NSS; CEQ)‘Your First College Year’ survey in the US doesn’t ask about itIn first-year experience surveys that did: Australia Australia UK FYE 2005 FYE 2010 FYE 2007 33% 33% 57% found found found feedback feedback feedback helpful helpful helpful
Good assessment and feedback should…1. Clarify goals, criteria, standards Sadler:2. Encourage time and effort on challenging tasks Students should3. Give good feedback that helps self-correction internalise4. Provide opportunities to act on feedback standards5. Ensure summative assessment assists learning6. Encourage dialogue about learning7. Facilitate self-assessment and reflection8. Encourage motivation and self-esteem9. Give students choice regarding assessment10. Involve students in policy and practice re assessment11. Support development of learning groupings12. Inform teachers about their teachingAdapted from Nicol, 2009
So what about students’ expectations and experience?
Perceptions of feedback Informing students about feedback is important since what staff consider to be feedback is not always appreciated as such by studentsStudents often or very often received it 38.6 Staff often or very often gave it 80.4 0 20 40 60 80 100 AUSSE 2008
Match of expectations and experience: assessment methods (A&D) Have the methods by which your work has been assessed Per been as you had expected? N cent No 55 8.1 Partly 226 33.1 Yes 402 58.9 Total respondents 683 100.0
Match of expectations and experience: assessment methods (A&D)70%60%50%40% Low prior inf30% High prior inf20%10%0% No Partly Yes
Expectations regarding feedbackIssue (abbreviated) 2006 2007 2010 N=979 N=1774 N=3091 % Agree % Agree % AgreeReady access to staff outside f2f important 87 87 87I expect teachers to read drafts 52Feedback on drafts important to learning 92 95Acceptable time-interval for return of work 1 week 2-3 weeks 4-6 weeks 2006 % 34 57 4 2007 % 38 55 3 2010 % 19 60Crisp et al (2009); Scutter et al (2011)
Expectations re feedback ... and experienceItem Item theme At End year 1 Year 2 Teachersno. (varies with orientation group studied) % Agree % Agree % Agree % Agree 7 ‘Ready’ access to lecturers, tutors 88 89 61 50 73 57 64 94 8 Feedback on submitted work 97 97 66 37 74 48 100 59 9 Feedback on DRAFTS of work 94 91 20 7 21 26 0 22Colour code: Humanities ScienceNote: phrasing adjusted to fit circumstances Brinkworth et al (2009)
Issues relating to the effectiveness of feedbackInadequate understanding of the task (Glover & Brown 2006)Student-staff differences in perception (Maclellan 2001)Feedback not understood (Chanock 2000; Weaver 2006)Tutors’ intentions re feedback not understood (Higgins et al 2002)‘Being told’ is not engaging (Crisp 2007)Disjuncts in the feedback loop (Hounsell et al 2006)Transferability not perceived (Carless 2006; Duncan 2007) ...... especially where assessment demands are diverse (Gibbs, TESTA)Timeliness (Brinkworth et al 2009)The grade’s the thing (Snyder 1971)Espoused theory & practice out of sync (Orrell 2006; Orsmond et al 2011)Where might enhancement efforts most usefully be targeted?
Worthwhile feedback?The feedback on my assignments comes back so slowly that weare already on the topic after next and I’ve already submittedthe next assignment. It’s water under the bridge, really.I just look at the mark and bin it.Collected by Graham GibbsRe a diagnostic essay:It took ages … about two months … by that time I guess youhad forgotten … we had already wrote (sic) another essay.Murphy and Cornell (2010, p.46)I found that I did not learn anything from my mistakes as I wasnever told what they were.Misko and Priest (2009, p.15)
What some students saidI don’t like these shorthand comments, and we get too many ofthem— ‘good analysis’—what does that mean? Where? Andwords like ‘weak’, ‘good’ and ‘strong’—I mean, is the whole thingweak … and blank, pointless comments like ‘wrong’ or ‘irrelevant’or random question marks, what use is that to anybody?Brown (2007, p.40)Terms like ‘check spellings’ and ‘more depth’ are unhelpful,just as are vague phrases like ‘this is unclear’.Orsmond et al (2005, p.377)... I got told that a piece of work was more like an essay than aliterature review. This is not helpful as it does not tell me whatshould be contained in a literature review or how it should bepresented.Weaver (2006, p.388)
When writing is not enoughWhen it (feedback) was written I didn’t understand it, butobviously you make the appointment to follow through onthat and she clarifies what she means …I didn’t know where I’d gone wrong, what she meant, andshe explained it, broke it down …Murphy and Cornell (2010, p.47)
Some comments from a survey of first-year students in Art & DesignYorke M and Vaughan D (2012) Deal or no deal? Expectations and experiencesof first-year students in Art and Design. Atwww.heacademy.ac.uk/resources/detail/disciplines/art-and-design/deal_or_no_deal_report
CriteriaThe assessment criteria are thoroughly explained at numerouspoints throughout the module so you know exactly what theyare looking for.We get a copy of the assessment criteria with our brief so wealways have it to refer to so we understand what exactly isexpected.To be honest, Im not entirely clear how my work is assessed.Its never been explained in simple terms, instead, paragraphsof descriptions, Im sure it must be simpler!
TimingThe written feedback was most helpful but was given too lateat the end of the projects meaning there was no opportunityto improve.My work is marked at several intervals throughout the year,rather than everything being marked at the end of the year,so I have several deadlines instead of one big final deadline.This reduces stress and helps me to keep on top of my work.Interim verbal and written feedback half way through anassignment has been the most useful to me as it has allowedme time to take on board what has been said and use (or not)the comments made.
OutcomesYou get given a piece of paper at the end where they haveticked some boxes and given you a percentage. That is notan assessment, it’s disgusting for the amount of work you do.Being given areas for improvement or development is themost useful [aspect of assessment] and that the fact that theformal aspect of this is written and you can reflect upon whatis written is very useful.
MotivationPedagogical approaches at tertiary level must motivatestudents to learn if deep, effective and engaged learningis to take place.(Kift and Field, 2010, p.4, original emphasis)There’s also ‘belongingness’, identified as important inPhase 1 of the ‘What works?’ programme.See: www.heacademy.ac.uk/assets/documents/retention/What_works_final_report.pdfWith striking consistency, studies show that innovative,active, collaborative, and constructivist instructionalapproaches shape learning more powerfully, in someforms by substantial margins, than do conventionallecture-discussion and text-based approaches.(Pascarella and Terenzini, 2005, p.646)
Motivation and engagementMotivation Engagement level level High HighModerate Teaching approach Low Low
Encouraging motivation Students observed that feedback was given in such a way that they did not feel it was rejecting or discouraging . . . [and] that feedback procedures assisted them in forming accurate perceptions of their abilities and establishing internal standards with which to evaluate their own workMentkowski and Associates (2000, p.82), emphasis addedBoud and Sadler (who argue for students’ internalisation ofstandards) would approve
AnxietyI was really, really scared because it was my first report.I had to hand it in just to see where I’m at. Where am Istanding?(McGinty, 2011)Psychological painThey are writing all over my work and it is like mangled upand most of the lecturers use red pen and I don’t know itkind of gets to me if I open it up and it’s covered with redcrosses and marks and it’s horrible. It’s like my work isbleeding.(McGinty, 2011)
Acknowledging emotionFor [first-year] students feedback goes beyond providinginformation on how to improve assessment marks. The‘effective feedback’ for these students is that whichprovides emotional support and facilitates integration intouniversity.(Poulos and Mahony, 2008)To what extent can ‘the personal’ be accommodated incontemporary HE?
Message not quite clear?The numbers and language used in assessment are opaque and need to be demystified
StandardsNormative (even when expressed in criterion-referenced terms).Discipline-driven, sometimes with external input.Difficult to define (like QAA Subject Benchmarks?).But ‘We know them when we see them’ – or we ought to).Or assessors may think they know standards, but notions such as• ‘graduateness’• ‘employability’• ‘wicked competences’• ‘soft skills’stretch assessors’ confidence to (beyond?) the limit.Learning outcomes are a kind of proxy, but are inherently fuzzy.Assessment criteria are likewise fuzzy.
The problem with criteria: ‘Conceptualisation’A/1st: Able to recognise consistency and reconcile inconsistencybetween information using cognitive and hypothesising skills.B+/2.1: Consistent understanding demonstrated in a logical andlucid manner.B/2.2: Demonstrate understanding in a style which is mostly logical,consistent and flowing.C/3rd: Attempts to demonstrate a logical and coherent understandingof the subject area but aspects become confused or are underdeveloped.Refer/Fail: Understanding of the assignment not apparent, or lacks alogical and coherent framework, or the subject is confused orunderdeveloped.Price & Rust (1999, p.134)How do you convey the meaning of all this to students?
Dealing with multiple learning outcomes: a hypothetical example Learning outcome Fully achieved Partly achieved Not achieved A B C ...Intended learning outcomes [ILOs] might be categorised asprimary or secondary, thereby influencing the overall assessment.Developed from Sadler (2005)Do students appreciate the relative importance of ILOs?
The potential for confusion: some examplesFormal statements of expectations are fairly meaninglesswithout exemplification, even for staff (e.g. Wolf, 1995).Students’ perceptions of what assessment is seeking may notalign with what staff think they ought to be perceiving(Maclellan, 2001).35 of 76 history and politics students did not understand themeaning of ‘more analysis, less description’ (Chanock, 2000).Staff themselves did not agree on what was meant by ‘analysis’and ‘evaluation’ (Webster et al, 2000).Whilst published assessment criteria are used, not all of themmay be applied – and some extraneous criteria may be invoked(Webster et al, 2000, regarding u/g dissertations).
Students need to understand the scoring systemJune 2010
How well do grades inform?The [honours degree] classifications are too general,there’s no way of showing if you were close to thegrade above.A straight percentage mark would be fairer.‘Maria’The Guardian RISE supplement, 18 December 2004Not so. For a start, there’s variation between subject areas
L1 Module means from 13 Owning Organisational Units80.070.060.050.040.030.0 CMP 5 LSS 31 ENR 8 PBS 19 NSP 29 ECL 75 BUE 16 LSA 26 SPS 11 HSS 37 HEA 16 LBS 34 LAW 10
So why the variation?Lots of variables may have exerted influence, including:• Student calibre (entry qualifications; commitment)• Nature of the subject (hard/soft; pure/applied)• Curriculum design• Pedagogic quality• Resourcing• Expected standards (intended learning outcomes)• Mode of assessment• Nature of the assessment demand• Marker variability (in some cases, due to differences in School)
What is the relationship between mode of assessment and marks awarded?
LJMU L1 module mean scores (N=317), by amount of coursework Mean CW = <35% CW = 35-75% CW = >75% Civil engineering surveying 1, CW=30% 65+ 60-64.99 55-59.99 50-54.99 <50 BUE HSS LBS LSA LSS NSP
LJMU uses ‘percentages’. If you are marking work out of 100,how do you arrive at the ‘percentage’ mark?(If marking work out of a smaller total, the same general question applies.)• You might tot up the marks from components of the student’s performance.• You might take a broad view of the overall performance (e.g. ‘this is of 2.1 standard’), and then decide where in the relevant band the actual mark should lie.• You may have some other approach.What approach do you use?And what assumptions underlie your markingmethodology?‘Fudges’? (e.g. Baume et al 2004; Bloxham et al 2011)
A question to ponder ...Should you finely grade work-based achievements,employability and ‘graduateness’?If ‘yes’, then how?Too complex an issue for this session, but a starting point isYorke M (2011) Assessing the complexity of professional achievementhttp://learningtobeprofessional.pbworks.com/f/CHAPTER+A10+FINAL.pdf
What assumptions are being made?(a)By staff(b)By students(c)And by others 40 35 30 25 The psychometric tradition would give 20 15 10 5 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 35 30 25 20 Using learning outcomes would give 15 10 5 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 But it’s rarely as cut and dried as that
Grades are not transparentThey can demoralise if they are awarded on a basis thatdiffers (negatively) from students’ previous experience.Grading at A-level, or in other countries’ systems, may notalign with UK HE practices.What does a mark of 64% mean?And what does the ‘missing’ 36% relate to?How different is 53% from 54%?And so on ...
Some other issuesGrade only on assignment and/or feedback boxes ticked (or not)Mark out of sync with commentFeedback comment on matters outside stated criteriaStudents not taught how to use feedbackThe difference between being positive and being constructiveThe need to provide emotional support (esp for 1st year students)The Higher Education Achievement Report [HEAR]
The Higher Education Achievement Report [HEAR] Official crest/seal/logo; introductory text SECTION 1 Confirms personal information about the student SECTION 2 Confirms the name of the qualification gained and the title conferred SECTION 3 Indicates the level of the qualification gained and its place in the relevant national qualifications framework SECTION 4 Details contents and results gained SECTION 5 Details the function of the qualification SECTION 6 Additional awards and recognised activities; prizes SECTION 7 Formal authentication of the HEAR SECTION 8 Weblink to a description of the national HE systemSeehttp://www.universitiesuk.ac.uk/Publications/Documents/BringingItAllTogetherIntroducingTheHEAR.pdf
The HEARSECTION 4Details contents and results gained4.1 the mode of study (e.g. ‘full time’)4.2 programme requirements4.3 details of all the components that have contributed to the final summary mark or grade4.4 details of the grading scheme used4.5 the overall classification of the final qualification.
In a nutshell ...1. Don’t assume that students will understand ‘assessment language’. Give explanations and, particularly, examples.2. Provide opportunities for students to develop their capacity to self-assess, without undue risk.3. Find ways to make feedback (and feedforward) effective.4. Clarify what grades signify, in the subject context.1 and 4 are the responsibility of teachers, and are fairlystraightforward.2 and 3 are much more challenging, in that they may requirea measure of innovation – creativity even – on the part ofteachers and certainly the active engagement of the students.
Some referencesAssessment and Evaluation in Higher Education is a particularly useful source of relevant articles, e.g.• Maclellan, 2001• Orsmond et al, 2005• Weaver, 2006• Webster et al, 2000Bloxham S and Boyd P (2007) Developing effective assessment in higher education: a practical guide.Maidenhead: Open University Press.Boud D & Falchikov N, eds (2007) Rethinking assessment in higher education: learning for the longer term.London: Routledge.Nicol D (2009) Transforming assessment and feedback: enhancing integration and empowermentin the first year. http://www.enhancementthemes.ac.uk/documents/firstyear/First_Year_Transforming_Assess.pdfYorke M (2008) Assessing student achievement in higher education: signals and shortcomings.Abingdon: Routledge (Contains a wide range of references.)Yorke M (2011) Assessing the complexity of professional achievementhttp://learningtobeprofessional.pbworks.com/f/CHAPTER+A10+FINAL.pdfYorke M (2011) Assessment and feedback in the first year: the professional and the personal.Available on request – web-based publication link is currently not working.