Fit-for-purpose assessment Sally Brown Provost and Pro-Vice-Chancellor, Assessment, Learning and Teaching Leeds Metropolitan University ATN Assessment Conference Adelaide November 2008

We work in a challenging context A rapidly-changing mass HE/FE environment with associated diversity issues; Challenges associated with Technology-Enhanced Learning; International challenges including UK and international competition (e.g. European nations teaching programmes in English), and worldwide recruitment.

A rapidly-changing mass HE/FE environment with associated diversity issues;

Challenges associated with Technology-Enhanced Learning;

International challenges including UK and international competition (e.g. European nations teaching programmes in English), and worldwide recruitment.

The role of HE in the coming decades Providing creative, powerful and positive ways of supporting student learning; Offering appropriate flexibility about mode, pace and content of programmes Providing a stronger focus on supporting information literacies; Incorporating coaching approaches.

Providing creative, powerful and positive ways of supporting student learning;

Offering appropriate flexibility about mode, pace and content of programmes

Providing a stronger focus on supporting information literacies;

Incorporating coaching approaches.

My prediction for HE The move away from HEIs and FEIs being the guardians of content, where everything is about delivery, towards them having two major functions: Recognising and accrediting achievement, where ever such learning has taken place (not necessarily in our college/university but from anywhere); Supporting student learning and engagement through a maieutic model.

The move away from HEIs and FEIs being the guardians of content, where everything is about delivery, towards them having two major functions:

Recognising and accrediting achievement, where ever such learning has taken place (not necessarily in our college/university but from anywhere);

Supporting student learning and engagement through a maieutic model.

The maieutic model Maieutics is a complex procedure of research introduced bySocrates, embracing the Socratic method in its widest sense. It is based on the idea that the truth is latent in the mind of every human being due to her/his innate reason but has to be "given birth" by answering questions (or problems) intelligently proposed.The word is derived from the Greek "μαιευτικός," pertaining to midwifery.

Maieutics is a complex procedure of research introduced bySocrates, embracing the Socratic method in its widest sense. It is based on the idea that the truth is latent in the mind of every human being due to her/his innate reason but has to be "given birth" by answering questions (or problems) intelligently proposed.The word is derived from the Greek "μαιευτικός," pertaining to midwifery.

Curriculum delivery, I argue… is less like a postman delivering a parcel and more like a midwife delivering a baby. Learning staff can advise, guide, help with advance preparation and intervene when things so wrong, but in the end only the student can bring learning into life!

is less like a postman delivering a parcel and more like a midwife delivering a baby.

Learning staff can advise, guide, help with advance preparation and intervene when things so wrong, but in the end only the student can bring learning into life!

Learning can change people’s lives by helping them to: Look over the horizon and discover opportunities they never new existed; Learn a great deal, often largely about themselves, to help use talents to the full; Become part of a lifelong learning community; Get qualified for graduate-level jobs.

Look over the horizon and discover opportunities they never new existed;

Learn a great deal, often largely about themselves, to help use talents to the full;

Become part of a lifelong learning community;

Get qualified for graduate-level jobs.

Problems can arise if students are allowed to believe that they: Don’t really belong in higher education; Can’t get the support they need to learn well; Are just a number; Are unique (and alone) in the problems they experience.

Don’t really belong in higher education;

Can’t get the support they need to learn well;

Are just a number;

Are unique (and alone) in the problems they experience.

Assessment shapes student behaviour by Giving them signals about what kinds of activity we really value; Influencing the extent to which they spend time on task productively; Providing structure for their learning activities; Helping them develop good learning patterns.

Giving them signals about what kinds of activity we really value;

Influencing the extent to which they spend time on task productively;

Providing structure for their learning activities;

Helping them develop good learning patterns.

How can we integrate assessment with learning? It needs to be built-in rather than bolt-on; Assignments need to be authentic, that is, assessing learning that is identified in the learning outcomes; Assessment should stimulate learning (not prevent it!); It should be fit for purpose.

It needs to be built-in rather than bolt-on;

Assignments need to be authentic, that is, assessing learning that is identified in the learning outcomes;

Assessment should stimulate learning (not prevent it!);

It should be fit for purpose.

A fit-for-purpose model of assessment: the key questions Why are we assessing? What is it we are actually assessing? How are we assessing? Who is best placed to assess? When should we assess?

Why are we assessing?

What is it we are actually assessing?

How are we assessing?

Who is best placed to assess?

When should we assess?

Why are we assessing? Choose reasons for assessment: Enabling students to get the measure of their achievement; Helping them consolidate their learning; Providing feedback so they can improve and remedy deficiencies;

Enabling students to get the measure of their achievement;

Helping them consolidate their learning;

Providing feedback so they can improve and remedy deficiencies;

More purposes... motivating students to engage in their learning; providing them with opportunities to relate theory and practice; Helping students make sensible choices about option alternatives and directions for further study;

motivating students to engage in their learning;

providing them with opportunities to relate theory and practice;

Helping students make sensible choices about option alternatives and directions for further study;

And more… demonstrating student employability; providing assurance of fitness to practice; giving feedback to teachers on effectiveness; providing statistics for internal and external agencies.

demonstrating student employability;

providing assurance of fitness to practice;

giving feedback to teachers on effectiveness;

providing statistics for internal and external agencies.

Choosing what we assess product or process? theory or practice? subject knowledge or application? what we’ve always assessed? what it’s easy to assess?

product or process?

theory or practice?

subject knowledge or application?

what we’ve always assessed?

what it’s easy to assess?

Approaches Self assessment peer assessment group-based assessment negotiated learning programmes computer-based assessment work-based assessment

Self assessment

peer assessment

group-based assessment

negotiated learning programmes

computer-based assessment

work-based assessment

Being imaginative by choosing diverse assessment methods? essays, unseen written exams, reports portfolios, projects, vivas, assessed seminars, poster presentations, annotated bibliographies, blogs, diaries, reflective journals, critical incident accounts, artefacts, productions, case studies, field studies, exhibitions, critiques, theses…….

essays, unseen written exams, reports

portfolios, projects, vivas, assessed seminars, poster presentations, annotated bibliographies, blogs, diaries, reflective journals, critical incident accounts, artefacts, productions, case studies, field studies, exhibitions, critiques, theses…….

Alternatives to traditional exams Open-book exams Take-away papers Case studies Simulations Objective Structured Clinical Examinations (OSCEs) Short answer questions In-tray exercises Live assignments Multiple choice Tests

Open-book exams Take-away papers

Case studies Simulations

Objective Structured Clinical Examinations (OSCEs)

Short answer questions

In-tray exercises Live assignments

Multiple choice Tests

Choosing diverse assessment methods is valuable because: all assessment methods disadvantage some students; we can access thereby a wider range of students abilities and skills; assessment can then become an integral purpose of learning; we can learn from the experiences of others.

all assessment methods disadvantage some students;

we can access thereby a wider range of students abilities and skills;

assessment can then become an integral purpose of learning;

we can learn from the experiences of others.

Choosing who is best placed to assess tutor assessment self-assessment peer assessment, (either inter or intra peer) employers, practice tutors and line managers client assessment

tutor assessment

self-assessment

peer assessment, (either inter or intra peer)

employers, practice tutors and line managers

client assessment

When should assessment take place? No sudden death. end point or incrementally? when students have finished learning or when there is still time for improvement? when it is convenient to our systems? when it is manageable for students? (avoiding week 7 blues).

No sudden death.

end point or incrementally?

when students have finished learning or when there is still time for improvement?

when it is convenient to our systems?

when it is manageable for students? (avoiding week 7 blues).

To integrate assessment we should realign it by: Exploring ways in which assessment can be made integral to learning; Constructively aligning assignments with learning outcomes; Providing realistic tasks: put more energy into authentic assignments.

Exploring ways in which assessment can be made integral to learning;

Constructively aligning assignments with learning outcomes;

Providing realistic tasks: put more energy into authentic assignments.

Feedback is a key means of supporting students’ learning Helping them know not just what’s wrong but how to make it better; Giving challenges to the most able students as well as those struggling; Providing feed-forward, guidance on how to prepare for future assignments.

Helping them know not just what’s wrong but how to make it better;

Giving challenges to the most able students as well as those struggling;

Providing feed-forward, guidance on how to prepare for future assignments.

But feedback can be problematic when it: makes students feel small, isolated, incapable; uses what Boud calls final language: ‘appalling’, ‘dreadful’, ‘hopeless’; doesn’t help students understand what to do next.

makes students feel small, isolated, incapable;

uses what Boud calls final language: ‘appalling’, ‘dreadful’, ‘hopeless’;

doesn’t help students understand what to do next.

So how can staff help learning happen through feedback? Offer innovative and flexible assessment opportunities; Make feedback really central to the assessment process; Ask students about their feedback needs and make use of the the answers; Focus productively on transitions.

Offer innovative and flexible assessment opportunities;

Make feedback really central to the assessment process;

Ask students about their feedback needs and make use of the the answers;

Focus productively on transitions.

Technology-Enhanced Learning Learning is the crucial bit; The technology is there to support and improve the learning through feedback; Enhancement means improving by doing things better and differently, not just replacing one means of giving feedback with another; TEL techniques can maximise the frequency and value of feedback, as well as efficiently helping students understand why things are right or wrong.

Learning is the crucial bit;

The technology is there to support and improve the learning through feedback;

Enhancement means improving by doing things better and differently, not just replacing one means of giving feedback with another;

TEL techniques can maximise the frequency and value of feedback, as well as efficiently helping students understand why things are right or wrong.

Personalised approaches can use feedback by: Enabling rapid and meaningful responses that focus on individual learning needs; Creating new coaching relationships between students and tutors through learning conversations; Offering greater opportunities for peer-to-peer coaching and feedback.

Enabling rapid and meaningful responses that focus on individual learning needs;

Creating new coaching relationships between students and tutors through learning conversations;

Offering greater opportunities for peer-to-peer coaching and feedback.

How can we make students take feedback more seriously? We can: Spend time and energy helping students to understand the importance of feedback and the value of spending some time after receiving work back to learn from the experience. Most students don’t do this at the moment, concentrating principally on the mark. Some withhold the mark until after the student has received and responded to feedback. Some provide assessed opportunities for reflection on previously marked work.

Spend time and energy helping students to understand the importance of feedback and the value of spending some time after receiving work back to learn from the experience. Most students don’t do this at the moment, concentrating principally on the mark.

Some withhold the mark until after the student has received and responded to feedback.

Some provide assessed opportunities for reflection on previously marked work.

We can make feedback timely Aim to get feedback on work back to students very quickly, while they still care and while there is till time for them to do something with it. The longer students have to wait to get work back, especially if they have moved into another semester by the time they receive their returned scripts, the less likely it is that they will do something constructive with lecturer’s hard-written comments.

Aim to get feedback on work back to students very quickly, while they still care and while there is till time for them to do something with it.

The longer students have to wait to get work back, especially if they have moved into another semester by the time they receive their returned scripts, the less likely it is that they will do something constructive with lecturer’s hard-written comments.

Concentrate formative feedback where it can do most good Don’t give detailed written feedback to students on final summative assignments where it is not likely to be useful: just give a mark or grade; Give more incremental feedback throughout the semester; Ensure that students at the top end of the ability range don’t feel short changed by minimal feedback.

Don’t give detailed written feedback to students on final summative assignments where it is not likely to be useful: just give a mark or grade;

Give more incremental feedback throughout the semester;

Ensure that students at the top end of the ability range don’t feel short changed by minimal feedback.

Use formative assessment to promote independence Investigate how learning can be advanced progressively (a ‘scaffolding’ approach); Provide support in the early stages when students don’t understand the ‘rules of the game’ and may lack confidence; Progressively removed support as students become more confident.

Investigate how learning can be advanced progressively (a ‘scaffolding’ approach);

Provide support in the early stages when students don’t understand the ‘rules of the game’ and may lack confidence;

Progressively removed support as students become more confident.

Modes of feedback: pros and cons Written feedback : time consuming, provides a permanent record, can be carefully worded, can be used in evidence against you, can be compared with other students’ comments and… Oral feedback : immediate, ephemeral, can be time-efficient, may be hasty or ill-considered, can be wrongly remembered, tone of voice and body language can add emphasis or soften the blow and…

Written feedback : time consuming, provides a permanent record, can be carefully worded, can be used in evidence against you, can be compared with other students’ comments and…

Oral feedback : immediate, ephemeral, can be time-efficient, may be hasty or ill-considered, can be wrongly remembered, tone of voice and body language can add emphasis or soften the blow and…

Other kinds of feedback Audio feedback : can feel artificial, equipment can let you down, provides a record, must be listened to in real time, time-efficient, can be shared by several students, can be quality audited and… Emailed feedback : practically instantaneous, students get feedback when they still care, needs marker to have a big computer desk, tone can be an issue and..

Audio feedback : can feel artificial, equipment can let you down, provides a record, must be listened to in real time, time-efficient, can be shared by several students, can be quality audited and…

Emailed feedback : practically instantaneous, students get feedback when they still care, needs marker to have a big computer desk, tone can be an issue and..

Engaging learners to use feedback thoughtfully requires: Understanding of the key purposes of feedback; A commitment to student-centredness; Apt use of Technology Enhanced Learning and Computer Assisted Assessment; A canny and highly systematic approach.

Understanding of the key purposes of feedback;

A commitment to student-centredness;

Apt use of Technology Enhanced Learning and Computer Assisted Assessment;

A canny and highly systematic approach.

As individuals we can… Avoid spending ages on writing idiosyncratic feedback to students who won’t read it; Be aware of the impact of our own fatigue and frustration on the type and level of feedback that late-marked students receive; Always ask ourselves “Is there an easier way I can be doing this?”

Avoid spending ages on writing idiosyncratic feedback to students who won’t read it;

Be aware of the impact of our own fatigue and frustration on the type and level of feedback that late-marked students receive;

Always ask ourselves “Is there an easier way I can be doing this?”

And also: Recognise that just because it’s always been done in a certain way doesn’t mean it always has to be. Consider the feedback needs of the recipient at the other end of the process.

Recognise that just because it’s always been done in a certain way doesn’t mean it always has to be.

Consider the feedback needs of the recipient at the other end of the process.

As course teams we can… Scrutinise the assessment approaches of modules/ programmes to ensure that they do what we need them to do; Take note of comments provided by course reps and student evaluations that tell us how well we are doing at assessment and feedback; Ensure that assessment is fit-for-purpose.

Scrutinise the assessment approaches of modules/ programmes to ensure that they do what we need them to do;

Take note of comments provided by course reps and student evaluations that tell us how well we are doing at assessment and feedback;

Ensure that assessment is fit-for-purpose.

As institutions we can… Ensure that our assessment regulations don’t result in perverse and inappropriate outcomes. Provide opportunities for staff to share better ways of assessing students with one another. Develop systems regularly to update assessment (not just same old same old!).

Ensure that our assessment regulations don’t result in perverse and inappropriate outcomes.

Provide opportunities for staff to share better ways of assessing students with one another.

Develop systems regularly to update assessment (not just same old same old!).

Conclusions Assessment matters: more than many have recognised previously; Marking and grading have become a major source of dissatisfaction for many teaching staff: we must remediate this; Good assessment practices can make the difference between success and failure, particularly for students from disadvantaged backgrounds.

Assessment matters: more than many have recognised previously;

Marking and grading have become a major source of dissatisfaction for many teaching staff: we must remediate this;

Good assessment practices can make the difference between success and failure, particularly for students from disadvantaged backgrounds.