• The whole series of 5 modules is called: “Assessment Professional Learning”. • This fifth (and final) Module is called: “Making Consistent Judgements”.
• The DE&T Assessment Advice centres on these three purposes of assessment. • The focus of this Module is making consistent judgements for all three purposes: assessment OF learning, assessment FOR learning and assessment AS learning.
• With the new Standards we are expected to be more consistent and valid in the judgements we make. • We therefore need to build some strategies and protocols to enable this to happen.
• “ Quality assurance” - means that students, parents and our teaching colleagues can be confident that the assessment decisions we make are trustworthy. • They must be both valid and consistent to be trustworthy.
• “ Valid” has several everyday meanings - and a technical one when talking about assessment.
Validity has several elements to it - including an assumption of fairness, accuracy and appropriateness. • We must try to be as fair, and unbiased, as possible in our judgements on students progression against the Standards. • The assessment ought to represent the whole of what we are trying to teach and assess - and not be skewed to a small section or standard (even if important). • The assessment will be valid if it requires the depth of learning we want - and students cannot skim lightly across the top.
• Our judgements need to accurately reflect the real performance and understandings of our students. This can sometimes be interfered with by an intervening skill that we have not realised is a pre-requisite. Such examples occur when students cannot read the instructions, or cannot google to do internet research, etc. We need to ensure that all students have all necessary pre-requisite skills - and not assume that because some have it, all do.
• This list can be used to check our assessment. • Whether the purpose is assessment FOR learning, assessment OF learning, or assessment AS learning - all judgements must be valid.
• Consistency is the everyday term for reliability (the technical term used in assessment literature).
• Reliability has many everyday meanings as well - some of which relate to the technical term. • Some of these meanings of reliability are about the concept of “validity”. For example, a clock that gives us the correct time is an accurate clock (i.e. a valid clock) and a set of scales that gives us the correct weight is also accurate (i.e. valid). But a clock that is consistently 5 minutes fast - is reliably wrong - but consistent.
• Put simply - consistency means that the results are the same no matter who fires the arrows, when, where, or what the weather is like.
• Two important aspects of reliability/consistency is being consistent with other teachers (inter-rater reliability) and being consistent with ourselves (intra-rater reliability. • For example, if I assess some students’ work late at night when I’m tired, I may not judge it the same as when I am fresh on a Saturday morning. • Or if I assess Student A’s work in the 1st three I “mark”, then get more generous as I proceed through the pile, I may not be consistent in my approach to all students’ work, and be more generous by the time I get to Student B’s work near the end of the pile. Some teachers always go back and re-mark the first few, to ensure they haven’t changed their interpretations on the way through the assessment process.
• The assessor need not be the teacher - it could be the student, a peer, some other external body (e.g. VCAA).
• Consistency/reliability is a technical matter - do we all give the same result for the same standard of work? • Validity has a technical dimension (accuracy) but also calls for judgement (is it fair and appropriate for these students)?
• We can all hit the same spot in our assessment of student work - but it is meaningless if what we are agreeing on is not the Standard we are trying to monitor. • Or: consistency is of no value if we are all consistently bad at assessing students progress against the Standards! • Being in solid agreement, but being inaccurate and unfair (invalid) is not a good thing in assessment!
• There are many traps in the assessment process. • We can build in collaborative planning so we know what the standards are about, we can build in collaborative design of the tasks and rubrics so we have a shared understanding of what the students are doing … we can be transparent with students about what they have to do and how we will judge their work … it all helps make a subjective process more objective.
• During this module you will develop some skills in these areas and be introduced to some practical strategies that will help you make more valid and consistent judgements.
Moderation is a process where by teachers discuss and agree on a final judgement of a piece of student work, it is a quality assurance for assessment. There are several stages in the moderation process. It does not have to be an ‘end’ judgement, but can can begin with the task The moderation process can be a form of professional learning for teachers. Think about current moderation processes in your school. With a partner brainstorm and discuss whether all the critical stages of moderation are present in your practice?
By the end of the moderation process assessors should feel assured that there assessment task is: Valid- it is reliable Consistent- the assessment of the student work assessment is consistent with other student work and with other assessors Complete activity 5-4A includes a reading providing and overview of Moderation Processes.
• Above all else we cannot let this take over our lives - we can work collaboratively to share the burden and build better consistency. • Our focus must be on using strategies that are efficient. • Do people have ideas of other ways we need to be efficient in our assessment processes?
• The students are at the centre. Their learning is what assessment using the Standards is about. • Whether the purpose is assessment for/as/of learning we need to make consistent AND valid judgements on their learning progression with the Standards.
Making consistent judgements
Assessment Professional Learning Module 5:Making Consistent Judgements 1
Assessment FOR learning:occurs when teachers use Assessment AS learning:inferences about student occurs when students reflectprogress to inform on and monitor their progresstheir teaching. to inform their future learning goals. Assessment OF learning: occurs when teachers use evidence of student learning to make judgements on student achievement against goals and standards. 2
As educators it is imperative that we make high quality judgements about student learning. What does that mean we should do? 3
EXAMPLE“ How can you possibly award prizes when everyone missed the target?” said Alice.“Well,” said the Queen, “Some missed by more than others and we have a fine normal distribution of misses, which means we can forget about the target.” Lewis Carroll, quoted in Rowntree, 1987, pp 181-2
Quality and trustworthiness in evidence of learning requires both:• valid (fair, accurate, appropriate) and • consistent (reliable) assessment judgements. 5
ValidEveryday synonyms:- truthful For valid assessment- appropriate fairness,- justified accuracy and- convincing appropriateness- accurate matter.- legally acceptable- fair 6
Validity is a matter of fairness, appropriateness and accuracy• Are the tasks constructed, presented and conducted so thatall students have an equal chance of demonstrating theirlearning?• Do the tasks represent all the valued learning you want thestudents to have undertaken? Do they assess a sample of allthe important concepts and objectives?• Are the assessment tasks probing the students thinking to thedepth you want - or are they able to “get away with” asuperficial understanding? 7
Validity is a matter of fairness, appropriateness and accuracy• Are the assessment tasks monitoring what you think they are(and not some intervening prerequisite skill, or conceptualunderstanding)?In particular, do the assessment tasks require specificknowledge or skills which some students may not have, andwhich have not been explicitly taught?(e.g. how to: read, draw a graph or ‘google’?) 8
Assessment is valid if it …• assesses appropriate content and objectives• provides information which is useful for some valuable purpose (for/as/of)• is assessed with sufficient accuracy• is fair to all students. 9
Consistency is a matter of reliabilityHow confident can you be that judgements that you havemade are not significantly affected by chance factors such as:• how the student was feeling on the day?• how the assessor was feeling on the day?• who the assessor was?• luck in being assessed on some things and not others?• luck in that the mode of assessment suited thestudent particularly well (or didn’t)? 10
Examples of everyday use of the term “reliability” A set of kitchen scales is reliable if ... A clock is reliable if … A train is reliable if … A friend is reliable if … An assessment task is reliable if … (what?) 11
Definition of consistencyThe consistency of an assessment is the reliability with which itassesses whatever it assesses.It is the reproducibility that is the focus. 12
The assessor matters in making consistent judgements.We are human and our judgements may be influenced by many factors besides the actual standard of work. • INTER-rater reliability: if another assessor judged the work, would the student be awarded the same result? • INTRA-rater reliability:if the same assessor judged the work on another day would the result be the same? 13
Consistent assessmentAn assessment is consistent if the scores that students get arereliable:• from one occasion to the next• from one form of assessment to another• from one assessor to another. 14
The difference between consistency and validityConsistency (or reliability) is a technical question. Will workbe judged in the same way every time, regardless of whoassesses it, or how, or when?Validity has both a technical element (does the assessmentaccurately judge what it says it judges?) and a philosophicalone (is the assessment appropriate and fair?).Validity is an indication of the value and correctness of whatwe do. 15
Consistency, alone, is not sufficient.Assessment must also be valid (fair, appropriate and accurate). 16
Each step in the assessment process involves:• the possibility of error• value-driven decisions• assessment processes that may disadvantagesome types of students.Use collaborative planning processes, transparency ofassessment tasks, criteria, rubrics and variety to maximisevalid and consistent judgements. 17
Strategies and ProtocolsFor valid and consistent judgements of student learningprogression, we need to put in place strategies such as:• common assessment tasks planned collaboratively• shared design processes for assessment tasks and forrubrics• comparing work with exemplars (e.g. assessment maps)• cross-marking of sample assessment tasks• moderation and consistency protocols. 18
Critical stages of the Moderation Process1. Developing a common understanding of what is to be assessed and the purpose of the assessment2. Drafting the assessment task requirements3. Drafting the criteria- or marking scheme4. Sample assessing5. Final assessing. 19
The final assessing stage of moderation may involve:• assessors meeting together• sharing of assessed work samples• discussion of queries• cross-marking each other’s work samples• discussion of borderline or special circumstance cases. 20
When deciding HOW to make more valid and consistent judgements “efficiency” issues must also be considered:• time efficient (yours and your students!)• learning efficient (maximises learning)• teaching efficient (reduces wastage)• cost efficient??? other factors 21
Making consistent judgements is important.AssessmentFOR learning Assessment AS Learning Assessment OF learning But consistency is not enough - we have to make valid judgements as well. 22