2. Assessment FOR learning:
occurs when teachers use Assessment AS learning:
inferences about student occurs when students reflect
progress to inform on and monitor their progress
their teaching. to inform their future
learning goals.
Assessment OF learning:
occurs when teachers use evidence
of student learning to make
judgements on student achievement
against goals and standards.
2
3. As educators it is imperative that we make high
quality judgements about student learning.
What does that mean we should do?
3
4. EXAMPLE
“ How can you possibly award prizes when
everyone missed the target?” said Alice.
“Well,” said the Queen, “Some missed by more
than others and we have a fine normal
distribution of misses, which means we can
forget about the target.”
Lewis Carroll, quoted in Rowntree, 1987, pp 181-2
5. Quality and trustworthiness in
evidence of learning requires both:
• valid (fair, accurate, appropriate) and
• consistent (reliable)
assessment judgements.
5
6. Valid
Everyday synonyms:
- truthful
For valid assessment
- appropriate fairness,
- justified accuracy and
- convincing appropriateness
- accurate matter.
- legally acceptable
- fair
6
7. Validity is a matter of
fairness, appropriateness and accuracy
• Are the tasks constructed, presented and conducted so that
all students have an equal chance of demonstrating their
learning?
• Do the tasks represent all the valued learning you want the
students to have undertaken? Do they assess a sample of all
the important concepts and objectives?
• Are the assessment tasks probing the students thinking to the
depth you want - or are they able to “get away with” a
superficial understanding?
7
8. Validity is a matter of
fairness, appropriateness and accuracy
• Are the assessment tasks monitoring what you think they are
(and not some intervening prerequisite skill, or conceptual
understanding)?
In particular, do the assessment tasks require specific
knowledge or skills which some students may not have, and
which have not been explicitly taught?
(e.g. how to: read, draw a graph or ‘google’?)
8
9. Assessment is valid if it …
• assesses appropriate content and objectives
• provides information which is useful for some
valuable purpose (for/as/of)
• is assessed with sufficient accuracy
• is fair to all students.
9
10. Consistency is a matter of reliability
How confident can you be that judgements that you have
made are not significantly affected by chance factors such as:
• how the student was feeling on the day?
• how the assessor was feeling on the day?
• who the assessor was?
• luck in being assessed on some things and not others?
• luck in that the mode of assessment suited the
student particularly well (or didn’t)?
10
11. Examples of everyday use of the term “reliability”
A set of kitchen scales is reliable if ...
A clock is reliable if …
A train is reliable if …
A friend is reliable if …
An assessment task is reliable if … (what?)
11
12. Definition of consistency
The consistency of an assessment is the reliability with which it
assesses whatever it assesses.
It is the reproducibility that is the focus.
12
13. The assessor matters in making consistent judgements.
We are human and our judgements may be influenced by many
factors besides the actual standard of work.
• INTER-rater reliability:
if another assessor judged the work, would the student be
awarded the same result?
• INTRA-rater reliability:
if the same assessor judged the work on another day would the
result be the same?
13
14. Consistent assessment
An assessment is consistent if the scores that students get are
reliable:
• from one occasion to the next
• from one form of assessment to another
• from one assessor to another.
14
15. The difference between consistency and validity
Consistency (or reliability) is a technical question. Will work
be judged in the same way every time, regardless of who
assesses it, or how, or when?
Validity has both a technical element (does the assessment
accurately judge what it says it judges?) and a philosophical
one (is the assessment appropriate and fair?).
Validity is an indication of the value and correctness of what
we do.
15
16. Consistency, alone, is not
sufficient.
Assessment must also be valid
(fair, appropriate and
accurate).
16
17. Each step in the assessment process involves:
• the possibility of error
• value-driven decisions
• assessment processes that may disadvantage
some types of students.
Use collaborative planning processes, transparency of
assessment tasks, criteria, rubrics and variety to maximise
valid and consistent judgements.
17
18. Strategies and Protocols
For valid and consistent judgements of student learning
progression, we need to put in place strategies such as:
• common assessment tasks planned collaboratively
• shared design processes for assessment tasks and for
rubrics
• comparing work with exemplars (e.g. assessment maps)
• cross-marking of sample assessment tasks
• moderation and consistency protocols.
18
19. Critical stages of the Moderation Process
1. Developing a common understanding of what is to be
assessed and the purpose of the assessment
2. Drafting the assessment task requirements
3. Drafting the criteria- or marking scheme
4. Sample assessing
5. Final assessing.
19
20. The final assessing stage of moderation may involve:
• assessors meeting together
• sharing of assessed work samples
• discussion of queries
• cross-marking each other’s work samples
• discussion of borderline or special circumstance cases.
20
21. When deciding HOW to make more valid and consistent
judgements “efficiency” issues must also be considered:
• time efficient (yours and your students!)
• learning efficient (maximises learning)
• teaching efficient (reduces wastage)
• cost efficient
??? other factors
21
22. Making consistent judgements is important.
Assessment
FOR learning
Assessment AS
Learning
Assessment OF
learning
But consistency is not enough -
we have to make valid judgements as well.
22
Editor's Notes
• The whole series of 5 modules is called: “Assessment Professional Learning”. • This fifth (and final) Module is called: “Making Consistent Judgements”.
• The DE&T Assessment Advice centres on these three purposes of assessment. • The focus of this Module is making consistent judgements for all three purposes: assessment OF learning, assessment FOR learning and assessment AS learning.
• With the new Standards we are expected to be more consistent and valid in the judgements we make. • We therefore need to build some strategies and protocols to enable this to happen.
• “ Quality assurance” - means that students, parents and our teaching colleagues can be confident that the assessment decisions we make are trustworthy. • They must be both valid and consistent to be trustworthy.
• “ Valid” has several everyday meanings - and a technical one when talking about assessment.
Validity has several elements to it - including an assumption of fairness, accuracy and appropriateness. • We must try to be as fair, and unbiased, as possible in our judgements on students progression against the Standards. • The assessment ought to represent the whole of what we are trying to teach and assess - and not be skewed to a small section or standard (even if important). • The assessment will be valid if it requires the depth of learning we want - and students cannot skim lightly across the top.
• Our judgements need to accurately reflect the real performance and understandings of our students. This can sometimes be interfered with by an intervening skill that we have not realised is a pre-requisite. Such examples occur when students cannot read the instructions, or cannot google to do internet research, etc. We need to ensure that all students have all necessary pre-requisite skills - and not assume that because some have it, all do.
• This list can be used to check our assessment. • Whether the purpose is assessment FOR learning, assessment OF learning, or assessment AS learning - all judgements must be valid.
• Consistency is the everyday term for reliability (the technical term used in assessment literature).
• Reliability has many everyday meanings as well - some of which relate to the technical term. • Some of these meanings of reliability are about the concept of “validity”. For example, a clock that gives us the correct time is an accurate clock (i.e. a valid clock) and a set of scales that gives us the correct weight is also accurate (i.e. valid). But a clock that is consistently 5 minutes fast - is reliably wrong - but consistent.
• Put simply - consistency means that the results are the same no matter who fires the arrows, when, where, or what the weather is like.
• Two important aspects of reliability/consistency is being consistent with other teachers (inter-rater reliability) and being consistent with ourselves (intra-rater reliability. • For example, if I assess some students’ work late at night when I’m tired, I may not judge it the same as when I am fresh on a Saturday morning. • Or if I assess Student A’s work in the 1st three I “mark”, then get more generous as I proceed through the pile, I may not be consistent in my approach to all students’ work, and be more generous by the time I get to Student B’s work near the end of the pile. Some teachers always go back and re-mark the first few, to ensure they haven’t changed their interpretations on the way through the assessment process.
• The assessor need not be the teacher - it could be the student, a peer, some other external body (e.g. VCAA).
• Consistency/reliability is a technical matter - do we all give the same result for the same standard of work? • Validity has a technical dimension (accuracy) but also calls for judgement (is it fair and appropriate for these students)?
• We can all hit the same spot in our assessment of student work - but it is meaningless if what we are agreeing on is not the Standard we are trying to monitor. • Or: consistency is of no value if we are all consistently bad at assessing students progress against the Standards! • Being in solid agreement, but being inaccurate and unfair (invalid) is not a good thing in assessment!
• There are many traps in the assessment process. • We can build in collaborative planning so we know what the standards are about, we can build in collaborative design of the tasks and rubrics so we have a shared understanding of what the students are doing … we can be transparent with students about what they have to do and how we will judge their work … it all helps make a subjective process more objective.
• During this module you will develop some skills in these areas and be introduced to some practical strategies that will help you make more valid and consistent judgements.
Moderation is a process where by teachers discuss and agree on a final judgement of a piece of student work, it is a quality assurance for assessment. There are several stages in the moderation process. It does not have to be an ‘end’ judgement, but can can begin with the task The moderation process can be a form of professional learning for teachers. Think about current moderation processes in your school. With a partner brainstorm and discuss whether all the critical stages of moderation are present in your practice?
By the end of the moderation process assessors should feel assured that there assessment task is: Valid- it is reliable Consistent- the assessment of the student work assessment is consistent with other student work and with other assessors Complete activity 5-4A includes a reading providing and overview of Moderation Processes.
• Above all else we cannot let this take over our lives - we can work collaboratively to share the burden and build better consistency. • Our focus must be on using strategies that are efficient. • Do people have ideas of other ways we need to be efficient in our assessment processes?
• The students are at the centre. Their learning is what assessment using the Standards is about. • Whether the purpose is assessment for/as/of learning we need to make consistent AND valid judgements on their learning progression with the Standards.
