Hunting assumptions about assessment; an up-dated replacement for a 2009 version.

1. Assessment
Recap…
… as ever with added complications!
James Atherton
Unit 3: 8 April 2014

2. The blog page to
which some notes
refer is at:
http://pce2013.blogspot.co.uk/2014/04/8-april-back-to-assessment.html

3. Why?
Requirements
Approaches
Assessment
08/04/2014 - v2
Feedback
(Formative)
Accreditation
(Summative)
Validity
Reliability
Fairness
(Security)
Norm-referenced
Criterion-referenced
Ipsative

4. • “Assessment is central because it frames
student learning”
(Oxford Brookes U. Position Paper on Assessment, n.d.)
• “From our students’ point of view,
assessment always defines the actual
curriculum”
(Ramsden P,1992)

5. • “Assessment is central because it frames
student learning”
(Oxford Brookes U. Position Paper on Assessment, n.d.)
• “From our students’ point of view,
assessment always defines the actual
curriculum”
(Ramsden P,1992)
Not for nothing do
students invariably
cry, “Are we being
assessed on this?”

6. • “Assessment is central because it frames
student learning”
(Oxford Brookes U. Position Paper on Assessment, n.d.)
• “From our students’ point of view,
assessment always defines the actual
curriculum”
(Ramsden P,1992)
Not for nothing do
students invariably
cry, “Are we being
assessed on this?”
And if the answer is
“No”, then the
learning is
automatically
worthless

7. It is assessment which turns;
• presenting
• story-telling
• demonstrating
• …
into teaching

8. • Assessment of
learning
• (summative)
• Assessment for
learning
• (formative)
Which is privileged?

9. • Assessment of
learning
• (summative)
• Assessment for
learning
• (formative)
Which is privileged?
Accreditation and
qualifications—
important but not
the whole story

10. • Assessment of
learning
• (summative)
• Assessment for
learning
• (formative)
Which is privileged?
Accreditation and
qualifications—
important but not
the whole story
Importance
increasingly
recognised in the
rhetoric of
education, if not
the practice…

11. • Assessment of
learning
• (summative)
• Assessment for
learning
• (formative)
Which is privileged?
Accreditation and
qualifications—
important but not
the whole story
Importance
increasingly
recognised in the
rhetoric of
education, if not
the practice…
See link in
notes

12. Formative
Summative
but…

13. Formative
Summative
but…
All assessment,
including most of
what is designed
for summative
purposes, can be
used formatively

14. Formative
Summative
but…
All assessment,
including most of
what is designed
for summative
purposes, can be
used formatively…
…but only if you can
get the students to
take their work back
and read the

15. PresenterPresenter AudienceLearner
Information
Feedback
Teacher
Teacher/
learner
Learner/
teacher

16. Experiential
environment
Theoretical
representation
Goal-oriented
behaviour
Conceptual
representation
Teacher Student
Articulation/
Re-articulation
Reflection/
Adaptation
Reflection/
Adaptation
Action/
Feedback
Based on Laurillard, 2001
This is Diana Laurillard’s
elaboration of the previous
graphic—link in the notes

17. Effect-size 2
• An effect-size of 1.0 indicates an increase of one standard
deviation, typically associated with advancing [students’]
achievement by one year, improving the rate of learning by 50%,
or a correlation between some variable (e.g., amount of
homework) and achievement of approximately .50. When
implementing a new program, an effect-size of 1.0
would mean that approximately 95% of
outcomes positively enhance
achievement, or average students
receiving that treatment would exceed
84% of students not receiving that
treatment.
Hattie, 1999:4 (my emphasis)
Just a reminder of
Hattie—now familiar
territory, with a link in
the notes.

18. Interventions and Effect Size
0
50
100
150
200
250
300
350
400
-0
.8
-0
.6
-0
.4
-0
.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Effect Size
No.ofstudies

19. Above
average
effect
size
Influence Effect Size
•Feedback 1.13
• Students' prior cognitive ability 1.04
• Instructional quality 1.00
• Direct instruction .82
• Remediation/feedback .65
• Students' disposition to learn .61
• Class environment .56
• Challenge of Goals .52
• Peer tutoring .50
• Mastery learning .50
• Homework .43
• Teacher Style .42
• Questioning .41

20. • Feed-back, or
• Feed-forward ?
Phil Race is the
principal advocate
for feed-forward.
Link in notes

21. Why?
Requirements
Approaches
Assessment
08/04/2014 - v2
Feedback
(Formative)
Accreditation
(Summative)
Validity
Reliability
Fairness
(Security)
Norm-referenced
Criterion-referenced
Ipsative

22. Requirements…
• Validity: the assessment measures what it
is supposed to.

23. •What mismatches have you come
across?
• (i.e. between desired outcomes and methods of
assessment)

24. The boundary
is constantly
moving
Common observation suggests the belief is unfounded; we
seldom find hard evidence of such correlations. Instead, we
rely on test results for want of anything better. In any event,
the skills required to perform well on school examinations
may not be the same skills required to perform adequately
in the situations the school trains people for.
(Becker, 1972)
We seldom argue that
conventional tests measure the
actual skills students are
supposed to have learned.
Rather, we believe that the test,
… is … highly correlated with
the ability to exercise those
skills, though the mechanism …
is seldom investigated or
demonstrated.
Link to full paper in the notes

25. Reference points for Assessment
• Competitive (“Norm-referenced”)
See link in notes

26. Norm-Referenced Assessment
0
5
10
15
20
25
0
35
45
55
65
75
85
95
105
115
125
135
145
155
165
%ofcohort
Mean
This curve is based on a real reference curve used for the
standardisation of IQ tests, which is why it is not quite as
graceful as a classic “bell-curve”. See link in notes.

27. •When might it be appropriate to
use norm-referenced assessment
in a PCE setting?

28. Reference points for Assessment (2)
• Competitive (“Norm-referenced”)
• Competence (Criterion-referenced)
[and
• Personal best (Ipsative)]

29. Criterion-Referencing
Threshold of competence
Object is to ensure minimum
level of competence attained
One problem is whether that
threshold effectively sets a cap
on the level of achievement: we
briefly discussed the question of
inability to deal with
exceptional genius in the arts.

30. A complication of “competence”
Threshold
of Competence
Competence is as much a matter of
consistency as of skill — what level
of consistency is “competent”?

31. Answer
Facts Stimulus
Idea
Convergent thinking
- science and technology(?)
Divergent thinking
- arts and humanities(?)
See link in notes

32. •How might you define
achievement of intended
outcomes in a divergent subject
area?

33. Requirements (2)…
• Validity: the assessment measures what it
is supposed to.
• Reliability: it produces consistent and
replicable results
 Between judges
 Between occasions
 Between cohorts

34. Why?
Requirements
Approaches
Assessment
08/04/2014 - v2
Feedback
(Formative)
Accreditation
(Summative)
Validity
Reliability
Fairness
(Security)
Norm-referenced
Criterion-referenced
Ipsative

35. • "Is an indistinct photograph a picture of a
person at all? Is it even always an advantage to
replace a blurred picture with a sharp one? Isn't
the blurred one often exactly what we need?"
Wittgenstein (1953) para. 71
• “OK, you’ve got your PhD; now don’t touch
anything!” (source unknown)
• “I claimed at the start that I would use the
term 'ambiguity' to mean anything I liked,”
(Empson, 1947; viii)16
Importance of clear criteria?

36. Criteria
a. Understanding and familiarity with the subject
b. Breadth of reading
c. Relevance of content to essay title
d. Structure of the argument
e. Independence of thought and critical evaluation
f. Presentation, including fluency of writing, grammar
and spelling
g. Citation and referencing
How would you weight these?
(in relation to an HE1/NQF 4 task in a “soft” discipline)

37. n
33223
erion
Without necessarily being
highly original, the work
shows consistent critical
evaluation of sources and
ideas, and the author's
own considered
judgments
The argument is pursued
clearly and in a balanced
manner, supported by
evidence and informed
judgement
Essay addresses title and
its underlying themes and
assumptions, clearly
explicated.
Evidence of extensive
reading beyond the
recommended texts,
including critical
evaluation of sources
Ability to contextualise
subject within wider
frameworks and critically
explore connections
Critical evaluation and
independent thought are
the norm, but are either
not consistent or coherent
throughout.
A clear and generally
structured argument let
down by some unsound
inferences or assertions
without evidence
Essay addresses title in a
focused and organised
way with some awareness
of any more general
issues raised by it.
Recommended reading
used, together with a
range of other sources
Familiarity with subject
demonstrated, including
complex connections
between elements
Evidence of occasional
independent and critical
thought, limited by either
inconsistent or incoherent
reasoning
Continuity of argument
evident throughout the
essay supported by
evidence
Content clearly addresses
set title as given.
Effective use made of
reading, largely confined
to reading list
Subject matter adequately
understood, with some
connections made
Critical evaluation limited
to expressions of opinion,
using mainly second-hand
ideas
Argument patchy,
inconsistent, or
proceeding by assertion
rather than evidence.
Essay generally
addresses title, but with
lapses, digressions or
padding.
Limited use made of texts
on set reading list
Evidence of knowledge,
but only basic
connections made
between items
Uncritical and unreflective
work, relying on
undigested sources.
Little clear structure or
argument evident.
Little or tenuous
connection between title
and essay content
Little or no reading
evident, even within the
set texts
Subject matter treated as
set of discrete bits of
knowledge
Independence of
thought and critical
evaluation
Structure of the
argument
Relevance of content to
essay title
Breadth of readingUnderstanding and
familiarity with the
subject
Criterion ECriterion DCriterion CCriterion BCriterion A
1Academic levelEducation, Individual and SocietyEduc 1002
5
4
3
2
1
More (including
download) via link in
notes.

39. Background
of topic

40. Background
of topic
Knowledge
of topic

41. Background
of topic
Knowledge
of topic
Research
skills

42. Background
of topic
Knowledge
of topic
Ability to
conduct
argument
Research
skills

43. Background
of topic
Knowledge
of topic
Ability to
conduct
argument
Research
skills
“Hard” and “soft”
objectives are
discussed in the notes.

44. Too clear and specific?
Task-2
An annotated diagram which illustrates how you contribute to the Quality
Assurance and Quality Improvement Processes, procedures and initiatives
in the area of work you are engaged in. A short report evaluating and
reflecting upon this contribution.
 Extended diagram(s) showing how quality framework systems work within own
teaching area making links to validity and reliability of the data collected.
 Comprehensive annotations including links to areas for improvements and
research.
 Excellent critical analysis of the current issues relating to quality assurance and
quality improvement processes in own area of teaching.
 Sound application of theory to practice in order to analyse and evaluate the
success of the quality procedures (both positive and negative) Applied theory
to practice to identify own CPD.
 Critical evaluation of the long term impact of QA and QI systems and
procedures.
 Detailed bibliography with substantial use of journal research sources. Quotes
are presented and fully engaged.
 Sound structure. Few errors of presentation.

45. Too clear and specific?
Task-2
An annotated diagram which illustrates how you contribute to the Quality
Assurance and Quality Improvement Processes, procedures and initiatives
in the area of work you are engaged in. A short report evaluating and
reflecting upon this contribution.
 Extended diagram(s) showing how quality framework systems work within own
teaching area making links to validity and reliability of the data collected.
 Comprehensive annotations including links to areas for improvements and
research.
 Excellent critical analysis of the current issues relating to quality assurance and
quality improvement processes in own area of teaching.
 Sound application of theory to practice in order to analyse and evaluate the
success of the quality procedures (both positive and negative) Applied theory
to practice to identify own CPD.
 Critical evaluation of the long term impact of QA and QI systems and
procedures.
 Detailed bibliography with substantial use of journal research sources. Quotes
are presented and fully engaged.
 Sound structure. Few errors of presentation.
This is an actual
rubric from a
supposedly Master’s
level course.

46. Too clear and specific?
Task-2
An annotated diagram which illustrates how you contribute to the Quality
Assurance and Quality Improvement Processes, procedures and initiatives
in the area of work you are engaged in. A short report evaluating and
reflecting upon this contribution.
 Extended diagram(s) showing how quality framework systems work within own
teaching area making links to validity and reliability of the data collected.
 Comprehensive annotations including links to areas for improvements and
research.
 Excellent critical analysis of the current issues relating to quality assurance and
quality improvement processes in own area of teaching.
 Sound application of theory to practice in order to analyse and evaluate the
success of the quality procedures (both positive and negative) Applied theory
to practice to identify own CPD.
 Critical evaluation of the long term impact of QA and QI systems and
procedures.
 Detailed bibliography with substantial use of journal research sources. Quotes
are presented and fully engaged.
 Sound structure. Few errors of presentation.
If you get this detailed
and prescriptive, can
you possibly hit the
level of critical
scholarship expected
of a graduate level
course?

47. Too clear and specific?
Task-2
An annotated diagram which illustrates how you contribute to the Quality
Assurance and Quality Improvement Processes, procedures and initiatives
in the area of work you are engaged in. A short report evaluating and
reflecting upon this contribution.
 Extended diagram(s) showing how quality framework systems work within own
teaching area making links to validity and reliability of the data collected.
 Comprehensive annotations including links to areas for improvements and
research.
 Excellent critical analysis of the current issues relating to quality assurance and
quality improvement processes in own area of teaching.
 Sound application of theory to practice in order to analyse and evaluate the
success of the quality procedures (both positive and negative) Applied theory
to practice to identify own CPD.
 Critical evaluation of the long term impact of QA and QI systems and
procedures.
 Detailed bibliography with substantial use of journal research sources. Quotes
are presented and fully engaged.
 Sound structure. Few errors of presentation.
If you get this detailed
and prescriptive, can
you possibly hit the
level of critical
scholarship expected
of a graduate level
course?
Despite fatuous usage
of the currently
fashionable phrases.

48. Requirements (4)…
• Validity: the assessment measures what it
is supposed to.
• Reliability: it produces consistent and
replicable results
• Fairness
• Security

49. Requirements (4)…
• Validity: the assessment measures what it
is supposed to.
• Reliability: it produces consistent and
replicable results
• Fairness
• Security
What does “inclusivity”
mean in the context of
assessment?

50. Why?
Requirements
Approaches
Assessment
08/04/2014 - v2
Feedback
(Formative)
Accreditation
(Summative)
Validity
Reliability
Fairness
(Security)
Norm-referenced
Criterion-referenced
Ipsative