Slides for 2016 British Educational Research Association (BERA) workshop. Workshop abstract: The term ‘rubric’ conflates many different practices, some of which are in opposition to each other. The same word ‘rubric’ represents both secret markers’ guides and co-created articulations of quality; it includes tools that support or supplant marker judgement; and it may contain detailed definitions of quality, or use broad qualitative terms. This workshop introduces a framework for more clearly and precisely discussing rubrics, which has been published in Assessment and Evaluation in Higher Education online-ahead-of-print (Dawson, 2015; remove for blind review). Through synthesising the literature on rubrics, that paper provides a framework of 14 different ‘design elements’ that make rubrics different from each other. The workshop will support participants to use the framework to share their current practices and design rubric interventions in response to scenarios. The problem with a lack of a common definition to describe rubrics is that they have been mandated, evaluated and implemented into software, often as if there was a shared understanding. Policy at many educational institutions requires the use of rubrics, but does not define if it is task-specific or generic rubrics; analytic rubrics or holistic rubrics; secret or shared. Many research papers evaluate particular rubrics, but omit important elements, such that even the most methodologically-robust evaluation study is of limited use as it is not clear what sort of rubric was evaluated. Software tools are developed to support ‘rubrics’ but it is often unclear to a potential customer what a rubric is assumed to be. A lack of clarity on rubrics may make us blind to their potential benefits and pitfalls. The framework presented in this workshop is the result of an extensive synthesis of existing research and practice. It aims to make participants aware of 14 decisions they are making about rubrics, whether they know it or not. On completion, participants will be able to apply the rubric framework to their own practice, to: (1) more precisely and comprehensively articulate what they currently do as an educator or researcher, and (2) design rubric interventions that take into account the diversity of potential practices. BERA attendees who feel enthusiastic, cautious or indifferent toward rubrics are very welcome to attend.

1. Deakin University CRICOS Provider Code: 00113B
deakin.edu.au/learning/enhancing-your-practice/cradle
The ‘unknown unknowns’ of
assessment rubrics in practice,
policy and research
Associate Professor Phillip Dawson
Associate Director
Centre for Research in Assessment & Digital Learning (CRADLE)
Deakin University, Melbourne, Australia
@phillipdawson

2. Deakin University CRICOS Provider Code: 00113B

3. Deakin University CRICOS Provider Code: 00113B
A couple of decades ago, rubric began to take
on a new meaning among educators.
Measurement specialists who scored
students’ written compositions began to use
the term to describe the rules that guided
their scoring.
They could have easily employed a more
readily comprehensible descriptor, such as
scoring guide, but scoring guide lacked
adequate opacity.
Rubric was a decisively more opaque, hence
technically attractive, descriptor.
(Popham 1997, p. 72)

4. Deakin University CRICOS Provider Code: 00113B
Hands up if rubrics make you feel:
• Confident
• Concerned
• Relieved
• Confused
• Excited
• Bored
• …another feeling
you’d like to share?

5. Deakin University CRICOS Provider Code: 00113B
0
1000
2000
3000
4000
5000
6000
7000
Pre-1997
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
Mentions on Google
Scholar
Mentions in books
Cumulative frequency, 28 April, 2015.

6. Deakin University CRICOS Provider Code: 00113B
What is a rubric?
• A tool used in the process of evaluating
student work, which usually includes:
– evaluative criteria
– quality definitions for those criteria at
particular levels
– a scoring strategy.
Per Popham (1997)

8. Deakin University CRICOS Provider Code: 00113B
This workshop
• Overview of framework
• Practice sharing
• Scenarios

9. Deakin University CRICOS Provider Code: 00113B
Framework overview
• 14 ‘design elements’ representing choices
made in rubric design
• Brief overview of each – stop me if you
want more detail

10. Deakin University CRICOS Provider Code: 00113B
Specificity: the particular object of assessment
• Generic
– Including faculty-wide
or course level
• Task-type
• Task-specific

11. Deakin University CRICOS Provider Code: 00113B
Secrecy: who the rubric is shared
with, and when it is shared
• Top-secret rubrics
• Rubrics returned with marks/feedback
• Rubrics shared with task instructions

12. Deakin University CRICOS Provider Code: 00113B
Exemplars: work samples
provided to illustrate quality
What does a good one
of these look like?
Why is this one good?

13. Deakin University CRICOS Provider Code: 00113B
Scoring strategy: procedures used to
arrive at marks and grades
Analytic
• Adding up
• Logic, conditions,
thresholds
• Combining scores from
multiple markers
Holistic
• Assessor uses all
information to make a
single overall quality
judgement

14. Deakin University CRICOS Provider Code: 00113B
Evaluative criteria: overall attributes
required of the student
• (usually the leftmost column)
• Mandatory (or not?)
• Connector between indicators of quality
and task/subject outcomes

15. Deakin University CRICOS Provider Code: 00113B
Quality levels: the number and type
of levels of quality
• Grade levels
(P/C/D/HD)
• Learning outcomes
taxonomies
• Levels of study
• Fail?
• Every criterion at
every level?

16. Deakin University CRICOS Provider Code: 00113B
Quality definitions: explanations of
attributes of different levels of quality
“Mixture of
qualitative and
quantitative
feedback
sources”

17. Deakin University CRICOS Provider Code: 00113B
Judgement complexity:
the evaluative expertise required of
users of the rubric
Pass Credit Distinction
Clarity of
expression
Bad Acceptable Good

18. Deakin University CRICOS Provider Code: 00113B
Judgement complexity:
the evaluative expertise required of
users of the rubric
Pass Credit Distinction
Clarity of
expression
Five or
fewer
grammatical
errors
Two or
fewer errors
Zero
grammatical
errors

19. Deakin University CRICOS Provider Code: 00113B
Users and uses: who makes use of the
rubric, and to what end
Teachers
• Marking
• Providing feedback
• Communicating
standards
• Understanding standards
Students
• Planning work
• Understanding standards
• Self assessment
• Peer assessment
• Grade grubbing
• Planning improvement to
future work

20. Deakin University CRICOS Provider Code: 00113B
Creators: the designers of the rubric
• Teachers
• Textbook publishers
• Internet communities
• Students
• Rubric specialists
• Researchers
• Professional bodies

21. Deakin University CRICOS Provider Code: 00113B
Quality processes: approaches to ensure
the reliability and validity of the rubric
• Formal reliability and validity processes
• Informal review, piloting and revision
• Haven’t got time for that, just see how it
goes

22. Deakin University CRICOS Provider Code: 00113B
Accompanying feedback information: comments,
annotation or other notes on student performance
• Stimulus for feedback discussion
• Focal point for written feedback
• A supplement to written feedback
• Replacement for written feedback

23. Deakin University CRICOS Provider Code: 00113B
Presentation: how the information in
the rubric is displayed

24. Deakin University CRICOS Provider Code: 00113B
Explanation: instructions or other
additional information provided to users
• Written instructions
• Discussion around rubric in class
• Marking workshop
• Video explanation

25. Deakin University CRICOS Provider Code: 00113B
How do you use rubrics?
• Do you feel strongly about any of these
elements?
• Discuss how you use rubrics in groups of
three or four
• Structure your conversation around the
framework

26. Deakin University CRICOS Provider Code: 00113B
Scenarios – choose one, in groups
Rubric advice
• 1st year educational
theory course
• 1000 students
• Lecturer wants your
advice on rubric for 2000
word essay
Faculty rubric
policy/procedures
• Education faculty at a
large university have
contracted you to write
rubric policy/procedures
• What would you
mandate?

27. Deakin University CRICOS Provider Code: 00113B
0
1000
2000
3000
4000
5000
6000
7000
Mentions on Google
Scholar
Mentions in books
@phillipdawson