1) The study sought to determine if an analytic or holistic rubric could reduce teacher bias in grading, and if student attainment level (SEN or HA) played a role. 2) Results found essays labeled as written by SEN students were graded significantly lower than HA essays without a rubric. 3) There was no main effect for rubric type, but there was a significant interaction between rubric type and attainment level, suggesting analytic rubrics may better reduce bias.

1. Analytic Rubric versus Holistic
Rubric in Reducing Teacher Bias
Regarding Attainment Level:
Special Educational Needs and High
Achiever
By Emma Kiely, Mary Immaculate College, Limerick
Supervised by Claire Griffin-O’Brien

2. Principles of the Curriculum as written in the Primary School Curriculum
(1999)
NCCA (1999), p.9
Importance of Assessment

3. Organisation for Economic Co-operation and
Development (2013)
The OECD Report “Synergies for
Better Learning” highlights the
importance of assessment, and the
need for improvement in many
areas of assessment - including the
fairness of student assessment

4. OECD, 2013

5. Teacher Bias
(Greenwald & Krieger, 2006)
... exists whereby a teacher
exhibits discriminatory
behaviour towards a student or
group of students as a result of
implicit attitudes the teacher
holds
Teachers may exhibit personal
bias when they ‘rate’ students
based on inappropriate and
irrelevant stereotypes

6. Bias often emerges regarding student
attainment level
Bias may present itself in the form of
grading variation
teacher assigns grades based on stereotyping of a
student rather than on that student’s performance
on a task
(Greenwald & Krieger, 2006; Rauschenberg, 2012)
Special Educational
Needs (SEN)
High Achiever (HA)
Teacher Bias

7. Analytic
Rubric
Holistic Rubric
Two types of rubric prevail:Solution?
“documents that articulate the
expectations for an assignment, or a set
of assignments, by listing the
assessment criteria and by describing
levels of quality in relation to each of
these criteria”
(Reddy & Andrade, 2010, p. 435)
Rubrics have been suggested in research
as an effective method of reducing bias in
grading.

8. Holistic vs. Analytic Rubric
(Mertler, 2001)

9. The reliability of
scoring of
performances is
enhanced through the
use of rubrics (Jonsson
& Svingby, 2007;
Whittaker, Salend &
Duhaney, 2001)
The more detailed and
specific the written scoring
criteria are, and the more
consistently these are
applied, the less bias in
grading is likely to occur
(Hardré, 2014; Jonsson &
Svingby, 2007; Rezaei &
Lovorn, 2010).
Hardré (2014)
Jonsson &
Svingby (2007)
Previous
Research

10. The current study sought to identify whether an analytic rubric
or a holistic rubric can reduce teacher bias in grading, and
whether student attainment level plays a role.
Hypothesis 2:
the analytic rubric
will result in a
reduction of bias
for both
attainment levels
(SEN and HA)
Current Study
Hypothesis 1:
Essays labelled as being
written by a child with
SEN will be graded
significantly lower than
those labelled HA in the
no rubric (control) group

11. Method: Design
Experimental
study
3x2
between-
subjects factorial
design
Independent Variables
Child attainment
level
SEN / High
Achiever
Rubric
Analytic/
Holistic/ None
Dependent Variable
Grade
Assigned to
Essay

12. Method: Participants
120 3rd & 4th
Year Student
Teachers B.Ed in Ed & Psych
B.Ed
Female
74%
Male
26%
Aged 20 - 23
(M = 21.29, SD
= .793)

13. Materials – Surveys on Google Forms

14. Materials & Procedure
No Rubric Groups
Rubric Groups
Priming passages,
giving details
about the child
who “wrote” the
essay to be graded

15. Procedure – Grade the child’s essay/ Analytic Rubric
National Council for Curriculum and Assessment (2016a).
Examples of children's language learning and
development: English and Irish. Retrieved September 17,
2016, from Curriculum Online:
http://www.curriculumonline.ie/Primary/Curriculum-
Areas/Language/Examples-of-Children-s-Learning-and-
Development

16. Analytic Rubric –
adapted from Saddler,
B., & Andrade, H.
(2004). The writing
rubric. Educational
Leadership, 62(2), 48–
52

17. Procedure – Grade the child’s essay/Holistic Rubric
National Council for Curriculum and Assessment (2016a).
Examples of children's language learning and
development: English and Irish. Retrieved September 17,
2016, from Curriculum Online:
http://www.curriculumonline.ie/Primary/Curriculum-
Areas/Language/Examples-of-Children-s-Learning-and-
Development

18. Holistic Rubric - adapted from Saddler,
B., & Andrade, H. (2004). The writing
rubric. Educational Leadership, 62(2),
48–52

19. Procedure – Grade the child’s essay/No Rubric
National Council for Curriculum and Assessment (2016a).
Examples of children's language learning and
development: English and Irish. Retrieved September 17,
2016, from Curriculum Online:
http://www.curriculumonline.ie/Primary/Curriculum-
Areas/Language/Examples-of-Children-s-Learning-and-
Development

20. Results: H1
2x3 between-groups ANOVA
IVs: child attainment level (SEN/High Achiever) & rubric
type (holistic/analytic/none)
DV: grade assigned to child’s essay
Hypothesis 1:Essays labelled
as being written by a child
with SEN will be graded
significantly lower than those
labelled HA in the no rubric
(control) group
F(1,114 ) = 27.78, p < .001

21. Results: H1
As can be seen here,
essays perceived as being
written by a child with
SEN were graded
significantly lower than
essays perceived to be
written by a HA in all
grading conditions,
including the condition of
no rubric
Interestingly, the essays
labelled as written by a
child with SEN were
graded particularly lower
than their HA counterpart
in the holistic rubric
conditions

22. Results: H2
F(2, 114) = 5.24, p =.007
2x3 between-groups ANOVA
IVs: child attainment level (SEN/High Achiever) & rubric
type (holistic/analytic/none)
DV: grade assigned to child’s essay
Hypothesis 2: the analytic
rubric will result in a
reduction of bias for both
attainment levels (SEN and
HA)
There was no main effect for
rubric type, p =.425.
However, there was a
significant Rubric ×
Attainment interaction effect

23. Discussion: Limitations
Absence of a condition with an essay written by a child with no label
as a control group
External validity: 3rd or 4th year student teachers as participants,
findings not generalizable to qualified/experienced teachers
Considerably more female participants than male participants
Fails to implement IAT to account for individual differences in
implicit cognition

24. Transitions theme

25. Findings highlight the need for the use of analytic rubrics to reduce grading
variation in the Primary School
Supports previous research which finds that detailed, analytic style rubrics
are successful in reducing grading variation as a result of teacher bias
(Hardré, 2014; Jonsson & Svingby, 2007; Rezaei & Lovorn, 2010).
Implications of the need for teacher training courses aimed at reducing or
eliminating bias
Implications of the need for CPD to ensure teachers are made aware of
potential biases they may develop throughout their career and ways to
address these biases
Conclusions & Implications

26. Bibliography
• Andrade, H. G. (2000). Using rubrics to promote thinking and learning. Educational Leadership,
57(5), 13–18.
• Andrade, H. L., Du, Y., & Wang, X. (2008). Putting Rubrics to the Test: The Effect of a Model,
Criteria Generation, and Rubric‐Referenced Self‐Assessment on Elementary School Students'
Writing. Educational Measurement: Issues and Practice, 27(2), 3-13.
• Hardré, P. (2014). Checked Your Bias Lately? Reasons and Strategies for Rural Teachers to Self-
Assess for Grading Bias. The Rural Educator, 35(2).
• Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational
consequences. Educational Research Review, 2(2), 130-144.
• Malouff, J. (2008). Bias in Grading. College Teaching, 56(3), 191-192.
• Mastergeorge, A., & Martinez, J. (2009). Rating Performance Assessments of Students With
Disabilities: A Study of Reliability and Bias. Journal Of Psychoeducational Assessment, 28(6),
536-550.
• Mertler, C. A. (2001). Designing scoring rubrics for your classroom. Practical Assessment,
Research and Evaluation, 7(25).
• National Council for Curriculum and Assessment (2007). Assessment in the Primary School:
Guidelines for Schools. Dublin: NCCA.
• National Council for Curriculum and Assessment, (1999). Primary School Curriculum. Dublin:
Government of Ireland Publications.

27. • National Council for Curriculum and Assessment, (2017). Examples of children's language
learning and development: English and Irish. (2017). Curriculumonline.ie. Available online:
http://curriculumonline.ie/Primary/Curriculum-Areas/Language/Examples-of-Children-s-
Learning-and-Development
• Organisation for Economic Co-operation and Development (2013). Synergies for Better
Learning: An International Perspective on Evaluation and Assessment. Retrieved from
http://www.oecd.org/edu/school/Synergies%20for%20Better%20Learning_Summary.pdf.
• Reddy, Y., & Andrade, H. (2010). A review of rubric use in higher education. Assessment &
Evaluation In Higher Education, 35(4), 435-448.
• Saddler, B., & Andrade, H. (2004). The writing rubric. Educational Leadership, 62(2), 48–52.