Presented at the UZRT 2016 conference

1. Inter-rater Reliability in
Assessing Oral Presentation
Skills
Kristína Kollárová and Krisztián Simon
University of Pécs
2016

2. Structure of the presentation
• I. Peer assessment
• II. Description of the study
• III. Implementation
• IV. Findings
• V. Answering the research questions
• VI. Further research
2

3. I. Peer assessment
• Defining peer assessment:
• educational arrangement (Dochy et al.,1999)
– students assess the quality of their fellow students’
– criteria in evaluation
• teaching, assessment and learning no longer
separated (Stoynof , 2012)
3

4. I. Peer assessment
• The merits of peer assessment
– complementing teacher assessment (Davison &
Leung, 2009)
– developing learning and life skills (Mok, 2011)
• The major limitation of peer assessment
– reliability (Brown, 2004)
– can be overcome with clear criteria (Patri, 2002)
4

5. II. Description of the study
• Context:
– Listening and Speaking Skills II (2014/15 spring)
– Listening and Speaking Skills I (2015/16 fall)
• Participants:
5
L&S II (2014/15 spring) L&S I (2015/16 fall)
15 student participants 20 student participants
12 female + 3 male 14 female + 6 male
average age: 19.6 years average age: 20.22 years
2 tutors: 1 female + 1 male, average age: 26.5 years

6. II. Description of the study
• Research questions
– RQ1: To what extent do the participants’ oral
presentation skill assessments differ?
– RQ2: To what extent do the evaluators’
assessments differ?
– RQ3: What is the level of agreement among the
two evaluators?
6

7. III. Implementation
• Data collection instruments
– students’ peer reviews based on the presentation tasks
– evaluator review sheets (average scores)
7
L&S II (2014/15 spring) L&S I (2015/16 fall)
1 presentation 1 presentation
1 picture talk 2 picture talks
1 pair presentation
ECL advanced level criteria Proficiency Exam criteria

8. III. Implementation
• Data analysis
– inter-rater reliability
• Krippendorff’s Alpha - reliability coefficient
(0 - disagreement , 1 - perfect reliability)
– level of agreement among the two raters
• Spearman’s Rho
8

9. IV. Findings – Inter-rater and student agreement
9

10. IV. Findings – Students’ qualitative answers
• Listening and Speaking Skills II (2014/15 spring)
– 'What I liked': 3,369 words
• top 3 keywords: pronunciation, powerpoint, fluent
– 'What I would recommend': 2,409 words
• top 3 keywords: pronunciation, nervy, monotone
10

11. IV. Findings – Students’ qualitative answers
• Listening and Speaking Skills I (2015/16 fall)
– 'What I liked': 4,919 words
• top 3 keywords: pronunciation, fluency, fluent
– 'What I would recommend': 4133 words
• top 3 keywords: pronunciation, intone, okay
11

12. V. Answering the research questions
• RQ1: To what extent do the participants’ oral presentation
skill assessments differ?
– highly different levels of agreement between students and tutors
• RQ2: To what extent do the evaluators’ assessments differ?
– significant differences
• RQ3: What is the level of agreement among the two
evaluators?
– ECL criteria - higher agreement among raters
12

13. VI. Further research
• Zhu (1995) - need for training in response
tasks
13

14. References
• Brown, H.D. (2004). Language assessment: Principles and
classroom practices. White Plains: Longman.
• Davison, C. & Leung, C. (2009). Current issues in English language
teacher-based assessment. TESOL Quarterly, 43 (3). 393–415.
• Dochy, F., Segers, M., & Sluijsmans, D. (1999). The use of self-,
peer and co-assessment in higher education: a review. Studies in
Higher Education, 24, 331-350.
• Mok, J. (2011). A case study of students' perceptions of peer
assessment in Hong Kong. ELT J, 65 (3). 230-239
• Stoynoff, S. (2012). Looking backward and forward at classroom-
based language assessment. ELT J, 66 (4). 523-532.
• Szabó, Sz. & Papp, E. (2013). How to pass the ECL exam? Level
B1, B2, C1. Nyíregyháza: Szabó Nyelviskola és Fordítóiroda.
• Zhu, W. (1995). Effects of training for peer response on students’
comments and interaction. Written Communication, 12 (4), 492-528.
14

15. Thank you for your attention!
15