Oral versus written assessments


Published on

These slides are from a journal club discussion at the Pedagogic Research in the Biosciences group at the University of Leicester (UK). The meeting was reflecting on the paper "Oral versus written assessments: a test of student performance and attitudes" by Mark Huxham and colleagues from Napier University, Edinburgh. The paper is due to appear in the February 2012 edition of Assessment and Evaluation in Higher Education

Published in: Education, Technology
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Oral versus written assessments

  1. 1. Bioscience PedR Journal Club, January 2012Oral versus written assessments: a test of student performance and attitudes Mark Huxham, Fiona Campbell & Jenny Westwood Assessment & Evaluation in Higher Education 37:125-136 (February 2012)Reflections led by Chris Willmott (cjrw2@le.ac.uk)
  2. 2. Why discuss this paper?• Interesting research question: - is there a difference between oral v written assignments in terms of: (a) student performance (b) student attitude?• Relevant research methodologies - quantitative test scores - qualitative: email survey, focus groups• Involves biologists
  3. 3. The oral exam (viva voce)• Has a noble history• Has largely been replaced by alternative assessments in undergraduate courses (in the UK )• Remains central to examination of PhDs (in the UK )
  4. 4. Possible benefits of oral assessment1. Development of oral communication skills2. More “authentic” assessment (i.e. relevant to „real life‟ post-graduation)3. More inclusive (e.g. for dyslexic students)4. Gauging understanding & Encouraging critical thinking5. Less potential for plagiarism6. Better at conveying nuances of meaning7. Easier to spot rote-learning
  5. 5. Possible issues with oral assessment1. Take a long time, impossible for n=300 yr 12. Reliability and bias across groups3. Anonymous marking is impossible4. Difficult to produce evidence for Externals5. Stressful (for students)6. Might favour extrovert and confident students7. Not appropriate for abstract reasoning tasks
  6. 6. Aims of the paper (p127)1. Do results in oral & written exams differ? Does it make any difference if questions are assessing (abstract) scientific questions or personal skills?2. Do students find oral exams more stressful?3. What do students think are the relative merits of oral v written assessments?
  7. 7. Students involved in studyThree cohorts:• Evolutionary biology (yr 1 of 4, Scottish system) n = 99 (72% , 28%  )• Applied terrestrial ecology (yr 3 of 4, 2008) n = 29 (21 , 8  )• Applied terrestrial ecology (yr 3 of 4, 2007) n = 18 (11 , 7  )
  8. 8. Cohort 1: Year 1_08 students (1)• Formative test - 7 x SAQ - understanding, not fact regurgitation• Allocated randomly to either “oral” or “written” - “oral” = 15 mins, 1-2-1 with examiner (n=10) standardised interview protocol & mark sheet interviewers met a priori re model answers - “written” = 30 mins, exam conditions - both tests same questions• Given 4 weeks notice of allocation, possible to swap - 2 swapped oral to written (non-English) - 2 swapped written to oral (dyslexic)
  9. 9. Cohort 1: Year 1_08 students (2)• 7 questions marked 0 to 2: 0 = no answer, completely wrong 1 = partially right 2 = correct, inc all key points t-test on mean scores• “oral” – short notes & marked before next candidate “written” – double-blind marked• single additional question: “how nervous were you about taking this test?” 1 = not at all nervous 4 = very nervous chi-squared test
  10. 10. Cohort 1: Year 1_08 students (3)• n=91 (45 oral, 46 written)• Overall difference mean for written = 6.24 (out of 14) mean for oral = 8.17, i.e 1.93 more two-sample t-test: t-value = 3.46, df = 89, p-value = 0.001• Gender differences female oral > written 2.03 male oral > written 1.50
  11. 11. Cohort 1: Year 1_08 students (4)• Nervousness• Students were more nervous about oral exam• 2 = 6.78, df = 3, p-value = 0.079
  12. 12. Cohort 2: Year 3_08 students (1)• Prior to this study “Applied terrestrial ecology” module already assessed by 4-question oral examination 2 x “scientific analysis” 2 x “personal and professional development” Questions tailored to individual students• For research, addition of 2 written questions 1 x “scientific analysis” 1 x “personal and professional development”• 8 mins written assessment before 15 mins oral exam
  13. 13. Cohort 2: Year 3_08 students (2)• All questions (oral and written) assessed on 7-point scale (0 to 6)• 1/3 of oral exams double-marked (i.e. 2 interviewers)• All written questions double-marked
  14. 14. Cohort 3: Year 3_07 students• Same 4-question oral examination as in 2008, i.e. 2 x “scientific analysis” 2 x “personal and professional development”• No written examination
  15. 15. Qualitative assessment (1)• 2007 Yr 3 students = post-test, post-feedback email to group• “Please describe how you felt the interview went. In particular, how did you perform compared to a more conventional assessment (such as a written exam)? What do you think the advantages an disadvantages of being assessed by interview are, and what lessons you learn from the experience?”• 15 out of 18 responded
  16. 16. Qualitative assessment (2)• 2008 Yr 1 students = selected students (number unspecified) invited to attend focus group• Intention was for focus group to include representation from diverse populations in cohort: - male & female - home & international - mature students & school leavers• Sandwich lunch offered• Conversation recorded, permission for anonymised quotation obtained• 3 students participated
  17. 17. Qualitative assessment (3)• 2008 Yr 3 students = selected students (number unspecified) invited to attend focus group• Same procedure as for Yr 1 focus group• 4 students participated• Recordings from both focus groups transcribed and subjected to thematic analysis
  18. 18. Summary of available data• Yr 1 2008 cohort - 99 in group - 91 took test (either written or oral) - 3 offered qualitative data (focus group)• Yr 3 2008 cohort - 29 in group - 24 took test (written and oral) - 4 offered qualitative data (focus group)• Yr 3 2007 cohort - ? in group - 18 took test (oral only) - 15 offered qualitative data (email survey)
  19. 19. Nervousness• Students reported being more nervous about oral exams (Yr 1 specific question, Yr 3 emails)• Are nerves just caused by lack of familiarity?• Authors note nerves do not necessarily hinder performance (and may bolster it)• David Cameron‟s “full-bladder” technique?
  20. 20. What aspect of the paperwere done well/less well?
  21. 21. Good aspects of paper• Posed an interesting and relevant question re contemporary assessment• Involved a combination of quantitative and qualitative data, including more than one cohort and/or year• Demonstrated a statistically-significant difference between performance oral > written test• Included randomised test (Yr 1 either/or) and a paired test (Yr 3 both/and)
  22. 22. Zadie Smith quote• Character using words „modern‟ and „science‟ “as if someone had lent him the words and made him swear not to break them ”• Reminiscent of final year library project where student who “owned” language in science chapter but phrasing of ethics chapter seemed “borrowed” (as if working in a foreign language)
  23. 23. Questions/Concerns about paper• Smallest cohort disproportionately represented in qualitative review?• 2007 cohort emailed responses – not anonymous? Fear of impact on (future) modules?• Question to Yr 3 2007 asked about “interview” not “oral examination”. Might this influence outcome?• What were the focus group questions? Appendix?• What themes emerged from “thematic analysis” other than anxiety?
  24. 24. Reflections re our context• Current Strategic Review of Bioscience – should we make more use of oral assessments?• We have questions as part of project seminar, but what other applications might be appropriate?• Would a switch to more oral assessment: - be better for students? - be better for us? - give a truer picture of abilities? - be a capitulation to student whim?• Oral assessment allows interaction that reveals if knowledge has depth or is a veneer
  25. 25. Acknowledgement & Disclaimer• These slides are from a presentation to the School of Biological Sciences Pedagogic Research group at the University of Leicester, 17th January 2012• With the exception of slides with a green border (like this one) all the credit for the content is due to Mark Huxham and colleagues, authors of the paper under discussionOriginal citation: Mark Huxham, Fiona Campbell & Jenny WestwoodAssessment and Evaluation in Higher Education 37:125-136 (February 2012)