Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Salt b2 van keulen

389 views

Published on

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

Salt b2 van keulen

  1. 1. Improving performance and the student assessment experience:multi-choice testing which rewards partial knowledge increases satisfaction and reduces anxiety without introducing gender biasDr Geertje van Keulen College of Medicine
  2. 2. MCQ assessments in higher education• MCQs are used to objectively measure: o Factual knowledge o Learning ability o complex, high-level learning outcomes• Often the sole assessment of theoretical knowledge forlevel 1 UG students in large classes• Variety of different types of mcq tests exist Ben-Simmon et al 1997 Bradbard et al 2004 Ng & Chan 2009a
  3. 3. MCQ with Single (best) Answering method Most commonly used MCQ test Student marks only one option, the correct answer Negative marking inhibits ‘pure guesswork’ Several issues: o Lower student confidence in answering o No chance to show partial knowledge o Reduced test score reliability through ‘educated guesswork’ o Reduced student satisfaction o Unfair? Gender bias?
  4. 4. MCQ with Elimination Answering method Discriminate between all possible levels of knowledge(full, partial, and absence of knowledge, fully and partiallymisinformed) Students asked to eliminate up to 4 possible incorrect answers Elimination of the correct answer incurs a penalty Test score reliability improved as guessing is not required, guessing isdiscouraged Student confidence improved? Improved tool for teaching staff? Jennings & Bush 2006, Ben-Simon et al 1997, Coombs et al 1956, Dressel & Schmidt 1953
  5. 5. Fairness of MCQ testing?Comparison of MCQ test performance in some subject areassuggested introduction of gender bias (Ng & Chan 2009b) However, introduction of gender bias not absolute but appearto depend on: o subject area o instruction/scoring condition o question difficulty Data missing for life sciences subject area
  6. 6. Objectives – Student related• To compare student performance in life sciences inidentically-worded single and elimination answer MCQ testswith negative marking• To assess whether single and elimination answer MCQ testsin life sciences result in gender bias in performance overall• To assess student satisfaction and anxiety/stress levels forand between each type of MCQ test
  7. 7. Approach• MCQ tests were voluntary• Students were examined on course material taught in the first 3 weeks of the module• Tested in week 5 of the module
  8. 8. Approach• Level 1 students consisted of the whole life science cohort• Level 2 cohort of biochemistry and genetics students• Student evaluation surveys ‐ Anxiety ‐ Stress ‐ Satisfaction• Incentives
  9. 9. Answer methods and Scoring Mechanisms• 1 test question sheet with 2 answer sheets o Single answer method o Elimination answer method o Sheets were scanned, electronically read, scored and input into Access database• Scoring grid for MCQ tests with 5 possible answers:
  10. 10. Datasets• 142 students (72%) participated in the Level 1 MCQ tests ‐ 138 paired answer sheets incorporating 25 questions ‐ 63 males and 75 females ‐ 138 unpaired student evaluation questionnaires• 40 students (80%) participated in the Level 2 MCQ tests ‐ 40 paired answer sheets incorporating 25 questions ‐ 26 males and 14 females ‐ 40 unpaired student evaluation questionnaires• Further data analysis with SSPSv19
  11. 11. MCQ scores distribution is normally distributed• Student performance betweenthe two types of tests isnormally distributed for level 1and level 2 students• Kolgomorov-Smirnov testconducted (p>0.05 for all)MCQ test P-valueL1 SA 0.2L1 ET 0.08L2 SA 0.2 Not significantly different from theL2 ET 0.2 normal distribution
  12. 12. Comparison of student cohort performance•Cohort average scores are 52 50 47.4relatively poor 48 44.0 % score 46 44•Students perform 42 40statistically significantly 38better in Elimination tests single answer elimination 40 MCQ testthan in Single Answer tests 39 38 38.0 37 36 34.6 % Score•L1=rel. 7% better 35 34 33•L2=rel. 9% better 32 31 30 Single Answer EliminationPaired t-tests MCQ TestL1: P<0.0005 and Mean ( SE) of the performanceL2: P=0.023
  13. 13. Does MCQ testing introduce gender bias in the life sciences subject area?Independent t-tests were conducted to determine if gendereffects performance:L1: males slightly better in both tests, but not significantL2: females slightly better in both tests, but not significantNO significant gender bias in Single Answer MCQ testingNO significant gender bias in Elimination MCQ testingNO significant gender bias in Level 1 Life Sciences studentcohorts (9 degrees)NO significant gender bias in Level 2 Biochemistry & Geneticscohorts
  14. 14. Student surveys on experiences• Immediately after sitting the MCQ tests, students completed anevaluation survey• MCQ evaluation questions investigated: o The students’ experience/attitude/emotion towards each type of MCQ test o Comparison of the two answering options• A second survey was completed by the students following theannouncement of test results
  15. 15. Student experience survey questions• The answering options were confusing• There is a high chance of getting answers right• I got distracted by thinking about the best tactics for getting a high mark• It makes you think more about your answers• It made me feel more relaxed, knowing that I can get a reasonable mark•I was scared to answer a question• I was confident to answer a question• It made me feel motivated• My stress levels were high• It is a fair test• Loosing marks for guessing detracted from the legitimate marks for knowingthe right answers to some questions• I prefer single answer testing• I prefer elimination testing• Many more....
  16. 16. Student satisfaction surveys• Students gave scores of: 1=strongly agree 2=agree 3=neutral 4=disagree 5=strongly disagree 6=not applicable/don’t know• One sample sign test performed with Null hypothesis: the median ofthe scores was 3 or neutral 60 50 Q14 & Q32 40 % Results 30 L1 ET 20 L2 ET 10 L1 NC 0 L2 NC
  17. 17. Selection of student experience responses (significant p<0.05)• Most students agreed they •prefered Elimination Testing over Single Answer testing •felt Elimination Testing better reflects their knowledge •felt both types of tests are fair •felt scared in Single Answer testing but were neutral for Elimination •enhanced their critical thinking skills by Elimination •felt confident to answer in both styles•Most students were neutral •Towards loosing marks for guessing detracted from the legitimate marks for knowing the right answers to some questions (except L2 for Single Answer) •about the questions being easy to answer •about being motivated by both types of answering methods •About Single Answer method enhancing their critical thinking skills
  18. 18. Conclusions• Both student cohorts get significant advantage answering eliminationMCQ tests compared to single answer tests with negative marking (L1 – P<0.0005 and L2 - P=0.023)• There is no significant difference in performance between genders ineither cohort for either MCQ test (L1 - P=0.309 and L2 - P=0.779)•Students prefer Elimination testing to Single answer MCQ and feelElimination testing reflects their knowledge better•Students feel scared to answer Single answer MCQs•In general, the student learning experience can be improved through theincorporation of the elimination answering methods in MCQ tests
  19. 19. Acknowledgements•Dr Liz Bond - Researcher•Dr Owen Bodger – (bio)statistician•Dr Hugh Jones - Examinations Coordinator•Dr Colin Restall – Biochemistry Module Coordinator• Dr Ed Dudley – Biochemistry Module Contributor• Dr Jess Murtagh – Learning Technologist• Support staff from Education Unit• 7 further academic staff from Genetics and Biochemistry• With many thanks to the tutors in BiosciencesContact us at: g.van.keulen@swansea.ac.uk or a.e.bond@swansea.ac.uk

×