Salt b2 van keulen


Published on

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Biology, zoology, ecology, marine biology, biochem, genetics – novice and general exam recapping alevel materialLevel 2 more experienced at answering mcqImmediately after completing the test and following the announcement of the results of the testsDue to the test being voluntary, incentives or freebies were used to promot participation in the tests – this included text books from staff and publishers, book tokens, bag for life, sweets and practice for the real exams in january
  • Remark Office OMR software package,
  • Null hypothesis is is the distribution of the l1 sa marks not different from the norm
  • A paired t-test showed that for the level 1 and level 2 student cohorts, performance was significantly higher for elimination test than for the single answer MCQ test (L1=7%,L2=9%)(t135 = 5.325, P <0.0005 and t39= 2.360, P = 0.023 respectively)
  • L1:The male students scored on average 8% more marks than the female students in the single answer test. The difference was not significant (t134 = 1.484, P = 0.140)The male students scored on average 7.9% more marks than the female students in the elimination test. The difference was not significant(t134 = 1.441, P = 0.152)L2:Single AnswerThe female students scored on average 2% higher marks than the male students in the single answer test. A difference that an independent t-test showed was not significant(t 38= 0.113, P = 0.520)Elimination TestThe female students scored on average 11% higher marks than the male students in the elimination test. A difference that an independent t-test showed was not significant(t38 = 0.722, P = 0.629)
  • Types of questions included:
  • 5 questions chosen from 43To test whether the sample median of one measurement taken on a single population is different from an expected value. One-sample t test
  • Salt b2 van keulen

    1. 1. Improving performance and the student assessment experience:multi-choice testing which rewards partial knowledge increases satisfaction and reduces anxiety without introducing gender biasDr Geertje van Keulen College of Medicine
    2. 2. MCQ assessments in higher education• MCQs are used to objectively measure: o Factual knowledge o Learning ability o complex, high-level learning outcomes• Often the sole assessment of theoretical knowledge forlevel 1 UG students in large classes• Variety of different types of mcq tests exist Ben-Simmon et al 1997 Bradbard et al 2004 Ng & Chan 2009a
    3. 3. MCQ with Single (best) Answering method Most commonly used MCQ test Student marks only one option, the correct answer Negative marking inhibits ‘pure guesswork’ Several issues: o Lower student confidence in answering o No chance to show partial knowledge o Reduced test score reliability through ‘educated guesswork’ o Reduced student satisfaction o Unfair? Gender bias?
    4. 4. MCQ with Elimination Answering method Discriminate between all possible levels of knowledge(full, partial, and absence of knowledge, fully and partiallymisinformed) Students asked to eliminate up to 4 possible incorrect answers Elimination of the correct answer incurs a penalty Test score reliability improved as guessing is not required, guessing isdiscouraged Student confidence improved? Improved tool for teaching staff? Jennings & Bush 2006, Ben-Simon et al 1997, Coombs et al 1956, Dressel & Schmidt 1953
    5. 5. Fairness of MCQ testing?Comparison of MCQ test performance in some subject areassuggested introduction of gender bias (Ng & Chan 2009b) However, introduction of gender bias not absolute but appearto depend on: o subject area o instruction/scoring condition o question difficulty Data missing for life sciences subject area
    6. 6. Objectives – Student related• To compare student performance in life sciences inidentically-worded single and elimination answer MCQ testswith negative marking• To assess whether single and elimination answer MCQ testsin life sciences result in gender bias in performance overall• To assess student satisfaction and anxiety/stress levels forand between each type of MCQ test
    7. 7. Approach• MCQ tests were voluntary• Students were examined on course material taught in the first 3 weeks of the module• Tested in week 5 of the module
    8. 8. Approach• Level 1 students consisted of the whole life science cohort• Level 2 cohort of biochemistry and genetics students• Student evaluation surveys ‐ Anxiety ‐ Stress ‐ Satisfaction• Incentives
    9. 9. Answer methods and Scoring Mechanisms• 1 test question sheet with 2 answer sheets o Single answer method o Elimination answer method o Sheets were scanned, electronically read, scored and input into Access database• Scoring grid for MCQ tests with 5 possible answers:
    10. 10. Datasets• 142 students (72%) participated in the Level 1 MCQ tests ‐ 138 paired answer sheets incorporating 25 questions ‐ 63 males and 75 females ‐ 138 unpaired student evaluation questionnaires• 40 students (80%) participated in the Level 2 MCQ tests ‐ 40 paired answer sheets incorporating 25 questions ‐ 26 males and 14 females ‐ 40 unpaired student evaluation questionnaires• Further data analysis with SSPSv19
    11. 11. MCQ scores distribution is normally distributed• Student performance betweenthe two types of tests isnormally distributed for level 1and level 2 students• Kolgomorov-Smirnov testconducted (p>0.05 for all)MCQ test P-valueL1 SA 0.2L1 ET 0.08L2 SA 0.2 Not significantly different from theL2 ET 0.2 normal distribution
    12. 12. Comparison of student cohort performance•Cohort average scores are 52 50 47.4relatively poor 48 44.0 % score 46 44•Students perform 42 40statistically significantly 38better in Elimination tests single answer elimination 40 MCQ testthan in Single Answer tests 39 38 38.0 37 36 34.6 % Score•L1=rel. 7% better 35 34 33•L2=rel. 9% better 32 31 30 Single Answer EliminationPaired t-tests MCQ TestL1: P<0.0005 and Mean ( SE) of the performanceL2: P=0.023
    13. 13. Does MCQ testing introduce gender bias in the life sciences subject area?Independent t-tests were conducted to determine if gendereffects performance:L1: males slightly better in both tests, but not significantL2: females slightly better in both tests, but not significantNO significant gender bias in Single Answer MCQ testingNO significant gender bias in Elimination MCQ testingNO significant gender bias in Level 1 Life Sciences studentcohorts (9 degrees)NO significant gender bias in Level 2 Biochemistry & Geneticscohorts
    14. 14. Student surveys on experiences• Immediately after sitting the MCQ tests, students completed anevaluation survey• MCQ evaluation questions investigated: o The students’ experience/attitude/emotion towards each type of MCQ test o Comparison of the two answering options• A second survey was completed by the students following theannouncement of test results
    15. 15. Student experience survey questions• The answering options were confusing• There is a high chance of getting answers right• I got distracted by thinking about the best tactics for getting a high mark• It makes you think more about your answers• It made me feel more relaxed, knowing that I can get a reasonable mark•I was scared to answer a question• I was confident to answer a question• It made me feel motivated• My stress levels were high• It is a fair test• Loosing marks for guessing detracted from the legitimate marks for knowingthe right answers to some questions• I prefer single answer testing• I prefer elimination testing• Many more....
    16. 16. Student satisfaction surveys• Students gave scores of: 1=strongly agree 2=agree 3=neutral 4=disagree 5=strongly disagree 6=not applicable/don’t know• One sample sign test performed with Null hypothesis: the median ofthe scores was 3 or neutral 60 50 Q14 & Q32 40 % Results 30 L1 ET 20 L2 ET 10 L1 NC 0 L2 NC
    17. 17. Selection of student experience responses (significant p<0.05)• Most students agreed they •prefered Elimination Testing over Single Answer testing •felt Elimination Testing better reflects their knowledge •felt both types of tests are fair •felt scared in Single Answer testing but were neutral for Elimination •enhanced their critical thinking skills by Elimination •felt confident to answer in both styles•Most students were neutral •Towards loosing marks for guessing detracted from the legitimate marks for knowing the right answers to some questions (except L2 for Single Answer) •about the questions being easy to answer •about being motivated by both types of answering methods •About Single Answer method enhancing their critical thinking skills
    18. 18. Conclusions• Both student cohorts get significant advantage answering eliminationMCQ tests compared to single answer tests with negative marking (L1 – P<0.0005 and L2 - P=0.023)• There is no significant difference in performance between genders ineither cohort for either MCQ test (L1 - P=0.309 and L2 - P=0.779)•Students prefer Elimination testing to Single answer MCQ and feelElimination testing reflects their knowledge better•Students feel scared to answer Single answer MCQs•In general, the student learning experience can be improved through theincorporation of the elimination answering methods in MCQ tests
    19. 19. Acknowledgements•Dr Liz Bond - Researcher•Dr Owen Bodger – (bio)statistician•Dr Hugh Jones - Examinations Coordinator•Dr Colin Restall – Biochemistry Module Coordinator• Dr Ed Dudley – Biochemistry Module Contributor• Dr Jess Murtagh – Learning Technologist• Support staff from Education Unit• 7 further academic staff from Genetics and Biochemistry• With many thanks to the tutors in BiosciencesContact us at: or