Your SlideShare is downloading. ×
Salt b2 van keulen
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Salt b2 van keulen

129
views

Published on

Published in: Education, Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
129
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Biology, zoology, ecology, marine biology, biochem, genetics – novice and general exam recapping alevel materialLevel 2 more experienced at answering mcqImmediately after completing the test and following the announcement of the results of the testsDue to the test being voluntary, incentives or freebies were used to promot participation in the tests – this included text books from staff and publishers, book tokens, bag for life, sweets and practice for the real exams in january
  • Remark Office OMR software package,
  • Null hypothesis is is the distribution of the l1 sa marks not different from the norm
  • A paired t-test showed that for the level 1 and level 2 student cohorts, performance was significantly higher for elimination test than for the single answer MCQ test (L1=7%,L2=9%)(t135 = 5.325, P <0.0005 and t39= 2.360, P = 0.023 respectively)
  • L1:The male students scored on average 8% more marks than the female students in the single answer test. The difference was not significant (t134 = 1.484, P = 0.140)The male students scored on average 7.9% more marks than the female students in the elimination test. The difference was not significant(t134 = 1.441, P = 0.152)L2:Single AnswerThe female students scored on average 2% higher marks than the male students in the single answer test. A difference that an independent t-test showed was not significant(t 38= 0.113, P = 0.520)Elimination TestThe female students scored on average 11% higher marks than the male students in the elimination test. A difference that an independent t-test showed was not significant(t38 = 0.722, P = 0.629)
  • Types of questions included:
  • 5 questions chosen from 43To test whether the sample median of one measurement taken on a single population is different from an expected value. One-sample t test
  • Transcript

    • 1. Improving performance and the student assessment experience:multi-choice testing which rewards partial knowledge increases satisfaction and reduces anxiety without introducing gender biasDr Geertje van Keulen College of Medicine
    • 2. MCQ assessments in higher education• MCQs are used to objectively measure: o Factual knowledge o Learning ability o complex, high-level learning outcomes• Often the sole assessment of theoretical knowledge forlevel 1 UG students in large classes• Variety of different types of mcq tests exist Ben-Simmon et al 1997 Bradbard et al 2004 Ng & Chan 2009a
    • 3. MCQ with Single (best) Answering method Most commonly used MCQ test Student marks only one option, the correct answer Negative marking inhibits ‘pure guesswork’ Several issues: o Lower student confidence in answering o No chance to show partial knowledge o Reduced test score reliability through ‘educated guesswork’ o Reduced student satisfaction o Unfair? Gender bias?
    • 4. MCQ with Elimination Answering method Discriminate between all possible levels of knowledge(full, partial, and absence of knowledge, fully and partiallymisinformed) Students asked to eliminate up to 4 possible incorrect answers Elimination of the correct answer incurs a penalty Test score reliability improved as guessing is not required, guessing isdiscouraged Student confidence improved? Improved tool for teaching staff? Jennings & Bush 2006, Ben-Simon et al 1997, Coombs et al 1956, Dressel & Schmidt 1953
    • 5. Fairness of MCQ testing?Comparison of MCQ test performance in some subject areassuggested introduction of gender bias (Ng & Chan 2009b) However, introduction of gender bias not absolute but appearto depend on: o subject area o instruction/scoring condition o question difficulty Data missing for life sciences subject area
    • 6. Objectives – Student related• To compare student performance in life sciences inidentically-worded single and elimination answer MCQ testswith negative marking• To assess whether single and elimination answer MCQ testsin life sciences result in gender bias in performance overall• To assess student satisfaction and anxiety/stress levels forand between each type of MCQ test
    • 7. Approach• MCQ tests were voluntary• Students were examined on course material taught in the first 3 weeks of the module• Tested in week 5 of the module
    • 8. Approach• Level 1 students consisted of the whole life science cohort• Level 2 cohort of biochemistry and genetics students• Student evaluation surveys ‐ Anxiety ‐ Stress ‐ Satisfaction• Incentives
    • 9. Answer methods and Scoring Mechanisms• 1 test question sheet with 2 answer sheets o Single answer method o Elimination answer method o Sheets were scanned, electronically read, scored and input into Access database• Scoring grid for MCQ tests with 5 possible answers:
    • 10. Datasets• 142 students (72%) participated in the Level 1 MCQ tests ‐ 138 paired answer sheets incorporating 25 questions ‐ 63 males and 75 females ‐ 138 unpaired student evaluation questionnaires• 40 students (80%) participated in the Level 2 MCQ tests ‐ 40 paired answer sheets incorporating 25 questions ‐ 26 males and 14 females ‐ 40 unpaired student evaluation questionnaires• Further data analysis with SSPSv19
    • 11. MCQ scores distribution is normally distributed• Student performance betweenthe two types of tests isnormally distributed for level 1and level 2 students• Kolgomorov-Smirnov testconducted (p>0.05 for all)MCQ test P-valueL1 SA 0.2L1 ET 0.08L2 SA 0.2 Not significantly different from theL2 ET 0.2 normal distribution
    • 12. Comparison of student cohort performance•Cohort average scores are 52 50 47.4relatively poor 48 44.0 % score 46 44•Students perform 42 40statistically significantly 38better in Elimination tests single answer elimination 40 MCQ testthan in Single Answer tests 39 38 38.0 37 36 34.6 % Score•L1=rel. 7% better 35 34 33•L2=rel. 9% better 32 31 30 Single Answer EliminationPaired t-tests MCQ TestL1: P<0.0005 and Mean ( SE) of the performanceL2: P=0.023
    • 13. Does MCQ testing introduce gender bias in the life sciences subject area?Independent t-tests were conducted to determine if gendereffects performance:L1: males slightly better in both tests, but not significantL2: females slightly better in both tests, but not significantNO significant gender bias in Single Answer MCQ testingNO significant gender bias in Elimination MCQ testingNO significant gender bias in Level 1 Life Sciences studentcohorts (9 degrees)NO significant gender bias in Level 2 Biochemistry & Geneticscohorts
    • 14. Student surveys on experiences• Immediately after sitting the MCQ tests, students completed anevaluation survey• MCQ evaluation questions investigated: o The students’ experience/attitude/emotion towards each type of MCQ test o Comparison of the two answering options• A second survey was completed by the students following theannouncement of test results
    • 15. Student experience survey questions• The answering options were confusing• There is a high chance of getting answers right• I got distracted by thinking about the best tactics for getting a high mark• It makes you think more about your answers• It made me feel more relaxed, knowing that I can get a reasonable mark•I was scared to answer a question• I was confident to answer a question• It made me feel motivated• My stress levels were high• It is a fair test• Loosing marks for guessing detracted from the legitimate marks for knowingthe right answers to some questions• I prefer single answer testing• I prefer elimination testing• Many more....
    • 16. Student satisfaction surveys• Students gave scores of: 1=strongly agree 2=agree 3=neutral 4=disagree 5=strongly disagree 6=not applicable/don’t know• One sample sign test performed with Null hypothesis: the median ofthe scores was 3 or neutral 60 50 Q14 & Q32 40 % Results 30 L1 ET 20 L2 ET 10 L1 NC 0 L2 NC
    • 17. Selection of student experience responses (significant p<0.05)• Most students agreed they •prefered Elimination Testing over Single Answer testing •felt Elimination Testing better reflects their knowledge •felt both types of tests are fair •felt scared in Single Answer testing but were neutral for Elimination •enhanced their critical thinking skills by Elimination •felt confident to answer in both styles•Most students were neutral •Towards loosing marks for guessing detracted from the legitimate marks for knowing the right answers to some questions (except L2 for Single Answer) •about the questions being easy to answer •about being motivated by both types of answering methods •About Single Answer method enhancing their critical thinking skills
    • 18. Conclusions• Both student cohorts get significant advantage answering eliminationMCQ tests compared to single answer tests with negative marking (L1 – P<0.0005 and L2 - P=0.023)• There is no significant difference in performance between genders ineither cohort for either MCQ test (L1 - P=0.309 and L2 - P=0.779)•Students prefer Elimination testing to Single answer MCQ and feelElimination testing reflects their knowledge better•Students feel scared to answer Single answer MCQs•In general, the student learning experience can be improved through theincorporation of the elimination answering methods in MCQ tests
    • 19. Acknowledgements•Dr Liz Bond - Researcher•Dr Owen Bodger – (bio)statistician•Dr Hugh Jones - Examinations Coordinator•Dr Colin Restall – Biochemistry Module Coordinator• Dr Ed Dudley – Biochemistry Module Contributor• Dr Jess Murtagh – Learning Technologist• Support staff from Education Unit• 7 further academic staff from Genetics and Biochemistry• With many thanks to the tutors in BiosciencesContact us at: g.van.keulen@swansea.ac.uk or a.e.bond@swansea.ac.uk