1. Advancing Online Assessment in Medical Education Greg Doyle, Faculty of Health Sciences Stephen Marquard, Centre for Educational Technology (Advised and enthused by Rachel Weiss, Faculty of Health Sciences) University of Cape Town June 2010

2. Overview Context About Extended Matching Items EMIs in Tests & Quizzes Managing MCQs in a Question Bank Sharing MCQs between institutions 2 11th Sakai Conference - June 15-17, 2010

4. Introducing the use of Extended Matching Item (R-type) MCQs

5. A Question Bank application

6. Author, manage, reuse and share MCQs

7. Collaborating to develop and share MCQs3 11th Sakai Conference - June 15-17, 2010

8. What is an EMI? Extended Matching Item, aka R-type MCQ EMI = R-Type MCQ (vs A-Type MCQ) An EMI has: A theme A list of options (text or labelled image) A lead-in statement A list of items 11th Sakai Conference - June 15-17, 2010 4

9. Sample EMI 11th Sakai Conference - June 15-17, 2010 5 This example shows options labelled1, 2, 3… but more conventionally should be labelled a, b, c …

10. EMI variants Items may require choosing: One option Multiple options (e.g. a, e, g) independent of order [in previous example, options 1,5,7] Options in a specific order (e.g. g, a, e) Implications for grading Limits the ability to use older question types (e.g. fill-in-the-blank) for EMIs 11th Sakai Conference - June 15-17, 2010 6

11. EMIs Advantages More discriminating Reproducibility using one-third fewer items Lower probability of guessing the correct answer Functional distractors Reusable by just writing a new stem/item Disadvantages Analysis, cannot test all content Not much software support yet

12. EMIs in Tests & Quizzes What we used to do Work around using fill-in-the-blank So…http://jira.sakaiproject.org/browse/SAK-17243 : Work underway to add support for EMIs as a new question type in Tests & Quizzes, including statistics and analysis. When complete and reviewed by Stanford, should become available in a future T&Q release.

13. Beyond Question Pools Lifecycle of a question Now: authored, lives in a Question Pool, delivered in assessments Should be: created, reviewed, tagged (metadata), selected for assessments, delivered, updated with delivery history (e.g. difficulty, discrimination), updated, shared, reused 11th Sakai Conference - June 15-17, 2010 9

14. We imagine a Question Bank which… Supports A-type (traditional) MCQs, EMIs, F-format questions (progressive sequence) Supports embedded rich media (image/video) Supports authoring/revision/approval workflow Includes flexible question metadata Enables blueprinting of assessments by selecting sets of questions to a target level of difficulty Records statistics from the delivery history of the question Enables internal and external reuse and sharing 11th Sakai Conference - June 15-17, 2010 10

15. Sharing MCQs 11th Sakai Conference - June 15-17, 2010 11 Question BankUniversity A Question Bank UCT Public OER Question Bank Assessment engine (e.g. Sakai T&Q) << Questions >> Delivery history and statistics Question Bank University B

16. We’d love to hear from you! If you are now or anticipate using Extended Matching Items for online assessment If you’d like to find ways to increase sharing and reuse of assessment items. If the type of Question Bank we’ve envisaged sounds attractive to you. If you want to help build a Question Bank tool / application. 11th Sakai Conference - June 15-17, 2010 12

17. Contact us stephen.marquard@uct.ac.za gregory.doyle@uct.ac.za 11th Sakai Conference - June 15-17, 2010 13