1. Jane Holland provides an overview of writing single best answer multiple choice questions, focusing on response formats, stimulus formats, and avoiding technical item flaws. 2. Key principles for writing high quality MCQ items include using a clinical vignette as the stem, homogeneous response options of similar length and format, and avoiding absolutes or negative questions. 3. Considerations for both authors and candidates are discussed to minimize construct irrelevant variance and support valid assessment of the intended construct.

1. Multiple Choice Questions
Writing Single Best Answer Qs
Moodlemoot UK & Ireland 2020
Jane Holland PgDipEd MD PhD MRCSI
RCSI Royal College of Surgeons in Ireland Coláiste Ríoga na Máinleá in Éirinn
including slides from Richard Arnett, PhD

2. Jane Holland
https://www.nbme.org/sites/default/files/2020-01/IWW_Gold_Book.pdf
➢ Item-writing – Single Best Answer (SBA) MCQs
❖ Response formats
❖ Stimulus formats
❖ Technical item flaws

3. MCQ response formats
• True / False
➢ The candidate is required to make a determination
as to whether each option provided is true or false
• One Best Answer
➢ The candidate is required to indicate a single
response i.e. the most likely diagnosis
Schuwirth: Med Educ. 2004 Sep;38(9):974-9.
https://www.nbme.org/sites/default/files/2020-01/IWW_Gold_Book.pdf Jane Holland

4. True / false
False True
Q1. The venae cordis minimae are small veins present in
the walls of all four chambers and open directly into
these respective chambers (T/F).
Q2. The great cardiac vein lies in the anterior
interventricular groove of the heart, alongside the
anterior interventricular artery (T/F).
Q3. The cystic artery is a branch of the common hepatic
artery, which arises from the coeliac trunk (T/F).
Jane Holland

5. True / false vs. Single Best Answer
Q. The gallbladder is typically supplied by the
cystic artery. The cystic artery is typically a
branch from which of the following arteries?
A. Common hepatic
B. Gastroduodenal
C. Left hepatic
D. Right hepatic
E. Superior mesenteric
Jane Holland
✓
DBACE

6. Q. Which of the following statements about Ireland
is true?
A. Tayto exports exceed € 1 Bn
B. The climate is sub-tropical
C. The main export is zinc
D. The population is 5 million
E. Rainy weather frequently occurs
True / false vs. Single Best Answer
Trade / exports
Weather
Demographics
D
C
B
A
E
Jane Holland

7. ✓
True / false vs. Single Best Answer
Q. What colour is the sky?
A. A hazy shade of winter
B. Blue
C. Grey
D. Orange
E. Red
Jane Holland

8. Q. Which of the following calls is a rugby union referee most
likely to make in the case of an intercepted knock-on?
A. Advantage
B. Line out
C. Penalty kick
D. Penalty try
E. Scrum
Q. Which of the following was an original recording by
Meatloaf?
A. Bat out of Hell
B. One out of three ain’t bad
C. Paradise by the Kitchen Light
D. Stand by me
E. Total Eclipse of the Heart
Jane Holland
The “cover test” – a guideline...

9. Q. Propranolol, a competitive antagonist at beta adrenergic receptors in
the heart:
A. Has affinity for its receptor but no intrinsic activity
B. Causes an increase in heart rate and contractility
C. Causes a reduction in the maximal response to adrenaline but
does not affect the Kd
D. Causes an apparent increase in agonist potency
E. Binds irreversibly to beta adrenergic receptors
Q. Propranolol, a competitive antagonist at beta adrenergic receptors in
the heart, causes a reduction in the maximal response to adrenaline
but does not affect the Kd (T/F)
Q. Propranolol has affinity for its receptor but no intrinsic activity (T/F)
Q. Propranolol is a competitive antagonist at beta adrenergic receptors in
the heart. Which of the following statements about this competitive
antagonist is correct? (2008)
Jane Holland
The “cover test” – a guideline...

10. MCQs – how many options?
Jane Holland
Swanson et al; Academic Medicine. 2005;80(10):S93-S6.
Tarrant & Ware; Nurse Education Today. 2010;30(6):539-43.

11. Stem /
vignette
Question
(Lead-in)
Options
Single Best Answer
Jane Holland

12. MCQ stimulus formats (stem or vignette)
➢ Context-rich
❖ Tests reasoning
or application
➢ Context -free
❖ Tests factual
knowledge
Schuwirth: Med Educ. 2004 Sep;38(9):974-9.
Kennedy, 2007
Jane Holland
Creating
Evaluating
Analysing
Applying
Understanding
Remembering

13. NBME Manual p 35 - 49
Q. Which dermatome is at the umbilicus?
A. T6
B. T8
C. T10
D. T12
E. L2
Q. A 65-year-old librarian presents with skin lesions
suggestive of a Herpes zoster infection affecting
the dermatome which includes the skin of the
umbilicus.
➢ Which segmental spinal nerve is associated with
that dermatome?
Jane Holland

14. Q. A 64-year-old man is brought to the emergency
department by his wife, who has noticed that his left
eyelid is drooping. On examination, you find that
he has abnormal ipsilateral pupillary reflexes and
abnormally dry skin.
Jane Holland
A. Ciliary
B. Geniculate
C. Otic
D. Pterygopalatine
E. Stellate
➢ These symptoms are most likely to due damage or
compression to which of the following ganglia?
➢ While attending an anatomy tutorial, you are asked to
name the ganglion that transmits sympthetic fibres to
the head & neck. What is the correct answer?

15. Jane Holland
Q. A 50-year-old man is seen in the gastroenterology
clinic with intermittent epigastric pains for the past 6
months. An endoscopy reveals an ulcer in the
cardia of his stomach. Blood supply to this area is
from which of the following arteries?
A. Left gastric
B. Right gastric
C. Short gastric
D. Left gastroepiploic
E. Right gastroepiploic
*Gastric acid secretion is increased by which hormone?
*Which treatment should the patient be prescribed?

16. Stem
Lead-in
Options
➢ Clinical (or contextual) vignette
• Age / gender
• Presenting Problem / history
• Examination findings / test results
➢ Which of the following is the most:
• Likely diagnosis
• Appropriate treatment
➢ Homogenous
• One best option & 4 distractors
• Alphabetical (or logical) order
• All plausible & same length (short!)
NBME Manual p 35 - 49
Jane Holland

17. Word
repeats
Long
correct
answer
Absolutes
Grammatical
clues
Logical
cues
Convergence
Technical
Item flaws
NBME Manual p 19 - 27
Jane Holland

18. Q. A 76-year-old diabetic is brought into the
Emergency Department with suspected diabetic
ketoacidosis. After confirming that she is
hyperglycaemic, the first step in management
should be intravenous administration of:
A. Chest X-ray
B. Electrocardiogram
C. Fasting blood glucose level
D. Insulin
E. Placement of an nasogastric tube
Grammatical clues
Jane Holland

19. Q. It is extremely important for to maintain a
healthy diet. In order to keep the doctor
away, every day you should eat an:
A. Apple
B. Banana
C. Bread roll
D. Chocolate bar
E. Mohito
Grammatical clues
Jane Holland
a / an
plural / singular

20. Word repeats
Q. A 58-year-old man with a history of heavy alcohol
use and previous psychiatric hospitalisation is
confused and agitated. He speaks of experiencing
the world as unreal. This symptom is called:
A. Depersonalisation
B. Derailment
C. Derealisation
D. Focal memory deficit
E. Signal anxiety
Jane Holland

21. Q. Which of the following Moodle activities
allows participants to select one choice
from a list of options?
Word repeats
Jane Holland
A. Checklist
B. Choice
C. Forum
D. Survey
E. Workshop

22. Q. Gavin is running a conference, at which there will
be a number of parallel workshops with limited
places. What Moodle activity could he use to
allow participants to preselect their preferred
workshop?
A. Checklist
B. Choice
C. Forum
D. Survey
E. Workshop
Jane Holland

23. Jane Holland
Q. A 50 year old man is seen in the gastroenterology
OPD with a 6 month history of epigastric pains. An
endoscopy reveals an ulcer in the cardia of his
stomach. Blood supply to this area is from which of
the following arteries?
A. Left gastric
B. Right gastric
C. Short gastric
D. Left gastroepiploic
E. Right gastroepiploic
Blood supply to this area is from what artery?
Blood supply to this area is from which arteries?

24. Logical clues
Q. Crime is:
A. Equally distributed among the social classes
B. Overrepresented among the poor
C. Overrepresented among the middle class &
rich
D. Primarily an indication of psychosocial
maladjustment
E. Reaching a plateau of tolerability for the nation
Jane Holland
Also be wary of overlapping numericals...

25. Long correct answer
Q. Secondary gain is:
A. Synonymous with malingering
B. A frequent problem in OCD
C. A complication of a variety of illnesses and
tends to prolong many of them
D. Never seen in organic brian damage
Jane Holland

26. Convergence Strategy
Q. Local anaesthetics are most effective in the:
A. Anionic form, acting from inside the
nerve membrane
B. Cationic form, acting from inside the
nerve membrane
C. Cationic form, acting from outside the
nerve membrane
D. Uncharged form, acting from inside the
nerve membrane
E. Uncharged form, acting from outside the
nerve membrane
Jane Holland

27. Negative question
Q. You examine a 42-year-old man with a known
history of ankylosing spondylitis. His disease has
progressed over the last number of years. Which
extra articular feature would you not expect to see
in this patient?
A. Achilles tendonitis
B. Anterior uveitis
C. Aortic incompetence
D. Plantar fasciitis
E. Respiratory fibrosis of lower lobes
Jane Holland

28. Absolutes...
Q. Which of the following shoes are never
worn by men?
A. Boots
B. Runners
C. Sandals
D. Stilettos
E. Wellies
Jane Holland

29. Absolutes
Holsgrove G, Medical Education 1998; 32: 343 - 350
Jane Holland
0
10
20
30
40
50
60
0% 1% 5%
“Never”
0
10
20
30
40
50
60
80% 95% 99% 100%
“Always”

30. Jane Holland

31. ➢ Shape  Long vignette & stem  short options
➢ Cover test
• Ideally can guess the best option (or potential options)
before seeing the option list
➢ Use homogenous distracters
• i.e. all arteries, all medications, all diagnoses
• Grammatically consistent
• Logically compatible
• List in logical or alphabetical order
➢ Don’t use vague frequency terms
➢ Never use absolutes!
➢ Do not write a negative question!!
NBME Manual p 35 - 49
Some useful principles
Jane Holland

32. ➢ For candidates
❖ If you don’t know the answer: Choose option ‘C’
❖ “Edge aversion”
❖ The longest option is most likely the correct one…
❖ If an option has a spelling mistake it is probably not
the correct answer
➢ For authors
❖ Follow the style guide (if it exists) and be consistent
❖ Provide normal value ranges (unless this is the
construct being tested)
❖ Be careful with acronyms and/or drug names
❖ Is language a factor?
❖ All to reduce Construct Irrelevant Variance… (CIV)
Final considerations
Richard Arnett

33. ➢ Minimal research / evidence available in assessment…
❖ No consistent influence on item difficulty or discrimination
❖ Inclusion - based upon the principles of constructive alignment
❖ Apply principles from multimedia learning
❖ No redundant or repeated information (i.e.
❖ Or can the question still be answered without the image?
❖ Do textual questions promote more internal visualisation?
❖ Do visual questions promote interpretation of images, or test additional abilities (i.e. spatial)
What about images?
Jane Holland
Mayer & Moreno; Educational Psychologist. 2003; 38(1), 43–52
Vorstenbosch et al. Anatomical Sciences Education. 2014;7:107-116
Holland et al. BMC Med Educ. 2015;15:184-192

34. Examples of Linguistic features (Avenia-Tapper & Llosa, 2015)
Jane Holland
Looking at language…

35. Jane Holland

37. References
➢ https://www.nbme.org/sites/default/files/2020-01/IWW_Gold_Book.pdf
➢ https://www2.nbme.org/nbme/IWTutorial/intro/index.html
➢ Bandaranayake : Med Teacher 2008; 30: 836 – 845
➢ Case SM. Academic Medicine. 1994;69(10):S4-6.
➢ Costello E, Holland J, Kirwan C. Int J of Educ Tech in Higher Ed.
2018;15(1):42.
➢ Holsgrove G, Medical Education 1998; 32: 343 – 350
➢ Norcini: Med Educ. 2003;37:464–469
➢ Pugh, De Champlain & Touchie. Med Teacher 2019; 41:5, 569-577
➢ Schuwirth: Med Educ. 2004 Sep;38(9):974-9.
➢ Schuwirth & van der Vleuten. Med Teacher 2011: 33; 783–797
➢ Swanson et al; Academic Medicine. 2005;80(10):S93-S6.
➢ Tarrant & Ware; Nurse Education Today. 2010;30(6):539-43.
Jane Holland