This document summarizes key points from a presentation on effective assessment methods in medical education. It discusses the importance of linking assessments to learning objectives, incorporating competencies, and minimizing construct underrepresentation. The presentation provides guidance on writing different types of questions, including essay questions, modified essay questions, short answer questions, and multiple choice questions. It emphasizes designing questions that clearly test intended objectives and competencies, avoiding ambiguity and bias.

1. • Revised Basic Course Workshop in
Medical Education and
Technologies.
• Medical Education and Technology Unit,
• Calcutta National Medical College,
• Kolkata

2. Assessment planning and
quality assurance:
Writing the correct
essay question, short
answer question and
MCQ
Dr. Anindya Dasgupta
Professor & Head,
Dept. of Biochemistry,
Calcutta National Medical

4. Objectives of assessment of knowledge is
to close the gap of knowledge
–From the view of the assessor
–From the view of the assesses.

5. Does the assessor always understand
what he actually wants to test?

6. Do the students understand what the
assessor wants to know?

7. Where is the problem then?
• Use of subjective criteria while assessing the
answers,
• Encourage students to memorize terms and
details:
– Their understanding remains incomplete

9. How to solve this problem?
• To incorporate the
competency component
into the assessment
process,
• To link the assessment with
learning objectives,
• Assessment should be clear
and transparent,
• Assessment should
minimize the construct
under-representation

10. Construct under-representation
The examination lacks validity because the examination content
is not reflective of relevant knowledge
• Trivial content
• Rote memorization for
factual recall
• Maldistribution of
examination items,
• Teaching to the test

11. Teaching to the test

12. • Construct under-representation can be
overcome by including sufficient exam items
requiring higher order of cognition to solve
clinically relevant problems.
• Aim is to assess the level of competency.

13. Searching for the solution

14. A blueprint is required

15. Blue printing
• For linking assessments with learning objectives
• For incorporating the competencies into the
assessment scheme according to their weightage
• To minimise the construct under representation
• All these things will make the assessment clear,
explicit and transparent and designing a better
paper

16. Learning objectives
• After the end of the session the audience will
be able to:
– How to plan an assessment
– How to prepare a question paper
– How to write an essay type question
– Advantages and disadvantages of essay type
question

17. Learning objectives
– How to write a short question,
– Advantage and disadvantages of a short answer
type question,
– Advantages and disadvantages of multiple choice
questions

18. Designing a question paper
• Choose the competencies and their objectives to
be included
• Decide the weightage to different objectives and
different areas of content
• Decide the form and type of questions to be set:
– Long questions,
– Short questions,
– MCQs

19. Designing paper
• Questions should be chosen from every competency
taught according to the weightage given.
• More stress on fundamental concepts and techniques
• All three domains of knowledge should be
incorporated as far as possible,
• However, always tests for conceptualization and higher
cognitive abilities should be given preferences

21. Final moderation and acceptance of
question paper
• Structure analysis,
• Content analysis,
• Difficulty analysis,
• Time analysis

23. Written examination
• Long answer type questions
• Short answer type questions
• Multiple choice questions.

24. Long answer type questions
• Essay type questions
– Structured
– Unstructured
• Modified essay type questions

25. Time for exercise

27. Written examination (Pen and Paper
test)

30. • Easy to set,
• Tests students comprehensively in:
– Understanding of the topic and Clarity of ideas
– Useful for testing the higher domains of cognition
– Organizing or planning an answer
– Capability of self expression in written form.

31. Disadvantages
• Low validity as only a small range of content may be covered,
• Time consuming from both students and teachers view,
• Examiner may be biased towards a particular point of view.
• Inter-rater difference in marks(Examiner tough/lenient)
• Permit and occasionally encourage bluffing
• Marking is not reliable:
– High degree of subjectivity involved in marking.

32. Extraneous factors influence marks
• Hand writing
• Grammar
• Diagrams
• Under linings
• Use of pens

33. Framing of an essay type of question

34. The question should be specific
• The question should clearly specify what is expected from
the student,
– Discuss protein energy malnutrition
• The student may give their responses in the context of:
• Epidemiology,
– Signs and symptoms
– Causes
– Management
– Or all of the above with different weightage on each context.

35. Better is a structured essay question
• Discuss protein energy malnutrition in
children of our country in the context of
– i) Common causes
– ii) Two major types and their epidemiology
– iii) Signs and symptoms differentiating them.
3 + 4 + 5 = 12
This type of question clearly specifies what is expected from the
student in which category.
It also indicates the weightage given for each part of the
question: How much is expected from him.

36. Writing a good essay type question
• The question should be specific,
– What, why or when
• If structured, marks should be provided for each
part,
– How much is needed for each part
• The language of question should be simple
– Effort from the teacher is to assess the scientific
acumen of the student more than his language skill.

37. Framing a good essay type question
• Reference points or supplementary
information should be provided where
necessary:
– Write the management of hemolytic jaundice in
a neonate
– Write the management of tuberculosis in a
pregnant female.

38. Evaluating essay questions
• Specific learning outcomes should be the sole criteria for
giving marks.
• Adequate timing for scoring all the answers should be
ensured,
• Marks should be given question to question rather than
student by student,
• Identity of the student should be concealed,
• It is always better if an average mark is taken after
checking the answer by more than one examiner.

39. Avoid bias against
• Poor presentation,
• Bad handwriting
• Spelling and grammar

40. Evaluating unstructured essay type questions

41. • These rules can make the long essay
type questions an effective tool for
assessment

43. •Modified Essay Questions
(MEQ)

44. MEQs are mainly problem solving type
of questions

45. • A problem or case scenario is given to the
students based on which questions are asked,
• The student has to apply the different levels of
his/her knowledge that he/she has acquired in
the context of the given problem/case.

46. Example of MEQ
• A person, aged 56 years is brought to the emergency ward at 8 am
with severe chest pain since early morning. The patient is conscious
and recalls episodes of profuse sweating during attacks of chest
pain. The patient party gives a history of heavy smoking and reports
show increased blood lipids on the basis of previous investigations.
On the basis of the above history answer the following question:
– What is the most probable diagnosis you will consider first (2)
– Describe the immediate management you will provide to the patient,
(3)
– Describe the immediate investigations you will perform to achieve a
definite diagnosis (3)
– Explain how different biochemical tests can help you in determining the
duration of such an attack. (2)

47. Example of MEQ

48. • So, an MEQ can help in the assessment of
– Recall,
– Comprehension,
– Analysis,
– Synthesis,
– Evaluation of knowledge

49. Advantages of MEQ
• Questions focus on more particular area,
• Tests the clinical competence based on the
student’s pre-clinical and para-clincical
knowledge more effectively,
• Minimum subjectivity in answers help in
assessments.

50. Disadvantages of MEQs
• If the student does not know the first answer, he may
be penalized many times for the next responses also,
– This is particularly evident where the answer to the no. 1
question determines the answer to the next questions,
– So, better to prepare independent sub-questions,
• Each case requires extensive time to prepare and test,
• Once used, it may be known to examinees.

51. Framing modified essay
type of questions
• Develop a realistic problem or scenario of clinical
importance
• Prepare a draft skeleton to outline the minimum
competency levels of the student in the context of that
problem,
• Define clearly the part of the knowledge or skill
judgment in each question,
• Prepare questions in an unambiguous way so that
student can give their responses with minimum
subjectivity.

52. Points to remember for both long type
essay questions and MEQs
• Whenever possible prepare a team of faculties to
develop such questions,
• Always prepare a model answer that should be
accredited by all participant faculties,
• Try to develop questions which are mutually
exclusive as far as practicable.

53. •Short answer questions
(SAQs)

54. Purpose of Small Answer Questions
• These are framed to avoid bias due to literary
presentation, style etc.
• They generally test lower domains of
knowledge,
• Are very useful to evaluate those aspects of
learning where
– Objectivity is more preferred,
– Reliability of scoring is more desired.

55. Different types of SAQs

56. Completion type SAQs
• Completion type:
– These are simple questions framed to test factual
recall.

57. Example: Label the following parts of
the given organ

58. Unique or one best answer type
SAQ
• Draw the structure of cholesterol,
• Write down the dosage of amoxycillin for a one year
baby weighing 10 kg.
• Classify the following antibiotics as bactericidal or
bacteriostatic:
– Rifampicin
– Penicillin
– Co-trimoxazole,
– Tetracyclines.

59. Open type of SAQs
• Provide some options regarding the choice of
correct answer.
• The student has the option of choosing his
responses among many.
• Examples:
– Enumerate three side effects of chloramphenicol
– Enumerate three causes of massive splenomegaly

60. Framing of SAQs
• TASK should be well defined
– Little scope of subjective interpretation
• Language should be simple
• Wording of questions should be unambiguous

61. Examples of wrongly formed SAQ
• A neonate is ________
• Possible answer may be
any of these:
– Male,
– Female,
– Live baby,
– Pre-term
– Post-term,
– Pink
– 2.5 kg.

62. Right way

63. Point of reference
• For any SAQ referring to any numerical value,
the point of reference should be given,
whenever possible:
– Infant mortality rate in India is ___________per
live birth according to 2018 estimation.
This will avoid ambiguity in both the student and the
teacher.

64. Avoid unintentional clues
• ‘A’ or ‘An’
– Imipramine is an _____________
– Imipramine fall in the class of ________drugs.
• An answer space which is too proportional to
the answer.

65. Advantages
• From the teacher’s point:
– The questions can be easily set and less time
consuming
• From the student’s point:
– Easy to score
– Marked fairly with less subjectivity,
• From both teacher’s and student’s points:
– Range of subject area covered is wide

66. Disadvantages
• 3-4 times more times required in comparison
to MCQs
– Less number of questions than MCQs in the test
paper,
– Less reliability than MCQs
• Marking of open SAQs involves some degree
of subjectivity

68. Construction of MCQs and
Item analysis

69. MULTIPLE CHOICE QUESTIONS
MCQs
• Multiple choice questions were first introduced
into medical examinations in 1950s,
• MCQ is an objective question for which there is a
prior agreement on what constitutes the correct
answer,
• The options given are close to that prior
agreement to variable extent with the key being
closest among all.

71. Advantages of MCQs.
• Screens for a wide range of knowledge
– Tests the concept in a most holistic way,
– Rote memorization can be minimized
• Easy to assess and mark
– For a large number of students,
– In a minimum amount of time,
– Least human intervention, so least biasness.
• All these properties make this pattern of questions
highly acceptable for written assessments in most vital
large scale assessments.

72. Limitations of MCQs
• Every good thing has also some limitations:
• Needs expertise and time for successful construction,
• Depends much on the instructors writing capability,
• Distracters are sometimes sufficient to get the correct
answer without sufficient knowledge about it,
• Security of questions are a matter of concern

73. Like all other assessments ideal MCQs
must have the following criteria.
• Validity
• Reliability
• Objectivity
• Feasibility
• Fairness.
• Which of the following
statements about mitral
valve is true?
– A) It has two cusps,
– B) It has three cusps,
– C) It regulates blood flow
between right atrium and
right ventricle,
– D) It regulates blood flow
between femoral vein and
great saphenous vein.

75. Ideal MCQs should analyze the
following
• Analysis at difficulty level
– Content,
– Structure, and
– Time.

76. Factors to be kept in mind during
designing MCQs
• Questions should be from every part/module of the
syllabus,
• Stress should be more on fundamental concepts and
techniques,
• Emphasis on applications of the concepts learnt and
use of techniques- competence based approach.

77. What are the different parts of an
MCQ?
• An MCQ is called an item.
• Anatomy of the item
– Statement of the problem: Stem,
– Followed by a single, distinct and unambiguous
question: Lead,
– Options: Choices,
• Correct answer: Key
• Incorrect answers: Distracters.

78. Example
• A middle aged male person is brought
to the emergency with chest pain and
sweating.
Which biochemical test you will give
most preference?
i) Serum cholesterol,
ii) Blood urea,
iii) Vitamin D level,
iv) CK-MB
Stem
Lead
Distracters
Key

79. Planning to write a correct MCQ

80. General cues
• The student population should be identified first
properly,
• Every topic in the syllabus should be considered,
• Specific areas of learning objectives or competency
levels should be identified,
• Always higher order of thinking should be assessed
with more preference,
• The language should be lucid, clear and unambiguous

81. Steps in writing an MCQ

82. A good stem is the hallmark of a good
MCQ
• Direction of the stem
should be very clear,
• A good stem is the best
hint to an intelligent
student,
• It can be easily assessed
by the hand cover test.

83. Example
• In pheochromocytoma some catecholamine end
products are excreted in urine in increased
amount.
Q. Which of them is extremely helpful for
diagnosing the disease?
i)Mono amine oxidase (MAO)
ii) Vanillyl mandelic acid (VMA)
iii) Homovalinic acid
iv) 3 methoxy tyramine.

84. •Writing down a good
stem

85. Very short stems should be avoided
• The mandibular nerve
– i) Sensory,
– ii) Supplies buccinator,
– iii) Is a branch of vagus nerve,
– iv) Passes through the foramen ovale.

86. The better option will be as follows
• Which of the following muscle is supplied by
the mandibular nerve?
– i) Platysma,
– ii) Buccinator,
– iii) Masseter,
– iv) Occipitalis

87. The central idea of the MCQ should be
included in the stem instead of the choice
• Very low density lipoprotein (VLDL) is:
– i) Synthesized from the intestine,
– ii) The major source of triglylceride in blood,
– iii) Metabolized to HDL,
– iv) Protective for heart.

88. The better form of this MCQ should be
• Which of the following lipoproteins is the
major source of triglyceride in circulating
plasma?
• i) LDL cholesterol,
• ii) VLDL cholesterol,
• iii) HDL cholesterol
• iv) Chylomicrons.

89. Negative assumptions should be
avoided in the stem
• Negative assumptions should be avoided in
the stem as far as practicable,
• When used, it should be:
– Capitalized,
– Underlined,
– Boldfaced.

90. Example
• All of the following are true for creatine kinase
EXCEPT:
– i) CK3 is raised in cerbral ischemia,
– ii) CK values are reduced generally in the
hypothyroidism,
– iii) Maternal total CK value rises in normal
childbirth,
– iv) Cardiac catheterisation can cause an increase
in the CK values,

91. Double negatives should be always
avoided
• Which of the following is not a loop diuretic
except:
• i) Furosemide,
• ii) Chlorthalidone,
• iii) Spironolactone,
• iv) Aminophylline,

93. Better is
• Which of the following is a loop diuretic?
– i) Furosemide,
– ii) Chlorthalidone
– iii) Spironolactone,
– iv) Aminophylline.

94. The stem of an MCQ should have a single
central theme instead of some diffuse ones
• Which of the following statements about
glucose transport in skeletal muscle is true:
– i) It is not dependent on insulin,
– ii) It is mediated by GLUT 4
– iii) It is dependent on catecholamine levels,
– iv) It is not dependent on the level of exercise.

95. Better one should be like this:
• Glucose is transported from the blood into
skeletal muscle by which of the following
glucose transporters?
– i) GLUT 1
– ii) GLUT 2
– iii) GLUT 3
– iv) GLUT 4

96. Irrelevant information should be eliminated
from the stem
• A 40 year old woman have a long history of heartburn. She told that
she has to get up in night and eat. Usually she gets relieved when
she takes cold milk. She is CEO in MNC and explained that her job is
very stressful. On examination it was observed that her pulse was
fast and blood pressure was raised. From last few days she was
suffering from severe pain in the epigastrium. She is diagnosed with
ruptured gastric ulcer in the posterior wall of stomach.
• Q. The contents of stomach will most likely collect in which
• part of peritoneal cavity?.
• a) Hepatorenal pouch
• b) Right paracolic gutter
• c) Greater sac
• d) Lesser sac

97. Better form of this MCQ will be:
• A 40 year old woman, having a long history of
heartburn, suffers from severe pain in the
epigastrium. She is diagnosed with ruptured
gastric ulcer in the posterior wall of stomach.
• The contents of stomach will most likely collect in
which part of peritoneal cavity?.
– a) Hepatorenal pouch
– b) Right paracolic gutter
– c) Greater sac
– d) Lesser sac

98. Clues should be avoided in the stem of
MCQ
• In which of the following conditions does
hemolysis play an important role in
pathophysiology?
– a) Albers-Schonberg disease
– b) Megaloblastic anemia
– c) Henoch-Schonlein purpura
– d) Hemolytic-uremic syndrome

99. • Facts to be followed for formulating
good options/choices for an MCQ.

100. Options should be grammatically
parallel
• Hemolytic anemia is suggested by
• (a) High alkaline phosphatase
• (b) Reticulocyte count is high
• (c) Only jaundice
• (d) Platelet count low

101. The better option should be:
• Which of the following suggests a hemolytic
picture
• (a) Jaundice
• (b) low platelet count
• (c) High alkaline phosphatase level
• (d) High reticulocyte count

102. Bogeys should be avoided for just
filling up the gaps for any choice
• Integrated child development program provides
services till the age of
• (a) 1 year
• (b) 3 year
• (c) 5 year
• (d) life long
• (d) is the bogey and can be easily excluded by any
sensible person.

103. Duplications should be avoided in
options
• The options should not have any duplications like
the following:
• Children of following age are covered under
universal immunization program
– (a) under 5 year
– (b) 0-1 year
– (c) 0-3 year
– (d) 0-5 year

104. The better option is always:
• (a) 0-1 year
• (b) 1-2 year
• (c) 2-3 year
• (d) 3-4 year

105. Choices should be better placed in
chronological order
• Q. In the course of a dark adaptation , the
eye’s best sensitivity to wavelength shifts to
• A. 580 millimicrons.
• B. 477 millimicrons.
• C. 505 millimicrons.
• D. 600 millimicrons.

106. Better option is like this:
• Q. In the course of a dark adaptation , the
eye’s best sensitivity to wavelength shifts to
• A. 600 millimicrons.
• B. 580 millimicrons.
• C. 505 millimicrons.
• D. 477millimicrons.

107. The stem should not contain any clue
for singular or plural choice
• Q. Imipramine is an:
– a) Antidepressant
– b) Bronchodilator
– c) Sedative
– d) Stimulant

108. Options should be of equal length as
far as possible
• In case of Ewing’s sarcoma the treatment of
choice would be-
– A) Surgery → chemotherapy
– B) Radiotherapy
– C) Chemotherapy → surgery / radiotherapy
→chemotherapy
– D) Surgery → Radiotherapy

109. All of the above and None of the above
should be avoided as far as practicable
• Segmental anatomy of the liver can be
observed with the help of
– a) X ray
– b) Ultrasound
– c) MRI
– d) None of the above

110. Some more important points for the
options
• Key should be clearly the best option,
• Excess adjectives like fairly high, too bad, never,
always should be avoided in the options. These
can confuse the brighter students.
• The order of key should be changed to avoid the
subconscious preference for answers.

111. Major aims to produce good MCQs.
• The MCQs should
always be prepared
according to Bloom’s
taxonomy of learning
objectives.
• They should always be
prepared to test higher
levels of skills and
competence.

112. Revised Bloom’s taxonomy: 2000):
Anderson, Krathwohl, Airasian, Cruikshank, Mayer, Pintrich, Raths,
Wittrock, 2000

113. MCQs on the same topic for different
cognition levels
• Q.1 . Cyanide inhibits which
of the following enzymes?
– A) Succindate dehydrogenase,
– B) Malate dehydrogenase
– C) Cytochrome oxidase,
– D) Cyrochrome reductase
• Q.2. An emergency antidote
for cyanide is nitrite
because,
– A) Nitrite regenerates the
cytochrome oxidase rapidly
– B) Nitrite converts ferro-
hemoglobin to ferri-
hemoglobin,
– C) Nitrite inhibits the cyanide
from binding cytochrome
oxidase,
– D) Nitrite increases the
binding affinity of oxygen to
heme.

114. How to assess that an MCQ has been
successfully created

115. •Item analysis

116. Item analysis is important for building up
the confidence in both student and teacher
• All the questions in all sets should be standardized,
• This will avoid the imprecision of the tests among the
students,
• This will reduce the assessment flaw,
• This will strengthen the faith and confidence among
students and will provide competent doctors.

117. • More than 50 percent of the MCQs can be
rectified by self and peer checking.
• By checking the specific indices for
formulating successful MCQs.

118. Item analysis: Post validation
• Post validation: The process of analyzing the
performance of an MCQ after it has appeared
in the question paper,
• To analyse whether the item was:
– Acceptable,
– Much easier
– Much difficult
– Defective

119. Parameters for item analysis
• Difficulty index/facility value (FV):
– Measures the optimum level of difficulty for a
batch of students,
• Discrimination index: (DI)
– Measures the ability of the item to discriminate
between the low performers and high
performers
• Distracter effectiveness or functionality:
– Measures the functionality of the distracters
– How much a distracter has been able to deviate
the concentration of the performer.
All these indices are to be calculated
for each individual item.

120. Steps of analysis
(For each item separately)
• All test items are scored.
• The students are ranked in order according to
their test scores,
• Depending on the scores the students are divided
into high achiever group and low achiever group,
• A table is prepared for each item as follows.

122. Facility value (FV)

123. Facility value calculation for three
groups
Item no. Upper group Middle group Lower group Facility value
5 10 9 6 25/36 = 69%
7 7 6 6 19/36 = 53%
9 4 4 5 13/36 = 36%

124. Interpretation of the difficulty index

128. Discrimination index should never be
negative
• A negative discriminator
index implies only:
– A defective item, or
– A wrong key.

132. Distracter index should not be too high
either.
• A too high distracter index indicates ambiguous
options.
– Example 1.
• 50% of the top rankers chose a distracter rather than the
correct option,
– Ambiguous distracter.
– Example 2.
• 2% of the top rankers chose a distracter
• 35% of the low rankers chose the same distracter,
– Good distracter.

133. Item analysis: Take home message
• The job of a teacher is not complete only after
construction of an MCQ.
• All MCQs should be post validated with full
expertise and all criteria prescribed for them.
• Only after full validation MCQs can be put in
exams for assessing high performers and low
performers.

134. So, an ideal MCQ must discriminate between high
performers and low performers unambiguously.

135. Good MCQs make an assessment complete
and authentic
• No single assessment type is sufficient to assess
completely the cognition, skill and attitude in
medical science,
• All types of assessment are needed in different
proportions in different types of assessments.
• Good and optimally written MCQs make the
assessments complete and authentic for both
formative and summative evaluation.

139. Group activity on formulating different types of
questions:
Group A: Prepare an essay type question, unstructured with
checklist.
Group B: Prepare a structured essay type question
Group C: Prepare one modified essay type question
Group D: Prepare one open type of short answer type question
Group E: Prepare an MCQ of your choice having higher cognitive
domain.