Design of multiple choice questions

L
P
U
Resource Person
Er. Vivek SRIVASTAVA
Assistant Professor
School of Agriculture
Lovely Professional
University
Workshop on Design of
Multiple Choice
Questions

Learning Outcomes
• To exemplify the principles of designing good
Multiple Choice Questions for assessing
Higher Order Thinking Skills (HOTS)
• To Identify the pitfalls in MCQ setting
• To prepare sample MCQs for testing memory,
understanding, application, analysis,
evaluation and creativity
© LPU :: Vivek Srivastava

Points to Ponder…
1. Recap on principles of designing MCQs
2. Identification of pitfalls in MCQ design and
rectification of errors
3. Design of sample Multiple Choice Questions

Well, that’s the whole point!
•In order to ask a question, you must
know something about the subject and
so you must study it.
•By studying and forming practice
questions in your mind and on paper,
you’ll begin to know that subject from a
different point of view and better retain
what you learn.
You may be wondering how you can
design up a classic MCQ’s?

Guidelines for Designing MCQ’s
“ Written properly, multiple choice exams
correlate strongly with assessments by
descriptive tests ”
(Brown, Robert, “Multiple Choice Versus Descriptive Tests”, Frontiers in Education Conference, Reno NC 2001.)

What is a multiple choice question(MCQ)?
• According to Clegg and Cashin (1986), a multiple
choice question contains three elements:
– Item stem: this is the problem statement. It contains
all the information necessary to answer the multiple
choice question.
– Correct option: this is the correct answer to the
multiple choice question.
– Distractors: these are the incorrect answers
presented as alternatives to the correct answer in a
multiple choice question.

Sample Multiple Choice Question
• Which of the following pH values represents a neutral
condition ?
a) 5.0
b)1.0
c) 6.0
d)7.0
• Item stem: “Which of the following pH values
represents a neutral condition ?
• Correct answer: d
• Distractors: a,b,c

Advantages of MCQs
• Easy to score
• Can tap in to different cognitive levels
• Useful for diagnosing student difficulties (e.g.,
when you use common errors as distractors)
• Provides an excellent basis for post-test
discussion (you should discuss why distractors
are wrong)
• Requires relatively little time to answer

Limitations of MCQs
• Items are open to misinterpretation by students
• It can be time-consuming to construct “good”
test items
• It is difficult to create questions that tap in to
higher levels of learning (i.e., Evaluation)
• Creativity cannot easily be tested
• They encourage guessing

Use of Multiple Choice Exams
Not limited to Factual Knowledge
– Making inferences
– Application of models
– Evaluation

Multiple Choice vs. Descriptive
Examinations
DescriptiveDescriptive
QuestionsQuestions
MultipleMultiple
ChoiceChoice
Setting the examSetting the exam EasyEasy DifficultDifficult
Grading TaskGrading Task DifficultDifficult EasyEasy
Grading TimeGrading Time LongLong ShortShort
Grade ConsistencyGrade Consistency VariesVaries HighHigh

MCQ design principles and
purpose(AV 1)

Structure of a
Multiple Choice Question
Stem
Answers

PRINCIPLES FOR MULTIPLE-
CHOICE QUESTIONS
A. The stem* of the question
should present a clear and
complete problem, (e.g., “Who
wrote the Declaration of
Independence?”)
B. The questions should not
clue the students to the correct
answer*:
1. The stem of the question
should agree grammatically
with each choice.
2. The options* should be
similar in length and amount of
detail.
3. All options should seem
plausible to the uninformed.

C. If the question is controversial, an
authority or theory should be referred
to as support, (e.g., “According to most
German historians, which of the
following caused World War II?”)
D. It is generally suggested that
negative words be avoided in the
question stem. If included, any
negative word should be capitalized
and/or underlined and bolded.

E. Each option should concisely state only one
idea.
F. The option “All of the above/None of the
above” should not be used,

G. All options should be as homogenous
or similar as possible.
H. Avoid complex multiple-choice
formats, (e.g., both A and B, both B and
C, etc.)
I. Order the options to each question
alphabetically
J. List options vertically, not horizontally.
K. The stem of a question should be
written as a question, not as an
incomplete sentence

• 1. Which of the following is not a mammal?
• a. Horse
• b. Whale
• c. Iguana
• d. Human

S. No Principles Violated Possible Corrections
1 ????? ??????

S. No Principles Violated Possible Corrections
1 Use of negative
words,
Options are not
homogenous ,
Order of the
options to each
question are not
alphabetically
arranged
Which of the Following is a Mammal?
a.Sea Horse
b.A Shark
c.A Whale

A verbal question:
The forbidden fruit that Eve offered Adam
was an:
(a) apple (b) pear
(c) banana (d) melon
(e) carrot

Scientific name of maize is?
a) Zea zea b) Xea mea
c) Zea mays d) Pommace
e) Prunello

Use related, but different terms….
Instrument use to measure intensity of
rainfall is:
(a) Rain meter
(b) Rain stick
(c) Rain scale
(d) Rain gauge
(e) Rainimeter

Formulating Good
Multiple Choice Questions
• Wording the stem so that there is a single
correct answer
• Finding plausible wrong answers

Stems
Statements
“A river that flows through a desert is called..”
Questions
“ Which of the following descriptions best fits
the character of asian grapes?”
Negative voice
“ Which of the following vitamins is not present in
guava?

Stems
Conditional Voice
“ If you want a to acquire vitamin C that
will assist your body requirement which fruit
you will select in your diet:”

Answer Options
• Single correct answer options
• Collective answer options:
“ (a) and (c) are both correct ”
“ all of the above are correct ”
“ none of the above are correct ”

Plausible Answer Strategies
• Good questions contain several plausible
answer options
• Grey area between designing “tricky”
questions and providing answer options that
test knowledge thoroughly.
• Inadvertently giving away the answers

MCQs…..Classifying (AV 2)

Creating variance in MCQs
© LPU ::
Remembering
&
Understandin
g
Applying &
Analysing
Evaluating &
Creating
Facts Remembering of
previously
learned material
Ability to use learned
material in new and
concrete situations
Ability to break down
material into its
component parts
Concepts Ability to grasp
the meaning of
material
Apply rules, methods,
concepts, principles,
laws, and/or theories
To understand
Organisational
Structure
Principles Memory based Apply laws and
theories to practical
situations
Distinguish between
facts and inferences
Procedures Know common
terms, Know
specific facts
Apply Knowledge and
mathematical
calculations
Evaluate the
relevancy of data

Creating variance in MCQs- Stems
© LPU ::
Remembering &
Understanding
Applying & Analysing Evaluating & Creating
Facts Who is the father of
genetics?
Referring to the chart below
which continent is most
populous?
Which scientist has greatest
role in green revolution?
Concepts When it is most
appropriate to use optical
sensor over NIR sensor for
crop stress modelling?
To set FCC in a satellite
image which band
combinations has to be
used?
Which of the following is
an essential plant
nutrient?
Principles Principle of
triangulation is used in
this precision
agriculture tool?
How does gravitational pull
of sun and moon affects the
ocean?
How can you optimize shutter
speed and aperture when
taking a picture in low light?
Procedures In precision farming for
acquiring maximum
information on crop water
content what wavelength
range a farmer has to
choose.
How would providing excess
irrigation affect rise in the
levels of electrical
conductivity?
Which amendment is best
suited for reclamation of
saline sodic soils?

MCQs…..Classifying further (AV 3)

MCQs… Writing Stems (AV 4)

Evaluating Multiple Choice Questions
Difficulty Factor = R/N
R = # of times the correct answer is
selected
N = # of people taking the exam
0.5 + 10
Too EasyToo Difficult

Difficulty Factor
Values greater than 0.8:
– The question is too easy
– Stem wording gives away the answer
– Not enough plausible incorrect answer options
Values less than 0.2:
– Guessing effect can be a factor
– The question is defective
– Material wasn’t covered
– Stem wording is unclear, confusing
– Answer options need work

Evaluating Multiple Choice Questions
Discrimination Index =
Proportion of capable students that select an
answer option – proportion of weak students
that select the answer option
i.e. D = Ftopquarter – Fbottomquarter
0 + 1- 1
Good
Discrimination
Reverse
Discrimination

Discrimination Index
For the Correct Answer Option
Values ~ 0.5 give good discrimination
< 0.2 consider dropping
negative… drop the question
For Incorrect Answer Options
Values should be negative.

Knowledge or Successful Guessing?
Multiple Choice Exam Strategies
– Improve odds by eliminating 1 or more infeasible
or unlikely answer options

Adjustment for Guessing
Negative Marking…
– Elimination strategy reduces odds of wrong
answer penalty
– Subtracting a percentage of the number of wrong
answers obtained from the final grade
– Give a grade of 4 for a correct answer and a
score of -1 for a wrong answer on a 4 choice
question

MCQs…. Designing Options (AV 5)

Sample Verbs
Knowledge
– Cite, define, identify, label, list, match, name, recognize,
reproduce, select, state
Comprehension
– Classify, convert, describe, distinguish between, explain,
extend, give examples, illustrate, interpret, paraphrase,
summarize, translate
Application
– Apply, arrange, compute, construct, demonstrate,
discover, modify, operate, predict, prepare, produce,
relate, show, solve, use

Sample Verbs continued
Analysis
– Analyze, associate, determine, diagram, differentiate,
discriminate, distinguish, estimate, infer, order, outline,
point out, separate, subdivide
Synthesis
– Combine, compile, compose, construct, create, design,
develop, devise, formulate, integrate, modify, organize,
plan, propose, rearrange, reorganize, revise, rewrite, tell,
write
Evaluation
– Appraise, assess, compare, conclude, contrast, criticize,
discriminate, evaluate, judge, justify, support, weigh

MOST IMPORTANT RULEMOST IMPORTANT RULE
• HAVE OTHERS REVIEW YOUR QUESTIONS – IF
POSSIBLE GET FEEDBACK FROM MORE THAN
ONE PERSON. WORK IN TEAMS AND
CRITIQUE EACH OTHERS QUESTIONS

The challenge…
Major Challenges
•Effective Question Design
•Errors/Special Characters etc.
•Wrong Question Types
•Question Duplication
•Disproportionate Question
Framing w.r.t marks

Design of sample test papers…
• All participants to design sample test paper of
the course assigned and on the format
provided
• Time Allotted :

Get Set Go!!!
Gear up
Fasten your seat belts
Explore your question
designing
abilities
Build futuristic
assessment solutions…

Queries if any…

Design of multiple choice questions

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (14)

Similar to Design of multiple choice questions

Similar to Design of multiple choice questions (20)

More from Vivek Srivastava

More from Vivek Srivastava (20)

Recently uploaded

Recently uploaded (20)

Design of multiple choice questions

Editor's Notes