This document discusses objective evaluation methods, specifically multiple choice questions (MCQs). It outlines the advantages and limitations of MCQs, including that they allow for objective, large-scale evaluation but can be time-consuming to create and may test trivial details. The document describes different types of MCQs and provides guidelines for writing high-quality questions at various cognitive levels. It also covers using item analysis to evaluate questions based on difficulty, discrimination, and distractor effectiveness to improve assessments over time.

1. Teaching technology:
Assessment-II
Dr Naresh T Chauhan
Objective methods of evaluation

2. • Advantage and limitations
• Enumerate the methods of objective evaluation
• Types of multiple choice question
• Advantage and disadvantage of MCQ
• The component of MCQ item
• Guideline for construction of an MCQ
• Item analysis
Objective

3. Dale's ‘Cone of Experience’

4. Advantage
• Objective in nature
• In depth evaluation of student !
• Easy to evaluate
• To test the candidate at the desired level of
difficulty
• Good for formative evaluation and for
screening a large number of examinees
• Standard of scoring can be kept constant for
many years

5. Disadvantage
• Poorly framed MCQ may be misleading
• Tendency among teacher to test trivial and
less important data
• Questions are difficult and time consuming to
construct
• Not very useful for summative evaluation

6. Types
• The single or one best response
• Multiple completion type
• Relationship analysis type
• Multiple true false completion type
• Matching type
• Case history type
• Pictorial/Graphical /Tabulated type
• Comparison type

8. Testing Item Content Levels
• Recall – testing knowledge of isolated facts
• Interpret* – asking to review and reach some
conclusion
• Problem solve* – asking to take some action after
reading a situation
*Higher order thinking skills.

9. Writing for Different Cognitive Levels
• Recall
– Which of the following is the process of mitosis?
• Interpret
– Which of the following can be classified as a
meiosis error?
• Problem-solving
– A maternal blood sample revealed decreased
levels of AFP and estriol and elevated hCG. What
would be the next logical test to perform?

10. Purposes…
• Motivate students
• Identify areas of deficiency or further learning
• Determine final grades or make promotion
decisions
• Identify areas where the course/curriculum is
weak

11. What Should Be Tested?
• Exam content should match course/clerkship
objectives
• Important topics should be weighted more
heavily
• The testing time devoted to each topic should
reflect the relative importance of the topic

12. Component of MCQ
Key
Distractor

13. Direction for framing MCQs
• Design each item to measure an important
learning outcome
• Present single, compact, clearly formulated
stem
• Use simple and clear language
• Stem should be a statement , not a word
• State the stem in positive form as far as possible
• The word ‘NOT’ and ‘EXCEPT’ should be
underlined

14. Direction …cont
• Grammar ,Style, length, plausible distractors
• Avoid clue that suggest answers, similar length
and form
• ‘All of the above’, ‘none of the above’
• Item with numerical alternatives , arrange
them in rank order
• Testwiseness
• Irrelevant Difficulty

15. Issues Related to Testwiseness
• Grammatical cues
• Logical cues
• Absolute terms
• Long correct answer
• Word repeats
• Convergence strategy
• B or C position overused

16. The forbidden fruit that Eve offered Adam was
an:
(a) apple (b) pear
(c) banana (d) melon
(e) carrot

17. Keep the alternatives simple
• When your body adapts to your exercise load,
a. you should decrease the load slightly.
b. you should increase the load slightly.*
c. you should change the kind of exercise you are doing.
d. you should stop exercising
• When your body adapts to your exercise
load, you should
• a. decrease the load slightly.
b. increase the load slightly.*
c. change the kind of exercise.
d. stop exercising

18. Implausible distractors:
• In a study of the effect of diet on risk of diabetes,
the researcher can manipulate a number of
variables including the amount of food,
carbohydrates, proteins or fats consumed. During
the experiment the amount of food, protein and
fat subjects consumed remained the same. Only
the amount of carbohydrates consumed
changed. What was the independent variable in
this study?
a. amount of food consumed
b. amount of carbohydrates consumed
c. amount of protein consumed
d. amount of fat consumed

19. Plausible distractors:
• In a study of the effect of diet on risk of diabetes, the
researcher measured how likely the subjects were to
get diabetes and how severe their symptoms were if
they developed the disease. To prevent amount of
exercise from influencing the results, the researcher
held it constant in the two groups he was studying.
What was the independent variable in this study?
a. likelihood of developing diabetes
b. severity of symptoms
c. diet
d. amount of exercise
dietary pattern

20. Possibility of a “Random Pass”
Depends on the number
of answer options per question
and the number of questions!

21. Number of
Questions
Percent Pass (≥50%) by Chance
2 choices 3 choices 4 choices 5 choices
1 50 33 25 20
2 75 56 44 36
4 69 41 26 18
6 66 32 17 10
10 62 21 8 3
20 59 9.2 1.4 .3
50 56 1 .01 .0004
from Brown, 2001

22. Adjustment for Guessing
Negative Marking…
– Elimination strategy reduces odds of wrong
answer penalty
– Subtracting a percentage of the number of wrong
answers obtained from the final grade
– Give a grade of 4 for a correct answer and a score
of -1 for a wrong answer on a 4 choice question

23. Item analysis
• Difficulty index
• Discrimination index
• Distractor effectiveness
• To create a question bank
• Used in classroom teaching to modify teaching content
or methodology

24. Purpose of Item Analysis
–Evaluates the quality of each item
–Rationale: the quality of items
determines the quality of test (i.e.,
reliability & validity)
– May suggest ways of improving the
measurement of a test

25. • Difficulty index
– The item difficulty for item i, pi , is defined as the
proportion of examinees who get that item correct.
• Method for Dichotomously Scored Item
• Method for Polytomously Scored Item
• Grouping Method
N
R
P 
maxX
X
P 
2
LU PP
P



26. Item Difficulty (cont.)
Interpreting the p-value...
example:
100 people take a test
15 got question 1 right
What is the p-value?
Is this an easy or hard item?

27. Item Difficulty (cont.)
Interpreting the p-value...
example:
100 people take a test
70 got question 1 right
What is the p-value?
Is this an easy or hard item?

28. 0.5 + 10
Too EasyToo Difficult
Level of Difficulty
Index Range Difficulty Level
0.00-0.20 Very Difficult
0.21-0.40 Difficult
0.41-0.60 Average/Moderately
Difficult
0.61-0.80 Easy
0.81-1.00 Very Easy

29. ITEM DISCRIMINATION
• The extent to which an item
differentiates people on the
behavior that the test is
designed to assess.
• the computed difference
between the percentage of
high achievers and the
percentage of low achievers
who got the item right.

30. Index of Discrimination
• (used for dichotomously scored items)
D = PH - PL
• We need to set one or two cutting scores to divide the
examinees into upper scoring group and lower
scoring group.
• PH is the proportion in the upper group who answer
the item correctly and PL is the proportion in the
lower group who answer the item correctly.
• Values of D may range from -1.00 to 1.00.

31. Example
There are 140 students attending a world history
test. If 18 examinees in upper group answer
item 5 correctly, and 6 examinees in lower
group answer it correctly, then calculate the
discrimination index for item 5.

32. Example 2
50 Examinees’Test Data on 8-Item Scale About Job
Stress.
Item 1 2 3 4 5 6 7 8
PH
PL
.54 .81 .47 .32 .51 .18 .63 .56
.32 .56 .11 .05 .10 . 23 .25 .19
D .18 .25 .36 .27 .41 -.05 .38 .37

33. Guidelines for Interpretation of D Value
• D≥.40, the item is functioning quite satisfactorily
• .30≤ D≤.39, little or no revision is required
• .20 ≤ D≤.29, the item is marginal and needs revision
• D≤.19, the item should be eliminated or completely
revised
0 + 1- 1
Good
Discrimination
Reverse
Discrimination

34. What’s a “Good” Distracter?
A good distracter is a plausible answer:
• CM = common student mistakes, a recurrent
misconception about topic
• TG = too general option, that is too broad to
answer the question
• TS = too specific option, option focuses on one
detail in stem
• OT = off topic, option misses the answer
completely

35. Guidelines for constructing a MCQ paper
• Include a sufficient number of items to have
validity
• Time allotted
• Group according to format
• Instructions for negative marking
• Place easy questions first, easy moderate and
difficult questions in sufficient numbers
• All part of an item should on same page

36. design
Pilot
study
Reliability
Internal consistency
– reliability
coefficients
(coefficient alpha or
KR 20)
Equivalence -
alternate form
correlation
Content validity -
expert panel
Face validity - expert
panel
Construct validity - 'key
check' / item
discrimination / item
difficulty

37. Use of machine !
• Computer
• OMR
Go green, Think green, Act green !
http://www.go-green.ae/

38. Further reading…
• http://www.teambasedlearning.org/
• Constructing Written Test Questions For the
Basic and Clinical Sciences Third Edition
(Revised)
• Multiple Choice Questions Not Considered
Harmful
by
Karyn Woodford, Peter Bancroft accessed from
http://crpit.com/confpapers/CRPITV42Woodford.pdf on 09 Aug14