Induction Training AEO

1. Item Development
for AEOs
By
Saad Salman

2. Types of Evaluation
• Formative - conducted to provide
• learners with evaluative information
• useful in improving
• Summative - provides
• information on the ability of the learner to do or know
• what he/she is supposed to do or know

3. Purposes of Testing -
Individual
• Evaluating learner progress toward achieving educational
goals
• Identifying areas that require remedial activity
• Assigning grades
• Identifying scholarship recipients
• Tool for program evaluation
• Identifying placement into a certain course
• Identifying strong or weak areas of curriculum

4. Learning Objectives
• Understand the pros and cons to various testing
questions for written examinations
• Learn how to determine
• Item difficulty and
• Item discrimination
• Understand the psychometrics of a high stakes test
• Validity
• Reliability
• Standard Setting

5. Question Types
• Essay Items
• Short Answer and Completion Items
• Matching Items
• True-False and Multiple-Choice Tests
• Interviews
• Portfolios
….all can be scored and can be subject to test development

6. Pros and Cons of MCQ’s
Pros
• Useful for measuring
learning outcomes at
almost any level
• Easy to understand
• Easy to score
• Easily analyzed for
effectiveness
• Allow broad coverage
efficiently
Cons
• Good questions
• Take a long time to
write
• Are difficult to write
• Constrain creative
responses from
learners
• May have more than
one correct answer

7. PREPARATION OF MCQ’s:
• Paper developer should read concept and learning outcome
and then write stem of Question .
• Instructions for writing stem; (AIMS UNICEF)
• Brief avoiding extra language in stem
• Comprehensive including the repetitive portion of options in
stem
• Meaningful as a complete statement
• Simple using easy language words

8. Instructions for writing stem
• Understandable avoiding negative statements.
• Neutral avoiding Gender, Medium, Area
• Independent so that student cannot get help from other
questions.
• Clear using reference of any statement
• Evaluator of learning outcome to be tested.
Free from content so that exact statement doesn’t matches with
content in book because in SOLO Taxonomy rote learned
responses have no level.

9. Instructions for writing
options
• All options should be written using logic.
• Options should be related to main concept.
• Order of correct option should be different in different
questions.
• Order of options can also be used for adjustment of difficulty
level.

10. Instructions for writing
options
• All options should be equally attractive for hit & trail maker.
• In options avoid lifting phrases directly from text .
• After writing stem write correct option and then search
plausible distracters. Finally shuffle correct option and
distracters.
• Answer options should be about the same length and parallel
in grammatical structure.

11. Instructions for writing
options
• Distracters must be incorrect, but plausible. Try to include
common errors in distracters.
• To make distracters more plausible, use words that should be
familiar to students
• If a recognizable key word appears in the correct answer, it
should appear in some or all of the distracters as well

12. Use Rarely:
– Extreme words like "all," "always" and "never" (generally a wrong answer).
– Vague words or phrases like "usually," "typically" and "may be" (generally a
correct answer).
– "All of the above" - eliminating one distracter immediately eliminates this, too.
– "None of the above" - use only when the correct answer can be absolutely
correct, such as in math, grammar, historical dates, geography, etc..
– Do not use with negatively-stated stems, as the resulting double-negative is
confusing. Studies do show that using "None of the above" does make a
question more difficult, and is a better choice when the alternative is a weak
distracter.

13. STUDENTS STRATEGIES ABOUT MULTIPLE
CHOICE ITEMS:
• Pick the longest answer.
• Pick the 'b' alternative.
• Never pick an answer which uses the word 'always' or 'never' in it.
• If there are two answers which express opposites, pick one or the other
and ignore other alternatives.
• If in doubt, guess.
• Pick the scientific-sounding answer.
• Don't pick an answer which is too simple or obvious.
• Pick a word which you remember was related to the topic.

14. Tips for writing discriminant MCQs
• Responses
• All answers should be plausible and homogenous
• Items need to be independent of one another
• Answer choices should be similar in length and grammatical form
• List answer choices in alphabetical or numerical order
• Avoid ‘all of the above’ as a response
• Avoid technical flaws (tense or plurality for example)

15. What Makes a Content-
Appropriate Item
• Be sure that each item reflects a clearly defined learning outcome
• Stem
• The stem of the item should be self-contained and written in clear and
precise language.
• Avoid ‘trigger’ words (e.g. pin-rolling tremor)
• Negatives, excepts, absolutes and qualifiers in question stems are no-no’s.
• Reflects specific content
• Able to be classified on the test blueprint
• Does not contain trivial content
• Independent from other items
• Not a trick question
• Vocabulary suitable for candidate population
• Not opinion-based

16. Guidelines for the Answer Options
• All answers should be plausible and homogenous
• Items need to be independent of one another
• Answer choices should be similar in length and grammatical form
• List answer choices in alphabetical or numerical order
• Avoid ‘all of the above’ as a response
• Avoid technical flaws (tense or plurality for example)
• Maintain similarity in the answer options.
• Keep the answer options relatively equal in length.
• Keep all answer options grammatically correct with the stem and
parallel.
• Avoid cueing the right answer.
• Avoid words such as “always” and “never”.
• Avoid repetitive wording in the answer options.
• Vary location of key.
• Place answer options in logical order.

17. Item Analysis
• Qualitative: looks at whether the content matches the
information, attitude, characteristic or behavior being
assessed
• Quantitative:
• Item difficulty
• Item discrimination

18. Determining item difficulty
• The percentage of participants who get that item
correct
• Item difficulty scores can range from 0 to 100%
• Low value = high difficulty
• High value = low difficulty
High
(Difficult)
Medium
(Moderate)
Low
(Easy)
<= 30%
>30% AND
< 80%
>=80
%
0 10 20 30 40 50 60 70 80 90 100
Number of Students achieving each Score
0
10
20
30
0 10 20 30 40 50 60 70 80 90 100
Hard Exam Normal Exam Easy Exam

19. Discrimination Index
 The Discrimination Index distinguishes for each item
between the performance of students who did well on the
exam and students who did poorly.

20. Discrimination Index
• Index of discrimination:
• The difference in the % of people in one extreme group
minus the % of people in the other extreme group
• Item discrimination scores can range from -1.00 to +1.00
• Example
• 100 test takers: 20 in top 25 were correct but only 5 in
the lowest 25 students were correct.
• DI = (20-5)/25 = 0.8

21. Test Validity
• Validity:
• The extent to which inferences made from a test are
appropriate, meaningful, or useful.
• Does my test measure what it is intended to
measure?
• Content validity
• Expert review
• Criterion validity – Predictive/Concurrent
• Scores can be related to another known metric
• Construct validity
• Successfully differentiates between levels of learners

22. Test Reliability
• Reliability: Measure the underlying construct consistently =
trustworthiness/stability
• Test-Retest Reliability
• Alternate forms reliability
• Internal consistency reliability (cronbach’s alpha)
• Inter-rater reliability

23. Your Turn!
Review the distributed questions and
identify strengths and weaknesses in
each.

24. Summary
• Utilize action verbs to write objectives
• Write your exam items based on the objectives
• Choose appropriate options with one best answer
• Avoid technical errors
• Utilize an item checklist to ensure that you have done
all you can to write the best items possible.
• Pretest your items

25. Item Discrimination: Examples
Item
No.
Number of Correct Answers in
Group
Item Discrimination
Index
Upper 1/4 Lower 1/4
1 90 20
2 80 70
3 100 0
4 100 100
5 50 50
6 20 60
0.7
0.1
1
0
0
-0.4
Number of students per group = 100

26. Distracter Analysis: Examples
Item 1 A* B C D E Omit
% of students in upper ¼ 20 5 0 0 0 0
% of students in the middle 15 10 10 10 5 0
% of students in lower ¼ 5 5 5 10 0 0
(*) marks the correct answer.
Item 2 A B C D* E Omit
% of students in upper ¼ 0 5 5 15 0 0
% of students in the middle 0 10 15 5 20 0
% of students in lower ¼ 0 5 10 0 10 0