2. MULTIPLE CHOICE TESTS
Multiple choice test questions, also known as
items, can be an effective and efficient way to
assess learning outcomes. Multiple choice test
items have several potential advantages:
3. Versatility
Multiple choice test items can be written to assess
various levels of learning outcomes, from basic recall to
application, analysis, and evaluation. Because students
are choosing from a set of potential answers, however,
there are obvious limits on what can be tested with
multiple choice items. For example, they are not an
effective way to test students’ ability to organize
thoughts or articulate explanations or creative ideas.
4. Reliability
Reliability is defined as the degree to which a test
consistently measures a learning outcome. Multiple choice
test items are less susceptible to guessing than true/false
questions, making them a more reliable means of
assessment. The reliability is enhanced when the number of
MC items focused on a single learning objective is increased.
In addition, the objective scoring associated with multiple
choice test items frees them from problems with scorer
inconsistency that can plague scoring of essay questions.
5. Validity
Validity is the degree to which a test measures the
learning outcomes it purports to measure. Because
students can typically answer a multiple choice item
much more quickly than an essay question, tests based
on multiple choice items can typically focus on a
relatively broad representation of course material, thus
increasing the validity of the assessment.
6. A multiple choice item consists of a problem,
known as the stem, and a list of suggested
solutions, known as alternatives. The
alternatives consist of one correct or best
alternative, which is the answer, and
incorrect or inferior alternatives, known as
distractors.
7.
8. CONSTRUCTING AN EFFECTIVE STEM
1. The stem should be meaningful by itself and should
present a definite problem. A stem that presents a definite
problem allows a focus on the learning outcome. A stem
that does not present a clear problem, however, may test
students’ ability to draw inferences from vague descriptions
rather serving as a more direct test of students’ achievement
of the learning outcome.
9.
10. 2. The stem should not contain irrelevant material, which can
decrease the reliability and the validity of the test scores (Haldyna and
Downing 1989).
11. 3. The stem should be negatively stated only when significant
learning outcomes require it. Students often have difficulty
items with negative phrasing (Rodriguez 1997). If a significant learning
outcome requires negative phrasing, such as identification of dangerous
laboratory or clinical practices, the negative element should be
emphasized with italics or capitalization.
12.
13. 4. The stem should be a question or a partial sentence. A question
stem is preferable because it allows the student to focus on answering
the question rather than holding the partial sentence in working
memory and sequentially completing it with each alternative (Statman
1988). The cognitive load is increased when the stem is constructed with
an initial or interior blank, so this construction should be avoided.
14.
15. CONSTRUCTING EFFECTIVE ALTERNATIVES
1. All alternatives should be plausible. The function of the
incorrect alternatives is to serve as distractors, which should
be selected by students who did not achieve the learning
outcome but ignored by students who did achieve the
learning outcome. Alternatives that are implausible don’t serve
as functional distractors and thus should not be used.
Common student errors provide the best source of distractors.
16.
17. 2. Alternatives should be stated clearly and concisely. Items that are
excessively wordy assess students’ reading ability rather than their
attainment of the learning objective
18. 3. Alternatives should be mutually exclusive. Alternatives with
overlapping content may be considered “trick” items by test-takers,
excessive use of which can erode trust and respect for the testing
process.
19. 4. Alternatives should be homogenous in content. Alternatives that
are heterogeneous in content can provide cues to student about the
correct answer.
20. 5. Alternatives should be free from clues about which response is
correct. Sophisticated test-takers are alert to inadvertent clues to the
correct answer, such differences in grammar, length, formatting, and
language choice in the alternatives. It’s therefore important that
alternatives
• have grammar consistent with the stem.
• are parallel in form.
• are similar in length.
• use similar language (e.g., all unlike textbook language or all like
textbook language).
21. 6. The alternatives “all of the above” and “none of
the above” should not be used. When “all of the
is used as an answer, test-takers who can identify more
than one alternative as correct can select the correct
answer even if unsure about other alternative(s). When
“none of the above” is used as an alternative, test-
takers who can eliminate a single option can thereby
eliminate a second option. In either case, students can
use partial knowledge to arrive at a correct answer.
22. 7. The alternatives should be presented in a logical order (e.g.,
alphabetical or numerical) to avoid a bias toward certain positions.
23. 8. The number of alternatives can vary among items
as long as all alternatives are plausible. Plausible
alternatives serve as functional distractors, which are
those chosen by students that have not achieved the
objective but ignored by students that have achieved
the objective. There is little difference in difficulty,
discrimination, and test score reliability among items
containing two, three, and four distractors.
24. ADDITIONAL GUIDELINES
1. Avoid complex multiple choice items, in which some or all of the
alternatives consist of different combinations of options. As with “all of
the above” answers, a sophisticated test-taker can use partial knowledge
to achieve a correct answer.
25. 2. Keep the specific content of items
independent of one another. Savvy test-takers
use information in one question to answer
another question, reducing the validity of the test.
26. ESSAY TESTS
Essay questions are used to assess student knowledge of the
subject content and writing ability. The question is followed by
a text box. Well written essay questions require students to
compose a response. Students use multiple sentences when
responding to an essay question. The responses are then
interrupted and manually scored by a subject-matter expert.
27. WHEN ARE ESSAY EXAMS APPROPRIATE?
• When you are measuring students’ ability to analyze,
synthesize, or evaluate
• When you have been teaching at these levels (i.e.
writing intensive courses, upper-division
undergraduate seminars, graduate courses) or the
content lends it self to more critical analysis as
opposed to recalling information
28. HOW DO YOU DESIGN ESSAY EXAMS?
• Be specific
• Use words and phrases that alert students to the kind of thinking you
expect; for example, identify, compare, or critique
• Indicate with points (or time limits) the approximate amount of time
students should spend on each question and the level of detail
expected in their responses
• Be aware of time; practice taking the exam yourself or ask a colleague
to look at the questions
29. HOW DO YOU GRADE ESSAY EXAMS?
• Develop criteria for appropriate responses to each essay question
• Develop a scoring guide that tell what you are looking for in each
response and how much credit you intend to give for each part of the
response
• Read all of the responses to question 1, then all of the responses to
question 2, and on through the exam. This will provide a more holistic
view of how the class answered the individual questions
30. HOW DO YOU HELP STUDENTS SUCCEED ON ESSAY
EXAMS?
• Use study questions that ask for the same kind of thinking you expect
on exams
• During lecture or discussion emphasize examples of thinking that
would be appropriate on essay exams
• Provide practice exams or sample test questions
• Show examples of successful exam answers
31. TYPES OF ESSAY QUESTIONS
• Extended Response Questions
• This type of essay question allows a student to determine the
content of the answer. Essay questions examine the student's
ability to organize, synthesize, and evaluate to compose a
response. One essay question topic could be, "How I Spent My
Holiday Break." This type of question works well to judge
writing skills, but does not necessarily have a right or wrong
answer.
32. TYPES OF ESSAY QUESTIONS
• Restricted Response Questions
• This type of essay question is limited in terms of content and
structure. For instance, you could ask, "Tell me about a time in
your life where you struggled to cope with the death of a
family member." This type of question would restrict the writer
to a limited time and scope enabling them to provide
sufficient detail on the topic. Restricted response questions are
helpful when testing students about content.
33. ADVANTAGES OF ESSAY QUESTIONS
• Allow learning objectives to be assessed through complex
thinking.
• Allows student reasoning skills to be assessed.
• Allows students to construct a response to a problem instead
of selecting one from a list.
34. DISADVANTAGES OF ESSAY QUESTIONS
• Samples limited content challenging validity and reliability.
• Produces questionable scores due to the subjective nature of the
questions.
• Scores can be influenced by factors outside the content being tested
such as a predetermined notion that a student is a "poor writer."
• May put poor writers at a disadvantage.
• Takes extra time to score.
35. CONSTRUCTING ESSAY QUESTIONS
• Allow adequate time to develop essay questions. Three well written essay
questions may take more time to develop than 40 multiple-choice
questions.
• Write questions to test a limited number of objectives.
• Limit essay questions to outcomes that cannot be measured in objective
terms.
• Never use essay questions to test facts or lower order thinking skills.
• Preference should be given to short versus the long essay questions to yield
the desired results; however, the short essay can also be flawed since it may
not employ complex thinking.
36. CONSTRUCTING ESSAY QUESTIONS
• Clearly write the task(s) the question is to address.
• State the time (minutes/hours) needed to complete a question.
• State the number of possible points for the question.
• Employ creativity when writing questions.
• Do not include optional questions.
• Do not provide a list of essay questions to prepare in advance of a test;
rather, consider allowing students to respond to a sample essay question
during class and provide examples of how their essay would be scored.
37. GOOD EXAMPLE OF AN ESSAY QUESTION
1. Explain in what way a person's failure to apply the smoking
cessation step process will impact his or her ability to stop
smoking. Provide an example to illustrate your point.
• This example requires student to compose a response. It also
requires the student to use reasoning skills and an example to
reinforce their argument.
38. BAD EXAMPLE OF AN ESSAY QUESTION
1. List all the steps to the smoking cessation process. Please list
them in chronological order.
• This example only requires the learner to access lower level
thinking skills which include memorization and recall.
40. CHARACTERISTICS OF TRUE-FALSE QUESTIONS
• Used to measure ability to identify whether statements of
facts, principles, generalizations, relationships, or evaluative
statements are correct
• Can be factual or can be a thought question that requires
reasoning
• Can be used in most disciplines
• Used to quickly poll a class as an introduction to a discussion
or determine knowledge of topic
41. ADVANTAGES OF TRUE-FALSE QUESTIONS
• Time-efficient administration and scoring:
– Easy to write.
– Easy to score.
– Scoring is objective.
• More information is sampled from a great deal of
content.
• Effective when assessing misconceptions, cause-effect
relations.
42. DISADVANTAGES OF TRUE-FALSE QUESTIONS
• Measures only low level of learning - remembering and
understanding.
• Need a larger number of items to distinguish stronger and
weaker knowledge levels.
• Students have a 50 percent chance of being correct, just by
chance.
• May be perceived as an unfair judgment of learning.
• Encourages guessing since there are only two alternatives.
44. CONSTRUCTING TRUE-FALSE QUESTIONS
True-False Topic
• Make questions require application of knowledge as much as
possible.
• Always make certain the statement is unequivocally true or
false.
• When using opinion statements, cite the source that will make
the complete statement true or false.
45. TRUE-FALSE PROTOCOL
• Develop approximately the same number of true and false
statements.
• Do not consistently make true statements longer than false
statements, or vice versa.
• Avoid negative and complicated statements.
• Words such as "all," "only," "never," and "always" give hints about the
answer and are usually false. Use them only when the question is to
measure understanding.
• Words such as "generally" and "usually" tend to be true, so avoid their
use.
• Do not randomize the response order in true/false questions.
46. GOOD EXAMPLE OF A TRUE-FALSE QUESTION
1. The true/false item is also called an alternative-
response item.
a. True
b. False
47. BAD EXAMPLE OF A TRUE-FALSE QUESTION
1. The true-false item, which is favored by all test
experts, is also called an alternative-response
item.
a. True
b. False
The question contains two ideas and therefore is confusing.
48. SHORT ANSWER QUESTIONS
• Short answer questions generally ask for brief, text-based responses
and may also be referred to as fill-in-the-blank;
or completion questions.
• Variations of the short answer question may request a list of terms or
rules in which the order is not important, or may require a numerical
or formula response.
49. Here is some general information about short answer questions:
• Does not measure interpretation.
• Can be used to check for preciseness such as correct spelling (good
when using computer grading), proper or specific names of things,
especially factual knowledge, and proper creation of formulas.
• Requires specific, definite, exact information.
• Can be used to discriminate whether errors can be detected in a
diagram, for example.
50. ADVANTAGES OF SHORT ANSWER QUESTIONS
• Easy to write.
• Reduces possibility of guessing.
• Can have a lengthy stem such as a paragraph. (Caution: You
generally should not expect an exact answer character-by-
character.)
• May be easy to score if the required answer is short.
51. DISADVANTAGES OF SHORT ANSWER QUESTIONS
• It can take time to create items with complex formulas.
• Can be turned into a measure of memorization ability.
• Grading can be subjective.
• Correct responses may appear incorrect due to minor errors
such as misspellings, order of words, etc.
• Difficult to machine score. Much work is being conducted in
this area, but it is still in early stages of development.
53. CONSTRUCTING SHORT ANSWER QUESTIONS
Short Answer Content
• Design for recall of specific names, facts, and basic knowledge.
• Focus the question so there a limited number of possible correct responses
(or only one) in order to discriminate achievement.
• Make sure that question provides enough information so that students
know what constitutes a correct response. For example, if you discussed
four books written by a certain author and you want students to name
them, be specific on how many constitute a correct answer (i.e., name two
of four, or name any three, etc.).
54. SHORT ANSWER PROTOCOL
• Make sure that the answer will require only a few words.
• If a list is expected, limit the number of items to be listed to 6 or so.
• Write the stem in your own words; that is, don't use wording directly
from a text or reading assignments.
• Make the stem complete enough so that the meaning is clear.
• When the stem is an incomplete statement, place the "blank" for the
correct choice at the end of the statement.
• When a negative item is used, emphasize the negative word or phrase,
that is, underline, capitalize, or italicize, for example, "DOES NOT."
55. GOOD EXAMPLE OF A SHORT ANSWER QUESTION
1. In our class discussion on regression, what two
terms were used interchangeably with the term
"independent variable"?
• Answer: predictor variable and explanatory
variable
56. BAD EXAMPLE OF A SHORT ANSWER QUESTION
1. What does the acronym ENIAC mean?
Answer: Electronic Numerical Integrator and Calculator
Main Errors
• Perhaps knowledge of the exact words is not important.
• It's not wise to expect an exact response when "numerical" could be
remembered as "numeric" and "calculator" could be remembered as
"calculation," etc.
57. ACTIVITY
Create a 40 point exam using 10 days meeting
• TOS
• 10 items Multiple choice
• 10 items True or False
• 10 items Short Answer
• 10 point Essay
