2. Questions Improve the Learning Process
Slide 2
Knowledge/Skill
Time
ReduceForgetting
ReduceForgetting
Creates
Intrigue
Strengthens
Memory
Questions….
3. Slide 3
Dr. Will Thalheimer research:
Learning Benefits from Questions Learning Benefit
Min Max
Pre-questions (to focus attention) 5% 40%
Repetitions
Initial
Subsequent
30%
15%
110%
40%
Feedback 15% 50%
Retrieval Practice 30% 100%
Questions Spaced Over Time 5% 40%
100% 380%
4. Guiding Principles for Evaluation
1. Match the right question type to the right skill
2. Carefully construct your questions
3. Provide feedback to guide learning
4. Organize questions in a centralized item bank
5. Randomize choices and questions
6. Align delivery environment to assessment stakes (*)
7. Accommodate participant needs: blended delivery (*)
8. Analyze results and sharing with stakeholders
5. Questions to Ask Yourself
• What objectives will (should) I be assessing?
• What types of items will be included in the test?
– 2i
– 2a
– 2b
– 2c
• How long will the test be in terms of time and number of items?
6. Common question types
• True/False / Yes/No - the participant selects one of two potential
choices
• Multiple choice - the participant selects one choice from a group of
possible answers
• Multiple response - Similar to multiple choice except the participant can
select one or more choices
• Numeric questions - a participant is prompted to enter a numeric value
• Fill in the blanks - participant is presented with a statement where one
or more words are missing
Slide 6
7. Question Types verses
Cognitive Levels of Learning
Knowledge
Comprehension
Application Analysis
Synthesis
Evaluation
2i, 2a 2b 2c
Multiple Choice (MC)
True/False (TF)
Matching
Completion
Short Answer
MC
Short Answer
Problems
Essay
Performance
MC
Short Answer
Essay
8. Planning
1. Outline learning objectives or major concepts to be covered by the
assessments
• Test should be representative of objectives and material covered
• Major Learner complaint: Tests don’t fairly cover the material that was supposed to be
canvassed on the test
2. Create a test blueprint – Currently working on
3. Create questions based on blueprint – Why we are here today
– Match the question type with the appropriate level of learning
4. Organize questions and/or ideas by item types
5. Eliminate similar questions and organize questions logically
6. Validate the tests
7. Analyze the results
9. Don’ts
Avoid… Why
Avoid negatives, especially double
negatives.
Can make questions confusing to the
participant.
Avoid "All of the above" and use of
"None of the above" with caution.
Make it much easier for a participant to
eliminate obvious choices and “guess”
an answer
Avoid vague pronouns (e.g. it, they). Can make questions ambiguous and
confusing to the participant.
Avoid unnecessary modifiers,
especially absolutes (e.g. always,
never, all, none).
“Always” and “Never”, etc., can make it
easier guess an answer
Slide 9
10. Don’ts
Avoid… Why
Avoid providing cues to correct answer
in the stem.
Giving the answer away means the
question will not effectively measure
the skill/knowledge it was intended to
measure
Avoid providing clues to the answer of
one question in another question.
Giving the answer away means the
question will not effectively measure
the skill/knowledge it was intended to
measure
Avoid conflicting alternatives. Can make questions ambiguous and
confusing to the participant.
Slide 10
11. Do’s
Do Why
Keep stems and statements as short
as possible and use clear, concise
language.
Provides clarity to what is being asked;
minimizes ambiguity
Use questions whenever possible
(What, Who, When, Where, Why and
How)
Provides clarity to what is being asked;
minimizes ambiguity
Maintain grammatical consistency to
avoid cueing (Parallel Structure).
Helps to avoid providing cues to the
correct answer
List choices in a logical order. Helps to avoid providing cues to the
correct answer
Optional – in many cases you may wish
to shuffle choices.
Slide 11
13. Multiple Choice Items
• Advantages:
– Extremely versatile-can measure the higher level mental processes (application,
analysis, synthesis and evaluation)
– A compromise between a short answer/essay and T/F item
– Can cover a wide range of content can be sampled by one test
• Disadvantages
– Difficult to construct plausible alternative responses
14. 14
Multiple Choice Question Construction
Stem: A 65-year-old man has difficulty rising
from a seated position and
straightening his trunk, but he has no
difficulty flexing his leg.
Lead-in: Which of the following muscles is most
likely to have been injured?
Options: A. Gluteus maximus* Answer
B. Gluteus minimus
C. Hamstrings
D. Iliopsoas
E. Obturator internus
Distractors
15. 15
Have a Distinctive Shape
Stem
Lead In
Option A (distractor)
Option B (answer)
Option C (distractor)
Option D (distractor)
Option E (distractor)
16. 16
Poorly Shaped MC Question
Short stem & lead-in
a. Long option
b. Long option
c. Long option
d. Long option
17. Consider this question:
• Which of the following are not disadvantages of hybrid cars when
compared to similar gasoline powered vehicles? (select all that apply)
A. Lower sticker price
B. Better fuel economy
C. Lower carbon emissions
D. Better audio systems
Slide 17
18. Consider this question:
• Which of the following are not disadvantages of hybrid cars when
compared to similar gasoline powered vehicles? (select all that apply)
A. Lower sticker price
B. Better fuel economy
C. Lower carbon emissions
D. Better audio systems
Slide 18
Double Negative
19. Better wording...
• Which of the following are advantages of hybrid cars when compared to
similar gasoline powered vehicles? (select all that apply)
A. Lower sticker price
B. Better fuel economy
C. Lower carbon emission
D. Better audio systems
Slide 19
20. Consider this question:
• The President and Vice President of the United States are
chosen by the Electoral College.
A. True
B. False
Slide 20
21. Consider this question:
• “The President and Vice President of the United States are
chosen by the Electoral College.”
– Ambiguous question because...
• This is true to some degree, BUT...
• You could make a strong argument for “False”, making the point that the
Electoral College is mainly a formality because electors are generally
“pledged” (or “instructed who to vote for”) based on the results of the popular
vote in their state.
Slide 21
22. Better wording:
• “The Electoral College meets every four years to cast the official votes for
President and Vice President of the United States.”
– There is no ambiguity here.
Slide 22
23. 1. Write the stem first:
a) Be sure the stem asks a clear question
b) Stems phrased as questions are usually easier to write
c) Stems should not contain a lot of irrelevant info.
d) Be sure the stem is grammatically correct
e) Avoid negatively stated stems (double negative risk)
How to Write Multiple Choice Questions
24. How to Write Multiple Choice Questions
2. Writing the correct response
1. Use same terms/reading level
2. Avoid too many qualifiers
3. Assign a random position in the answer sequence
3. Read the stem and correct response together
4. Generate the distractors/alternative responses
25. Multiple Choice Items
• Other Tips for Constructing MC Items:
– Items should have 3-4 alternatives.
– Stem should present a single, clearly formulated problem
– Simple, understandable, exclude extraneous words from
both stem and alternatives
– Include in the stem any word that are repeated in each
response
– Avoid all of the above (can answer based on partial
information)
– Avoid none of the above
26. 26
Consider this question:
• The primary purpose of the stam is to remove the
– carm
– denton
– menace
– stam bar
27. 27
Consider this question:
• The primary purpose of the stam is to remove the
– carm
– denton
– menace
– stam bar
“Word repeats” in stem and answer
28. 28
Consider this question:
• Which of the following pairs has won the greatest
number of Abby awards?
– Jones and Smith
– Smith and Taylor
– Smith and White
– White and Allen
29. 29
Consider this question:
• Which of the following pairs has won the greatest
number of Abby awards?
– Jones and Smith
– Smith and Taylor
– Smith and White
– White and Allen
Convergence: correct answer has the
most in common with other choices
31. 31
Consider this question:
• How many pounds of pressure are exerted by a
callam?
– 2.6
– 150
– 260
– 2600
Convergence: Correct answer has the most
in common with other choices
33. 33
Consider this question:
• The stanon is aided by a
– instel
– immon
– octal
– port
Grammatical cues: only “port” follows
grammatically from the stem
34. 34
Consider this question:
• The stanon frequently overheats because
– all grestels are bilious
– no immon are directly fectitious
– octals are usually casable
– ports are always critical
35. 35
• The stanon frequently overheats because
– all grestels are bilious
– no immon are directly fectitious
– octals are usually casable
– ports are always critical
Absolute terms: never found in the correct answer
Consider this question:
36. 36
Consider this question:
• Stammation normally occurs when the
– anstels rupture
– immon falls and the denton is in place
– octal rotates easily
– ports pass over the carm
37. 37
Consider this question:
• Stammation normally occurs when the
– anstels rupture
– immon falls and the denton is in place
– octal rotates easily
– ports pass over the carm
Longest answer is usually the correct answer
38. True-False Test Quedstions
• Best suited for testing 3 kinds of info.:
• Knowledge level learning
• Understanding of misconceptions
• When there are two logical responses
• Advantages:
– Sample a large amount of learning per unit of Learner
testing time
• Disadvantages:
– Tends to be very easy
– 50-50 chance of guessing
– Tends to be low in reliability
39. Tips for Constructing True/False Items
–Tips for constructing True-False Items
• Avoid double negatives
• Avoid long or complex sentences
• Specific determiners (always, never, only, etc.) should be
used with caution
• Include only one central idea in each statement
• Avoid emphasizing the trivial
• Exact quantitative (two, three, four) language is better
than qualitative (some, few, many)
• Avoid a pattern of answers
40. Short-Answer Items
• Two Types: (Question and Incomplete Statement)
• Advantages:
– Easy to construct
– Excellent format for measuring who, what, when, and where info.
– Guessing in minimized
– Student must know the material- rather than simply recognize the
answer
• Disadvantages:
– Grading can be time consuming/difficult
– More than one answer can be correct
41. Short Answer Items
• Tips for Constructing Short Answer Items
– Better to supply the term and require a definition
– For numerical answers, indicate the degree of precision
expected and the units in which they are to be expressed.
– Use direct questions rather than incomplete statements
– Try to phrase items so that there is only one possible
correct response
– When incomplete statements are used, do not use more
than one blank within an item.
42. Essay Questions
Types of Essay Questions
• Extended Response Question
– Great deal of latitude on how to respond to a question.
– Example: Discuss essay and multiple-choice type tests.
• Restricted Response Question
– More specific, easier to score, improved reliability and validity
– Example: Compare and contrast the relative advantages of
disadvantages of essay and multiple choice tests with respect to:
reliability, validity, objectivity, & usability.
43. Essay Items
• Advantages:
– Measures higher learning levels (synthesis, evaluation) and
is easier to construct than an objective test item
– Learners are less likely to answer an essay question by
guessing
– Require superior study methods
– Offer Learners an opportunity to demonstrate their abilities
to:
• Organize knowledge
• Express opinions
• Foster creativity
44. Essay/Scenario Items
• May limit the sampling of material covered
• Tends to reduce validity of the test
• Disadvantages
– Subjective unreliable nature of scoring
• “halo effect” – good or bad student’s previous level of performance
• Written expression
• Handwriting legibility
• Grammatical and spelling errors
• Time Consuming
45. Essay/Scenario Items
• Give Learners a clear idea of the scope & direction intended
for the answer
– Might help to start the question with the description of the required
behavior (e.g., compare, analyze)
• Appropriate language level for Learners
• Construct questions that require Learners to demonstrate a
command of background info, but do not simply repeat that
info.
• If question calls for an opinion, be sure that the emphasis is
not on the opinion but on the way its presented or argued.
• Use a larger number of shorter, more specific questions
rather than one or two longer questions so that more
information can be assessed.
46. Essay/Scenario Items
• You might
– Give Learners a pair of sample answers to a question of the type you
will give on the test.
– Sketch out a rubric (grading scheme) for each question before reading
the papers OR randomly select a few to read and make up the grading
scheme based on those answers
– Give Learners a writing rubric
– Detach identifying information and use code numbers instead to avoid
letting personality factors influence you.
– After grading all the papers on one item, reread the first few to make
sure you maintained consistent standards
– Be clear to student the extend to which factors other than content (e.g.,
grammar, handwriting, etc.) will influence the grade.
47. Essay Questions
• Tips for constructing Essay Questions
– Provide reasonable time limits for each question
• “thinking and writing time”
– Avoid permitting Learners a choice of questions
• Will not necessarily get a representative sample of student
achievement. Only be requiring all Learners to answer all questions
can their achievement be compared
– A definite task should be put forth to the student
• Critical words: compare, contrast, analyze, evaluate, etc.
48. Scoring Essay Items
• Write an outline of the key points (use outline to design a
rubric)
• Determine how many points are to be assigned to the
question as a whole and to the various parts within it.
• If possible, score the test without knowledge of the student’s
name
– Face Sheet
• Score all of the answers to one question before proceeding to
the next question
– Consistent standard
