Generating Adequate Distractors for Multiple-Choice Questions

1. Learning with PurposeLearning with Purpose
Generating Adequate Distractors for
Multiple-Choice Questions
Authors: Cheng Zhang, Yicheng Sun,
Hejia Chen and Jie Wang
Presenter: Cheng Zhang
University of Massachusetts Lowell, USA

2. Learning with Purpose
1. Introduction
2. Distractor Generation
3. Evaluations
4. Conclusions
Overview

3. Learning with Purpose
An approach to automatic generation of adequate distractors for
a given question answer pair (QAP) to form an adequate
multiple-choice question (MCQ).
Combination of part-of-speech tagging, named-entity tagging,
semantic-role labeling, regular expressions, domain knowledge
bases, word embeddings, word edit distance, WordNet, and
other algorithms.
Evaluations by human judges, each MCQ has at least one
adequate distractor and 84% of MCQs have three adequate
distractors
Introduction
Abstract

4. Learning with Purpose
Methods of generating adequate distractors are typically
following two directions (Pho et al., 2014; Rao and Saha, 2018):
• 1. Domain specific knowledge bases
• 2. Semantic similarity
Previous efforts have focused on finding some forms of
distractors, instead of making them look more distracting.
Introduction
Background

5. Learning with Purpose
The Generated adequate distractor must satisfy the following
requirements:
• It is an incorrect answer to the question.
• It is grammatically correct.
• It is semantically related to the correct answer.
• It must provide enough distraction.
Introduction
Our Goals

6. Learning with Purpose
Input:
• Original article
• Answer in QAP
The fixed order of distractor generation for each target word:
1. Subjects,
2. Objects,
3. Adjectives for subjects,
4. Adjectives for objects,
5. Predicates,
6. Adverbs
Distractor Generation
Output:
• Distractors

7. Learning with Purpose
Three type of target word:
• Type-1: time point, time range,
numerical number, ordinal
number.
• Type-2: person, location,
organization.
• Type-3: others.
Distractor Generation

8. Learning with Purpose
Distractor candidates for Type-3:
• Semantic similarly words
• Hypernyms
• Antonyms
Filter out unsuitable candidates:
• Distractor candidates that contain the target word.
• Distractor candidates that have the same prefix of the target word
with edit distance less than three.
• E.g. Misspelled: “knowledge” vs “knowladge”
• Different tense: “try” vs “tries”
Distractor Generation
Target word in Type-3

9. Learning with Purpose
For each distractor candidate 𝑊𝑐 with target word 𝑊𝑡 :
𝑆 𝑣 = Word embedding cosine similarity score.
𝑆 𝑛 = WordNet WUP (Wu and Palmer, 1994) similarity score.
𝑆 𝑑 = Edit distance score.
where E is the edit distance.
Distractor Generation
Ranking Algorithm

10. Learning with Purpose
R = Ranking score.
if 𝑊𝑐 is an antonym of 𝑊𝑡
otherwise
Note that 𝑆 𝑣, 𝑆 𝑛, 𝑆 𝑑 are each between 0 and 1, and so 𝑅′(𝑊𝑐, 𝑊𝑡)
is between 0 and 1, which implies that log 𝑅′
𝑊𝑐, 𝑊𝑡 > 0 .
Also note that we give more weight to antonyms.
Distractor Generation
Ranking Algorithm

11. Learning with Purpose
U.S. SAT practice reading tests as a dataset.
Total of 303 distractors for evaluation.
Evaluated by human judgment.
Evaluation result:
• All distractors generated by our method are grammatically correct.
• 98% distractors are relevant to the QAP with distraction.
• 96% distractors provide sufficient distraction.
• 84% MCQs are adequate.
• All MCQs are acceptable (i.e., with at least one adequate distractor).
Evaluation

12. Learning with Purpose
What did Chie hear? (SAT practice test 1 article 1)
• her soft scuttling footsteps, the creak of the driveway.
• her soft scuttling footsteps, the creak of the stairwell.
• her soft scuttling footsteps, the knock of the door.
• her soft scuttling footsteps, the creak of the door. (Correct answer)
When should ethics apply? (SAT practice test 2 article 2)
• when someone makes an economic request.
• when someone makes an economic proposition.
• when someone makes a political decision.
• when someone makes an economic decision. (Correct answer)
Evaluation
Examples

13. Learning with Purpose
We presented a novel method using various NLP tools for
generating adequate distractors.
Improve the ranking measure to help select a better distractor
for a target word from a list of candidates.
Explore how to produce generative distractors using neural
networks, instead of just replacing a few target words in a given
answer.
Conclusions

14. Learning with PurposeLearning with Purpose
Thank you