Question Generation paper.

1. Yanghoon Kim, Hwanhee Lee, Joongbo Shin and Kyomin Jung
Improving Neural Question Generation
using Answer Separation
김양훈

2. Background
Neural question generation (NQG)
- Generating a question from a given text passage with deep neural networks.
Importance of NQG
- Generating questions for educational materials.
- Generating questions for improving QA systems.
2
Original passage: John Francis O’hara was elected president of Notre Dame in 1934.
Generated question 1: Who was elected president of Notre Dame in 1934?
Generated question 2: When was John Francis O’hara elected president of Notre Dame?

3. Problem
Previous NQG systems suffer from a critical problem
- Some models don’t take the question target into account.
- RNNs often follow a shallow generation process.
- Some models can’t well grasp the target answer(question target) .
- A sophisticated proportion of generated questions include word in the target
answer.
3
Original passage: John Francis O’Hara was elected president of Notre Dame in 1934.
Given target answer: John Francis O’hara
Correctly Generated question: Who was elected president of Notre Dame in 1934?
Incorrectly generated question: Who was elected John Francis?

4. Contribution
We propose answer-separated seq2seq
- Treats the target answer(question target) and the passage separately.
- Prevent the generated question from including words in the target answer.
- Better capture the information from both the target answer and the passage
- We propose keyword-net
- Model is consistently aware of the target answer.
- Extract the key information in the target answer.
- We use retrieval style word generator
- Take the word meaning into account when generating words.
4

5. Task Definition
5

6. Model
Base model
- We use RNN encoder-decoder with attention
Answer-separated seq2seq consist of
- Answer-separated passage encoder
- Target answer encoder
- Answer-separated decoder
- keyword-net
- Retrieval style word generator
6

7. Model
Answer-separated passage encoder
- A simple preprocessing of the input passage
- Original passage: Steve Jobs is the founder of Apple.
- Masked passage: Steve Jobs is the <a> .
- A one-layer bi-LSTM
Answer encoder
- A one-layer bi-LSTM
7

8. Model
Answer-separated decoder
- A one-layer LSTM
- keyword-net
- Let the model consistently be aware of the target answer.
- Extract key information.
- Passage: Steve Jobs is the founder of Apple
- Target answer: founder of Apple
8

9. Model
Answer-separated decoder
- Retrieval style word generator by (Ma et al. 2018)*
- seq2seq has tendency to memorize the sequence pattern rather than
reflecting word meanings
- The word generator produces words by querying the distributed word
representations.
*Query and Output: Generating Words by Querying Distributed Word Representations for Paraphrase Generation 9

10. Experiment
Data
- Processed version of SQuAD 1.1
- Data split 1: 70,484/10,570/11,877 (train/dev/test)
- Data split 2: 86,635/8,965/8,964
Evaluation
Our model(Ass2s) outperform the previous state-of-the-art model
10

11. Experiment
Impact of answer separation
- Ability to capture target answer
- We checked if the target answer is included in the generated question
- AP : Answer position Feature (BIO scheme)
- (Song et al. 2018) used the copy mechanism.
11
Our model has better ability to generate the right question given the target answer

12. Experiment
Impact of answer separation
- Interrogative word prediction
- “What” takes up more than half of the whole training set
- “Which” : “Which year” can be represented as “When”
- “why”, “yes/no” : only takes up 1.5% and 1.2% of the training set.
12
Our model has better ability to predict the question type for the given target answer

13. Experiment
Impact of answer separation
- Attention from <answer>
- (a) is the attention matrix from our model
- (b) is the attention matrix from seq2seq + AP
- <a> token gives the highest attention weights to the interrogative word “who” in (a)
13

14. Experiment
Question generation for machine comprehension
- Use named entities as target answers, generate synthetic data for machine
comprehension system(QA net by Google).
- ALL : Evaluation result of SQuAD dev set(10k)
- NER : Evaluation result of partial SQuAD dev set(4k)
- answers of single named entity
14

15. Conclusion
We propose Answer-separated seq2seq for NQG
- Separate utilization of target answer and the passage(without target answer)
- By masking the target answer inside the passage
- By using keyword-net to extract key feature from target answer
- By using retrieval style word generator to capture word meaning information
- Our model can
- Reduce the probability that the target answer is included by the generated question
- Generate fluent and right question for the given passage and the target answer
- Better inference the type of question

16. Thank you for listening!
Code, paper: https://yanghoonkim.github.io
Questions: ad26kr@snu.ac.kr