Successfully reported this slideshow.
Your SlideShare is downloading. ×

Reading Comprehension Quiz Generation using Generative Pre-trained Transformers

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 19 Ad

Reading Comprehension Quiz Generation using Generative Pre-trained Transformers

Download to read offline

Recent advances in AI have resulted in large pre-trained language models with superior performance on text generation tasks, prompting the question of whether we can use them to generate edu- cationally useful text completions. This holds the potential to generate relevant quizzes for any educational text, greatly complementing current formative and summative tests from education professionals. We explore pre-trained language models for quiz generation on reading comprehen- sion texts and propose EduQuiz, an end-to-end quiz generator based on a GPT-3 model fine-tuned on text-quiz pairs, able to generate a com- plete multiple-choice question, with the correct and distractor answers. We observed that the majority of generated quizzes is reasonable, and that generation of high-quality distractors is more challenging than ques- tion and answer generation. More generally, while it may be too early to replace manually generated tests for summative feedback and grad- ing with automatic quiz generation, EduQuiz already has potential value for formative feedback and to increase engagement during the learning phase by enhancing textbooks with assessments.

Recent advances in AI have resulted in large pre-trained language models with superior performance on text generation tasks, prompting the question of whether we can use them to generate edu- cationally useful text completions. This holds the potential to generate relevant quizzes for any educational text, greatly complementing current formative and summative tests from education professionals. We explore pre-trained language models for quiz generation on reading comprehen- sion texts and propose EduQuiz, an end-to-end quiz generator based on a GPT-3 model fine-tuned on text-quiz pairs, able to generate a com- plete multiple-choice question, with the correct and distractor answers. We observed that the majority of generated quizzes is reasonable, and that generation of high-quality distractors is more challenging than ques- tion and answer generation. More generally, while it may be too early to replace manually generated tests for summative feedback and grad- ing with automatic quiz generation, EduQuiz already has potential value for formative feedback and to increase engagement during the learning phase by enhancing textbooks with assessments.

Advertisement
Advertisement

More Related Content

More from Sergey Sosnovsky (15)

Recently uploaded (20)

Advertisement

Reading Comprehension Quiz Generation using Generative Pre-trained Transformers

  1. 1. Reading Comprehension Quiz Generation using Generative Pre-trained Transformers Ramon Dijkstra, Zülküf Genç, Subhradeep Kayal and Jaap Kamps The 23th International Conference on Artificial Intelligence in Education (AIED’2022) Fourth Workshop on Intelligent Textbooks (iTextbooks) 27 July 2022
  2. 2. Agenda • Background • Goal • Demo • Approach • Experimental Setup • Experimental Results • Analysis • Revisiting the demo • Main Takeaways • Q&A
  3. 3. Background Quiz Generation • Question Generation • Question Answering • Distractor Generation
  4. 4. Goal Educational Text → Multiple-choice quiz Why? • Enhance intelligent textbooks with assessments • Students could test themselves during the learning phase • Teachers could use the tool to generate assessments
  5. 5. Demo
  6. 6. Approach Large pre-trained transformers have shown superior perfomances on several text generation tasks. Generative Pre-trained Transformer 3 (GPT-3) can be finetuned to downstream tasks using the API of OpenAI: • Train on prompt-completion pairs • Give a never-seen before prompt during inference
  7. 7. Approach Prompt: Educational Text Completion: Quiz End-to-End Quiz Generation Template Question: . . . True answer: . . . False answer: . . . False answer: . . . False answer: . . . We will call this finetuned model EduQuiz.
  8. 8. Experimental Setup – quiz generation techniques Two quiz generation techniques: • Step-Wise Quiz Generation (SWQG) • End-to-End Quiz Generation (EEQG)
  9. 9. Experimental Setup – models Two models: • GPT-3 • Macaw-11b Macaw-11b is a general-purpose model trained on various text generation tasks.
  10. 10. Experimental Setup – dataset EQG-RACE dataset: • 18.501 train • 1.035 validation • 950 test Processed RACE dataset where only examination questions are kept.
  11. 11. Experimental Setup – automatic evaluation Metrics: • BLEU-4: measures the 4-gram similarity between a prediction and ground truth instances • ROUGE-L: measures the longest common sub-sequence between the prediction and ground truth instances • METEOR: is similar in comparison to BLEU-4 but also takes synonyms, stemming, and paraphrasing into account
  12. 12. Experimental Results Macaw-11b did have the configurations to perform EEQG but our experiments showed that only 14% of the generated quizzes contained all the elements of the quiz. Therefore, we will not use Macaw-11b for EEQG.
  13. 13. Experimental Results Original quiz: Question: What is Edell’s strongest character? True answer: Liking to try new things. False answer: Liking to remember things. False answer: Liking to help others. False answer: Liking to think of problems differently. Macaw-11b SWQG generated quiz (BLEU-4 = 0.0): Question: The writer’s mother told her that if she wanted to be successful in life, she should be prepared for any kind of change. What did Edell do to prepare for the Asian MTV Awards? True answer: she read the biographies of the guests False answer: she practiced her English with her friends False answer: she made a list of the names of the guests False answer: she made a plan to talk with the guests GPT-3 SWQG generated quiz (BLEU-4 = 0.0): Question: What advice does Edell give us? True answer: Get ready for any opportunity in life. False answer: Try to live a colorful life. False answer: Take any chance that comes up. False answer: Explore your potential talents. EduQuiz-generated quiz (BLEU-4 = 0.0): Question: What advice does Edell give to young people? True answer: Try to get yourself well-prepared in life. False answer: Have a rich collection of CDs. False answer: Never miss an opportunity to learn ballet. False answer: Be a hostess of the Asian MTV Awards.
  14. 14. Analysis - metrics
  15. 15. Analysis - results
  16. 16. Revisiting the demo
  17. 17. Revisiting the demo
  18. 18. Main Takeaways • Already useful for formative feedback and to increase engagement during the learning phase • Currently only limited to English language and reading comprehension texts • Too early to replace educational professionals • Current performances require a human-in-the-loop to check the quality
  19. 19. Q&A

×