Your SlideShare is downloading. ×
Varga ha
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Varga ha

423
views

Published on

This paper presents the question generation system used by the University of Wolverhampton in QGSTEC 2010 task B. We have modified our multiple-choice question generation (MCQ) system in order to …

This paper presents the question generation system used by the University of Wolverhampton in QGSTEC 2010 task B. We have modified our multiple-choice question generation (MCQ) system in order to generate a new type of questions as requested by this task. We have also removed several constraints from our original system in order to generate more questions. In this paper, we will describe our approach and manual evaluation results on the development data #QGSTEC #WLV


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
423
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. WLV: a question generation system for QGSTEC 2010 task B Andrea Varga and Le An Ha 1 Research Group in Computational Linguistics University of Wolverhampton 18 June 2010 / QGSTEC 2010
  • 2. OutlineTask B: Question generation from a single sentenceOur previous experience in question generationOur method used to solve task BEvaluation results on development data setConclusions
  • 3. Task B: Question generation from a single sentence: Input: a sentences from Wikipedia, OpenLearn, Yahoo!Answers or similar data sources a specific target question type (which, what, who, when, where, why, how many/long, yes/no) Output: 2 questions generated per question type Example: <instance ="4"> <source>K100_2</source> <text>In 1996, the trust employed over 7,000 staff and managed another six sites in Leeds and the surrounding area.</text> <question type="where">Where did the trust employ over 7,000 staff and manage another six sites?</question> <question type="where" /> <question type="when">When did the trust employ over 7,000 staff and manage another six sites in Leeds and the surrounding area?</question> <question type="when" /> <question type="how many">In 1996, the trust employed how many staff and managed another six sites in Leeds and the surrounding area?</question> <question type="how many">In 1996, the trust employed 7,000 staff and managed how many sites in Leeds and the surrounding area?</question> </instance>
  • 4. Our previous experience in question generation:Initial multiple-choice question (MCQ) generation system Our previous work: Mitkov and Ha (2003) Mitkov et al. (2006) Input: instructive text (textbook chapters and encyclopaedia entries) Performed tasks: term extraction -noun phrases satisfying the [AN]+N or [AN]*NP[AN]*N regular expression question generation sentence filtering constraints -the terms occur in the main clauses or subordinate clauses -the sentence has coordinate structure -the sentence contains negations distractor selection Resources: Corpora and ontologies (WordNet) Question types: which, how many
  • 5. Our method used to solve task B:Modified question generation system Input: single sentence Performed tasks: identification of key phrases -noun phrases satisfying the [AN]+N or [AN]*NP[AN]*N regular expression - preporsitional phrases - adverbial phrases assignment of semantic types - a named entity recognition (NER) module assigns for the head of each phrase a semantic type: location; person; time; number; other identification of question type question generation - we added few more syntactic rules for the missing question types - we removed several constraints Question types: which, what, who, when, where, why, how many
  • 6. Our method used to solve task B:Question generation: "WhichH VO"; "WhichH do-support SV" (1) Input: source clauses are: finite contain at least one key phrase of subject-verb-object (SVO) or SV structure Which and What questions -key phrases: all the NPs S(key phrase)VO => "WhichH VO" where WhichH is replaced by: -"Which" + head of NP (in case of multi-word phrase) -"Which" + hypernym of the word from WordNet (in case of single-word phrase) S(key phrase)VO => "What VO" SVO(key phrase) => "WhichH do-support SV" SVO(key phrase) => "What do-support SV"
  • 7. Our method used to solve task B:Question generation: "WhichH VO"; "WhichH do-support SV" (2) Who, Whose and Whom -key phrases: NPs recognised as person names for NP in subject position S(key phrase)VO => "Who VO" for NP in possessive structure S(key phrase)VO => "Whose VO" for NP in any other position S(key phrase)VO => "Whom VO" When and Where -key phrases for the when questions:NPs, PPs and AdvPs (being the extent of a temporal expression) -key phrases for where questions: NPs, PPs (the head of the phrases recognised as location) S(key phrase)VO => When VO S(key phrase)VO => Where VO SVO(key phrase) => When do-support SV SVO(key phrase) => Where do-support SV subclauses containing the answer are ignored
  • 8. Our method used to solve task B:Question generation: "WhichH VO"; "WhichH do-support SV" (3) Why -key phrases: NPs Why do-support VO; ignoring the subclause containing the answer How many -key phrases: NPs containing numeric expressions S(key phrase)VO => "How many H VO" SVO(key phrase) => "How many H do-support SV" S(key phrase)VO => "How many percent VO" SVO(key phrase) => "How many percent do-support SV"
  • 9. Evaluation results on development data set:Manual evaluation results 115 questions has been generated out of 180 because we have not built a model to generate yes/no questions the transformational rules are not able to deal with sentences that are too complicated some of the sentences were incorrectly parsed the system failed to identify any source clause for some sentences kappa agreement on Relevance was 0.21 kappa agreement on Syntactic Correctness and Fluency was 0.22 Human One Human Two Relevance(180 questions) 2.45 2.85 Relevance(115 questions) 1.57 2.20 Syntactic(180 questions) 2.85 3.10 Syntactic(115 questions) 2.20 2.64 Table: average Relevance and Syntactic Correctness and Fluency values
  • 10. Conclusions : we presented our question generation system used to generate questions from a single sentence: 115 questions were generated out of the target 180 questions for the different question types: which, what, who, when, where, how many the generated questions do not score well in both relevancy and syntactic correctness measures the agreement between two human judges is quite low