Download as PDF, PPTX
More Related Content
Similar to Story generation-Sarah Saneei
Story generation-Sarah Saneei
- 1. SNLP Presentation Sarah Saneei Studyingmaster of Computational Linguistics
- 2. Neural Text Generationin Stories using Entity Representation as Context Elizabeth Clark Yangfeng Ji Noah A. Smith
- 3. How was theidea born? Neural Text Generation in Stories using Entity Representation as Context RNN Topical inf Neural Models for Text Generation S2SA Story Generation Inspired by Ref. exp. Generation & Entity Prediction Defined by authors Mention Generation entity ~ coherence Centering Theory EntityNLMEntity-related Generation Used for coref. , Lang. Model & scoring
- 4. How was theidea born? Neural Text Generation in Stories using Entity Representation as Context Topical inf Neural Models for Text Generation Story Generation Inspired by Ref. exp. Generation & Entity Prediction Defined by authors Mention Generation ~ coherence Centering Theory Entity-related Generation Used for coref. , Lang. Model & scoring
- 5. Model Description • Entity-basedgeneration model (ENGEN) • Combines 3 sources of contextual information for text generation • Each of these types of info is encoded in vector form ENGEN Context from entities Content of current sentence Context from previous sentence
- 6. Model Description • Entity-basedgeneration model (ENGEN) • Combines 3 sources of contextual information for text generation • Each of these types of info is encoded in vector form ENGEN Context from entities Content of current sentence Context from previous sentenceℎ 𝑡,𝑖 𝑒𝑡,𝑖 𝑝𝑡,𝑖
- 7. Model Description ENGEN Context from entities Content of current sentence Context from previous sentenceℎ𝑡,𝑖 𝑒𝑡,𝑖 𝑝𝑡,𝑖 • Every entity is assigned a vector representation • Update every time the entity mentioned • Appropriate for generating narrative stories • Characters develop and change • Each time m+1 options (why?)
- 8. Model Description ENGEN Context from entities Content of current sentence Context from previous sentenceℎ𝑡,𝑖 𝑒𝑡,𝑖 𝑝𝑡,𝑖 • When an entity is selected, its vector is assigned to • : • Should not refer to an entity • Rep. of most recently mentioned entity • Should refer to a new entity • Generate from a normal dist. • has been generated • Entity rep. update base on 𝑒 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑤𝑡 𝑒 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑤𝑡 ℎ 𝑡
- 9.
- 10. Model Description ENGEN Context from entities Content of current sentence Context from previous sentence S2SA ENTITYNLM Capturelocal Contextual features • No extra param • : Combined context vector use to generate by calculating probability of each word type in the vocabulary 𝑐𝑡 𝑤𝑡 Eq. 6
- 11. Learning • Training objective: maximize All the model’s paramAll decisions at timestep t
- 12. Learning • Training objective: maximize • The model • Predicts the word • The entity info. associated with that word • Same training method used for ENTITYNLM • Requires training data annotated with mention and coref. Info. (entity cluster) All the model’s paramAll decisions at timestep t (whether it is part of a entity mention, if so the entity the mention refers to, the length of the mention, and the word itself)
- 13. Data • Training data:312 adventure books from the Toronto Book Corpus • Development: 39 books • Test: 39 books • Divide to segments (include up to 50 sentences) • Used Stanford CoreNLP system • In coref. results: Entity mentions: even more than 70 keep only 3 word or fewer(95% of mentions) others replace by head word All tokens were down cased Numbers > NUM Frequency less than 10 > UNK
- 14. 1 Mention Generation 2 Pairwise SentenceSelection 3 Sentence Generation EVALUATIONS
- 15. 1/ Experiment: MentionGeneration • Given a text and a slot to be filled with an entity mention, • A model must choose among all preceding entity mentions and the correct mention • To perform well: • Choose both the entity and the words used to refer to it • Requires the greatest precision: • Possible to select the correct mention • But not the correct cluster and vice versa select between all the previous entity mentions (Emily, the dragon, Seth, and her) and the correct mention (she).
- 16. 1/ Experiment: MentionGeneration • Given a text and a slot to be filled with an entity mention, • A model must choose among all preceding entity mentions and the correct mention S2SA does not model entities
- 17. 1/ Experiment: MentionGeneration • Size of candidate list can exceed 100 • Reporting MAP (Mean Average Precision) to aggregate across contexts of all lengths • Using language model scores to rank candidates
- 18. 1/ Experiment: MentionGeneration • Size of candidate list can exceed 100 • Reporting MAP (Mean Average Precision) to aggregate across contexts of all lengths • Using language model scores to rank candidates Baselines: • S2SA • ENTITYNLM • Reverse order (rank mentions by recency)
- 19. 1/ Experiment: MentionGeneration Baselines: • S2SA • ENTITYNLM • Reverse order (rank mentions by recency) Higher MAP > better system
- 20. 1/ Experiment: MentionGeneration • Line 1: distance alone not an effective heuristic for mention generation (but useful in coref.) • Line 4 and 2: benefit of adding entity representations for text generation • Line 3 and 4: local context also gives small boost
- 21. 1 Mention Generation 2 Pairwise SentenceSelection 3 Sentence Generation EVALUATIONS
- 22. 2/ Experiment: PairwiseSentence Selection • Inspired by tests of coherence • to assess text generation automatically • without human evaluation • Model is generative • Can assign scores to candidate sentences, given a context
- 23. 2/ Experiment: PairwiseSentence Selection Inputs: • 49 sentences of preceding context • Two choices: • The actual 50th • A distractor sentence (randomly chosen from next 50 sentences) • Random baseline achieve 50% accuracy • Not a trivial task: • Distractor: similar lang., chars, topics • Relatively nearby (in 2% cases, the very next sentence)
- 24. 2/ Experiment: PairwiseSentence Selection Inputs: • 49 sentences of preceding context • Two choices: • The actual 50th • A distractor sentence (randomly chosen from next 50 sentences) 48 lines away 10 lines away
- 25. 2/ Experiment: PairwiseSentence Selection To select • Model scores each of two based on • Its probability on words • All entity-related info. (Eq 6) 48 lines away 10 lines away
- 26. 2/ Experiment: PairwiseSentence Selection To select • Model scores each of two based on • Its probability on words • All entity-related info. (Eq 6) Ran this pairwise decision 5 times and average the performance • Different set of random distractors
- 27. 2/ Experiment: PairwiseSentence Selection To select • Model scores each of two based on • Its probability on words • All entity-related info. (Eq 6) Ran this pairwise decision 5 times and average the performance • Different set of random distractors
- 28. 2/ Experiment: PairwiseSentence Selection To select • Model scores each of two based on • Its probability on words • All entity-related info. (Eq 6) Ran this pairwise decision 5 times and average the performance • Different set of random distractors • Unlike the mention generation task • S2SA beats ENTITYNLM • Importance of local context
- 29. 2/ Experiment: PairwiseSentence Selection To select • Model scores each of two based on • Its probability on words • All entity-related info. (Eq 6) Ran this pairwise decision 5 times and average the performance • Different set of random distractors • Unlike the mention generation task • S2SA beats ENTITYNLM • Importance of local context Consistency (regardless of the distance of distractors)
- 30. 1 Mention Generation 2 Pairwise SentenceSelection 3 Sentence Generation EVALUATIONS
- 31. 3/ Human evaluation:Sentence Generation • Best measure of the quality • Among AMTs (Amazon Mechanical Turkers) • Americans • Completed over 1000 tasks • Over 95% task acceptance rate • 11 selected
- 32. 3/ Human evaluation:Sentence Generation • Best measure of the quality • Among AMTs (Amazon Mechanical Turkers) • Americans • Completed over 1000 tasks • Over 95% task acceptance rate • 11 selected • Input: • Short excerpt from a story • Two generated sentences (ENGEN & entity-unaware S2SA) • Asking them to “Choose a sentence to continue the story” and explain why • Not primed by “focus on entities”
- 33. 3/ Human evaluation:Sentence Generation • Subset of 50 randomly selected text • Final 60 words of segments selected • Same for model • Generated 100 sentences and choose the best one (ranked with 5-gram)
- 34. 3/ Human evaluation:Sentence Generation • Subset of 50 randomly selected text • Final 60 words of segments selected • Same for model • Generated 100 sentences and choose the best one (ranked with 5-gram) • Results • 27 of the passages > ENGEN • 23 > S2SA • Many cases of same scores (both would have worked)
- 35. 3/ Human evaluation:Sentence Generation • Reasons: • ENGEN > Connection between pronouns • ENGEN> Mismatch in entities (starting with “she” while no female character exists) • S2SA > new proper noun (gives some context of characters of the story) • Importance the ability to generate new entities • Move the plot forward • Fit better with “the theme” or “the tone” • Dialogue vs. descriptive sentence • Statement vs. question • Social knowledge
- 36. 3/ Human evaluation:Sentence Generation
- 37. 3/ Human evaluation:Sentence Generation “the introduction makes the man sound like he is a stranger, so ‘I’m proud of you’ seems out of place.”
- 38. 1 Mention Generation 2 Pairwise SentenceSelection 3 Sentence Generation EVALUATIONS
- 39.
- 40. Thanks for yourattention :)