Recommended
More Related Content
What's hot
What's hot (20)
Similar to SNLP Presentation on Neural Text Generation in Stories using Entity Representation as Context
Similar to SNLP Presentation on Neural Text Generation in Stories using Entity Representation as Context (20)
More from SRah Sanei
Recently uploaded
Recently uploaded (20)
SNLP Presentation on Neural Text Generation in Stories using Entity Representation as Context
- 1. SNLP Presentation Sarah Saneei Studying master of Computational Linguistics
- 2. Neural Text Generation in Stories using Entity Representation as Context Elizabeth Clark Yangfeng Ji Noah A. Smith
- 3. How was the idea born? Neural Text Generation in Stories using Entity Representation as Context RNN Topical inf Neural Models for Text Generation S2SA Story Generation Inspired by Ref. exp. Generation & Entity Prediction Defined by authors Mention Generation entity ~ coherence Centering Theory EntityNLMEntity-related Generation Used for coref. , Lang. Model & scoring
- 4. How was the idea born? Neural Text Generation in Stories using Entity Representation as Context Topical inf Neural Models for Text Generation Story Generation Inspired by Ref. exp. Generation & Entity Prediction Defined by authors Mention Generation ~ coherence Centering Theory Entity-related Generation Used for coref. , Lang. Model & scoring
- 5. Model Description • Entity-based generation model (ENGEN) • Combines 3 sources of contextual information for text generation • Each of these types of info is encoded in vector form ENGEN Context from entities Content of current sentence Context from previous sentence
- 6. Model Description • Entity-based generation model (ENGEN) • Combines 3 sources of contextual information for text generation • Each of these types of info is encoded in vector form ENGEN Context from entities Content of current sentence Context from previous sentenceℎ 𝑡,𝑖 𝑒𝑡,𝑖 𝑝𝑡,𝑖
- 7. Model Description ENGEN Context from entities Content of current sentence Context from previous sentenceℎ 𝑡,𝑖 𝑒𝑡,𝑖 𝑝𝑡,𝑖 • Every entity is assigned a vector representation • Update every time the entity mentioned • Appropriate for generating narrative stories • Characters develop and change • Each time m+1 options (why?)
- 8. Model Description ENGEN Context from entities Content of current sentence Context from previous sentenceℎ 𝑡,𝑖 𝑒𝑡,𝑖 𝑝𝑡,𝑖 • When an entity is selected, its vector is assigned to • : • Should not refer to an entity • Rep. of most recently mentioned entity • Should refer to a new entity • Generate from a normal dist. • has been generated • Entity rep. update base on 𝑒 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑤𝑡 𝑒 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑤𝑡 ℎ 𝑡
- 9. Model Description ENGEN Context from entities Content of current sentence Context from previous sentence S2SA ENTITYNLM Capture local Contextual features
- 10. Model Description ENGEN Context from entities Content of current sentence Context from previous sentence S2SA ENTITYNLM Capture local Contextual features • No extra param • : Combined context vector use to generate by calculating probability of each word type in the vocabulary 𝑐𝑡 𝑤𝑡 Eq. 6
- 11. Learning • Training objective : maximize All the model’s paramAll decisions at timestep t
- 12. Learning • Training objective : maximize • The model • Predicts the word • The entity info. associated with that word • Same training method used for ENTITYNLM • Requires training data annotated with mention and coref. Info. (entity cluster) All the model’s paramAll decisions at timestep t (whether it is part of a entity mention, if so the entity the mention refers to, the length of the mention, and the word itself)
- 13. Data • Training data: 312 adventure books from the Toronto Book Corpus • Development: 39 books • Test: 39 books • Divide to segments (include up to 50 sentences) • Used Stanford CoreNLP system • In coref. results: Entity mentions: even more than 70 keep only 3 word or fewer(95% of mentions) others replace by head word All tokens were down cased Numbers > NUM Frequency less than 10 > UNK
- 14. 1 Mention Generation 2 Pairwise Sentence Selection 3 Sentence Generation EVALUATIONS
- 15. 1/ Experiment: Mention Generation • Given a text and a slot to be filled with an entity mention, • A model must choose among all preceding entity mentions and the correct mention • To perform well: • Choose both the entity and the words used to refer to it • Requires the greatest precision: • Possible to select the correct mention • But not the correct cluster and vice versa select between all the previous entity mentions (Emily, the dragon, Seth, and her) and the correct mention (she).
- 16. 1/ Experiment: Mention Generation • Given a text and a slot to be filled with an entity mention, • A model must choose among all preceding entity mentions and the correct mention S2SA does not model entities
- 17. 1/ Experiment: Mention Generation • Size of candidate list can exceed 100 • Reporting MAP (Mean Average Precision) to aggregate across contexts of all lengths • Using language model scores to rank candidates
- 18. 1/ Experiment: Mention Generation • Size of candidate list can exceed 100 • Reporting MAP (Mean Average Precision) to aggregate across contexts of all lengths • Using language model scores to rank candidates Baselines: • S2SA • ENTITYNLM • Reverse order (rank mentions by recency)
- 19. 1/ Experiment: Mention Generation Baselines: • S2SA • ENTITYNLM • Reverse order (rank mentions by recency) Higher MAP > better system
- 20. 1/ Experiment: Mention Generation • Line 1: distance alone not an effective heuristic for mention generation (but useful in coref.) • Line 4 and 2: benefit of adding entity representations for text generation • Line 3 and 4: local context also gives small boost
- 21. 1 Mention Generation 2 Pairwise Sentence Selection 3 Sentence Generation EVALUATIONS
- 22. 2/ Experiment: Pairwise Sentence Selection • Inspired by tests of coherence • to assess text generation automatically • without human evaluation • Model is generative • Can assign scores to candidate sentences, given a context
- 23. 2/ Experiment: Pairwise Sentence Selection Inputs: • 49 sentences of preceding context • Two choices: • The actual 50th • A distractor sentence (randomly chosen from next 50 sentences) • Random baseline achieve 50% accuracy • Not a trivial task: • Distractor: similar lang., chars, topics • Relatively nearby (in 2% cases, the very next sentence)
- 24. 2/ Experiment: Pairwise Sentence Selection Inputs: • 49 sentences of preceding context • Two choices: • The actual 50th • A distractor sentence (randomly chosen from next 50 sentences) 48 lines away 10 lines away
- 25. 2/ Experiment: Pairwise Sentence Selection To select • Model scores each of two based on • Its probability on words • All entity-related info. (Eq 6) 48 lines away 10 lines away
- 26. 2/ Experiment: Pairwise Sentence Selection To select • Model scores each of two based on • Its probability on words • All entity-related info. (Eq 6) Ran this pairwise decision 5 times and average the performance • Different set of random distractors
- 27. 2/ Experiment: Pairwise Sentence Selection To select • Model scores each of two based on • Its probability on words • All entity-related info. (Eq 6) Ran this pairwise decision 5 times and average the performance • Different set of random distractors
- 28. 2/ Experiment: Pairwise Sentence Selection To select • Model scores each of two based on • Its probability on words • All entity-related info. (Eq 6) Ran this pairwise decision 5 times and average the performance • Different set of random distractors • Unlike the mention generation task • S2SA beats ENTITYNLM • Importance of local context
- 29. 2/ Experiment: Pairwise Sentence Selection To select • Model scores each of two based on • Its probability on words • All entity-related info. (Eq 6) Ran this pairwise decision 5 times and average the performance • Different set of random distractors • Unlike the mention generation task • S2SA beats ENTITYNLM • Importance of local context Consistency (regardless of the distance of distractors)
- 30. 1 Mention Generation 2 Pairwise Sentence Selection 3 Sentence Generation EVALUATIONS
- 31. 3/ Human evaluation: Sentence Generation • Best measure of the quality • Among AMTs (Amazon Mechanical Turkers) • Americans • Completed over 1000 tasks • Over 95% task acceptance rate • 11 selected
- 32. 3/ Human evaluation: Sentence Generation • Best measure of the quality • Among AMTs (Amazon Mechanical Turkers) • Americans • Completed over 1000 tasks • Over 95% task acceptance rate • 11 selected • Input: • Short excerpt from a story • Two generated sentences (ENGEN & entity-unaware S2SA) • Asking them to “Choose a sentence to continue the story” and explain why • Not primed by “focus on entities”
- 33. 3/ Human evaluation: Sentence Generation • Subset of 50 randomly selected text • Final 60 words of segments selected • Same for model • Generated 100 sentences and choose the best one (ranked with 5-gram)
- 34. 3/ Human evaluation: Sentence Generation • Subset of 50 randomly selected text • Final 60 words of segments selected • Same for model • Generated 100 sentences and choose the best one (ranked with 5-gram) • Results • 27 of the passages > ENGEN • 23 > S2SA • Many cases of same scores (both would have worked)
- 35. 3/ Human evaluation: Sentence Generation • Reasons: • ENGEN > Connection between pronouns • ENGEN> Mismatch in entities (starting with “she” while no female character exists) • S2SA > new proper noun (gives some context of characters of the story) • Importance the ability to generate new entities • Move the plot forward • Fit better with “the theme” or “the tone” • Dialogue vs. descriptive sentence • Statement vs. question • Social knowledge
- 36. 3/ Human evaluation: Sentence Generation
- 37. 3/ Human evaluation: Sentence Generation “the introduction makes the man sound like he is a stranger, so ‘I’m proud of you’ seems out of place.”
- 38. 1 Mention Generation 2 Pairwise Sentence Selection 3 Sentence Generation EVALUATIONS
- 40. Thanks for your attention :)