Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Generating Natural-Language Text with Neural Networks

237 views

Published on

Automatic text generation enables computers to summarize text, to have conversations in customer-service and other settings, and to customize content based on the characteristics and goals of the human interlocutor. Using neural networks to automatically generate text is appealing because they can be trained through examples with no need to manually specify what should be said when. In this talk, we will provide an overview of the existing algorithms used in neural text generation, such as sequence2sequence models, reinforcement learning, variational methods, and generative adversarial networks. We will also discuss existing work that specifies how the content of generated text can be determined by manipulating a latent code. The talk will conclude with a discussion of current challenges and shortcomings of neural text generation.

Published in: Data & Analytics
  • Be the first to comment

Generating Natural-Language Text with Neural Networks

  1. 1. Generating Natural-Language Text with Neural Networks Jonathan Mugan @jmugan Austin Data Geeks July 19, 2018
  2. 2. 2 We are also hiring Take our deep learning algorithms and turn them into cool beta software
  3. 3. 3 Computers are Illiterate Reading requires mapping the words on a page to shared concepts in our culture. Writing requires mapping those shared concepts into other words on a page. We currently don’t know how to endow computer systems with a conceptual system rich enough to represent even what a small child knows. However, AI researchers have developed a method that can 1. encode representations of our world, such as images or words on a page, into a space of meanings (proto-read) 2. convert points in that meaning space into language (proto-write) This method uses neural networks, so we call it neural text generation.
  4. 4. 4 Overview of neural text generation world state encoder meaning space decoder n generated text decoder 1 generated text ... Neural text generation is useful for • Machine translation • Image captioning • Style manipulation
  5. 5. 5 Overview of neural text generation world state encoder meaning space decoder n generated text decoder 1 generated text ... • Imagine having the text on your website customized for each user. • Imagine being able to have your boring HR manual translated into the style of Cormac McCarthy. The coffee in the break room is available to all employees, like some primordial soup handed out to the huddled and wasted damned.
  6. 6. 6 Limited by a lack of common sense world state encoder meaning space decoder n generated text decoder 1 generated text ... • It lacks precision: if you ask it to make you a reservation on Tuesday, it may make it on Wednesday because the word vectors are similar (also see [0]). • It also makes mistakes that no human would not make: in image captioning it can mistake a toothbrush for a baseball bat [1]. [0] A Random Walk Through EMNLP 2017 http://approximatelycorrect.com/2017/09/26/a-random-walk-through-emnlp-2017/ [1] Deep Visual-Semantic Alignments for Generating Image Descriptions http://cs.stanford.edu/people/karpathy/deepimagesent/ The beautiful: as our neural networks get richer, the meaning space will get closer to being able to represent the concepts a small child can understand, and we will get closer to human-level literacy. We are taking baby steps toward real intelligence.
  7. 7. 7 Outline • Introduction • Learning from pairs of sentences • Mapping to and from meaning space • Moving beyond pairs of sentences • Conclusion
  8. 8. 8 Outline • Introduction • Learning from pairs of sentences • Overview • Encoders • Decoders • Attention • Uses • Mapping to and from meaning space • Adversarial text generation • Conclusion
  9. 9. 9 Sequence-to-sequence models Both the encoding and decoding are often done using recurrent neural networks (RNNs). The initial big application for this was machine translation. For example, where the source sentences are English and the target sentences are Spanish. A sequence-to-sequence (seq2seq) model 1. encodes a sequence of tokens, such as a sentence, into a single vector 2. decodes that vector into another sequences of tokens
  10. 10. 10 The ℎ0 woman ℎ1 ate ℎ2 pizza ℎ3 . ℎ4 As each word is examined, the encoder RNN integrates it into the hidden state vector h. Encoding a sentence for machine translation
  11. 11. 11 La ℎ4 mujer ℎ5 comió ℎ6 pizza ℎ7 . ℎ8 The model begins with the last hidden state from the encoder and generates words until a special stop symbol is generated. Decoding the sentence into Spanish Note that the lengths of the source and target sentences do not need to be the same, because it goes until it generates the stop symbol.
  12. 12. 12 Generalized view text meaning space generated text ℎ4
  13. 13. 13 Outline • Introduction • Learning from pairs of sentences • Overview • Encoders • Decoders • Attention • Uses • Mapping to and from meaning space • Adversarial text generation • Conclusion
  14. 14. 14 The ℎ0 woman ℎ1 ate ℎ2 pizza ℎ3 . ℎ4 Let’s look at a simple cell so we can get a grounded understanding. Concrete example of an RNN encoder aardvark 1.32 1.56 0.31 –1.21 0.31 ate 0.36 –0.26 0.31 –1.99 0.11 ... –0.69 0.33 0.77 0.22 –1.29 zoology 0.41 0.21 –0.32 0.31 0.22 Vocabulary table • 𝑉: vocabulary table of size 50,000 by 300 • 𝑣𝑡: word vector is row from 𝑉 • 𝑊𝑇: transition matrix of size 300 by 300 • 𝑊𝐼: input matrix of size 300 by 300 • 𝑏 is a bias vector of size 300 • ℎ 𝑡 = tanh(ℎ 𝑡−1 𝑊𝑇 + 𝑣𝑡 𝑊𝐼 + 𝑏) See Christopher Olah’s blog post for great presentation of more complex cells like LSTMs http://colah.github.io/posts/2015-08-Understanding-LSTMs/ 𝑣 𝑡 ℎ 𝑡 (each row is actually 300 dimensions) Parameters start off as random values and are updated through backpropagation.
  15. 15. 15 Tom ate with his brother 0.23 0.36 0.23 0.23 0.23 1.42 –0.26 1.32 1.27 –1.42 –0.33 0.31 0.33 0.59 0.33 1.23 –1.99 1.23 –1.31 1.23 1.11 0.11 0.11 2.19 1.83 0.21 0.56 0.21 1.12 –1.12 1.12 0.55 0.26 0.55 0.81 1.71 –0.31 –0.66 9.66 0.66 • 128 filters of width 3, 4, and 5 • Total of 384 filters • 2) Slide each filter over the sentence and element- wise multiply and add (dot product); then take the max of the values seen. • Result is a sentence vector of length 384 that represents the sentence. • Analogous to • For classification: each class has a vector of weights of length 384 • 3) To decide which class, the class vector is multiplied element wise by the sentence vector and summed (dot product) • The class with the highest sum wins aardvark 1.32 1.56 0.31 –1.21 0.31 ate 0.36 –0.26 0.31 –1.99 0.11 ... –0.69 0.33 0.77 0.22 –1.29 zoology 0.41 0.21 –0.32 0.31 0.22 Vocabulary table 1) For each word in the sentence, get its word vector from the vocabulary table. Convolutional Neural Network (CNN) Text Encoder (each row is actually 300 dimensions) • See the code here by Denny Britz https://github.com/dennybritz/cnn-text-classification-tf • README.md contains reference to his blog post and the paper by Y. Kim ℎ4 (example filter three words wide; size 300 by 3)
  16. 16. 16 Outline • Introduction • Learning from pairs of sentences • Overview • Encoders • Decoders • Attention • Uses • Mapping to and from meaning space • Adversarial text generation • Conclusion
  17. 17. 17 Concrete example of RNN decoder • 𝑉: vocabulary table of size 50,000 by 300 • 𝑣 𝑡−1: word vector row from 𝑉; 𝑣5 is “mujer” • 𝑊𝑇 𝐷 : transition matrix of size 300 by 300 • 𝑊𝐼 𝐷 : input matrix of size 300 by 300 • 𝑏 is a bias vector of size 300 • ℎ 𝑡 = tanh(ℎ 𝑡−1 𝑊𝑇 𝐷 + 𝑣 𝑡−1 𝑊𝐼 𝐷 + 𝑏) ℎ6 La ℎ4 mujer ℎ5 comió ℎ6 pizza ℎ7 . ℎ8 • 𝑊𝑂 𝐷 : output matrix of size 300 by 50,000 • A probability distribution over next words • 𝑝 𝑤𝑜𝑟𝑑𝑠 = 𝑠𝑜𝑓𝑡𝑚𝑎𝑥(ℎ 𝑡 𝑊𝑂 𝐷 ) • The softmax function makes them all sum to 1 • 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 𝑥𝑗 = 𝑒 𝑥 𝑗/ 𝑖 𝑒 𝑥 𝑖
  18. 18. 18 Models are trained with paired sentences to minimize cost At time 𝑡 = 6 we know the correct word is “comió”, so the cost is 𝑐𝑜𝑠𝑡 = −log(𝑝(comió)) So the more likely “comió” is, the lower the cost. Also, note that regardless of what the model says at time 𝑡 − 1, we feed in “mujer” at time 𝑡. This is called teacher forcing. La ℎ4 mujer ℎ5 comió ℎ6 pizza ℎ7 . ℎ8 • 𝑊𝑂 𝐷 : output matrix of size 300 by 50,000 • A probability distribution over next words • 𝑝 𝑤𝑜𝑟𝑑𝑠 = 𝑠𝑜𝑓𝑡𝑚𝑎𝑥(ℎ 𝑡 𝑊𝑂 𝐷 ) • The softmax function makes them all sum to 1 • 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 𝑥𝑗 = 𝑒 𝑥 𝑗/ 𝑖 𝑒 𝑥 𝑖
  19. 19. 19 When it comes time to generate text A compromise is beam search • Start generating with the first word • Keep the 𝑘 (around 10) most promising sequences When you are actually using the model to generate text, you don’t know what the right answer is. You could do a greedy decoder • Take the most likely first word, then given that word, takes the most likely next word, and so on. • But that could lead you down a suboptimal path. You could look at all sequences of words up to a given length • Choose the sequence with the lowest cost • But there are far too many. too cheap too expensive La ℎ4 mujer ℎ5 comió ℎ6 pizza ℎ7 . ℎ8 Beam search can sometimes lack diversity, and there have been proposals to address this; see Li and Jurafsky, https://arxiv.org/pdf/1601.00372v2.pdf and Ashwin et al., https://arxiv.org/pdf/1610.02424v1.pdf
  20. 20. 20 Outline • Introduction • Learning from pairs of sentences • Overview • Encoders • Decoders • Attention • Uses • Mapping to and from meaning space • Adversarial text generation • Conclusion
  21. 21. 21 The network learns how to determine which previous memory is most relevant at each decision point [Bahdanau et al., 2014]. The ℎ0 woman ℎ1 ate ℎ2 tacos ℎ3 . ℎ4 La ℎ4 mujer ℎ5 comió ℎ6 tacos ℎ7 . ℎ8 Attention helps the model consider more information
  22. 22. 22 RNN seq2seq model with attention text meaning space generated text ℎ4 Attention
  23. 23. 23 Outline • Introduction • Learning from pairs of sentences • Overview • Encoders • Decoders • Attention • Uses • Mapping to and from meaning space • Adversarial text generation • Conclusion
  24. 24. 24 Seq2seq works on all kinds of sentence pairs For example, take a bunch of dialogs and use machine learning to predict what the next statement will be given the last statement. • Movie and TV subtitles • OpenSubtitles http://opus.lingfil.uu.se/OpenSubtitles.php • Can mine Twitter • Look for replies to tweets using the API • Ubuntu Dialog Corpus • Dialogs of people wanting technical support https://arxiv.org/abs/1506.08909 Another example, text summarization. E.g., learn to generate headlines for news articles. See https://memkite.com/deeplearningkit/2016/04/23/deep-learning-for-text-summarization/
  25. 25. 25 Style and more [0] Seq2seq style transfer: Dear Sir or Madam, May I introduce the YAFC Corpus: Corpus, Benchmarks and Metrics for Formality Style Transfer https://arxiv.org/pdf/1803.06535.pdf • You can even do style transfer [0]. • They had Mechanical Turk workers write different styles of sentences for training. • We need to get them on the Cormac McCarthy task.
  26. 26. 26 ℎ4 Convolutional neural network Weird ℎ4 dude ℎ5 on ℎ6 table ℎ7 . ℎ8 Karpathy and Fei-Fei, 2015 http://cs.stanford.edu/people/karpathy/deepimagesent/ Vinyals et al., 2014 http://googleresearch.blogspot.com/2014/11/a-picture-is- worth-thousand-coherent.html Can even automatically caption images Chavo protects my office from Evil spirits. Seq2seq takeaway: works well if you have lots of pairs of things to train on.
  27. 27. 27 Outline • Introduction • Learning from pairs of sentences • Mapping to and from meaning space • Meaning space should be smooth • Variational methods ensure smoothness • Navigating the meaning space • Moving beyond pairs of sentences • Conclusion
  28. 28. 28 Outline • Introduction • Learning from pairs of sentences • Mapping to and from meaning space • Meaning space should be smooth • Variational methods ensure smoothness • Navigating the meaning space • Moving beyond pairs of sentences • Conclusion
  29. 29. 29 Meaning Space Should be Smooth Meaning space: mapping of concepts and ideas to points in a continuous, high-dimensional grid. We want: • Semantically meaningful things to be near each other. • To be able to interpolate between sentences. • Could be the last state of the RNN or the output of the CNN • But we want to use the space as a representation of meaning. • We can get more continuity in the space when we use a bias encourage the neural network to represent the space compactly [0]. Meaning Space This is an outrage! tacos are me I love tacos we are all tacos [0] Generating Sentences from a Continuous Space, https://arxiv.org/abs/1511.06349
  30. 30. 30 Meaning Space Should be Smooth • Get bias by adding prior probability distribution as organizational constraint. • Prior wants to put all points at (0,0,...0) unless there is a good reason not to. • That “good reason” is the meaning we want to capture. • By forcing the system to maintain that balance, we encourage it to organize such that concepts that are near each other in space have similar meaning. Meaning Space This is an outrage! tacos are me I love tacos we are all tacos
  31. 31. 31 Outline • Introduction • Learning from pairs of sentences • Mapping to and from meaning space • Meaning space should be smooth • Variational methods ensure smoothness • Navigating the meaning space • Moving beyond pairs of sentences • Conclusion
  32. 32. 32 Variational methods provide structure to meaning space Variational methods can provide this organizational constraint in the form of a prior. • In the movie Arrival, variational principles in physics allowed one to know the future. • In our world, variational comes from the calculus of variations; optimization over a space of functions. • Variational inference is coming up with a simpler function that closely approximates the true function. meaning space This is an outrage! we are all tacos tacos are me I love tacos we are all tacos
  33. 33. 33 Variational methods provide structure to meaning space meaning space This is an outrage! Our simpler function for 𝑝(ℎ|𝑥) is 𝑞 ℎ 𝑥 = 𝑁(𝜇, 𝜎). We use a neural network (RNN) to output 𝜇 and 𝜎. To maximize this approximation, it turns out that we maximize the evidence lower bound (ELBO): ℒ 𝑥, 𝑞 = 𝐸 𝑞 ℎ 𝑥 [log 𝑝 𝑥 ℎ ] − 𝐾𝐿(𝑞(ℎ|𝑥)||𝑝 ℎ ). See Kingma and Welling https://arxiv.org/pdf/1312.6114.pdf In practice, we only take one sample and we use the reparameterization trick to keep it differentiable. tacos are me I love tacos we are all tacos
  34. 34. 34 Variational methods provide structure to meaning space Note that the equation ℒ 𝑥, 𝑞 = 𝐸 𝑞 ℎ 𝑥 [log 𝑝 𝑥 ℎ ] − 𝐾𝐿(𝑞(ℎ|𝑥)||𝑝 ℎ ) has the prior 𝑝 ℎ . The prior 𝑝 ℎ = 𝑁(0, 𝐼) forces the algorithm to use the space efficiently, which forces it to put semantically similar latent representations together. Without this prior, the algorithm could put ℎ values anywhere in the latent space that it wanted, willy-nilly. 𝑁(0, 𝐼) This is an outrage! tacos are me I love tacos we are all tacos
  35. 35. 35 Trained using an autoencoder The prior acts like regularization, so we are 1. maximizing the reconstruction fidelity 𝑝 𝑥 ℎ and 2. minimizing the difference of ℎ from the prior 𝑁(0, 𝐼). Can be tricky to balance these things. Use KL annealing, see Bowman et al., https://arxiv.org/pdf/1511.06349.pdf text 𝑥 encoder decoder text 𝑥𝑁(0, 𝐼) 𝑞(ℎ|𝑥)
  36. 36. 36 We can also do classification over meaning space When we train with variational methods to regenerate the input text, this is called a variational autoencoder. classifier This can be the basis for learning with less labeled training data (semi-supervised learning). See Kingma et al., https://arxiv.org/pdf/1406.5298.pdf 𝑁(0, 𝐼) This is an outrage! tacos are me I love tacos we are all tacos
  37. 37. 37 We can place a hierarchical structure on the meaning space Hierarchical Meaning Space tacos Food In the vision domain, Goyal et al. allows one to learn a hierarchy on the prior space using a hierarchical Dirichlet process https://arxiv.org/pdf/1703.07027.pdf Videos on understanding Dirichlet processes and the related Chinese restaurant processes by Tamara Broderick part I https://www.youtube.com/watch?v=kKZkNUvsJ4M part II https://www.youtube.com/watch?v=oPcv8MaESGY part III https://www.youtube.com/watch?v=HpcGlr19vNk candy GPUs computers VIC-20 old cars Cars self- drivingThe Dirichlet process has allows for an infinite number of clusters. As the world changes and new concepts emerge, you can keep adding clusters.
  38. 38. 38 Outline • Introduction • Seq2seq models and aligned data • Representing text in meaning space • Meaning space should be smooth • Variational methods ensure smoothness • Navigating the meaning space • Adversarial text generation • Conclusion
  39. 39. 39 We can learn functions to find desired parts of the meaning space meaning space In the vision domain, Engel et al. allows one to translate a point in meaning space to a new point with desired characteristics https://arxiv.org/pdf/1711.05772.pdf Bob Bob w/ glasses 𝑓(𝐵𝑜𝑏)
  40. 40. 40 We can learn functions to find desired parts of the meaning space meaning space Bob Bob w/ glasses 𝑓(𝐵𝑜𝑏) In the text domain, [0] provides a way to rewrite sentences to change them according to some function, such as • making the sentiment more positive or • making a Shakespearean sentence sound more modern. [0] Sequence to Better Sequence: Continuous Revision of Combinatorial Structures, http://www.mit.edu/~jonasm/info/Seq2betterSeq.pdf
  41. 41. 41 We can learn functions to find desired parts of the meaning space meaning space Bob Bob w/ glasses 𝑓(𝐵𝑜𝑏) Since the space is smooth, we might be able to hill climb to get whatever digital artifacts we wanted, such as computer programs. [see Liang et al., https://arxiv.org/pdf/1611.00020.pdf and Yang et al., https://arxiv.org/pdf/1711.06744.pdf] This movie looks pretty good, but make it a little more steampunk ... [0] Sequence to Better Sequence: Continuous Revision of Combinatorial Structures, http://www.mit.edu/~jonasm/info/Seq2betterSeq.pdf Method [0] 1. Encode sentence 𝑠 using a variational autoencoder to get ℎ. 2. Look around the meaning space to find the point ℎ∗ of the nearby space that maximizes your evaluation function. 3. Decode ℎ∗ to create a modified version of sentence 𝑠
  42. 42. 42 Outline • Introduction • Learning from pairs of sentences • Mapping to and from meaning space • Moving beyond pairs of sentences • Conclusion
  43. 43. 43 Outline • Introduction • Learning from pairs of sentences • Mapping to and from meaning space • Moving beyond pairs of sentences • Bridging meanings through shared space • Adversarial text generation • Learning through reinforcement learning • Conclusion
  44. 44. 44 You have pairs of languages, but not the right ones • You may have pairs of languages or styles but not the pairs you want. • For example, you have lots of translations from English to Spanish and from English to Tagalog but you want to translate directly from Spanish to Tagalog. Google faced this scenario with language translation [0]. • Use a single encoder and decoder for all languages. • Use word segments so there is one vocabulary table; and the encoder can encode all languages to the same space. • Add the target language as the first symbol of the source text, so the decoder knows the language to use. [0] Zero-Shot Translation with Google’s Multilingual Neural Machine Translation System, https://research.googleblog.com/2016/11/zero-shot-translation-with-googles.html Source lang. text & target symbol encoder meaning space decoder Target language After, they could translate pairs of languages even when there was no pairwise training data.
  45. 45. 45 You have no pairs at all decoder generated text meaning space decoder generated text But how do you associate your language model with a meaning space? How do you get the probability of the first word? Imagine you have text written in two styles, but it is not tied together in pairs. For each style, you can learn a language model, which gives a probability distribution over the next word given the pervious ones. Language models are the technology behind those posts of the form: “I forced my computer to watch 1000 hours of Hee Haw and this was the result.”
  46. 46. 46 Assign a meaning space using an autoencoder! You can associate a language model with a meaning space using an autoencoder. But how can you line up the meaning spaces of the different languages and styles so you can translate between them? text 𝑥 encoder decoder text 𝑥𝑁(0, 𝐼) 𝑞(ℎ|𝑥)
  47. 47. 47 1. Train languages 𝐴 and 𝐵 with autoencoders using same encoder and different decoders. 2. Start with a sentence 𝑏 ∈ 𝐵 (start with 𝐵 because we want to train in the other direction) 3. Encode 𝑏 and decode it into 𝑎 ∈ 𝐴 4. You now have a synthetic pair (𝑎, 𝑏) 5. Train using the synthetic pairs Bridging meaning spaces with synthetic pairs [0] Lang. 𝐴 or 𝐵 encoder meaning space decoder B generated B decoder 𝐴 generated 𝐴 [0] Unsupervised Neural Machine Translation https://arxiv.org/pdf/1710.11041.pdf Creating pairs this way is called backtranslation Improving Neural Machine Translation Models with Monolingual Data, http://www.aclweb.org/anthology/P16-1009
  48. 48. 48 Bridging meaning spaces with round-trip cost [0] 1. Start with sentence 𝑎 ∈ 𝐴 2. Backprop with autoencoder cost 𝑎 → 𝑎 3. Backprop with round trip cost 𝑎 → 𝑏 → 𝑎 4. Repeat with a sentence 𝑏 ∈ 𝐵 [0] Improved Neural Text Attribute Transfer with Non- parallel Data https://arxiv.org/pdf/1711.09395.pdf • For example, for a Yelp review, they automatically converted the positive sentiment phrase “love the southwestern burger” to the negative sentiment phrase “avoid the grease burger.” • Notice that the word “burger” is in both, as it should be. Maintaining semantic meaning can sometimes be a challenge with these systems. • In their work [0], they add an additional constraint that nouns should be maintained. Lang. 𝐴 or 𝐵 encoder meaning space decoder B generated B decoder 𝐴 generated 𝐴
  49. 49. 49 Making the most of data with dual learning [0] Text 𝐴 Encoder 𝐴 meaning space Decoder 𝐵 Text 𝐵 Decoder 𝐵 Text 𝐵 Model from 𝐴 to 𝐵. Language model for 𝐵. Text 𝐵 Encoder 𝐵 meaning space Decoder 𝐴 Text 𝐴 Decoder 𝐴 Text 𝐴 Model from 𝐵 to 𝐴. Language model for 𝐴. Policy gradient methods to maximize 1. communication reward, which is they fidelity of a round-trip translation of 𝐴 back to 𝐴 (and 𝐵 back to 𝐵) 2. language model reward, which for 𝐴 → 𝐵 is how likely 𝐵 is in the monolingual language model for 𝐵 (and likewise for 𝐵 → 𝐴). [0] Achieving Human Parity on Automatic Chinese to English News Translation https://arxiv.org/pdf/1803.05567.pdf
  50. 50. 50 Outline • Introduction • Learning from pairs of sentences • Mapping to and from meaning space • Moving beyond pairs of sentences • Bridging meanings through shared space • Adversarial text generation • Learning through reinforcement learning • Conclusion
  51. 51. 51 Examples on movie reviews: (“negative”, “past”) the acting was also kind of hit or miss . (“positive”, “past”) his acting was impeccable (“negative”, “present”) the movie is very close to the show in plot and characters (“positive”, “present”) this is one of the better dance films Using a discriminator [0] Specify desired feature text encoder 𝑁(0,1) decoder generated text with feature Learned Discriminator on that feature [0] Hu et al. allow one to manipulate single values of the meaning vector to change what is written https://arxiv.org/abs/1703.00955
  52. 52. 52 Generative Adversarial networks (GANs) Generator Synthetic Data Discriminator Real Data Guidance to improve generator • Generative Adversarial Networks (GANs) are built on deep learning. • A GAN consists of a generator neural network and a discriminator neural network training together in an adversarial relationship. Goodfellow et al., Generative Adversarial Networks, https://arxiv.org/abs/1406.2661
  53. 53. 53 GAN for text generation text encoder meaning space decoder generated text Learned Discriminator on text Problem: Generating text in this way is not differentiable. Three solutions 1. Soft selection (take weighted average of words) (Hu does this.) 2. Run discriminator on continuous meaning space or continuous decoder state. E.g., Professor Forcing [0] 3. Use reinforcement learning. The discriminator could try to differentiate generated from real text. [0] Professor Forcing: A New Algorithm for Training Recurrent Networks https://arxiv.org/pdf/1610.09038.pdf [1] Adversarial Learning for Neural Dialogue Generation https://arxiv.org/pdf/1701.06547.pdf [Note, encoding could be a line in dialog, like in [1]].
  54. 54. 54 Outline • Introduction • Sequence-to-sequence models: learning form pairs of sentences • Mapping meaning to and from meaning space • Moving beyond pairs of sentences • Bridging meanings through shared space • Adversarial text generation • Learning through reinforcement learning • Conclusion
  55. 55. 55 Reinforcement learning is a gradual stamping in of behavior • Reinforcement learning (RL) was studied in animals by Thorndike [1898]. • Became the study of Behaviorism [Skinner, 1953] (see Skinner box on the right). • Formulated into artificial intelligence; see leading book by Sutton and Barto, 1998. (new draft available online http://incompleteideas.net/book/bookdraft2018jan1.pdf) By Andreas1 (Adapted from Image:Boite skinner.jpg) [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)], via Wikimedia Commons A “Skinner box”
  56. 56. 56 Beginning with random exploration In reinforcement learning, the agent begins by randomly exploring until it reaches its goal.
  57. 57. 57 • When it reaches the goal, credit is propagated back to its previous states. • The agent learns the function 𝑄 𝜋(𝑠, 𝑎), which gives the cumulative expected discounted reward of being in state 𝑠 and taking action 𝑎 and acting according to policy 𝜋 thereafter. Reaching the goal
  58. 58. 58 Eventually, the agent learns the value of being in each state and taking each action and can therefore always do the best thing in each state. Learning the behavior
  59. 59. 59 RL is now like supervised learning over actions With the rise of deep learning, we can now more effectively work in high dimensional and continuous domains. When domains have large action spaces, the kind of reinforcement learning that works best is policy gradient methods. • D. Silver http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/pg.pdf • A. Karpathy http://karpathy.github.io/2016/05/31/rl/ Policy gradient methods learn a policy directly on top of a neural network. Basically softmax over actions on top of a neural network. Reinforcement learning starts to look more like supervised learning, where the labels are actions. Instead of doing gradient descent on the error function as in supervised learning, you are doing gradient ascent on the expected reward.
  60. 60. 60 Searching forward to get this reward You have to do a search forward to get the reward. La ℎ4 mujer ℎ5 comió ℎ6 Since we are not doing teacher forcing where we know what the next word should be, we do a forward search: If we were to write “comió” and followed our current policy after that, how good would the final reward be? That “how good” is the estimate we use to determine whether writing “comió” should be the action this network takes in the future. [see Lantao et al., https://arxiv.org/pdf/1609.05473v3.pdf] Called Monte Carlo search This is similar how game playing AI is done, for example with the recent success in Go [see Silver et al., https://arxiv.org/pdf/1712.01815.pdf ] In the language domain, this kind of training can be very slow. There are 50,000 choices.
  61. 61. 61 Outline • Introduction • Learning from pairs of sentences • Mapping to and from meaning space • Moving beyond pairs of sentences • Conclusion
  62. 62. 62 Conclusion: when should you use neural text generation? If you can enumerate all of the patterns for the things you want the computer to understand and say: use context-free grammars for understanding and templates for language generation. If you can’t enumerate what you want the computer to say and you have a lot of data, neural text generation might be the way to go. • Machine translation is an excellent example. • Or, if you want your text converted to different styles. • Machines will be illiterate until we can teach them common sense. • But as algorithms get better at controlling and navigating the meaning space, neural text generation has the potential to be transformative. • Few things are more enjoyable than reading a story by your favorite author, and if we could have all of our information given to us with a style tailored to our tastes, the world could be enchanting. Text can’t be long, and it won’t be exact.
  63. 63. 6500 River Place Blvd. Building 3, Suite 120 Austin, TX 78730 www.DeUmbra.com Thanks for listening! @jmugan

×