This document discusses context-based text generation using LSTM networks. It describes how traditional LSTM models have poor semantic context, and proposes a solution that extracts context using TF-IDF topics from sentences or word clustering. Context is then provided to the LSTM network to generate text related to that context. Applications mentioned include question answering systems, chatbots, and structured text generation.

1. Contextbased TextgenerationusingLSTM networks
Sivasurya Santhanam
Intelligentand Distributed systems
GermanAerospace Center.
sivasurya.santhanam@dlr.de
@ahamsiva
21.11.2018

2. Content of the presentation
• Sequence models in Natural language processing
• Text generation using Long short-term memory (LSTM) networks
• Drawbacks & Proposed solution
• Context LSTM using Tf-idf & word clustering
• Applications

3. Sequence models in Natural language processing(NLP)
• Recentdevelopments in NLP
• Bigdata+ GPUs+ ML Algorithms
• Sequentialmodelswithmemoryunitsin
Speech recognitionandLanguagetranslation
• RecurrentNeuralnetworks(RNN) vs LSTM
Image sources: me.me/i/2456240, machine translation(shutterstock), soundwriter(efv-solutions)
I lived in France for the past 7 years, I speak fluent
_________

4. Text generation using LSTM
The quick brown fox jumps over
quick brown fox jumps over the
brown fox jumps over the lazy
fox jumps over the lazy dog
The king is going to the
king is going to the castle
Input words Target word
Sequence
model
(LSTM)

5. Text generation using LSTM
'Hobbits lived in the woods happily and the story begins to climb out of the North to be seen in the light of the old man. The Dark Lord were
bent in the dark and the light of the great ship of the Road and the night and the dwarves were drawing off to the ground and the grass and the
main door was gone. The foremost were flung with candles and as the Sun and the horns leapt from the hill and came out of the mist and the
sun was in the sky and the sun was shining and a white figure ran down into a wide shallow sky above them and they were in the air. There
was a great deal of ancient and smooth and the mountains was coming. Now there were no sign of the Enemy. There was a ford in the great
light of the great range of Dale and the Riders of the Eagles and on the borders of the Shire and the great mountains was thrown down and the
road and the River was in the East and the travellers was shut. The men were already in a cloud of gold. There was no longer. He could not
see him to see‘
(Network configuration: Bidirectional LSTM. 256 hidden units. ‘relu’ activation.)

6. Drawbacks & Proposal
• The language structure of the LSTM model is impressive
• Semantic context of the text is so poor
Proposed Solution
• Context based LSTM models for text generation
• Methods adopted for context extraction
• Tf-idf topics from each sentences
• Topics extracted from clustering word-embedding

7. Proposed solution

8. 1. Context extraction using
Term frequency – Inverse document frequency (Tf-idf)
Context provided: “Hobbits – gollum – adventure – king – ring – war – friends – war – book – home”
Hobbits lived in the woods and once an elf came all his There was Tom to go on by this If he meant all the game but there was a bit of Moria of
and Bilbo was running a black eyebrows at the Mountain pretty of and and as you played up about and he said up about the hall in all this They
will not come down in it. He had been going up a bit and crept in into the side of a magic beard and into the prisoners of our If you have get him
or I will get my dear sir and Bilbo felt he must have to find this adventures but you are cook the question they all are yes. Dont me on to Bilbo
my sudden understanding a pity mixed with various noticing down and their parlour as well as he could. No trouble He was in a boat and a
breakfast and their sun had been and one of his second wealth as Gandalf dared their ponies behind they had just had a fair breakfast as they
had been gone further he had been adventures many He had and been very He had come a ring of their wild Well all this That

9. 2. Context extraction using word cluster centers
w
• Word embedding of each word
k-means
• Clustering of word vector space using k-means
𝑠
• Summed-up word vectors for the words in a
sentence representing a sentence vector
Cosine
similarity
• Context extracted from the nearest cluster center
to each sentence vector
𝑠 ∈ 𝑚 c ∈ 𝐶
Such that,
Cosine ( 𝑤=0
𝑛
𝑠, c) is minimum

10. Word vector space
Image source: shanelynn.ie (Example 2-D word vector space) Image source: samyzaf.com (NLP with gensim)
Semantic similarities Semantic relationships

11. Context extraction using word cluster centers
(a) Word-embedding clusters (n=10) visualized under t-SNE (b) Word-embedding clusters (n=100) visualized under t-SNE

12. • gandalf - characters hobbits lived in the woods and an elf said that please try down they all that been had only been so they had only been
much in the meantime in the deal of the and and they had brought the dwarves of the dwarves but they had brought him from the road and
the goblins went out and again that they could hear him on their chief and the great Master of the town that of the great wooden spur which
they had had been as the Great Goblin and his little sword that had was in a way but he had picked up in the dark
• ring - Some ago it was so all that so fitted off he found he more one all his second staff on the floor and the laughter of the others of the
door and the laughter of the line of the dwarves and he had never a shocking of the below
• war - end of their way had would be and the most of the dwarves were built with guards
• friends - It was like a pull for the jug
• snake - smash and but the various birds that is the more and horses now he never a cave and was not to be done in the obscurest words
• book - Bilbo I have a little time for the terrible hours and in fact and besides of the town were me that the track had come from a good way
that was too I dont know but I am your last Burglar said a share of the rate
• home - The heart of the hobbit and made of yellow wood
• king - Bilbo was more and a long deal of the dwarves they had been busy in the direction of the Mountain
• hobbit - The dwarves that was in a story of the chief of the dwarves and they were all alone and that they were all alone and whether they
had brought of them the dwarves were not in the valley and they were forced to pull the frightful Eagle the Great Mountain and coming
Context extraction using word cluster centers

13. Applications
• Question answering systems
• Chat-bots (keeping track of topics)
• Generating text for a defined structure
• Introduction-Literature-method-Conclusion
• Hints development