A walkthrough of a Deep Learning based technique which would generate TV scripts using Recurrent Neural Network. The model will generate a completely new TV script for a scene, after being training from a dataset. One will learn the concepts around RNN, NLP and various deep learning techniques. Technologies to be used: Python 3, Jupyter, TensorFlow Source code: https://github.com/syednasar/talks/tree/master/synthetic-dialog

1. Synthetic Dialogue
Generation with Deep
Learning
Using TensorFlow
Nasar, Syed | August 2017

2. About me
Machine Learning and Big Data Architect
MS Computer Science, Machine Learning, Georgia Tech
Deep Learning Nanodegree, Udacity
Artificial Intelligence Nanodegree, Udacity
Connect
 Twitter : @techmazed
 LinkedIn : https://www.linkedin.com/in/knownasar
 Founder: Nashville Artificial Intelligence Society (twitter @AINashville)

3. What we’ll cover
The synthetic data generation challenge
Introducing Deep Learning
Introducing RNN & Sequence to Sequence model
Introducing NLP concepts
Brief introduction to TensorFlow & FloydHub
The solution approach
Code walkthrough
What next!
Question time

4. TheSyntheticDataGeneration Challenge

5. The problem
The challenge is to generate our own TV dialogue scripts by training a Neural Network with an existing
corpus of dialogues.
Dialogues from Simpson TV show:
The sentences 0 to 10:
[YEAR DATE 1989] © Twentieth Century Fox Film Corporation. All rights reserved.
Moe_Szyslak: (INTO PHONE) Moe's Tavern. Where the elite meet to drink.
Bart_Simpson: Eh, yeah, hello, is Mike there? Last name, Rotch.
Moe_Szyslak: (INTO PHONE) Hold on, I'll check. (TO BARFLIES) Mike Rotch. Mike Rotch. Hey, has anybody
seen Mike Rotch, lately?
Moe_Szyslak: (INTO PHONE) Listen you little puke. One of these days I'm gonna catch you, and I'm gonna
carve my name on your back with an ice pick.
Moe_Szyslak: What's the matter Homer? You're not your normal effervescent self.
Homer_Simpson: I got my problems, Moe. Give me another one.
Moe_Szyslak: Homer, hey, you should not drink to forget your problems.
Barney_Gumble: Yeah, you should only drink to enhance your social skills.

6. Introduction to
the important
concepts
Concepts
A brief discussion on the various
technologies, techniques and
concepts that we apply towards
the solution.
Deep
Learning
Recurrent
Neural
Network
(RNN)
Natural
Language
Processing
(NLP)
Technologies

7. What is Deep
Learning?
Neural networks with large number of
parameters and layers.
The layers belong to one of four
fundamental network architectures:
 Unsupervised Pre-Trained Networks
 Convolutional Neural Networks
 Recurrent Neural Networks
 Recursive Neural Networks
 Also Generative Adversarial
Networks
Introducing Deep Learning
The behavior of neural networks is shaped by its network architecture.
A network’s architecture can be defined, in part, by:
• number of neurons
• number of layers
• types of connections between layers
A Neural Network is an algorithm
that learns to identify patterns in
data.
Backpropagation is a technique to
train a Neural Net by updating
weights via gradient descent.
Deep Learning:
many layer neural net + massive
data (big data) + massive compute
(GPU)
Multi-Layer Neural Network topology

8. Backpropagation
Learning
Backpropagation is an important part of
reducing error in a neural network
model.
General Neural Network Training
Pseudocode
function neural-network-learning( training-records ) returns network
network <- initialize weights (randomly)
start loop
for each example in training-records do
network-output = neural-network-output( network, example )
actual-output = observed outcome associated with example
update weights in network based on { example, network-output, actual-output }
end for
end loop when all examples correctly predicted or hit stopping conditions
return network

9. IntroducingRecurrentNeural Network

10. What is RNN?
Recurrent Neural Networks have
loops.
In the diagram (right), a chunk of neural
network, A, looks at some input xt and
outputs a value ht.
A loop allows information to be passed
from one step of the network to the next.
Consider what happens if we unroll the
loop:

11. The flow
Recurrent Neural Network

12. IntroducingNatural LanguageProcessing(NLP) concepts

13. Tokenize
Given a character sequence and a defined document unit, tokenization is the task of
chopping it up into pieces, called tokens, also sometimes throwing away certain
characters, such as punctuation.
Here is an example:
Input: Friends, Romans, Countrymen, lend me your ears;
Output: Friends || Comma || Romans || Comma || Countrymen || Comma || lend || me || your || ears || semicolon ||
bye bye!

14. One-Hot Encoding
One hot encoding technique is used to encode categorical integer features using a one-
hot scheme.

15. NeuralNetworktips& tricks

16. LSTMs
This basically handles the problem of vanishing and exploding gradients. LSTMs help
preserve the error that can be backpropagated through time and layers.
Image courtesy - http://deeplearning.net/tutorial/lstm.html

17. Logits
Logistic Function:
The logit function is the inverse of the
logistic function.

18. SoftMax
MulticlassClassificationModelOutputLayer

19. Cross Entropy & Embeddings

20. Brief introductionto technologies used

21. TensorFlow
TensorFlow is an open source software library by Google
To design, build, and train deep learning models
Why use:
Thoughtful design & Ease of use
Deploy computation to one or more CPUs or GPUs
TensorFlow alternatives: Theano, Torch, Caffe, Neon, and Keras

22. FloydHub
“Floydhub is Heroku for Deep Learning. A Platform-as-a-Service for training
and deploying your models in the cloud with a few simple commands.”
# Install locally and login
$pip install -U floyd-cli
$floyd login
# Run a Jupyter notebook on FloydHub
$ floyd run --mode jupyter --gpu
#Run a python code on GPU
$ floyd run --gpu "python mnist_cnn.py"

23. Demo Time
Let us look at the code

24. Get the helper function.
Load the data
Summary of the data
Sample Data

25. Create a word embedding
Transform the words to ids.
Two dictionaries:
• Dictionary to go from the words to
an id, we'll call vocab_to_int
• Dictionary to go from the id to
word, we'll call int_to_vocab
Look at vocab-to-int

26. Tokenize Punctuation
This dictionary will be used to token
the symbols and add the delimiter (||)
around it.
This separates the symbols as it's own
word, making it easier for the neural
network to predict on the next word.
Look at token dictionary

27. Let’s preprocess the data
Look at the pre-processed data

28. Build the Neural Network
We'll build the components
necessary to build a RNN by
implementing these functions
• get_inputs
• get_init_cell
• get_embed
• build_rnn
• build_nn
• get_batches Placeholders for Inputs

29. Build the Neural Network
• get_inputs
• get_init_cell
• get_embed
• build_rnn
• build_nn
• get_batches
Build RNN Cell and Initialize

30. Build the Neural Network
• get_inputs
• get_init_cell
• get_embed
• build_rnn
• build_nn
• get_batches
Word Embedding

31. Build the Neural Network
• get_inputs
• get_init_cell
• get_embed
• build_rnn
• build_nn
• get_batches
Build RNN
Name for the
operation
A tensor

32. Apply embedding Build RNN
Apply a fully
connected layer
Return the logits and
final state
Build the Neural Network
• get_inputs
• get_init_cell
• get_embed
• build_rnn
• build_nn
• get_batches
Build the Neural Network

33. Build the Neural Network
• get_inputs
• get_init_cell
• get_embed
• build_rnn
• build_nn
• get_batches
Create Batches
•The first element is a single batch of input with the shape [batch size, sequence length]
•The second element is a single batch of targets with the shape [batch size, sequence length]

34. Neural Network Training
Number of epochs = 50
Batch size = 256
Size of the RNNs = 256
Length of sequence = 10
Learning rate = 0.01
Tuning Hyperparameters

35. Get Inputs
Initialize cells
Build NN => logits
Apply Softmax
Calculate Loss
Compute Gradients
Apply Optimizer
Build the Graph
Neural Network Training
Loss function
Optimizer

36. Get batches
Initialize
variables
Initialize state
Enumerate
batches
Get inputs and
hyperparameters
Compute loss
Show batch
results every 100
Save trained
model
Train the neural network
Neural Network Training

37. Training output
Neural Network Training

38. Get Tensors
Script Generation
Choose Word

39. Generate TV Script
Script Generation
Load Model
Get Tensors
Setup Sentence
generation
Generate sentences
Dynamic input
Get prediction

40. Generated Script
Neural Network Training
Larger Dataset: https://www.kaggle.com/wcukierski/the-simpsons-by-the-data

41. Resources
The code associated with this presentation is available
here:
https://github.com/syednasar/talks/tree/master/synthetic-dialog
In case of any questions on the code or the
implementation, reach out at:
Email: techtalk@nasars.com or
Twitter: @techmazed

42. Whatcanyoudo from here?

43. References
&
Recommendations
•Visualizing and Understanding Recurrent Networks
(paper): https://arxiv.org/abs/1506.02078
•Blog by Andrej Karpathy:
http://karpathy.github.io/2015/05/21/rnn-
effectiveness/

44. Keep(deep)learning!
@techmazed