This document outlines steps to build a natural language processing deep neural network (NLP DNN) in 5 minutes. It discusses creating an embedding layer to tokenize and vectorize text, using pre-trained word embeddings. It then shows how to add other layers like bidirectional GRUs for encoding and attention layers for weighting encodings. The embedding layer takes only one line of code in Keras. The document provides an example of building and training an NLP DNN that embeds, encodes, applies attention, and predicts outputs.

1. Building a NLP DNN in 5 minutes!!Building a NLP DNN in 5 minutes!!
Isuru GunasekaraIsuru Gunasekara
isuru@imrsv.ai

2. Steps in creating a typical NLP DNNSteps in creating a typical NLP DNN
1. Create embedding layer
2. Add other layers same as any DNN
3. Get results!!

3. Creating the embedding layerCreating the embedding layer
Tokenize train & test setsTokenize train & test sets
A language toolkit such as NLTK can be used for this.
Find all the words in the train & test setsFind all the words in the train & test sets
["this","is","a","sentence","another",...]
["This is a sentence", "this is another sentence",...]
===>>>
[["This", "is", "a", "sentence"],["This", "is", "another", "sentence"],...]

4. Find all the word vectors for the words in the embeddingFind all the word vectors for the words in the embedding
1.Create a dictionary with indexes for all the words in the train and test sets
2.Create a list with the vectors corresponding to those words from the embedding.
This list is used to create the rst layer of the DNN
3.If the embedding does not contain a vector for a word, give it a prede ned vector. eg:
[0,0,....]
embedding_dict = {"this":0, "is":1, "a":2, "sentence":3, "another":4....}
embedding_matrix = [[300d vector corresponding to "this"], [300d vector corres
ponding to "is"],......]

5. Embed->Encode->Attend->PredictEmbed->Encode->Attend->Predict

6. First layer of the DNNFirst layer of the DNN

7. This takes only one line of code thanks to Keras!!!This takes only one line of code thanks to Keras!!!
Embed->Encode->Attend->Predict
from keras.layers import Embedding
embedding_layer = Embedding(embedding_matrix.shape[0],
embedding_matrix.shape[1],
weights=[embedding_matrix],
trainable=False)(input_layer)

8. Embed->Encode->Attend->Predict
x = Bidirectional(CuDNNGRU(recurrent_units,
return_sequences=True)
)(embedding_layer)

9. Embed->Encode->Attend->Predict
Keras doesn't have a Attention layer as of yet. However, there are many implementations
of different attention layers implemented for both Tensor ow and Theano backends for
Keras.
eg: MIT Deepmoji -> (
)
x = AttentionWeightedAverage(name='attlayer', return_attention=False)(x)
https://github.com/bfelbo/DeepMoji
(https://github.com/bfelbo/DeepMoji)

10. Embed->Encode->Attend->Predict
output_layer = Dense(6, activation="sigmoid")(x)

11. In [24]: Image(filename = "presentation/img/model_tb.png", width=400, height=300, unconfi
ned=True)
Out[24]:

12. In [25]: Image(filename = "presentation/img/model_att.png", width=400, height=300, unconf
ined=True)
Out[25]:

13. Thank you!Thank you!