Deep Learning for Natural Language Processing by Roopal Garg
•
2 likes•260 views
Abstract:- The talk will focus on how Neural Networks are applied in the field of NLP for tasks like classification. Building blocks like Word Embeddings, Recurrent NN, LSTM, GRU, Convolutional NN, Sentence Representation and how they are applied to a piece of text in Tensorflow will be covered. These building blocks can be stacked together in various ways to form deeper network architectures. We will discuss one such architecture which is used within GumGum Inc to do Sentiment Analysis on web pages using NN in Tensorflow.
Recommended
Recommended
More Related Content
Similar to Deep Learning for Natural Language Processing by Roopal Garg
Similar to Deep Learning for Natural Language Processing by Roopal Garg (20)
More from Data Con LA
More from Data Con LA (20)
Recently uploaded
Recently uploaded (20)
Deep Learning for Natural Language Processing by Roopal Garg
- 1. WHAT YOU SEE IS WHAT WE GET Deep Learning for Natural Language Processing
- 3. GumGum Inc 3 Artificial Intelligence Company 9 year old, 225 Employees Based in Santa Monica, CA Offices in London, Sydney, New York, Chicago Thousands of Publishers and Advertisers Billions of Impressions per day
- 7. Brain’s interpretation... 7 Hi, what’s up? Hi, how are you? Trying to understand DL! Mr. Cool Brain
- 8. Brain’s mapping to Deep Learning 8 USC is the oldest private research university in California. USCFight On!! 0.58 1.60 0.10 -0.85 -0.18 -0.97 -0.14 1.22 1.89 1.91 0.78 0.33 -0.66 -0.19 1.59 Embeddings
- 9. Deep Learning in NLP... 9 Machine Learning Bags of n-grams Index Weight Deep Learning Word Sequence Vector Index
- 11. Word Embeddings : Word2Vec 11 CBOW PROJECTIONS in speak French France When OUTPUTINPUTS
- 12. Word Embeddings : Word2Vec 12 Skip-gram PROJECTIONS OUTPUT in speak French When France INPUTS
- 13. Word Embeddings : Glove 13 https://nlp.stanford.edu/projects/glove/ Co-occurrence Matrix
- 14. Sentences 14 I like playing soccer -7.76 -0.28 -4.12 -1.67 -5.02 x -6.97 -9.9 -1.94 -0.03 -5.19 -6.66 9.84 -5.44 -3.38 -8.06 -7.54 -8.74 -6.46 -9.61 -3.70-9.87 7.68 -2.66 6.35 -1.49
- 15. Sentences : Recurrent Neural Network (RNN) 15 x RNN y1
- 16. Sentences : Recurrent Neural Network (RNN) 16 I RNN y1 x1 h1 WX Wh Wo INPUT HIDDEN OUTPUT i1 = Wx x1 h0 Wh h0 h1 = tanh( i1 + Wh h0 ) y1 = ( Wo h1 )
- 17. Sentences : Recurrent Neural Network (RNN) 17 I like playing soccer RNN RNN RNN RNN y1 y2 y3 y4 x1 x2 x3 x4 h1 h2 h3 h4 WX WX WX WX Wh Wh Wh Wo Wo Wo Wo h0 Wh
- 18. Sentences : RNN : Long Short Term Memory (LSTM) 18 x LSTM y1 I LSTM y1 x1 h1 WX Wh Wo h0
- 19. LSTM Sentences : RNN : Long Short Term Memory (LSTM) 19 xt tanh tanh ht Ct Ct-1 ht-1
- 20. Sentences : RNN : Long Short Term Memory (LSTM) 20 xt tanh tanh ht Ct Ct-1 ht-1 forget gate ft ft = ( Wxf xt + Whf ht-1 ) Whf Wxf
- 21. Sentences : RNN : Long Short Term Memory (LSTM) 21 xt tanh tanh ht Ct Ct-1 ht-1 input gate ft it = ( Wxi xt + Whi ht-1 ) Whf Wxf Whi Wxi it Ct ~ Whc Wxc = tanh ( Wxc xt + Whc ht-1 )Ct ~
- 22. Sentences : RNN : Long Short Term Memory (LSTM) 22 xt tanh tanh ht Ct Ct-1 ht-1 cell state update ft Whf Wxf Whi Wxi it Ct ~ Whc Wxc Ct = ft * Ct-1 + it *Ct ~
- 23. Sentences : RNN : Long Short Term Memory (LSTM) 23 xt tanh tanh ht Ct Ct-1 ht-1 output gate ft Whf Wxf Whi Wxi it Ct ~ Whc Wxc ot = ( Wxo xt + Who ht-1 ) Who Wxo ht = ot * tanh( Ct ) ot
- 24. GRU Sentences : RNN : Gated Recurrent Unit (GRU) 24 xt ht ht-1 1 - zt tanh rt zt = ( Wxz xt + Whz ht-1 ) rt = ( Wxr xt + Whr ht-1 ) ct = tanh( Wxc xt + Whc ( rt * ht-1 ) ) ht = (1 - zt ) * ht-1 + zt * ct Whz Wxz Whr Wxr Whc Wxc ct
- 25. Sentences : Convolutional Neural Network (CNN) 25 I like playing soccer -5.75 -0.59 2.32 -8.76 -8.12 Wl1 Wr1 Wl1 Wr1 Wl1 Wr1 Wl2 Wr2 Wl2 Wr2 Wl3 Wr3
- 26. Paragraphs 26 I like playing soccer . I wish I was any good though . RNN RNN y h0 h13 h5 RNN RNN h6
- 27. Paragraphs 27 I like playing soccer . I wish I was any good though . RNN RNN h0 h8 h5 RNN RNN h1 y RNN RNN h0 h0 h1 h2
- 28. Classification Task 28 I like playing soccer . I wish I was any good . LSTM LSTM LSTM LSTM LSTM LSTM LSTM LSTM LSTM LSTM LSTM LSTM ATTENTION LSTM LSTMATTENTION ATTENTION SOFTMAX Hierarchical Attention Networks for Document Classification https://www.cs.cmu.edu/~diyiy/docs/naacl16.pdf
- 30. Tensorflow: Workflow 30 Train Data Embedding Layer xy Ops Loss Gradients Batch Predictions Updates word2idx tag2idx
- 31. Tensorflow: CODE!!!! 31 Part of Speech Tagging using Recurrent Neural Networks https://github.com/roopalgarg/recurrent-neural-nets-tensorflow
- 33. Reading material... 33 ● http://karpathy.github.io/2015/05/21/rnn-effectiveness ● http://colah.github.io/posts/2015-08-Understanding-LSTMs ● http://www.wildml.com/2016/08/rnns-in-tensorflow-a-practical-guide-and-undocumented -features/ ● http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp ● http://www.wildml.com/category/nlp/ ● http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/ ● http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/ Courses: ● DL for NLP: http://cs224d.stanford.edu ● Udacity Deep Learning: https://classroom.udacity.com/courses/ud730