Slides for Introduction to Neural Networks for Language Modeling Tasks

1. Neural Network
Language Modeling
ADEL RAHIMI

2. Table of Contents
•Introduction to neural networks
•Neurons
•Activation Functions
•Types of neural networks
•Learning in neural networks
•Applications of neural networks
•Advantages of neural networks
•Disadvantages of neural networks
•Introduction to Language modeling
•Neural Networks Language Modeling

3. Neural Networks
Computing based on interaction of multiple connected processing elements
Powerful in doing many processes
Ability to adapt and learn
Ability to deal with incomplete data

4. Basics of neural networks
Developed from biological approaches
Developed in 1943
Has one or more layers
Has different types of feeding: including feedforward and feedback networks

5. neurons
Artificial
Biological

6. Perceptron
A mathematical model of biological neuron

7. Neural network neurons
Receives input
Multiplies input by weight
applies activation function to the sum of results
Output results

8. Neural network advantages
Neural networks can perform better than normal linear programs
As neural networks are parallel they can continue without failing
Can be implemented in variety of applications
Ability to derive meaning from complicated or imprecise data

9. Disadvantages
Neural networks need training data
Requires high processing time

10. Activation function
Control the activity of the unit
Threshold function outputs 1 when it is active and 0 when it is inactive
Some examples of activation functions:
Sigmoid = 1 / (1 + e-x)
Tanh = 2/(1+e-2x)-1 (-1, 1)

11. Types of neural networks
Connection type:
◦ Static
◦ Dynamic
Typology:
◦ Single layer
◦ Multiple layers
◦ Recurrent
Learning method:
◦ Supervised
◦ Unsupervised
◦ Reinforcement

12. Connections
Static:
Like feedforward networks
Dynamic:
Feedback networks; which their connection changes when the network is going backwards

13. Topology
Single layer Multilayer Recurrent

14. Learning methods
Supervised learning
◦ Each learning pattern: input + desired output
◦ At each presentation: adapts weights
◦ After many epochs convergence to a local minimum

15. Unsupervised learning
No help from outside
Learning by doing
Pick out structures in the input:
◦ Clustering
◦ Dimensionality reduction

16. Reinforcement learning
Inspired by behaviorist psychologists
Teacher: Training data
The teacher scores the performance of the training examples
Use performance to shuffle weights randomly
Relatively slow in learning due to randomness
Examples: Robotic tasks

17. Neural network applications
Pattern recognition
Investments analysis
Control systems and monitoring
Mobile computing
Natural language processing
Forecasting sales, market, meteorology

18. Language modeling
Filtering out bad sentences
Model the sentences via probability distribution over sequences of words:
Pr(w1,w2,…,wn)
Assign a probability to a given sentence:
S1 = “The cat jumped over the dog”, Pr(S1) ~1
S2 = “The Over cat dog jumped”, Pr(S2) ~0

19. Language modeling applications
Machine translation:
◦ P(high winds tonight) > P(large winds tonight)
Spell correction:
◦ The office is about fifteen minuets from my house
◦ P(about fifteen minutes from) > P(about fifteen minuets from)
Speech Recognition:
◦ P(I saw a van) > P(eyes awe of an)
Summarization, question-answering, etc.

20. N-gram models
A sentence s = (x1, x2, ... , xT)
How likely is s?
p(x1, x2, ... , xT)

21. N-gram models
Data sparsity
Lack of generalization:
[ride a horse], [ride a llama]
[ride a zebra]

22. One-hot encoding
V = {zebra, horse, school, summer}
V(zebra) = [1,0,0,0]
V(Horse) = [0,1,0,0]
V(school) = [0,0,1,0]
V(summer) = [0,1,0,1]
One hot encoding is simple and yet word similarity is undefined

23. Distributional Representation
Stars: shining, bright, light, dark
Stars: cucumber, computer, lava, Harvard

24. Distributional Representation
It is simple
It has the notion of word similarity
But it can be computationally and memory-wise inefficient

25. Distributional Representation
V is a vocabulary
wi ∈ V
v(wi) ∈ Rn
v(wi) is a low-dimensional, learnable, dense word vector

26. Distributed Representation

27. Deep learning + language modeling
Traditionally uses architecture such as Recurrent neural networks
Advancements In neural networks: LSTM
LSTM is a recurrent network that can be part of an eventually bigger recurrent network.
LSTM contains: Cell, input gate, output gate, forget gate
The Cell is the memory, the three other gates are conventional artificial neurons

28. LSTM
it= g(wxixt+whiht-1)
ft=g(wxfxt+whfht-1)
ot=g(wxoxt+whoht-1)

29. Neural Network Language Model

30. Neural Network Language Model

31. Convolutional networks
CNNs are breakthrough in image processing; however they can used in language modeling as
well.

32. Convolutional networks
The input of convolutional neural nets for language modelng could either be word2vec to one-
hot embedding.

33. Evaluation
Based on:
Jozefowicz, Rafal, et al. "Exploring the limits of language modeling." arXiv preprint
arXiv:1602.02410 (2016).
Best performance of LSTM in language modeling yields 23 in perplexity and RNN resulted 41 in
perplexity.