Foundation of RecurrentNeural Networks
Goal
model long term dependencies
connect previous information to the present task
model sequence of events with loops, allowing information to
persist
punching
3.
Foundation of RecurrentNeural Networks
Goal
model long term dependencies
connect previous information to the present task
model sequence of events with loops, allowing information to persist
Feed Forward NNets can not take time dependencies into
account. Sequential data needs a Feedback Mechanism.
feedback mechanism
or internal state loop
o
x
A
x0
…
…
FF-net / CNN
xt
…
…
o0
ot
…
…
x0
Recurrent Neural Network (RNN)
xt-1 xt xT
… …
o0
ot-1
ot oT
Unfold
in time
tim
e
Whh Whh Whh
Whh
Foundation of Recurrent Neural Networks
4.
1. Sequential Input:RNNs process input data sequentially, one
element at a time, where each element corresponds to a specific
time step in the sequence.
2. Hidden State: At each time step, the RNN maintains a hidden
state vector. This hidden state captures information from
previous time steps and serves as a form of memory, allowing the
network to consider context from earlier elements in the
sequence.
3. Weight Sharing: RNNs use the same set of weights and biases for
each time step, allowing them to share parameters across the
entire sequence. This weight sharing is a key feature that enables
RNNs to maintain memory of past inputs.
4. Update Rule: The hidden state at each time step is updated
based on the current input and the previous hidden state. The
update rule involves a combination of the input, the previous
hidden state, and the network's parameters (weights and biases).
5. Output: The updated hidden state at each time step is then used
to generate the output for that time step. This output can be
used for various tasks, such as classification, regression, or
generating the next element in the sequence.
6.
The sigmoid layeroutputs numbers between 0-1 determine how much each
component should be let through. Pink X gate is point-wise multiplication.
