5. State of the art
• Learning feature detectors for faces and
cats unsupervised from videos.
• Billions of units, >9 layers
• Better than human recognition of traffic
sings
7. The good
• State of the art results in many fields
• Unsupervised, semi-supervised
• (at least somewhat) online learning and
adaptation
• multi-task learning
• (close to) linear scalability
8. The bad
• Expensive to train
• Hard to inspect/visualize progress for non-
visual tasks
9. The ugly
• Hyperparameter and topology selection
still critical
• Dependance on tricks for practical results
on real-life datasets
11. DBN key ideas:
network stacking
• Greedy layer-wise learning
• Hidden units of level k as visible units of
level k+1
• (Use backpropagation on whole stack)
13. DBN key ideas: RBM
• generative stochastic neural network
• the network has an energy function and we
are searching for thermal equilibrium
• binary units; weights are state change
probabilities
• learning via contrastive divergence
15. DBN key ideas:
auto-encoder
• Denoising auto-encoder
(corrupt and reconstruct
the input)
• Sparse coding
(each item is encoded by
strong activation of a
small set of neurons)
17. LSTM
• RNN with explicit state
• Combination of BPTT and RTRL learning
• Online learning
• Can retain information over arbitrarily long
periods of time
• Can be trained by artificial evolution
• Can combine LSTM blocks with regular
units
