https://www.youtube.com/watch?v=5ZUlVlumIQo&list=PLqJzTtkUiq54DDEEZvzisPlSGp_BadhNJ&index=10 Over the last years, deep learning is rapidly advancing with impressive results obtained in several areas including computer vision, machine translation and speech recognition. Deep learning attempts to learn complex function through learning hierarchical representation of data. A deep learning model is composed of non-linear modules that each transforms the representation from lower layer to the higher more abstract one. Very complex functions can be learned using enough composition of the non-linear modules. Furthermore, the need for manual feature engineering can be obviated by learning features themselves through the representation learning. In this talk, we first explain how deep learning architecture in particular and neural networks in general are loosely inspired by mammalian visual cortex and nervous system respectively. We also discuss about the reason for big and successful comeback of neural networks with the deep learning models. Finally, we give a brief introduction of various deep structures and their applications to several domains. References: LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." Nature 521.7553 (2015): 436-444. Socher, Richard, Yoshua Bengio, and Chris Manning. "Deep learning for NLP." Tutorial at Association of Computational Logistics (ACL), 2012, and North American Chapter of the Association of Computational Linguistics (NAACL) (2013). Lee, Honglak. "Tutorial on deep learning and applications." NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning. 2010. LeCun, Yann, and M. Ranzato. "Deep learning tutorial." Tutorials in International Conference on Machine Learning (ICML’13). 2013. Socher, Richard, et al. "Recursive deep models for semantic compositionality over a sentiment treebank." Proceedings of the conference on empirical methods in natural language processing (EMNLP). Vol. 1631. 2013. https://www.youtube.com/channel/UC9OeZkIwhzfv-_Cb7fCikLQ https://www.udacity.com/course/deep-learning--ud730 http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/

1. Brief Overview of Deep Networks
Monireh Ebrahimi
Semantic Cognitive Perceptual Computing Course, July 2016.
Ohio Center of Excellence in Knowledge-enabled Computing(Kno.e.sis),
Wright State University, USA
1

2. • “Representation-learning methods with
multiple levels of representation, obtained by
composing simple but non-linear modules
that each transform the representation at one
level (starting with the raw input) into a
representation at a higher, slightly more
abstract level. “
What is deep learning?
2
LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." Nature 521.7553
(2015): 436-444.

3. Successive model layers learn deeper intermediate representations.
Lee, Honglak. "Tutorial on deep learning and applications." NIPS 2010
Workshop on Deep Learning and Unsupervised Feature Learning.
2010.
3
What is deep learning? Learning Hierarchical Representations
Socher, Richard, Yoshua Bengio, and Chris Manning. "Deep learning
for NLP." Tutorial at Association of Computational Logistics (ACL),
2012, and North American Chapter of the Association of
Computational Linguistics (NAACL) (2013).

4. • Image recognition: Pixel → edge → texton →
motif → part → object
• Text: Character → word → word group →
clause → sentence → story
• Speech: Sample → spectral band → sound →
… phone → phoneme → word
What is deep learning? Learning Hierarchical Representations
4
LeCun, Yann, and M. Ranzato. "Deep learning tutorial." Tutorials in International
Conference on Machine Learning (ICML’13). 2013.

5. • Does not require any manual Feature
Engineering
• Deep architectures work well (vision, audio,
NLP, etc.)!
– Speech Recognition(2009)
– Computer Vision (2012)
• Early in 2015, a machine was able to beat the human at
an object recognition challenge for the first time in the
history of AI.
– Machine Translation (2014)
Why go deep?
5
LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." Nature 521.7553
(2015): 436-444.

6. • Loosely inspired by biological neural networks
(the central nervous system of animals),
particularly brain
Biologically inspired: how does the cortex learn perception?
6

7. • which details are important?
• For airplanes, feathers and wing flapping
weren't crucial
• What is the equivalent of aerodynamics for
understanding intelligence?
“Let's be inspired by nature, but not too much”
7
LeCun, Yann, and M. Ranzato. "Deep learning tutorial." Tutorials in International Conference
on Machine Learning (ICML’13). 2013.

8. • Retina - LGN - V1 - V2 - V4 - PIT – AIT
• Lots of intermediate representations
Biologically Inspired: The Mammalian Visual Cortex is
Hierarchical.
8
[picture from Simon Thorpe]
LeCun, Yann, and M. Ranzato. "Deep learning tutorial." Tutorials in International
Conference on Machine Learning (ICML’13). 2013.

9. All models are WRONG, but some are USEFUL.
9

10. Neural Networks
10

11. Why now?
11
• Vanishing Gradient Problem
• Lots of data
• GPUs

12. RBM (Restricted Boltzman Machine)
12
• Solution to Vanishing
Gradient Problem
• Reconstruct the input and
learn the features in this
process.
https://www.youtube.com/channel/UC9OeZkIwhzfv-
_Cb7fCikLQ

13. Autoencoders
13
https://www.youtube.com/channel/UC9OeZkIwhzfv-_Cb7fCikLQ
• Kind of Autoencoder (Feature
Extractor Neural Net)
• Detects inherent patterns in
data
• Unsupervised
• Good for real-world
problems
• Both Shallow and deep

14. Deep Learning for NLP
14
• Use of vectors
– dense low-dimensional real-valued vectors
• Continuous Bag of Words
• Skip Gram Model
• Two popular tools: Word2Vec, Glove
– One-hot vector
• Size of the entire vocabulary
• Very large sparse vector
https://www.youtube.com/channel/UC9OeZkIwhzfv-_Cb7fCikLQ

15. Continuous Bag of Words
15
Context Words Target word

16. Skip Gram Model
16
Target Word Context Words

17. Deep Belief Net
17
• Stack of RBMs
• Identical to MLP in terms of network structure
• Different Training:
– Pre-training
– Fine-tuning
• Small labeled dataset
• Reasonable training time
• Very accurate
• Image Recognition

18. Convolutional Neural Networks
18
1. Convolutional layer
2. ReLU layer
3. Pooling Layer
4. Fully Connected Layer
• Supervised
• Large amount of labeled
data for training

19. Convolutional Neural Networks
19
– CNN performs quite well on NLP problems.
• Although we do not have the nice intuition that we have for
image recognition
– Text Processing (Sentiment Analysis and Text
Categorization)
• Word-level
• Character-level:
– Very attractive for user-generated contents with typos and new
vocabularies
– Models can be fine-tuned from a task A with large corpus to a
more targeted task with smaller corpus
– Learning directly from character-level input (needs millions of
examples)
– Learning from pre-trained character embeddings

20. Recurrent Neural Nets
20
http://www.wildml.com/2015/09/recurrent-neural-networks-
tutorial-part-1-introduction-to-rnns/
• Not Feedforward
• Sequence of values as input
• Sequence of values as output
• Stacking RNNs on top of each other

21. Recurrent Neural Nets
21
• Extremely difficult to train
– Exponential Vanishing Gradient Problem
• RNN with n time steps = n layers MLP
– Solution:
• LSTM/GRU: Helps the net to decide when to forget the current
input and when to remember it for the future time steps.
• Good for:
– Time Series Analysis (Forecasting)
– Machine Translation
– Text Processing (Parsing, NER, Sentiment Analysis)
• Word-level
• Character-level

22. Recursive Neural Tensor Network
22
• Leave group:
• input
• Root group:
• class and score
Socher, Richard, et al. "Recursive deep models for semantic compositionality
over a sentiment treebank." Proceedings of the conference on empirical
methods in natural language processing (EMNLP). Vol. 1631. 2013.

23. References
23
1. LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." Nature
521.7553 (2015): 436-444.
2. Socher, Richard, Yoshua Bengio, and Chris Manning. "Deep learning for NLP."
Tutorial at Association of Computational Logistics (ACL), 2012, and North American
Chapter of the Association of Computational Linguistics (NAACL) (2013).
3. Lee, Honglak. "Tutorial on deep learning and applications." NIPS 2010 Workshop
on Deep Learning and Unsupervised Feature Learning. 2010.
4. LeCun, Yann, and M. Ranzato. "Deep learning tutorial." Tutorials in International
Conference on Machine Learning (ICML’13). 2013.
5. Socher, Richard, et al. "Recursive deep models for semantic compositionality over
a sentiment treebank." Proceedings of the conference on empirical methods in natural
language processing (EMNLP). Vol. 1631. 2013.
6. https://www.youtube.com/channel/UC9OeZkIwhzfv-_Cb7fCikLQ
7. https://www.udacity.com/course/deep-learning--ud730
8. http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-
introduction-to-rnns/

24. Thank you 
Thank you, and please visit us at http://knoesis.org
monireh@knoesis.org
24