This document discusses deep learning and neural networks. It begins with a brief history of neural networks, from the earliest Perceptron algorithm in 1958 to modern developments enabled by increased computational power and data. Deep learning uses neural networks with multiple hidden layers to automatically learn representations of data and hierarchical feature detectors. Examples are given of applying deep learning to tasks like image recognition. The document outlines challenges of deep learning like the large amount of training required and complexity of modeling real-world behaviors.

1. 1
KOTA BARU PARAHYANGAN BANDUNG
Machine Learning
Deep Learning

2. 2
Deep Learning (DL)

3. 3
References
• Jared Dean, Big Data, Data Mining, and
Machine Learning, Willey
• Brett Lantz, Machine Learning with R,
Packt
• M S Ram, Introduction to Deep
Learning, Indian Institute of Technology
Kanpur
• Papers, Website, tutorial

4. • The term “deep learning” has become
widely used to describe machine learning
methods that can learn more abstract
concepts, primarily in the areas of image
recognition and text processing than
methods previously available
• Deep Learning is another name for a set
of algorithms that use a neural network as
an architecture, and learn the features
automatically
Deep Learning Overview

5. • The earliest example of artificial neural networks is the
Perceptron algorithm developed by Rosenblatt in 1958
• In the late 1970’s, researchers discovered that Perceptron
cannot approximate many nonlinear decision functions
• In 1980’s, researchers found a solution to that problem by
stacking multiple layers of linear classifiers (hence the name
“multilayer perceptron”) to approximate nonlinear decision
functions
• the lack of computational power and labeled data etc., neural
networks were left out of mainstream research in late 1990’s
and early 2000’s
• Since the late 2000’s, neural networks have recovered and
become more successful thanks to the availability of
inexpensive, parallel hardware (graphics processors,
computer clusters) and a massive amount of labeled data
Deep Learning - Brief History

6. Perceptron
• It was the first algorithmically described neural
network. Its invention by Rosenblatt in 1958
• It is the simplest form of a neural network used
for the classification of patterns said to be
linearly separable
• The perceptron built around a single neuron is
limited to performing pattern classification with
only two classes (hypotheses)

7. • The most important reason is that neural networks have
a lot of parameters, and can approximate very nonlinear
functions. So if the problem is complex, and has a lot of
data, neural networks are good approximators for it
– For instance, neural networks with three or more hidden layers
have proven to do quite well at tasks such as recognizing
handwritten digits or selecting images with dogs in them
• The second reason is that neural networks are very
flexible: we can change the architecture fairly easily to
adapt to specific problems/domains (such as
convolutional neural networks and recurrent neural
networks
Neural Network – Why Success ?

8. Deep Learning Application

11. Neural Network & Deep Learning
• Ideas drawn from neural networks and
machine learning are hybridized to
perform improved learning tasks beyond
the capability of either one operating on its
own, and
• ideas inspired by the human brain lead to
new perspectives wherever they are of
particular importance

12. Deep Learning- Example

13. Deep Learning- Example

14. Deep Learning- Example 1

20. Deep Learning- Example 2

23. Deep Learning- Example 3

26. • Create algorithms
– that can understand scenes and describe them in
natural language
– that can infer semantic concepts to allow machines to
interact with humans using these concepts
• Requires creating a series of abstractions
– Image (Pixel Intensities) -> Objects in Image ->
Object Interactions -> Scene Description
• Deep learning aims to automatically learn these
abstractions with little supervision
Deep Learning - Aim

27. • Inspiration from mammal brain
• Multiple Layers of “neurons” (Rumelhart et al
1986)
• Train each layer to compose the representations
of the previous layer
• to learn a higher level abstraction
– Ex: Pixels -> Edges -> Contours -> Object parts ->
Object categories
– Local Features -> Global Features
• Train the layers one-by-one (Hinton et al 2006)
– Greedy strategy
How do we train?

28. Neuron vs Artificial Network

29. Neuron vs Artificial Network

30. Neuron vs Artificial Network

31. Element of Neural Network

32. Element of Neural Network
1. Inputs are fed into the perceptron
2. Weights are multiplied to each input
3. Summation and then add bias
4. Activation function is applied. Note that here
we use a step function, but there are other
more sophisticated activation functions like
sigmoid, hyperbolic tangent (tanh), rectifier
(relu) and more. No worries, we will cover many
of them in the future!
5. Output is either triggered as 1, or not, as 0.
Note we use y hat to label output produced by
our perceptron model

37. Deep Learning Model

38. First DL Model

41. • Training neural networks
takes a long time,
especially when the
training set is large
• It therefore makes sense
to use many machines
to train our neural
networks.
Challenge

42. • Too many concepts to learn
– Too many object categories
– Too many ways of interaction between objects
categories
• Behaviour is a highly varying function underlying
factors
– f: L -> V
– L: latent factors of variation
• low dimensional latent factor space
– V: visible behaviour
• high dimensional observable space
– f: highly non-linear function
Challenge