Raise of deep learning

TheRiseofDeep
Learning
Wenjin Gu, August 2018

Year1 ofDL Era
Genesys confidential and proprietary information.
Unauthorized disclosure is prohibited. 2

Whathappened–AlexNet
3
Unauthorized disclosure is prohibited.
◦ Winner of ImageNet LSVRC in 2012
◦ Designed by Alex Krizhevsky, Geoffrey Hinton, and Ilya
Sutskever (the SuperVision group,UofT)
◦ Achieved a top-5 error of 15.3% (2nd place - 26.2%)

Whymachinelearning
4
◦ Conventional programming is logic driven
◦ Human thoughts are limited by number of logical arguments
thus have troubles to handle high dimensional data
◦ NP- hard problems cannot be solved in polynomial time

5
◦ Machine Learning is data driven and self-fitting thus can
handle high dimensional data
◦ Machine Learning gives approximation of an unknown target
function up to a given accuracy thus can tackle np-hard
problems (Heuristic algorithm)
Why machinelearning

6
◦ Assuming dogs and cats are distinguishable, there must be a
hyperplane divides them in a high dimensional feature space.
However the space is crumpled and folded in its raw data
representation
What does deep learningdo

How DeepLearningunfoldsthespace
7
◦ Connections cause linear transformation and feature interaction
◦ More nodes increases feature dimensionality
◦ Less nodes reduces dimensionality (drops irrelevant features)
◦ Activation function unfolds the space by nonlinearity
φ(∑wx+b)

Abriefhistory
8

Abriefhistory– thetrouble
9
◦ Feature engineering + classifier > Shallow NN > Deep NN
◦ Believe domain expertise is mandatory
◦ Assumes gradient descent would get trapped in poor local
minima

Localminimum andSaddle Point
10

Abriefhistory– therealproblem
11
◦ Vanishing of gradients
In the attempt to normalize data, activation functions are usually in the (0,1)
range. Backpropagation computes gradients by the chain rule. This has the effect of
multiplying n of these small numbers to compute gradients. Error signal eventually
disappears in the front layer, effectively preventing the further training.

Abriefhistory– thebreakthrough
12
◦ “A Fast Learning Algorithm for Deep Belief Nets” by Hinton, Geoffrey E., Simon
Osindero, and Yee-Whye Teh. Neural computation 18.7 (2006): 1527-1554
◦ Different from Deep Forward
Network, Deep Belief Network
connections are bi-direction

Abriefhistory– thebreakthrough
13
◦ Unsupervised learning to train layer by layer in a deep forward network view
◦ Increases the selectivity and the
invariance of the representation

Abriefhistory– newtechniques(RectifiedLinearUnit)
14
Short cut

Goingdeeperanddeeper
15

16
Buildingblock - Layers

17
Buildingblock - Layers
◦ Dense - fully connected layer often used for simple vector data
◦ LSTM - recurrent layer often used for Sequence data (NLP, stock)
◦ Conv2D - 2D convolution layer often used for image process
◦ MaxPooling2D - 2D pooling layer often used for image process down sizing

18
Buildingblock - Lossfunction

19
Buildingblock - Lossfunction
◦ Lie at the heart of machine learning. The network will take whatever shortcut it
can, to minimize the loss. Take an example, anything wrong with this objective?
“Maximizing the average well-being of all humans”
◦ Common Loss functions
• Binary_crossentropy for a binary classification,
• Categorical_crossentropy for multi-class classification problem,
• Mean Squared Error for a regression problem.

20
Buildingblock – Input tensorshape

21
Buildingblock – Input tensorshape
◦ Vector data—2D tensors of shape (samples, features)
◦ Timeseries data or sequence data—3D tensors of shape (samples, timesteps,
features)
◦ Images—4D tensors of shape (samples, height, width, channels) or (samples,
channels, height, width)
◦ Video—5D tensors of shape (samples, frames, height, width, channels) or
(samples, frames, channels, height, width)
* Batch dimension (samples) is not included in ‘input_shape’

22
Buildingblock – Activationfunction

23
Buildingblock – Activationfunction
Reason
• Continuously differentiable – required by gradient-based optimization
• Nonlinearity – Map to new hypothesis space. Without nonlinearity,
multi-layer collapse to single layer
Common activation function
• Relu – hidden layer
• Sigmoid – last layer for Binary classification, Multiclass multilabel
classification
• Softmax – last layer for Multiclass single-label classification

24
Buildingblock – Gradient descentoptimizer

25
Buildingblock – Gradient descentoptimizer
◦ Overcome local minima and saddle points (Momentum, Look-ahead)
SGD optimization on saddle point SGD optimization on loss surface contours

26
Buildingblock – Metrics

27
Buildingblock – Metrics
◦ Question 1: why we need metrics since there is already a loss function?
 Loss function is for machine, metrics is for human
◦ Question 2: Can we use metrics as a loss function?
 Loss function needs to be continuous and differentiable in order for gradient
descent to work
◦ Question 3: Do we need any metrics other than ‘accuracy’?
 Email spam detection – false positive is expensive, false negative is probably ok
 Airport terrorist detection – false negative is fatal, false positive is less expensive

DoesDeepLearningsurpasshumanalready?
28
◦ Image Recognition – 2015 Microsoft, Google Beat Humans at Image Recognition
Really? try this one -
Tagging image is simple, but to tell a story
from a image requires knowledges outside of
the picture. Same goes with Natural language
understanding

29
◦ Game – 2016 AlphaGo beats Lee Sedol in a five-game match
◦ 2018 August Dota 2 Pro players beats OpenAI (founded by
Elon Musk) bot in a best of three game match
MOBA games take place in complex, ever-changing environment, which is closer to the problems we
want AI to tackle in real life. To make the life easer for the machine, 18 of more than 100 heroes are
allowed to be used in the game.

30
Deep learning in different fields are unbalanced. While it exceeds human
level in fields with clear rules and sufficient training data, it lags behind in
area requires external knowledge or with more uncertainties.
The Real World Is Far More Complicated than a board game and human
learn things much fast and efficiently with fewer examples.

Nextstep
31
◦ Short term – blend deep learning and machine learning with
human knowledge
◦ Long term – Unsupervised learning, multi-task multi-stage
learning to create real AI

Questions?
Unauthorized disclosure is prohibited. 32

Copyright ©2017 Genesys.
2001 Junipero Serra Blvd., Daly City, CA 94014
All Rights reserved. Genesys and the Genesys logo are registered trademarks of Genesys. All other company names and logos may be registered
trademarks or trademarks of their respective companies.
Thank You
Visit www.genesys.com or call +1.855.821.0932 for more information

Raise of deep learning

Recommended

Recommended

More Related Content

Similar to Raise of deep learning

Similar to Raise of deep learning (20)

Recently uploaded

Recently uploaded (20)

Raise of deep learning