Dmytro Fishman, profile picture
Uploaded byDmytro Fishman
PDF, PPTX1,775 views

5 Introduction to neural networks

This document provides an introduction to machine learning, specifically focusing on neural networks and deep learning. It discusses the evolution of machine learning methods, the architecture and functioning of artificial neural networks, and the concept of training these networks through backpropagation. Key historical references and concepts are presented, detailing the process of modifying weights to improve predictions.

Embed presentation

Download as PDF, PPTX
1 / 104
2 / 104
3 / 104
4 / 104
5 / 104
6 / 104
7 / 104
8 / 104
9 / 104
10 / 104
11 / 104
12 / 104
13 / 104
14 / 104
15 / 104
16 / 104
17 / 104
18 / 104
19 / 104
20 / 104
21 / 104
22 / 104
23 / 104
24 / 104
25 / 104
26 / 104
27 / 104
28 / 104
29 / 104
30 / 104
31 / 104
32 / 104
33 / 104
34 / 104
35 / 104
36 / 104
37 / 104
38 / 104
39 / 104
40 / 104
41 / 104
42 / 104
43 / 104
44 / 104
45 / 104
46 / 104
47 / 104
48 / 104
49 / 104
50 / 104
51 / 104
52 / 104
53 / 104
54 / 104
55 / 104
56 / 104
57 / 104
58 / 104
59 / 104
60 / 104
61 / 104
62 / 104
63 / 104
64 / 104
65 / 104
66 / 104
67 / 104
68 / 104
69 / 104
70 / 104
71 / 104
72 / 104
73 / 104
74 / 104
75 / 104
76 / 104
77 / 104
78 / 104
79 / 104
80 / 104
81 / 104
82 / 104
83 / 104
84 / 104
85 / 104
86 / 104
87 / 104
88 / 104
89 / 104
90 / 104
91 / 104
92 / 104
93 / 104
94 / 104
95 / 104
96 / 104
97 / 104
98 / 104
99 / 104
100 / 104
101 / 104
102 / 104
103 / 104
104 / 104
Introduction to Machine Learning (Neural networks) Dmytro Fishman (dmytro@ut.ee)
The following material was adopted from:
Evolution of ML methods Rule-based Adopted from Y.Bengio http://videolectures.net/deeplearning2015_bengio_theoretical_motivations/ Input Fixed set of rules Output
Evolution of ML methods Rule-based Classic Machine Learning Adopted from Y.Bengio http://videolectures.net/deeplearning2015_bengio_theoretical_motivations/ Input Input Fixed set of rules Output Hand designed features Learning Output
Evolution of ML methods Rule-based Classic Machine Learning Representation Learning Adopted from Y.Bengio http://videolectures.net/deeplearning2015_bengio_theoretical_motivations/ Input Input Input Fixed set of rules Output Hand designed features Learning Automated feature extraction Output Learning Output
Evolution of ML methods Rule-based Classic Machine Learning Representation Learning Deep Learning Adopted from Y.Bengio http://videolectures.net/deeplearning2015_bengio_theoretical_motivations/ Input Input Input Input Fixed set of rules Output Hand designed features Learning Automated feature extraction Output Learning Output Low level features High level features Learning Output
What is deep learning? Many layers of adaptive non-linear processing to model complex relationships among data Space 1 Space 2
What is deep learning? Many layers of adaptive non-linear processing to model complex relationships among data Space 1 Space 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 155 255 255 255 155 0 255 255 255 255 255 255 255 255 155 78 78 155 255 255 255 0 0 0 0 155 255 0,1,2,3,…9
What is deep learning? Many layers of adaptive non-linear processing to model complex relationships among data Space 1 Space 2 species
What is deep learning? Many layers of adaptive non-linear processing to model complex relationships among data Space 1 Space 2 “We love you”
In practice DL = Artificial Neural Networks with many layers
A Logical Calculus of the Ideas Immanent in Nervous Activity McCulloch & Pitts (1943)
Rosenblatt (1957) Perceptron NewYorkTimes: “(The perceptron) is the embryo of an electronic computer that is expected to be able to walk, talk, see, write, reproduce itself and be conscious of its existence” A Logical Calculus of the Ideas Immanent in Nervous Activity McCulloch & Pitts (1943)
Minsky & Papert (1969) Perceptrons: an introduction to computational geometry Rosenblatt (1957) Perceptron A Logical Calculus of the Ideas Immanent in Nervous Activity McCulloch & Pitts (1943)
Blum & Rivest (1992) Training a 3-node neural network is NP- complete Minsky & Papert (1969) Perceptrons: an introduction to computational geometry Rosenblatt (1957) Perceptron A Logical Calculus of the Ideas Immanent in Nervous Activity McCulloch & Pitts (1943)
Rumelhart, Hinton & Williams (1986) Learning representations by back- propagating errors Blum & Rivest (1992) Training a 3-node neural network is NP- complete Minsky & Papert (1969) Perceptrons: an introduction to computational geometry Rosenblatt (1957) Perceptron A Logical Calculus of the Ideas Immanent in Nervous Activity McCulloch & Pitts (1943)
Artificial neural network • A collection of simple trainable mathematical units, which collaborate to compute a complicated function • Compatible with supervised, unsupervised, and reinforcement • Brain inspired (loosely)
x2w2 x1w1 x2w2 w0 X 0 x1w1 x0 Artificial Neuron
x2w2 x1w1 x2w2 w0 X 0 x1w1 x2w2 x0 Artificial Neuron
x2w2 x1w1 x2w2 w0 X 0 x1w1 x2w2 x0 Artificial Neuron x1w1
x2w2 x1w1 x2w2 w0 X 0 x1w1 x2w2 x0 Artificial Neuron x1w1 x0w0
x2w2 x1w1 x2w2 w0 X 0 x1w1 x2w2 x0 Artificial Neuron x1w1 x0w0
X i 0 x2w2 x1w1 x0 Input layer Fully connected layer Output layer Single neuron Feedforward Neural Network
X i 0 x2w2 x1w1 x0 Input layer Fully connected layer Output layer Single neuron Feedforward Neural Network
X i 0 x2w2 x1w1 x0 Input layer Fully connected layer Output layer Single neuron Feedforward Neural Network
x2w2 x1w1 x0 Input layer Fully connected layer Output layer Single neuron Feedforward Neural Network X x x2w2 x1w1 x0w0 0
x2w2 x1w1 x0 Input layer Fully connected layer Output layer Single neuron Feedforward Neural Network X x x2w2 x1w1 x0w0 0
x2w2 x1w1 x0 Input layer Fully connected layer Output layer Single neuron Feedforward Neural Network X i 0
Learning algorithm • modify connection weights to make prediction closer to y • run neuronal network on input x • while not done • pick a random training instance (x, y)
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error neth1 = w1 ⇤ i1 + w2 ⇤ i2 + b1 ⇤ 1 INPUT TARGET
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error neth1 = w1 ⇤ i1 + w2 ⇤ i2 + b1 ⇤ 1 neth1 = 0.15 ⇤ 0.05 + 0.2 ⇤ 0.1 + 0.35 ⇤ 1 = 0.3775 INPUT TARGET
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error neth1 = w1 ⇤ i1 + w2 ⇤ i2 + b1 ⇤ 1 neth1 = 0.15 ⇤ 0.05 + 0.2 ⇤ 0.1 + 0.35 ⇤ 1 = 0.3775 INPUT TARGET f(x) = 1 1 + e x outh1 = 1 1 + e neth1 = 1 1 + e 0.3775 = 0.5933
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error neth1 = w1 ⇤ i1 + w2 ⇤ i2 + b1 ⇤ 1 neth1 = 0.15 ⇤ 0.05 + 0.2 ⇤ 0.1 + 0.35 ⇤ 1 = 0.3775 INPUT TARGET f(x) = 1 1 + e x outh1 = 1 1 + e neth1 = 1 1 + e 0.3775 = 0.5933 Repeat for h2 = 0.596; o1 = 0.751; o2 = 0.773
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error INPUT TARGET We have o1, o2
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error INPUT TARGET We have o1, o2 Etotal = X 1 2 (target output) 2
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error INPUT TARGET We have o1, o2 Etotal = X 1 2 (target output) 2 Eo1 = 1 2 (targeto1 outo1) 2 = 1 2 (0.01 0.7514) 2 = 0.2748
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error INPUT TARGET We have o1, o2 Etotal = X 1 2 (target output) 2 Eo1 = 1 2 (targeto1 outo1) 2 = 1 2 (0.01 0.7514) 2 = 0.2748 Eo2 = 0.02356
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error INPUT TARGET We have o1, o2 Etotal = X 1 2 (target output) 2 Eo1 = 1 2 (targeto1 outo1) 2 = 1 2 (0.01 0.7514) 2 = 0.2748 Eo2 = 0.02356 Etotal = Eo1 + Eo2 = 0.2748 + 0.02356 = 0.29836
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET We want to know how much a change in w5 affects the total error
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET Etotal = X 1 2 (target output) 2 @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET Etotal = X 1 2 (target output) 2 @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error @Etotal @outo1 = 2 ⇤ 1 2 (targeto1 outo1) ⇤ 1 + 0 = (0.01 0.751) = 0.741
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error outo1 = 1 1 + e neto1
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error @outo1 @neto1 = outo1 (1 outo1) = 0.1868 outo1 = 1 1 + e neto1
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error neto1 = w5 ⇤ outh1 + w6 ⇤ outh2 + b2 ⇤ 1
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error neto1 = w5 ⇤ outh1 + w6 ⇤ outh2 + b2 ⇤ 1 @neto1 @w5 = outh1 = 0.5933
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error @Etotal @w5 = 0.7414 ⇤ 0.1868 ⇤ 0.5933 = 0.0821
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error @Etotal @w5 = 0.7414 ⇤ 0.1868 ⇤ 0.5933 = 0.0821 wnew 5 = wold 5 ⌘ ⇤ @Etotal @w5 = 0.4 0.5 ⇤ 0.0821 = 0.3589
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 • Repeat for w6, w7, w8 INPUT TARGET
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 • Repeat for w6, w7, w8 INPUT TARGET • In analogous way for w1, w2, w3, w4
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 • Repeat for w6, w7, w8 INPUT TARGET • In analogous way for w1, w2, w3, w4 • Compute the total error before: 0.298371109
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 • Repeat for w6, w7, w8 INPUT TARGET • In analogous way for w1, w2, w3, w4 • Compute the total error before: now: 0.298371109 0.291027924
Lets backpropagate i1 = 0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 • Repeat for w6, w7, w8 INPUT TARGET • In analogous way for w1, w2, w3, w4 • Compute the total error before: now: 0.298371109 0.291027924 • Repeat x10000: 0.000035085
http://www.emergentmind.com/neural-network Training Neural Networks
http://playground.tensorflow.org/ Training Neural Networks (part II)
Deep networks were difficult to train Overfitting DimensionalityVanishing gradients Complex landscape w2 w1 E(w1,w2)
Why DL revolution did not happen in 1986?
Not enough data (datasets 1000 too small) Why DL revolution did not happen in 1986?
Computers were too slow (1000000 times) Why DL revolution did not happen in 1986? Not enough data (datasets 1000 too small)
1.2 million images 1000 categories
Errors 2010 2011 28% 26% http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/
Errors 2010 2011 2012 28% 26% 16% http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/ AlexNet (A. Krizhevsky et al. 2012)
Errors 2010 2011 2012 2013 2014 2015 2016 28% 26% 16% 12% 7% 3% <3% AlexNet (A. Krizhevsky et al. 2012) http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/
Errors 2010 2011 2012 2013 2014 2015 2016 28% 26% 16% 12% 7% 3% <3% Hypothetical super- dedicated fine- grained expert ensemble of human labelers AlexNet (A. Krizhevsky et al. 2012) http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/
Convolutional Neural Network Let’s consider the following image
Convolutional Neural Network Let’s consider the following image
Convolutional Neural Network Convolutional layer works as a filter applied to the original image
Convolutional Neural Network Convolutional layer works as a filter applied to the original image There are many filters in the convolutional layer, they detect different patterns 4 filters
Convolutional Neural Network 4 filters Each filter applied to all possible 2x2 patches of the original image produces one output value
Convolutional Neural Network 4 filters Each filter applied to all possible 2x2 patches of the original image produces one output value
Convolutional Neural Network 4 filters Each filter applied to all possible 2x2 patches of the original image produces one output value
Convolutional Neural Network 4 filters Each filter applied to all possible 2x2 patches of the original image produces one output value
Convolutional Neural Network Each filter applied to all possible 2x2 patches of the original image produces one output value Repeat this process for all filters in this layer
Convolutional Neural Network Each filter applied to all possible 2x2 patches of the original image produces one output value Repeat this process for all filters in this layer and the next
Flattening The output of the last convolutional layer is flattened into a single vector (like we did with images) Convolutional Neural Network
Flattening The output of the last convolutional layer is flattened into a single vector (like we did with images) Convolutional Neural Network 0 1 2 7 8 9 This vector is fed into fully connected layer with as many neutrons as possible classes
Flattening The output of the last convolutional layer is flattened into a single vector (like we did with images) Convolutional Neural Network 0 1 2 8 9 This vector is fed into fully connected layer with as many neutrons as possible classes Each neuron outputs probabilities 7
Training Neural Networks (part III) http://scs.ryerson.ca/~aharley/vis/conv/
http://www.asimovinstitute.org/wp-content/uploads/2016/09/neuralnetworks.png
• Pre-training (weights initialization) (complex landscape)
• Pre-training (weights initialization) • Efficient descent algorithms (complex landscape) (complex landscape)
• Pre-training (weights initialization) • Efficient descent algorithms • Activation (complex landscape) (complex landscape) (vanishing gradient)
• Pre-training (weights initialization) • Efficient descent algorithms • Dropout • Activation (complex landscape) (complex landscape) (vanishing gradient) (overfitting)
• Pre-training (weights initialization) • Efficient descent algorithms • Dropout • Domain Prior Knowledge • Activation (complex landscape) (complex landscape) (vanishing gradient) (overfitting)
Now that we are deep... • Powerful function approximation • Instead of hand-crafted features, let the algorithm build the relevant features for your problem • More representational power for learning
False positives
False negatives
Karpathy, Fei-Fei,“DeepVisual-Semantic Alignments for Generating Image Descriptions” (2014)
Style transferring Texture Networks by Dmitry Ulyanov et al.
Visual and Textual Question Answering
Visual and Textual Question Answering
Visual and Textual Question Answering http://cloudcv.org/vqa/
References • Machine Learning by Andrew Ng (https://www.coursera.org/learn/machine- learning) • Introduction to Machine Learning by Pascal Vincent given at Deep Learning Summer School, Montreal 2015 (http://videolectures.net/ deeplearning2015_vincent_machine_learning/) • Welcome to Machine Learning by Konstantin Tretyakov delivered at AACIMP Summer School 2015 (http://kt.era.ee/lectures/aacimp2015/1-intro.pdf) • Stanford CS class: Convolutional Neural Networks for Visual Recognition by Andrej Karpathy (http://cs231n.github.io/) • Data Mining Course by Jaak Vilo at University of Tartu (https://courses.cs.ut.ee/ MTAT.03.183/2017_spring/uploads/Main/DM_05_Clustering.pdf) • Machine Learning Essential Conepts by Ilya Kuzovkin (https:// www.slideshare.net/iljakuzovkin) • From the brain to deep learning and back by Raul Vicente Zafra and Ilya Kuzovkin (http://www.uttv.ee/naita?id=23585&keel=eng)
www.biit.cs.ut.ee www.ut.ee www.quretec.ee

More Related Content

PPTX
Artificial Neural Network in Medical Diagnosis
byAdityendra Kumar Singh
 
PDF
Lecture 2: Artificial Neural Network
byMohamed Loey
 
PDF
Deep learning
byKuppusamy P
 
PPTX
Backpropagation And Gradient Descent In Neural Networks | Neural Network Tuto...
bySimplilearn
 
PPTX
Introduction to Artificial Neural Networks
byAdri Jovin
 
PPTX
Neural Networks
byAdri Jovin
 
PPT
Multi-Layer Perceptrons
byESCOM
 
PDF
Lecture 4 neural networks
byParveenMalik18
 
Artificial Neural Network in Medical Diagnosis
byAdityendra Kumar Singh
 
Lecture 2: Artificial Neural Network
byMohamed Loey
 
Deep learning
byKuppusamy P
 
Backpropagation And Gradient Descent In Neural Networks | Neural Network Tuto...
bySimplilearn
 
Introduction to Artificial Neural Networks
byAdri Jovin
 
Neural Networks
byAdri Jovin
 
Multi-Layer Perceptrons
byESCOM
 
Lecture 4 neural networks
byParveenMalik18
 

What's hot

PPT
backpropagation in neural networks
byAkash Goel
 
PPTX
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
bySimplilearn
 
PPTX
What Is A Neural Network? | How Deep Neural Networks Work | Neural Network Tu...
bySimplilearn
 
PPTX
Introduction to deep learning
byAmr Rashed
 
PPTX
CNN Tutorial
bySungjoon Choi
 
PPTX
Convolutional Neural Networks
byAshray Bhandare
 
PDF
Supervised vs Unsupervised vs Reinforcement Learning | Edureka
byEdureka!
 
PPTX
Mathematics Foundation Course for Machine Learning & AI By Eduonix
byNick Trott
 
PDF
CNN
bygeorgejustymirobi1
 
PDF
Training Neural Networks
byDatabricks
 
PDF
Uncertainty Quantification in AI
byFlorian Wilhelm
 
PPTX
Decision Trees
byCloudxLab
 
PPTX
Artificial Neural Network | Deep Neural Network Explained | Artificial Neural...
bySimplilearn
 
PPTX
Naive Bayes
byAbdullah al Mamun
 
PPTX
Autoencoders in Deep Learning
bymilad abbasi
 
PPTX
The world of loss function
by홍배 김
 
PDF
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...
byEdureka!
 
PDF
Autoencoders
byCloudxLab
 
PDF
Data Visualization in Python
byJagriti Goswami
 
PPTX
Artifical Neural Network and its applications
bySangeeta Tiwari
 
backpropagation in neural networks
byAkash Goel
 
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
bySimplilearn
 
What Is A Neural Network? | How Deep Neural Networks Work | Neural Network Tu...
bySimplilearn
 
Introduction to deep learning
byAmr Rashed
 
CNN Tutorial
bySungjoon Choi
 
Convolutional Neural Networks
byAshray Bhandare
 
Supervised vs Unsupervised vs Reinforcement Learning | Edureka
byEdureka!
 
Mathematics Foundation Course for Machine Learning & AI By Eduonix
byNick Trott
 
CNN
bygeorgejustymirobi1
 
Training Neural Networks
byDatabricks
 
Uncertainty Quantification in AI
byFlorian Wilhelm
 
Decision Trees
byCloudxLab
 
Artificial Neural Network | Deep Neural Network Explained | Artificial Neural...
bySimplilearn
 
Naive Bayes
byAbdullah al Mamun
 
Autoencoders in Deep Learning
bymilad abbasi
 
The world of loss function
by홍배 김
 
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...
byEdureka!
 
Autoencoders
byCloudxLab
 
Data Visualization in Python
byJagriti Goswami
 
Artifical Neural Network and its applications
bySangeeta Tiwari
 

Similar to 5 Introduction to neural networks

PPTX
NNs5_6_MLhhbhbhhhbhjbhjhbjhhbhjP (1).pptx
bysyedtahaali4947
 
PPTX
Backpropagation and computational graph.pptx
bytintu47
 
PPTX
DL unit 2 for engineering students1.pptx
bymbadepartment5
 
PPTX
Artificial Neural Network
byDessy Amirudin
 
PPTX
Neural Networks
byMakerere Unversity School of Public Health, Victoria University
 
PDF
Levenberg marquardt-algorithm-for-karachi-stock-exchange-share-rates-forecast...
byCemal Ardil
 
PPTX
Machine Learning Essentials Demystified part2 | Big Data Demystified
byOmid Vahdaty
 
PPTX
CS532L4_Backpropagation.pptx
byMFaisalRiaz5
 
PDF
Machine Learning.pdf
bynikola_tesla1
 
PPT
Neural networks,Single Layer Feed Forward
byRohiniRajaramPandian
 
PPTX
Deep learning from scratch
byEran Shlomo
 
PDF
Introduction to Neural Networks.pptx.pdf
byjatt66016
 
PPT
Artificial Neural Network
byPratik Aggarwal
 
DOCX
Backpropagation
byariffast
 
PPT
Artificial Neural Networks_Bioinsspired_Algorithms_Nov 20.ppt
byAnonymous9etQKwW
 
PPTX
Artificial neural networks
byShwethaShreeS
 
PPTX
Nimrita deep learning
byNimrita Koul
 
PPTX
Artificial neural network
byIldar Nurgaliev
 
PDF
Capstone paper
byMuhammad Saeed
 
PPT
Annintro
bykaushaljha009
 
NNs5_6_MLhhbhbhhhbhjbhjhbjhhbhjP (1).pptx
bysyedtahaali4947
 
Backpropagation and computational graph.pptx
bytintu47
 
DL unit 2 for engineering students1.pptx
bymbadepartment5
 
Artificial Neural Network
byDessy Amirudin
 
Neural Networks
byMakerere Unversity School of Public Health, Victoria University
 
Levenberg marquardt-algorithm-for-karachi-stock-exchange-share-rates-forecast...
byCemal Ardil
 
Machine Learning Essentials Demystified part2 | Big Data Demystified
byOmid Vahdaty
 
CS532L4_Backpropagation.pptx
byMFaisalRiaz5
 
Machine Learning.pdf
bynikola_tesla1
 
Neural networks,Single Layer Feed Forward
byRohiniRajaramPandian
 
Deep learning from scratch
byEran Shlomo
 
Introduction to Neural Networks.pptx.pdf
byjatt66016
 
Artificial Neural Network
byPratik Aggarwal
 
Backpropagation
byariffast
 
Artificial Neural Networks_Bioinsspired_Algorithms_Nov 20.ppt
byAnonymous9etQKwW
 
Artificial neural networks
byShwethaShreeS
 
Nimrita deep learning
byNimrita Koul
 
Artificial neural network
byIldar Nurgaliev
 
Capstone paper
byMuhammad Saeed
 
Annintro
bykaushaljha009
 

More from Dmytro Fishman

PDF
DOME: Recommendations for supervised machine learning validation in biology
byDmytro Fishman
 
PDF
Tips for effective presentations
byDmytro Fishman
 
PDF
Autonomous Driving Lab - Simultaneous Localization and Mapping WP
byDmytro Fishman
 
PDF
Introduction to Deep Learning
byDmytro Fishman
 
PDF
Introduction to Machine Learning for Taxify/Bolt
byDmytro Fishman
 
PDF
Introduction to Gaussian Processes
byDmytro Fishman
 
PDF
Biit group 2018
byDmytro Fishman
 
PDF
Detecting Nuclei from Microscopy Images with Deep Learning
byDmytro Fishman
 
PDF
Deep Learning in Healthcare
byDmytro Fishman
 
PDF
4 Dimensionality reduction (PCA & t-SNE)
byDmytro Fishman
 
PDF
3 Unsupervised learning
byDmytro Fishman
 
PDF
1 Supervised learning
byDmytro Fishman
 
PDF
What does it mean to be a bioinformatician?
byDmytro Fishman
 
PDF
Machine Learning in Bioinformatics
byDmytro Fishman
 
DOME: Recommendations for supervised machine learning validation in biology
byDmytro Fishman
 
Tips for effective presentations
byDmytro Fishman
 
Autonomous Driving Lab - Simultaneous Localization and Mapping WP
byDmytro Fishman
 
Introduction to Deep Learning
byDmytro Fishman
 
Introduction to Machine Learning for Taxify/Bolt
byDmytro Fishman
 
Introduction to Gaussian Processes
byDmytro Fishman
 
Biit group 2018
byDmytro Fishman
 
Detecting Nuclei from Microscopy Images with Deep Learning
byDmytro Fishman
 
Deep Learning in Healthcare
byDmytro Fishman
 
4 Dimensionality reduction (PCA & t-SNE)
byDmytro Fishman
 
3 Unsupervised learning
byDmytro Fishman
 
1 Supervised learning
byDmytro Fishman
 
What does it mean to be a bioinformatician?
byDmytro Fishman
 
Machine Learning in Bioinformatics
byDmytro Fishman
 

Recently uploaded

PDF
Tetracycline Class of Antibiotics: SAR, MOA and Uses
bymominzarinamdnajeeb
 
PPTX
A brief introduction to Minor vegetable crops.pptx
byUmeshTimilsina1
 
PDF
Types of Foot & Foot Modifications in Birds
byDr. Ramzan Virani
 
PPTX
Lesson_1 Acceleration.pptx_Science 8-4th quarter
byAnnabelAlarcon
 
PPTX
How to Configure Dispatch Management System in Odoo 18 Inventory
byCeline George
 
PPTX
Fluorimetric Analysis- Theory, Instrumentation and Application
bySayali Powar
 
PPTX
ENGLISH-7-Quarter-4-Week-3 Matatag .pptx
byKimberlyJadeCuyos
 
PDF
Family and Marriage __ Sociology __ Jhasketan Kuanar __ NURSING VIBE ONLY .pdf
byJK NURSING VIBES ONLY JHASKETAN KUANAR
 
PDF
PPT Daftar Bahan Kelas Bedah Kitab Matius Khotbah di Bukit
bySABDA
 
PPTX
Modeling Simple Equation using Bar Model.pptx
byshahanieabbat3
 
PPTX
GRADE 8_WEEK 2_QUARTER 4_ENGLISH_MATATAG.pptx
byRoinelReyes1
 
PPTX
How to Track & Manage My Time Section in Odoo 18 Time Off
byCeline George
 
PPTX
Biological source, chemical constituents, and therapeutic efficacy of the fol...
bySai Meer College of Pharmacy
 
PPTX
Problem and Solution (PowerPoint Game Template)
byacpaulite
 
PPTX
How to Change Shipping Label Size in Odoo 18 Inventory
byCeline George
 
PDF
Artificial Intelligence in Research and Academic Writing, Workshop on Researc...
byProf. Vinod Kumar Kanvaria
 
PDF
Unit Plan and Unit Test-pdf-Dr. Rajashekhar Shirvalkar, Principal, SMRS B.Ed ...
byRajashekhar Shirvalkar
 
PPTX
EYE IRRIGATION AND INSTILLATION....pptx
byAneetaSharma15
 
PPTX
Chapter 1: Introduction to Economics - Macroeconomics and Microeconomics.pptx
byFJ M
 
PDF
Pharmaceutical Quality Assurance Unit 1 (BP606T)
byChetan Mali
 
Tetracycline Class of Antibiotics: SAR, MOA and Uses
bymominzarinamdnajeeb
 
A brief introduction to Minor vegetable crops.pptx
byUmeshTimilsina1
 
Types of Foot & Foot Modifications in Birds
byDr. Ramzan Virani
 
Lesson_1 Acceleration.pptx_Science 8-4th quarter
byAnnabelAlarcon
 
How to Configure Dispatch Management System in Odoo 18 Inventory
byCeline George
 
Fluorimetric Analysis- Theory, Instrumentation and Application
bySayali Powar
 
ENGLISH-7-Quarter-4-Week-3 Matatag .pptx
byKimberlyJadeCuyos
 
Family and Marriage __ Sociology __ Jhasketan Kuanar __ NURSING VIBE ONLY .pdf
byJK NURSING VIBES ONLY JHASKETAN KUANAR
 
PPT Daftar Bahan Kelas Bedah Kitab Matius Khotbah di Bukit
bySABDA
 
Modeling Simple Equation using Bar Model.pptx
byshahanieabbat3
 
GRADE 8_WEEK 2_QUARTER 4_ENGLISH_MATATAG.pptx
byRoinelReyes1
 
How to Track & Manage My Time Section in Odoo 18 Time Off
byCeline George
 
Biological source, chemical constituents, and therapeutic efficacy of the fol...
bySai Meer College of Pharmacy
 
Problem and Solution (PowerPoint Game Template)
byacpaulite
 
How to Change Shipping Label Size in Odoo 18 Inventory
byCeline George
 
Artificial Intelligence in Research and Academic Writing, Workshop on Researc...
byProf. Vinod Kumar Kanvaria
 
Unit Plan and Unit Test-pdf-Dr. Rajashekhar Shirvalkar, Principal, SMRS B.Ed ...
byRajashekhar Shirvalkar
 
EYE IRRIGATION AND INSTILLATION....pptx
byAneetaSharma15
 
Chapter 1: Introduction to Economics - Macroeconomics and Microeconomics.pptx
byFJ M
 
Pharmaceutical Quality Assurance Unit 1 (BP606T)
byChetan Mali
 

5 Introduction to neural networks

  • 1.
    Introduction to Machine Learning (Neuralnetworks) Dmytro Fishman (dmytro@ut.ee)
  • 2.
    The following materialwas adopted from:
  • 3.
    Evolution of MLmethods Rule-based Adopted from Y.Bengio http://videolectures.net/deeplearning2015_bengio_theoretical_motivations/ Input Fixed set of rules Output
  • 4.
    Evolution of MLmethods Rule-based Classic Machine Learning Adopted from Y.Bengio http://videolectures.net/deeplearning2015_bengio_theoretical_motivations/ Input Input Fixed set of rules Output Hand designed features Learning Output
  • 5.
    Evolution of MLmethods Rule-based Classic Machine Learning Representation Learning Adopted from Y.Bengio http://videolectures.net/deeplearning2015_bengio_theoretical_motivations/ Input Input Input Fixed set of rules Output Hand designed features Learning Automated feature extraction Output Learning Output
  • 6.
    Evolution of MLmethods Rule-based Classic Machine Learning Representation Learning Deep Learning Adopted from Y.Bengio http://videolectures.net/deeplearning2015_bengio_theoretical_motivations/ Input Input Input Input Fixed set of rules Output Hand designed features Learning Automated feature extraction Output Learning Output Low level features High level features Learning Output
  • 7.
    What is deeplearning? Many layers of adaptive non-linear processing to model complex relationships among data Space 1 Space 2
  • 8.
    What is deeplearning? Many layers of adaptive non-linear processing to model complex relationships among data Space 1 Space 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 155 255 255 255 155 0 255 255 255 255 255 255 255 255 155 78 78 155 255 255 255 0 0 0 0 155 255 0,1,2,3,…9
  • 9.
    What is deeplearning? Many layers of adaptive non-linear processing to model complex relationships among data Space 1 Space 2 species
  • 10.
    What is deeplearning? Many layers of adaptive non-linear processing to model complex relationships among data Space 1 Space 2 “We love you”
  • 11.
    In practice DL =Artificial Neural Networks with many layers
  • 12.
    A Logical Calculusof the Ideas Immanent in Nervous Activity McCulloch & Pitts (1943)
  • 13.
    Rosenblatt (1957) Perceptron NewYorkTimes: “(The perceptron)is the embryo of an electronic computer that is expected to be able to walk, talk, see, write, reproduce itself and be conscious of its existence” A Logical Calculus of the Ideas Immanent in Nervous Activity McCulloch & Pitts (1943)
  • 14.
    Minsky & Papert(1969) Perceptrons: an introduction to computational geometry Rosenblatt (1957) Perceptron A Logical Calculus of the Ideas Immanent in Nervous Activity McCulloch & Pitts (1943)
  • 15.
    Blum & Rivest(1992) Training a 3-node neural network is NP- complete Minsky & Papert (1969) Perceptrons: an introduction to computational geometry Rosenblatt (1957) Perceptron A Logical Calculus of the Ideas Immanent in Nervous Activity McCulloch & Pitts (1943)
  • 16.
    Rumelhart, Hinton &Williams (1986) Learning representations by back- propagating errors Blum & Rivest (1992) Training a 3-node neural network is NP- complete Minsky & Papert (1969) Perceptrons: an introduction to computational geometry Rosenblatt (1957) Perceptron A Logical Calculus of the Ideas Immanent in Nervous Activity McCulloch & Pitts (1943)
  • 17.
    Artificial neural network •A collection of simple trainable mathematical units, which collaborate to compute a complicated function • Compatible with supervised, unsupervised, and reinforcement • Brain inspired (loosely)
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
    X i 0 x2w2 x1w1 x0 Input layerFully connected layer Output layer Single neuron Feedforward Neural Network
  • 24.
    X i 0 x2w2 x1w1 x0 Input layerFully connected layer Output layer Single neuron Feedforward Neural Network
  • 25.
    X i 0 x2w2 x1w1 x0 Input layerFully connected layer Output layer Single neuron Feedforward Neural Network
  • 26.
    x2w2 x1w1 x0 Input layer Fullyconnected layer Output layer Single neuron Feedforward Neural Network X x x2w2 x1w1 x0w0 0
  • 27.
    x2w2 x1w1 x0 Input layer Fullyconnected layer Output layer Single neuron Feedforward Neural Network X x x2w2 x1w1 x0w0 0
  • 28.
    x2w2 x1w1 x0 Input layer Fullyconnected layer Output layer Single neuron Feedforward Neural Network X i 0
  • 29.
    Learning algorithm • modifyconnection weights to make prediction closer to y • run neuronal network on input x • while not done • pick a random training instance (x, y)
  • 30.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error neth1 = w1 ⇤ i1 + w2 ⇤ i2 + b1 ⇤ 1 INPUT TARGET
  • 31.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error neth1 = w1 ⇤ i1 + w2 ⇤ i2 + b1 ⇤ 1 neth1 = 0.15 ⇤ 0.05 + 0.2 ⇤ 0.1 + 0.35 ⇤ 1 = 0.3775 INPUT TARGET
  • 32.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error neth1 = w1 ⇤ i1 + w2 ⇤ i2 + b1 ⇤ 1 neth1 = 0.15 ⇤ 0.05 + 0.2 ⇤ 0.1 + 0.35 ⇤ 1 = 0.3775 INPUT TARGET f(x) = 1 1 + e x outh1 = 1 1 + e neth1 = 1 1 + e 0.3775 = 0.5933
  • 33.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error neth1 = w1 ⇤ i1 + w2 ⇤ i2 + b1 ⇤ 1 neth1 = 0.15 ⇤ 0.05 + 0.2 ⇤ 0.1 + 0.35 ⇤ 1 = 0.3775 INPUT TARGET f(x) = 1 1 + e x outh1 = 1 1 + e neth1 = 1 1 + e 0.3775 = 0.5933 Repeat for h2 = 0.596; o1 = 0.751; o2 = 0.773
  • 34.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error INPUT TARGET We have o1, o2
  • 35.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error INPUT TARGET We have o1, o2 Etotal = X 1 2 (target output) 2
  • 36.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error INPUT TARGET We have o1, o2 Etotal = X 1 2 (target output) 2 Eo1 = 1 2 (targeto1 outo1) 2 = 1 2 (0.01 0.7514) 2 = 0.2748
  • 37.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error INPUT TARGET We have o1, o2 Etotal = X 1 2 (target output) 2 Eo1 = 1 2 (targeto1 outo1) 2 = 1 2 (0.01 0.7514) 2 = 0.2748 Eo2 = 0.02356
  • 38.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 1.The Forward pass - Compute total error INPUT TARGET We have o1, o2 Etotal = X 1 2 (target output) 2 Eo1 = 1 2 (targeto1 outo1) 2 = 1 2 (0.01 0.7514) 2 = 0.2748 Eo2 = 0.02356 Etotal = Eo1 + Eo2 = 0.2748 + 0.02356 = 0.29836
  • 39.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET We want to know how much a change in w5 affects the total error
  • 40.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error
  • 41.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error
  • 42.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET Etotal = X 1 2 (target output) 2 @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error
  • 43.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET Etotal = X 1 2 (target output) 2 @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error @Etotal @outo1 = 2 ⇤ 1 2 (targeto1 outo1) ⇤ 1 + 0 = (0.01 0.751) = 0.741
  • 44.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error
  • 45.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error outo1 = 1 1 + e neto1
  • 46.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error @outo1 @neto1 = outo1 (1 outo1) = 0.1868 outo1 = 1 1 + e neto1
  • 47.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error
  • 48.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error neto1 = w5 ⇤ outh1 + w6 ⇤ outh2 + b2 ⇤ 1
  • 49.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error neto1 = w5 ⇤ outh1 + w6 ⇤ outh2 + b2 ⇤ 1 @neto1 @w5 = outh1 = 0.5933
  • 50.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error
  • 51.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error @Etotal @w5 = 0.7414 ⇤ 0.1868 ⇤ 0.5933 = 0.0821
  • 52.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 2.The Backward pass - Updating weights INPUT TARGET @Etotal @w5 = @Etotal @outo1 ⇤ @outo1 @neto1 ⇤ @neto1 @w5 We want to know how much a change in w5 affects the total error @Etotal @w5 = 0.7414 ⇤ 0.1868 ⇤ 0.5933 = 0.0821 wnew 5 = wold 5 ⌘ ⇤ @Etotal @w5 = 0.4 0.5 ⇤ 0.0821 = 0.3589
  • 53.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 • Repeat for w6, w7, w8 INPUT TARGET
  • 54.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 • Repeat for w6, w7, w8 INPUT TARGET • In analogous way for w1, w2, w3, w4
  • 55.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 • Repeat for w6, w7, w8 INPUT TARGET • In analogous way for w1, w2, w3, w4 • Compute the total error before: 0.298371109
  • 56.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 • Repeat for w6, w7, w8 INPUT TARGET • In analogous way for w1, w2, w3, w4 • Compute the total error before: now: 0.298371109 0.291027924
  • 57.
    Lets backpropagate i1 =0.05 i2 = 0.10 o1 = 0.01 o2 = 0.99 • Repeat for w6, w7, w8 INPUT TARGET • In analogous way for w1, w2, w3, w4 • Compute the total error before: now: 0.298371109 0.291027924 • Repeat x10000: 0.000035085
  • 58.
  • 59.
  • 60.
    Deep networks were difficultto train Overfitting DimensionalityVanishing gradients Complex landscape w2 w1 E(w1,w2)
  • 61.
    Why DL revolutiondid not happen in 1986?
  • 62.
    Not enough data (datasets1000 too small) Why DL revolution did not happen in 1986?
  • 63.
    Computers were tooslow (1000000 times) Why DL revolution did not happen in 1986? Not enough data (datasets 1000 too small)
  • 64.
  • 65.
  • 66.
  • 67.
    Errors 2010 2011 2012 2013 2014 2015 2016 28% 26% 16% 12% 7% 3% <3% AlexNet (A.Krizhevsky et al. 2012) http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/
  • 68.
    Errors 2010 2011 2012 2013 2014 2015 2016 28% 26% 16% 12% 7% 3% <3% Hypothetical super- dedicatedfine- grained expert ensemble of human labelers AlexNet (A. Krizhevsky et al. 2012) http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/
  • 69.
    Convolutional Neural Network Let’sconsider the following image
  • 70.
    Convolutional Neural Network Let’sconsider the following image
  • 71.
    Convolutional Neural Network Convolutionallayer works as a filter applied to the original image
  • 72.
    Convolutional Neural Network Convolutionallayer works as a filter applied to the original image There are many filters in the convolutional layer, they detect different patterns 4 filters
  • 73.
    Convolutional Neural Network 4filters Each filter applied to all possible 2x2 patches of the original image produces one output value
  • 74.
    Convolutional Neural Network 4filters Each filter applied to all possible 2x2 patches of the original image produces one output value
  • 75.
    Convolutional Neural Network 4filters Each filter applied to all possible 2x2 patches of the original image produces one output value
  • 76.
    Convolutional Neural Network 4filters Each filter applied to all possible 2x2 patches of the original image produces one output value
  • 77.
    Convolutional Neural Network Eachfilter applied to all possible 2x2 patches of the original image produces one output value Repeat this process for all filters in this layer
  • 78.
    Convolutional Neural Network Eachfilter applied to all possible 2x2 patches of the original image produces one output value Repeat this process for all filters in this layer and the next
  • 79.
    Flattening The output ofthe last convolutional layer is flattened into a single vector (like we did with images) Convolutional Neural Network
  • 80.
    Flattening The output ofthe last convolutional layer is flattened into a single vector (like we did with images) Convolutional Neural Network 0 1 2 7 8 9 This vector is fed into fully connected layer with as many neutrons as possible classes
  • 81.
    Flattening The output ofthe last convolutional layer is flattened into a single vector (like we did with images) Convolutional Neural Network 0 1 2 8 9 This vector is fed into fully connected layer with as many neutrons as possible classes Each neuron outputs probabilities 7
  • 82.
    Training Neural Networks (partIII) http://scs.ryerson.ca/~aharley/vis/conv/
  • 83.
  • 85.
    • Pre-training (weightsinitialization) (complex landscape)
  • 86.
    • Pre-training (weightsinitialization) • Efficient descent algorithms (complex landscape) (complex landscape)
  • 87.
    • Pre-training (weightsinitialization) • Efficient descent algorithms • Activation (complex landscape) (complex landscape) (vanishing gradient)
  • 88.
    • Pre-training (weightsinitialization) • Efficient descent algorithms • Dropout • Activation (complex landscape) (complex landscape) (vanishing gradient) (overfitting)
  • 89.
    • Pre-training (weightsinitialization) • Efficient descent algorithms • Dropout • Domain Prior Knowledge • Activation (complex landscape) (complex landscape) (vanishing gradient) (overfitting)
  • 90.
    Now that weare deep... • Powerful function approximation • Instead of hand-crafted features, let the algorithm build the relevant features for your problem • More representational power for learning
  • 96.
  • 97.
  • 98.
    Karpathy, Fei-Fei,“DeepVisual-Semantic Alignmentsfor Generating Image Descriptions” (2014)
  • 99.
    Style transferring Texture Networksby Dmitry Ulyanov et al.
  • 100.
    Visual and TextualQuestion Answering
  • 101.
    Visual and TextualQuestion Answering
  • 102.
    Visual and TextualQuestion Answering http://cloudcv.org/vqa/
  • 103.
    References • Machine Learningby Andrew Ng (https://www.coursera.org/learn/machine- learning) • Introduction to Machine Learning by Pascal Vincent given at Deep Learning Summer School, Montreal 2015 (http://videolectures.net/ deeplearning2015_vincent_machine_learning/) • Welcome to Machine Learning by Konstantin Tretyakov delivered at AACIMP Summer School 2015 (http://kt.era.ee/lectures/aacimp2015/1-intro.pdf) • Stanford CS class: Convolutional Neural Networks for Visual Recognition by Andrej Karpathy (http://cs231n.github.io/) • Data Mining Course by Jaak Vilo at University of Tartu (https://courses.cs.ut.ee/ MTAT.03.183/2017_spring/uploads/Main/DM_05_Clustering.pdf) • Machine Learning Essential Conepts by Ilya Kuzovkin (https:// www.slideshare.net/iljakuzovkin) • From the brain to deep learning and back by Raul Vicente Zafra and Ilya Kuzovkin (http://www.uttv.ee/naita?id=23585&keel=eng)
  • 104.