1. Neural networks are a type of machine learning model that can learn highly non-linear functions to map inputs to outputs. They consist of interconnected layers of nodes that mimic biological neurons. 2. Backpropagation is an algorithm that allows neural networks to be trained using gradient descent by efficiently computing the gradient of the loss function with respect to the network parameters. It works by propagating gradients from the output layer back through the network using the chain rule. 3. There are many design decisions that go into building a neural network architecture, such as the number of hidden layers and nodes, choice of activation functions, objective function, and training algorithm like stochastic gradient descent. Common activation functions are the sigmoid, tanh, and rectified linear

1. Neural Networks
1
Matt Gormley
Lecture 15
October 19, 2016
School of Computer Science
Readings:
Bishop Ch. 5
Murphy Ch. 16.5, Ch. 28
Mitchell Ch. 4
10-601B Introduction to Machine Learning

2. Reminders
2

3. Outline
• Logistic Regression (Recap)
• Neural Networks
• Backpropagation
3

4. RECALL: LOGISTIC REGRESSION
4

5. Using gradient ascent for linear
classifiers
Key idea behind today’s lecture:
1. Define a linear classifier (logistic regression)
2. Define an objective function (likelihood)
3. Optimize it with gradient descent to learn
parameters
4. Predict the class with highest probability under
the model
5

6. Using gradient ascent for linear
classifiers
6
Use a differentiable
function instead:
logistic(u) º
1
1+e-u
This decision function isn’t
differentiable:
sign(x)

7. Using gradient ascent for linear
classifiers
7
Use a differentiable
function instead:
logistic(u) º
1
1+e-u
This decision function isn’t
differentiable:
sign(x)

8. Logistic Regression
8
Learning: finds the parameters that minimize some
objective function.
Data: Inputs are continuous vectors of length K. Outputs
are discrete.
Prediction: Output is the most probable class.
Model: Logistic function applied to dot product of
parameters with input vector.

9. NEURAL NETWORKS
9

10. Learning highly non-linear functions
f: X  Y
 f might be non-linear function
 X (vector of) continuous and/or discrete vars
 Y (vector of) continuous and/or discrete vars
© Eric Xing @ CMU, 2006-2011 10
The XOR gate Speech recognition

11. Perceptron and Neural Nets
 From biological neuron to artificial neuron (perceptron)
 Activation function
 Artificial neuron networks
 supervised learning
 gradient descent
© Eric Xing @ CMU, 2006-2011 11
Soma Soma
Synapse
Synapse
Dendrites
Axon
Synapse
Dendrites
Axon
Threshold
Inputs
x1
x2
Output
Y

Hard
Limiter
w2
w1
Linear
Combiner




n
i
i
iw
x
X
1 











X
X
Y
if
,
if
,
1
1
Input Layer Output Layer
Middle Layer
I
n
p
u
t
S
i
g
n
a
l
s
O
u
t
p
u
t
S
i
g
n
a
l
s

12. Connectionist Models
 Consider humans:
 Neuron switching time
~ 0.001 second
 Number of neurons
~ 1010
 Connections per neuron
~ 104-5
 Scene recognition time
~ 0.1 second
 100 inference steps doesn't seem like enough
 much parallel computation
 Properties of artificial neural nets (ANN)
 Many neuron-like threshold switching units
 Many weighted interconnections among units
 Highly parallel, distributed processes
© Eric Xing @ CMU, 2006-2011 12

13. Why is everyone talking
about Deep Learning?
• Because a lot of money is invested in it…
– DeepMind: Acquired by Google for $400
million
– DNNResearch: Three person startup
(including Geoff Hinton) acquired by Google
for unknown price tag
– Enlitic, Ersatz, MetaMind, Nervana, Skylab:
Deep Learning startups commanding millions
of VC dollars
• Because it made the front page of the
New York Times
13
Motivation

14. Why is everyone talking
about Deep Learning?
Deep learning:
– Has won numerous pattern recognition
competitions
– Does so with minimal feature
engineering
14
Motivation
1960s
1980s
1990s
2006
2016
This wasn’t always the case!
Since 1980s: Form of models hasn’t changed much,
but lots of new tricks…
– More hidden units
– Better (online) optimization
– New nonlinear functions (ReLUs)
– Faster computers (CPUs and GPUs)

15. A Recipe for
Machine Learning
1. Given training data:
15
Background
2. Choose each of these:
– Decision function
– Loss function
Face Face Not a face
Examples: Linear regression,
Logistic regression, Neural Network
Examples: Mean-squared error,
Cross Entropy

16. A Recipe for
Machine Learning
1. Given training data: 3. Define goal:
16
Background
2. Choose each of these:
– Decision function
– Loss function
4. Train with SGD:
(take small steps
opposite the gradient)

17. A Recipe for
Machine Learning
1. Given training data: 3. Define goal:
17
Background
2. Choose each of these:
– Decision function
– Loss function
4. Train with SGD:
(take small steps
opposite the gradient)
Gradients
Backpropagation can compute this
gradient!
And it’s a special case of a more
general algorithm called reverse-
mode automatic differentiation that
can compute the gradient of any
differentiable function efficiently!

18. A Recipe for
Machine Learning
1. Given training data: 3. Define goal:
18
Background
2. Choose each of these:
– Decision function
– Loss function
4. Train with SGD:
(take small steps
opposite the gradient)
Goals for Today’s Lecture
1. Explore a new class of decision functions
(Neural Networks)
2. Consider variants of this recipe for training

19. Linear Regression
19
Decision
Functions
…
Output
Input
θ1
θ2 θ3 θM

20. Logistic Regression
20
Decision
Functions
…
Output
Input
θ1
θ2 θ3 θM

21. Logistic Regression
21
Decision
Functions
…
Output
Input
θ1
θ2 θ3 θM
Face Face Not a face

22. Logistic Regression
22
Decision
Functions
…
Output
Input
θ1
θ2 θ3 θM
1 1 0
x1
x2
y

23. Logistic Regression
23
Decision
Functions
…
Output
Input
θ1
θ2 θ3 θM

24. Inputs
Weights
Output
Independent
variables
Dependent
variable
Prediction
Age 34
2
Gender
Stage 4
.6
.5
.8
.2
.1
.3
.7
.2
Weights
HiddenL
ayer
“Probability of
beingAlive”
0.6
S
S
.4
.2
S
Neural Network Model
© Eric Xing @ CMU, 2006-2011 24

25. Inputs
Weights
Output
Independent
variables
Dependent
variable
Prediction
Age 34
2
Gender
Stage 4
.6
.5
.8
.1
.7
Weights
HiddenL
ayer
“Probability of
beingAlive”
0.6
S
“Combined logistic models”
© Eric Xing @ CMU, 2006-2011 25

26. Inputs
Weights
Output
Independent
variables
Dependent
variable
Prediction
Age 34
2
Gender
Stage 4
.5
.8
.2
.3
.2
Weights
HiddenL
ayer
“Probability of
beingAlive”
0.6
S
© Eric Xing @ CMU, 2006-2011 26

27. © Eric Xing @ CMU, 2006-2011 27
Inputs
Weights
Output
Independent
variables
Dependent
variable
Prediction
Age 34
1
Gender
Stage 4
.6
.5
.8
.2
.1
.3
.7
.2
Weights
HiddenL
ayer
“Probability of
beingAlive”
0.6
S

28. Weights
Independent
variables
Dependent
variable
Prediction
Age 34
2
Gender
Stage 4
.6
.5
.8
.2
.1
.3
.7
.2
Weights
HiddenL
ayer
“Probability of
beingAlive”
0.6
S
S
.4
.2
S
Not really,
no target for hidden units...
© Eric Xing @ CMU, 2006-2011 28

29. Jargon Pseudo-Correspondence
 Independent variable = input variable
 Dependent variable = output variable
 Coefficients = “weights”
 Estimates = “targets”
Logistic Regression Model (the sigmoid unit)
© Eric Xing @ CMU, 2006-2011 29
Inputs Output
Age 34
1
Gende
r
Stage 4
“Probability of
beingAlive”
5
8
4
0.6
S
Coefficients
a, b, c
Independent variables
x1, x2, x3
Dependent variable
p Prediction

30. Neural Network
30
Decision
Functions
…
…
Output
Input
Hidden Layer

31. Neural Network
31
Decision
Functions
…
…
Output
Input
Hidden Layer

32. Building a Neural Net
…
Output
Features
32

33. Building a Neural Net
…
…
Output
Input
Hidden Layer
D = M
33
1 1 1

34. Building a Neural Net
…
…
Output
Input
Hidden Layer
D = M
34

35. Building a Neural Net
…
…
Output
Input
Hidden Layer
D = M
35

36. Building a Neural Net
…
…
Output
Input
Hidden Layer
D < M
36

37. Decision Boundary
• 0 hidden layers: linear classifier
– Hyperplanes
y
1
x 2
x
Example from to Eric Postma via Jason Eisner 37

38. Decision Boundary
• 1 hidden layer
– Boundary of convex region (open or closed)
y
1
x 2
x
Example from to Eric Postma via Jason Eisner 38

39. Decision Boundary
• 2 hidden layers
– Combinations of convex regions
Example from to Eric Postma via Jason Eisner
y
1
x 2
x
39

40. Multi-Class Output
40
Decision
Functions
…
…
Output
Input
Hidden Layer
…

41. Deeper Networks
Next lecture:
41
Decision
Functions
…
…
Output
Input
Hidden Layer 1

42. Deeper Networks
Next lecture:
42
Decision
Functions
…
…
Input
Hidden Layer 1
…
Output
Hidden Layer 2

43. Deeper Networks
Next lecture:
Making the
neural
networks
deeper
43
Decision
Functions
…
…
Input
Hidden Layer 1
…
Hidden Layer 2
…
Output
Hidden Layer 3

44. Different Levels of
Abstraction
• We don’t know
the “right”
levels of
abstraction
• So let the model
figure it out!
44
Decision
Functions
Example from Honglak Lee (NIPS 2010)

45. Different Levels of
Abstraction
Face Recognition:
– Deep Network
can build up
increasingly
higher levels of
abstraction
– Lines, parts,
regions
45
Decision
Functions
Example from Honglak Lee (NIPS 2010)

46. Different Levels of
Abstraction
46
Decision
Functions
Example from Honglak Lee (NIPS 2010)
…
…
Input
Hidden Layer 1
…
Hidden Layer 2
…
Output
Hidden Layer 3

47. ARCHITECTURES
47

48. Neural Network Architectures
Even for a basic Neural Network, there are
many design decisions to make:
1. # of hidden layers (depth)
2. # of units per hidden layer (width)
3. Type of activation function (nonlinearity)
4. Form of objective function
48

49. Activation Functions
49
…
…
Output
Input
Hidden Layer
Neural Network with sigmoid
activation functions

50. Activation Functions
50
…
…
Output
Input
Hidden Layer
Neural Network with arbitrary
nonlinear activation functions

51. Activation Functions
So far, we’ve
assumed that the
activation function
(nonlinearity) is
always the sigmoid
function…
51
Sigmoid / Logistic Function
logistic(u) º
1
1+e-u

52. Activation Functions
• A new change: modifying the nonlinearity
– The logistic is not widely used in modern ANNs
Alternate 1:
tanh
Like logistic function but
shifted to range [-1, +1]
Slide from William Cohen

53. AI Stats 2010
sigmoid
vs.
tanh
depth 4?
Figure from Glorot & Bentio (2010)

54. Activation Functions
• A new change: modifying the nonlinearity
– reLU often used in vision tasks
Alternate 2: rectified linear unit
Linear with a cutoff at zero
(Implementation: clip the gradient
when you pass zero)
Slide from William Cohen

55. Activation Functions
• A new change: modifying the nonlinearity
– reLU often used in vision tasks
Alternate 2: rectified linear unit
Soft version: log(exp(x)+1)
Doesn’t saturate (at one end)
Sparsifies outputs
Helps with vanishing gradient
Slide from William Cohen

56. Objective Functions for NNs
• Regression:
– Use the same objective as Linear Regression
– Quadratic loss (i.e. mean squared error)
• Classification:
– Use the same objective as Logistic Regression
– Cross-entropy (i.e. negative log likelihood)
– This requires probabilities, so we add an additional
“softmax” layer at the end of our network
56

57. Multi-Class Output
57
…
…
Output
Input
Hidden Layer
…

58. Multi-Class Output
58
…
…
Output
Input
Hidden Layer
…
Softmax:

59. Cross-entropy vs. Quadratic loss
Figure from Glorot & Bentio (2010)

60. A Recipe for
Machine Learning
1. Given training data: 3. Define goal:
60
Background
2. Choose each of these:
– Decision function
– Loss function
4. Train with SGD:
(take small steps
opposite the gradient)

61. Objective Functions
Matching Quiz: Suppose you are given a neural net with a
single output, y, and one hidden layer.
61
1) Minimizing sum of squared
errors…
2) Minimizing sum of squared
errors plus squared Euclidean
norm of weights…
3) Minimizing cross-entropy…
4) Minimizing hinge loss…
5) …MLE estimates of weights assuming
target follows a Bernoulli with
parameter given by the output value
6) …MAP estimates of weights
assuming weight priors are zero mean
Gaussian
7) …estimates with a large margin on
the training data
8) …MLE estimates of weights assuming
zero mean Gaussian noise on the output
value
…gives…
A. 1=5, 2=7, 3=6, 4=8
B. 1=5, 2=7, 3=8, 4=6
C. 1=7, 2=5, 3=5, 4=7
E. 1=8, 2=6, 3=5, 4=7
F. 1=8, 2=6, 3=8, 4=6

62. BACKPROPAGATION
62

63. A Recipe for
Machine Learning
1. Given training data: 3. Define goal:
63
Background
2. Choose each of these:
– Decision function
– Loss function
4. Train with SGD:
(take small steps
opposite the gradient)

64. Backpropagation
• Question 1:
When can we compute the gradients of the
parameters of an arbitrary neural network?
• Question 2:
When can we make the gradient
computation efficient?
64
Training

65. Chain Rule
65
Training
Chain Rule:
Given:
…

66. Chain Rule
66
Training
Chain Rule:
Given:
…
Backpropagation
is just repeated
application of the
chain rule from
Calculus 101.

67. Chain Rule
67
Training
Chain Rule:
Given:
…
Backpropagation:
1. Instantiate the computation as a directed acyclic graph, where each
intermediate quantity is a node
2. At each node, store (a) the quantity computed in the forward pass
and (b) the partial derivative of the goal with respect to that node’s
intermediate quantity.
3. Initialize all partial derivatives to 0.
4. Visit each node in reverse topological order. At each node, add its
contribution to the partial derivatives of its parents
This algorithm is also called automatic differentiation in the reverse-

68. Backpropagation
68
Training

69. Backpropagation
69
Training

70. Backpropagation
70
Training
…
Output
Input
θ1
θ2 θ3 θM
Case 1:
Logistic
Regression

71. Backpropagation
71
Training
…
…
Output
Input
Hidden Layer

72. Backpropagation
72
Training
…
…
Output
Input
Hidden Layer

73. Backpropagation
73
Training
Case 2:
Neural
Network
…
…

74. Chain Rule
74
Training
Chain Rule:
Given:
…
Backpropagation:
1. Instantiate the computation as a directed acyclic graph, where each
intermediate quantity is a node
2. At each node, store (a) the quantity computed in the forward pass
and (b) the partial derivative of the goal with respect to that node’s
intermediate quantity.
3. Initialize all partial derivatives to 0.
4. Visit each node in reverse topological order. At each node, add its
contribution to the partial derivatives of its parents
This algorithm is also called automatic differentiation in the reverse-

75. Chain Rule
75
Training
Chain Rule:
Given:
…
Backpropagation:
1. Instantiate the computation as a directed acyclic graph, where each
node represents a Tensor.
2. At each node, store (a) the quantity computed in the forward pass
and (b) the partial derivatives of the goal with respect to that node’s
Tensor.
3. Initialize all partial derivatives to 0.
4. Visit each node in reverse topological order. At each node, add its
contribution to the partial derivatives of its parents
This algorithm is also called automatic differentiation in the reverse-

76. Case 2:
Neural
Network
…
…
Module 1
Module 2
Module 3
Module 4
Module 5
Backpropagation
76
Training

77. A Recipe for
Machine Learning
1. Given training data: 3. Define goal:
77
Background
2. Choose each of these:
– Decision function
– Loss function
4. Train with SGD:
(take small steps
opposite the gradient)
Gradients
Backpropagation can compute this
gradient!
And it’s a special case of a more
general algorithm called reverse-
mode automatic differentiation that
can compute the gradient of any
differentiable function efficiently!

78. Summary
1. Neural Networks…
– provide a way of learning features
– are highly nonlinear prediction functions
– (can be) a highly parallel network of logistic
regression classifiers
– discover useful hidden representations of the
input
2. Backpropagation…
– provides an efficient way to compute gradients
– is a special case of reverse-mode automatic
differentiation
78