Explanation for Stanford Online Machine Learning class, chapter 8 neural network

1. Stanford ML Lecture #8
Neural Network
Representation
S. Takei @shtaag
2012 1 26

2. Reference
Stanford Online ML class #8
”Machine Learning: An Algorithmic
Perspective” (Marsland, 2009)
2012 1 26

3. Why NN is necessary?
in non-linear classification problem...
polynomial regression is not efficient!!
ex. 50 x 50 pixels picture
n features = 2500
quadratic features = 50C2 = 3000000
O(Tn2) : where T = iterations
2012 1 26

4. Neuron Model
logistic activation function
O(T m n) : where m = input n = output T = iterations
more efficient than polynomial regression!!
2012 1 26

5. Simple Type : PERCEPTRON
1 layer of Input nodes
1 layer of Output nodes
weighted connections
only forward propagation
learning :
update weights based on error function
error function = difference between output values
2012 1 26

6. Represents logical operation
AND
linear boundary
In2
Input1 Input2 out
0 0 0
0 1 0
1 0 0
1 1 1
In1
2012 1 26

7. Represents logical operation
OR
In2
Input1 Input2 out
0 0 0
0 1 1
1 0 1
1 1 1
In1
linear boundary
2012 1 26

8. Represents logical operation
XOR
Can’t separate linearly!!
In2
Input1 Input2 out
?
0 0 0
0 1 1
1 0 1
1 1 0
In1
2012 1 26

9. Perceptron (2D) can’t represent XOR
( Minsky & Papert, “Perceptrons”, 1969)
1, add dimension -> Support Vector Machine
2, add layers -> Neural Network
2012 1 26

10. Neural Network
Input layers + output layers + hidden layers
learning :
backward propagation
2012 1 26

11. Neural Network can represent XOR
XOR = (In1 AND In2) OR (In1 NAND In2)
2012 1 26