The document discusses different methods for initializing weights in neural networks. It first mentions that all zero initialization causes every neuron to compute the same output, lacking asymmetry. Xavier initialization scales the variance of the weights so that the variance of the input and output are the same. Initialization for ReLU networks is discussed in a paper by Kaiming He et al. that achieved human-level performance on ImageNet classification.

1. Weights Initialization
Kong
2017.1.6

2. Weights and bias
All Zero InitializationEvery neuron in the network computes the sameoutputThere is no source of asymmetryInitialization with Small Random NumbersSmall random numbers which are very close tozero, and it is treated as symmetry breaking

3. Weights and bias

5. Weights and bias

6. Xavier initialization
The variance of the output is the variance of
the input, but scaled by nVar(Wi).
So if we want the variance of the input and
output to be the same, that means nVar(Wi)
should be 1.
Which means the variance of the weights
should be

7. Xavier initialization

8. Initialization for ReLU
Kaiming He, et al. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

9. Initialization for ReLU
Kaiming He, et al. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

10. Thank you for your attention!