This document summarizes the paper "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift". It introduces batch normalization, which normalizes layer inputs to speed up training of neural networks. Batch normalization reduces internal covariate shift by normalizing layer inputs. It computes normalization statistics over each mini-batch and applies them to the inputs. This allows higher learning rates and acts as a regularizer. Experiments show batch normalization stabilizes and accelerates the training of neural networks on ImageNet classification.

1. Batch Normalization:
Accelerating Deep Network Training
by Reducing Internal Covariate Shift
#17
2019/02/06
@iiou16_tech

2. abstract
Deep Neural Networks
Batch Normalization
dropOut
Batch Normalization 14 1
ImageNet
4.9 5 4.8

3. outline
1. Introduction
2. Towards Reducing Internal Covariate Shift
3. Normalization via Mini-Batch Statistics
1. Training and Inference with Batch-Normalized
Networks
2. Batch-Normalized ConvolutionalNetworks
3. Batch Normalization enables higher learning rates
4. Batch Normalization regularizes the model
4. Experiments
1. Activations over time
2. ImageNet classification
5. Conclusion

4. Introduction
• Deep Learning SGD
– x θ l
θ
–
• ( ) ( m
)
• 1
•

5. Introduction
• covariate shift
–
DNNx
DNNx’
x DNN x’

6. Introduction
• covariate shift
x
F1 F2
DNN
F2 F1
→F1 F2 x

7. 2 Towards Reducing Internal Covariate
Shift
•
–
(DNN)
– DNN
– 0 1
( )
– 1

8. 2 Towards Reducing Internal Covariate
Shift
•
•
• itr
• (SGD)
•

9. 3 Normalization via Mini-Batch Statistics
•
• 1
• 0 1
•
• γ β x
• 2

10. 3 Normalization via Mini-Batch Statistics
•
• 2
• 0 1 SGD
DNN
•
/
• itr

11. 3.1 Training and Inference with
BatchNormalized Networks
•
• /
•
• /
•
activation
/
/

12. 3.2 Batch-Normalized Convolutional
Networks
• Convolutionarl
• ( )
•
BN
• m
• Conv BN p*q
m*p*q
• Conv BN 2*

13. 3.3 Batch Normalization enables higher
learning rates
•
• BN
a
1/a

14. 3.4 Batch Normalization regularizes the
model
•
• BN
• DropOut

15. 4 Experiments
4.1 Activations over time
•
BN
• MNIST
• 3 NN ( )
• 60 50000
BN BN

16. 4.2 ImageNet classification
• Inception ImageNet
• Relu
• CNN layer 5*5( )→3*3 ×2
• batch size = 32
• Optimiser : Momentum SGD
https://arxiv.org/pdf/1409.4842.pdf

17. 4.2.1 Accelerating BN Networks
• BASE Inception BN
• BN
•
• DropOut
• L2 Weight regularization 1/5
• 6
•
• 1%
• photometric distortion
•
• Local Response Normalization
https://arxiv.org/pdf/1409.4842.pdf

18. 4.2.2 Single-Network Classification
BN-x5LSVRC2012
lr=0.0015
BN
4.2.1
lr=0.0075
4.2.1
lr=0.045
BN-x5
Leru→sigmoid

19. 4.2.3 Ensemble Classification
• ImageNet Best Result
• BN-x30 6
SoTA
• DropOut (5% or 10%)

20. Conclusion(1/2)
•
• NN
• activation
DNN
• SGD
BN 2
• BN
• BN
• dropOut
• BN ImageNet

21. Conclution(2/2)
• Standardization layer
• BN
• future work
• Recurrent Neural Networks BN
• / BN
• domain adaptation
•
Batch Normalization BN→ , 2
Standardization layer SL no paramater
activation
activation

22. 1
• BN google
• https://patents.google.com/patent/US20160217368A1/en
A neural network system implemented by one or more computers, the neural network system comprising:
a batch normalization layer between a first neural network layer and a second neural network layer, wherein the
first neural network layer generates first layer outputs having a plurality of components, and wherein the batch
normalization layer is configured to, during training of the neural network system on a batch of training examples:
receive a respective first layer output for each training example in the batch;
compute a plurality of normalization statistics for the batch from the first layer outputs;
normalize each component of each first layer output using the normalization statistics to generate a respective
normalized layer output for each training example in the batch;
generate a respective batch normalization layer output for each of the training examples from the normalized layer
outputs; and
provide the batch normalization layer output as an input to the second neural network layer.
• ※
• https://www.slideshare.net/YosukeShinya/ss-125937523
• by 50 @2018/12/15

23. 2
• BN
• Group Normalization
• fixup