The document presents the deep residual learning framework proposed in the paper "Deep Residual Learning for Image Recognition". The framework aims to make it easier to optimize extremely deep convolutional neural networks. It does this by introducing "skip connections" that allow layers to learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. This addresses issues like vanishing gradients in very deep networks. The authors demonstrate that residual networks are easier to optimize and can gain accuracy from increased depth, outperforming standard networks.

1. Deep Residual Learning for
Image Recognition
Authors: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun
Presented by – Sanjay Saha, School of Computing, NUS
CS6240 – Multimedia Analysis – Sem 2 AY2019/20

2. Objective | Problem Statement
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

3. Motivation
Performance of plain networks in a deeper architecture
Image source: paper
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

4. Main Idea
• Skip Connections/ Shortcuts
• Trying to avoid:
‘Vanishing Gradients’
‘Long training times’
Image source: Wikipedia
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

5. Contributions | Problem Statement
• These extremely deep residual nets are easy to optimize, but the
counterpart “plain” nets (that simply stack layers) exhibit higher
training error when the depth increases.
• These deep residual nets can easily enjoy accuracy gains from greatly
increased depth, producing results substantially better than previous
networks.
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu)
A residual learning framework to ease the training of networks that
are substantially deeper than those used previously.
Perfor
mance
Depth
School of Computing

6. Literature
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

7. Literature Review
• Partial solutions for vanishing
• Batch Normalization – To rescale the weights over some batch.
• Smart Initialization of weights – Like for example Xavier initialization.
• Train portions of the network individually.
• Highway Networks
• Feature residual connections of the form
𝑌 = 𝑓 𝑥 × 𝑠𝑖𝑔𝑚𝑜𝑖𝑑(𝑊𝑥 + 𝑏) + 𝑥 × (1 − 𝑠𝑖𝑔𝑚𝑜𝑖𝑑 𝑊𝑥 + 𝑏 )
• Data-dependent gated shortcuts with parameters
• When gates are ‘closed’, the layers become ‘non-residual’.
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

8. ResNet | Design | Architecture
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

9. Plain Block
𝑎[𝑙] 𝑎[𝑙+2]
𝑎[𝑙+1]
𝑧[𝑙+1]
= 𝑊[𝑙+1]
𝑎[𝑙]
+ 𝑏[𝑙+1]
“linear”
𝑎[𝑙+1] = 𝑔(𝑧[𝑙+1])
“relu”
𝑧[𝑙+2] = 𝑊[𝑙+2] 𝑎[𝑙+1] + 𝑏[𝑙+2]
“output”
𝑎[𝑙+2]
= 𝑔 𝑧 𝑙+2
“relu on output”
Image source: deeplearning.ai
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

10. Residual Block
𝑎[𝑙] 𝑎[𝑙+2]
𝑎[𝑙+1]
𝑧[𝑙+1]
= 𝑊[𝑙+1]
𝑎[𝑙]
+ 𝑏[𝑙+1]
“linear”
𝑎[𝑙+1] = 𝑔(𝑧[𝑙+1])
“relu”
𝑧[𝑙+2] = 𝑊[𝑙+2] 𝑎[𝑙+1] + 𝑏[𝑙+2]
“output”
𝑎[𝑙+2]
= 𝑔 𝑧 𝑙+2
+ 𝑎 𝑙
“relu on output plus input”
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu)
Image source: deeplearning.ai
School of Computing

11. Skip Connections
• Skipping immediate connections!
• Referred to as residual part of the network.
• Such residual part receives the input as an amplifier to its output –
The dimensions usually are the same.
• Another option is to use a projection to the output space.
• Either way – no additional training parameters are used.
Image source: towardsdatascience.com
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

12. ResNet Architecture
Image source: paper
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

13. ResNet Architecture
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu)
Image source: paper
Stacked Residual Blocks
School of Computing

14. ResNet Architecture
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu)
Image source: paper
3x3 conv
layers
2x # of filters
2 strides to down-sample
Avg. pool after the
last conv layer
FC layer to
output classes
School of Computing

15. ResNet Architecture
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu)
Image source: paper
3x3 conv
layers
2x # of filters
2 strides to down-sample
Avg. pool after the
last conv layer
FC layer to
output classes
School of Computing

16. ResNet Architecture
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu)
Image source: paper
3x3 conv
layers
2x # of filters
2 strides to down-sample
Avg. pool after the
last conv layer
FC layer to
output classes
School of Computing

17. ResNet Architecture
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu)
1x1 conv with 64 filters
28x28x64
Input:
28x28x256
Image source: paper
School of Computing

18. ResNet Architecture
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu)
1x1 conv with 64 filters
28x28x64
Input:
28x28x256
3x3 conv on 64 feature
maps only
Image source: paper
School of Computing

19. ResNet Architecture
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu)
1x1 conv with 64 filters
28x28x64
Input:
28x28x256
3x3 conv on 64 feature
maps only
1x1 conv with 256 filters
28x28x256
BOTTLENECK
Image source: paper
School of Computing

20. Summary | Advantages
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

21. Benefits of Bottleneck
• Less training time for deeper networks
• By keeping time complexity same as
two-layer conv.
• Hence, allows to increase # of layers.
• And, model converges faster: 152-
layer ResNet has 11.3 billion FLOPS
while VGG-16/19 nets has 15.3/19.6
billion FLOPS.
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu)
Input:
28x28x256
Image source: paper
School of Computing

22. Summary – Advantages of ResNet over Plain
Networks
• A deeper plain network tends to perform bad because of the
vanishing and exploding gradients
• In such cases, ResNets will stop improving rather than decrease in
performance: 𝑎[𝑙+2] = 𝑔 𝑧 𝑙+2 + 𝑎 𝑙 = 𝑔(𝑤 𝑙+1 𝑎 𝑙+1 + 𝑏 𝑙 + 𝑎[𝑙])
• If a layer is not ‘useful’, L2 regularization will bring its parameters very
close to zero, resulting in 𝑎[𝑙+2]
= 𝑔 𝑎[𝑙]
= 𝑎[𝑙]
(when using ReLU)
• In theory, ResNet is still identical to plain networks, but in practice
due to the above the convergence is much faster.
• No additional training parameters and complexity introduced.
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

23. Results
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

24. Results
• ILSVRC 2015 classification winner (3.6% top 5 error) -- better than “human performance”!
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu)
Error rates (%) of ensembles. The top-5 error is on the
test set of ImageNet and reported by the test server
School of Computing

25. Results
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu)
Error rates (%, 10-crop testing) on ImageNet
validation set
Error rates (%) of single-model results on
the ImageNet validation set
School of Computing

26. Plain vs. ResNet
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu)
Image source: paper
School of Computing

27. Plain vs. Deeper ResNet
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu)
Image source: paper
School of Computing

28. Conclusion | Future Trends
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

29. Conclusion
•Easy to optimize deep neural networks.
•Guaranteed Accuracy gain with deeper layers.
•Addressed: Vanishing Gradient and Longer
Training duration.
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

30. Conclusion
•Easy to optimize deep neural networks.
•Guaranteed Accuracy gain with deeper layers.
•Addressed: Vanishing Gradient and Longer
Training duration.
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

31. Conclusion
•Easy to optimize deep neural networks.
•Guaranteed Accuracy gain with deeper layers.
•Addressed: Vanishing Gradient and Longer
Training duration.
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

32. Conclusion
•Easy to optimize deep neural networks.
•Guaranteed Accuracy gain with deeper layers.
•Addressed: Vanishing Gradient and Longer
Training duration.
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

33. Future Trends
• Identity Mappings in Deep Residual Networks suggests to pass the
input directly to the final residual layer, hence allowing the network
to easily learn to pass the input as identity mapping both in forward
and backward passes. (He et. al. 2016)
• Using the Batch Normalization as pre-activation improves the
regularization
• Reduce Learning Time with Random Layer Drops
• ResNeXt: Aggregated Residual Transformations for Deep Neural
Networks. (Xie et. al. 2016)
Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing

34. Presented by – Sanjay Saha (sanjaysaha@u.nus.edu) School of Computing
Questions?