A class presentation in soft computing course, June 2018

1. DENSELY
Prof. Mohammad-R.Akbarzadeh-T
Ferdowsi University of Mashhad
A Presentation by:
• Hosein Mohebbi
• M.-Sajad Abavisani

2. CVPR 2017 BEST PAPER AWARD
2
Gao Huang
Cornell University
h-index: 12
Zhuang Liu
Tsinghua University
h-index: 5
Laurens van der Maaten
Facebook AI Research
h-index: 29
Kilian Weinberger
Associate Professor
Cornell University
h-index: 41

3. 3

4.  Normalized initialization and intermediate normalization layers
 The main culprit :Vanishing/exploding gradients
 Not caused by overfitting
4

5. 5
RESNET

6. STOCHASTIC DEPTH
 Deep network during testing,but shallower network during training.
6
𝐻𝑙 = 𝑅𝑒𝐿𝑈 𝑏𝑙 𝑓𝑙(𝐻𝑙−1 + 𝑖𝑑 𝐻𝑙−1 ) 𝑏𝑙 ∈ 0,1
They all share a key characteristic:
They create short paths from early layers to later layers

7. DENSE
7
𝑙 (𝑙+1)
2
direct connections

8. DENSE
8
 The growth rate regulates how much new information each
layer contributes to the global state.

9.  Traditional Convolutional feed-forward networks :
9
𝑥𝑙 = 𝐻𝑙 (𝑥𝑙−1)
𝑥𝑙 = 𝐻𝑙 (𝑥𝑙−1) + 𝑥𝑙−1
 ResNets :
 DenseNets :
𝑥𝑙 = 𝐻𝑙 ([𝑥0, 𝑥1, … , 𝑥𝑙−1])
Where [𝑥0, 𝑥1, … , 𝑥𝑙−1] refers to the concatenation of the feature-maps produced
in layers 0….. 𝑙-1.

10. 10

11. DENSENET
11

12. DENSENET
WITH BOTTLENECK LAYER
12

13. DENSENET
13
Compression in transition layer
Pooling reduces
Feature map sizes
Feature map sizes match
Within each block

14. DENSE

15. 15
 Direct access: Deep Supervision with single classifier
 Reduces overfitting on tasks with smaller training set sizes

16. 16

17. 17
ResNet connectivity:
DenseNet connectivity:
#parameters:
k: Growth rate

18. 18
Standard Connectivity :

19. 19
Feature visualization of convolutional net trained on ImageNet from [Zeiler & Fergus 2013]
 Remember feature visualization

20. 20
“Collective Knowledge”

21.  Feature reuse
 Information flow from the first to the last layers of the block
 Compression in transition layer
 Concentrate on high level feature for final classification
21

23. CIFAR-10
23

24. DENSENET IMAGENET
24

25. IMAGENET
25

26. 26
 Kaiming He,et al."Deep residual learning for image recognition" CVPR
2016
 Chen-Yu Lee,et al."Deeply-supervised nets" AISTATS 2015
 Gao Huang,et al."Deep networks with stochastic depth" ECCV 2016
 CS231n:Convolutional Neural Networks forVisual Recognition
 Gao Huang,Zhuang Liu,Kilian QWeinberger,and Laurens van der
Maaten. Densely connected convolutional networks.Conference on
ComputerVision and Pattern Recognition,2017
 Geoff Pleiss,et al."Memory-Efficient Implementation of DenseNets”,
arXiv preprint arXiv:1707.06990 (2017)