Reading "Swapout: Learning an ensemble of deep architectures" and introduced at Waseda Univ. Also introducing some deep stochastic techniques ex) dropout, ResNet, etc.

1. Swapout: Learning an ensemble
of deep architecture
Saurabh Singh, Derek Hoiem, David Forsyth
Department of Computer Science
University of Illinois, Urbana-Champaign
B4
1
NIPS 2016

2. Deep Learning
2
Deep Neural Network
wij

3. Deep Learning
3
Deep Neural Network
wij

4. Deep Learning
4
Deep Neural Network
wij

5. Deep Learning
5
Deep Neural Network
Dropout
Residual structure
Stochastic depth
Batch Normalization etc…
wij

6. Deep Learning
6
Deep Neural Network
Dropout
Residual structure
Stochastic depth
Swapout
wij

7. 7
a
Swapout

8. Dropout
8
DNN
Dropout
wij

9. Dropout
9
DNN
Dropout
wij

10. Dropout
10
DNN
Dropout
wij

11. Dropout
11
DNN
Dropout
wij

12. Dropout
12
DNN
Dropout
wijwijwij

13. Dropout
13
DNN
Dropout
wijwijwij

14. Residual structure
14
DNN
the deeper, the better
Residual Nets[4]

15. Residual structure
15
DNN
wij

16. Residual structure
16
DNN
wij

17. Residual structure
17
DNN
wij

18. Residual structure
18
DNN
0
wij

19. Residual structure
19
wij
DNN

20. Residual structure
20
DNN
FX F(X)

21. Residual structure
21
DNN
Residual structure
0
FX
X
F(X)
+X

22. Residual structure
22
Residual Net[4]
Residual Block

23. Stochastic depth
23
Residual structure
FX
X
F(X)
+X

24. Stochastic depth
24
Residual structure
FX
X
F(X)
+X
FX
X
X

25. Stochastic depth
25
Residual structure
Dropout
X
X
F
F(X) + X
or X

26. Swapout
26
DNN
Dropout, Residual structure, Stochastic depth
Swapout

27. Swapout
27
A
wij
27

28. Swapout
28
A
FX F(X)

29. Swapout
29
A
X F(X)
F

30. Swapout
30
A
Dropout
F
X 0 or F(X)

31. Swapout
31
A
Residual structure
F
X X + F(X)

32. Swapout
32
A
Stochastic depth
F
X X or F(X) + X

33. Swapout
33
A
Dropout
Residual Structure
Stochastic depth
0 X + 1 F(X)
0 X + (0 or 1) F(X)
1 X + 1 F(X)
1 X + (0 or 1) F(X)

34. Swapout
34
A
Swapout
F
X
0 : orX : or
F(X) : orX + F(X) :

35. Swapout
35
A
Dropout, Residual structure, …
Y = 1 X + 2 F(X)

36. Inference in Networks
36
A
wijwijwij

37. Inference in Networks
37
a
wijwijwij
wij

38. Inference in Networks
38
Swapout Networks A
30

39. 39
CIFAR-10 A
CIFAR-10
60000 10
Swapout

40. 40
b
20 CNN
ResNet v1[4], v2[5]
Residual structure
Dropout
Stochastic depth

41. 41
N
Conv

42. 42
20
ResNet

43. 43

44. 44
Y = 1 X + 2 F(X)

45. 45
Y = 1 X + 2 F(X)

46. 46
Swapout
Y = 1 X + 2 F(X)

47. 47
Y = 1 X + 2 F(X)
30

48. 48
Swapout
DNN

49. Appendix
49

50. Residual structure
50
[5] ResNet
Batch Normalization
ReLU

51. Swapout
51
A
Skip forward
F
X XorF(X)

52. Swapout
52
A
Dropout, Residual structure, …
M. Hardt et al.[3]
Y = 1 X + 2 F(X)

53. 53
20
ResNet v1[4]
[4] ResNet
ResNet v2[5]
Batch Normalization
ResNet Ours v1/v2
depth
average pooling projection shortcut

54. 54
20
batch size : 128
momentum 0.9
weight decay : 0.0001
epoch : 256
learning rate : 0.1 → 196, 224 epoch 1/10
data augmentation
: Linear(1, 0.5)

55. 55
a
ResNet

56. 1000 ResNet
56
1000 ResNet
32 Swapout 1000 ResNet

57. CIFAR-100
57
CIFAR-100
32 Swapout 1000 ResNet

58. 58
Swapout: Learning an ensemble of deep architectures
Deep residual learning for image recognition
Identity mappings in deep residual networks
Deep networks with stochastic depth
[Survey]Deep Networks with Stochastic Depth
Residual Network(ResNet)
Microsoft (?)152