A presentation introducting DeepLab V3+, the state-of-the-art architecture for semantic segmentation. It also includes detailed descriptions of how 2D multi-channel convolutions function, as well as giving a detailed explanation of depth-wise separable convolutions.

1. DeepLabV3+
Encoder-Decoder with Atrous Separable Convolution
for Semantic Image Segmentation

2. Background
▪ DeepLabV3+ is the latest version of the DeepLab models.
▪ DeepLab V1: Semantic Image Segmentation with Deep Convolutional Nets and
Fully Connected CRFs. ICLR 2015.
▪ DeepLab V2: DeepLab: Semantic Image Segmentation with Deep Convolutional
Nets, Atrous Convolution, and Fully Connected CRFs. TPAMI 2017.
▪ DeepLab V3: Rethinking Atrous Convolution for Semantic Image Segmentation.
arXiv 2017.
▪ DeepLab V3+: Encoder-Decoder with Atrous Separable Convolution for Semantic
Image Segmentation. arXiv 2018.

3. Semantic Segmentation
▪ Classifying all pixels in an image
into classes.
▪ Classification at the pixel level.
▪ Does not have to separate different
instances of the same class.
▪ Has important applications in
Medical Imaging.

4. Current Results on Pascal VOC 2012

5. Motivation and Key Concepts
▪ Use Atrous Convolution and Separable Convolutions to reduce computation.
▪ Combine Atrous Spatial Pyramid Pooling Modules and Encoder-Decoder
Structures.
▪ ASPPs capture contextual information at multiple scales by pooling features at
different resolutions.
▪ Encoder-Decoders can obtain sharp object boundaries.

6. Architecture Overview

7. Advanced Convolutions

8. Convolution (Cross-Correlation) for 1 Channel
Convolution with Zero-Padding Display with Convolution Kernel
Blue maps: inputs, Cyan maps: outputs, Kernel: not displayed

9. Other Convolutions (Cross-Correlations)
Strided Convolution with Padding Atrous (Dilated) Convolution with r=2
Blue maps: inputs, Cyan maps: outputs, Kernel: not displayed

10. Atrous Convolution
▪ à trous is French for “with holes”
▪ Atrous Convolution is also known as
Dilated Convolution.
▪ Atrous Convolution with r=1 is the
same as ordinary Convolution
▪ The image on the left shows 1D
atrous convolution

11. Receptive Field of Atrous Convolutions
▪ Left: r=1, Middle: r=2, Right: r=4
▪ Atrous Convolution has a larger receptive field than normal convolution
with the same number of parameters.

12. Depth-wise Separable Convolution
▪ A special case of Grouped Convolution.
▪ Separate the convolution operation along the depth (channel) dimension.
▪ It can refer to both (depth -> point) and (point -> depth).
▪ It only has meaning in multi-channel convolutions (cross-correlations).

13. Review: Multi-Channel 2D Convolution

14. Exact Shapes and Terminology
▪ Filter: A collection of 𝑪𝒊𝒏 Kernels of shape (𝑲 𝑯, 𝑲 𝑾) concatenated channel-wise.
▪ Input Tensor Shape: 𝑵, 𝑪𝒊𝒏, 𝑯, 𝑾 𝑜𝑟 (𝑵, 𝑯, 𝑾, 𝑪𝒊𝒏)
▪ Filters are 3D, Kernels are 2D, All filters are concatenated to a single 4D array in 2D CNNs.
(𝑵: 𝐵𝑎𝑡𝑐ℎ 𝑁𝑢𝑚𝑏𝑒𝑟, 𝑯: 𝐹𝑒𝑎𝑡𝑢𝑟𝑒 𝐻𝑒𝑖𝑔ℎ𝑡, 𝑾: 𝐹𝑒𝑎𝑡𝑢𝑟𝑒 𝑊𝑖𝑑𝑡ℎ,
𝑪𝒊𝒏: #𝐼𝑛𝑝𝑢𝑡 𝐶ℎ𝑎𝑛𝑛𝑒𝑙𝑠, 𝑪 𝒐𝒖𝒕: #𝑂𝑢𝑡𝑝𝑢𝑡 𝐶ℎ𝑎𝑛𝑛𝑒𝑙𝑠, 𝑲 𝑯: 𝐾𝑒𝑟𝑛𝑒𝑙 𝐻𝑒𝑖𝑔ℎ𝑡, 𝑲 𝑾: 𝐾𝑒𝑟𝑛𝑒𝑙 𝑊𝑖𝑑𝑡ℎ)

15. Step 1: Convolution on Input Tensor Channels

16. Step 2: Summation along Input Channel Dimension

17. Step 3: Add Bias Term
▪ Each kernel of a filter iterates only
1 channel of the input tensor.
▪ The number of filters is 𝐶 𝑜𝑢𝑡. Each
filter generates one output channel.
▪ Each 2D kernel is different from all
other kernels in the 3D filter.
Key Points

18. Normal Convolution
▪ Top: Input Tensor
▪ Middle: Filter
▪ Bottom: Output Tensor

19. Depth-wise Separable Convolution
▪ Replace Step 2.
▪ Instead of summation, use point-wise convolution (1x1 convolution).
▪ There is now only one (𝑪𝒊𝒏, 𝑲 𝑯, 𝑲 𝑾) filter.
▪ The number of 1x1 filters is 𝑪 𝒐𝒖𝒕.
▪ Bias is usually included only at the end of both convolution operations.
▪ Usually refers to depth-wise convolution -> point-wise convolution.
▪ Xception uses point-wise convolution -> depth-wise convolution.

20. Depth-wise Separable Convolution

21. Characteristics
▪ Depth-wise Separable Convolution can be used as a drop-in replacement for
ordinary convolution in DCNNs.
▪ The number of parameters is reduced significantly (sparse representation).
▪ The number of flops is reduced by several orders of magnitude
(computationally efficient).
▪ There is no significant drop in performance (performance may even improve).
▪ Wall-clock time reduction is less dramatic due to GPU memory access patterns.

22. Example: Flop Comparison (Padding O, Bias X)
Ordinary Convolution
▪ 𝐻 ∗ 𝑊 ∗ 𝐾 𝐻 ∗ 𝐾 𝑊 ∗ 𝐶𝑖𝑛 ∗ 𝐶 𝑜𝑢𝑡
▪ For a 256x256x3 image with 128 filters
with kernel size of 3x3, the number of
flops would be
256 ∗ 256 ∗ 3 ∗ 3 ∗ 3 ∗ 128 = 226,492,416
▪ There is an 8-fold reduction in the
number of flops.
Depth-wise Separable Convolution
▪ 𝐻 ∗ 𝑊 ∗ 𝐾 𝐻 ∗ 𝐾 𝑊 ∗ 𝐶𝑖𝑛 + 𝐻 ∗ 𝑊 ∗ 𝐶𝑖𝑛 ∗ 𝐶 𝑜𝑢𝑡
▪ Left: Depth Conv, Right: Point Conv
▪ For a 256x256x3 image with 128 filters
and a 3x3 kernel size, the number of
flops would be
256 ∗ 256 ∗ 3 ∗ 3 ∗ 3 + 256 ∗ 256 ∗ 3 ∗ 128
= 1,769,472 + 25,165,824 = 26,935,296

23. Example: Parameter Comparison (Excluding Bias Term)
Ordinary Convolution
▪ 𝐾 𝐻 ∗ 𝐾 𝑊 ∗ 𝐶𝑖𝑛 ∗ 𝐶 𝑜𝑢𝑡
▪ For a 256x256x3 image with 128 filters
and 3x3 kernel size, the number of
weights would be
3 ∗ 3 ∗ 3 ∗ 128 = 3,456
Depth-wise Separable Convolution
▪ 𝐾 𝐻 ∗ 𝐾 𝑊 ∗ 𝐶𝑖𝑛 + 𝐶𝑖𝑛 ∗ 𝐶 𝑜𝑢𝑡
▪ For a 256x256x3 image with 128 filters
and 3x3 kernel size, the number of flops
would be
3 ∗ 3 ∗ 3 + 3 ∗ 128 = 411
▪ There is also an 8-fold reduction in
parameter numbers.

24. Atrous Depth-wise Separable Convolution

25. Architecture Overview

26. Encoder-Decoder Structures
▪ The Encoder reduces the spatial sizes of feature maps, while extracting higher-
level semantic information.
▪ The Decoder gradually recovers the spatial information.
▪ UNETs are a classical example of encoder-decoder structures.
▪ In DeepLabV3+, DeepLabV3 is used as the encoder.

27. Architecture Overview

28. Decoder Layer
Structure
1. Apply 4-fold bilinear up-sampling on the
ASPP outputs.
2. Apply 1x1 Convolution with reduced filter
number on a intermediate feature layer.
3. Concatenate ASPP outputs with
intermediate features.
4. Apply two 3x3 Convolutions.
5. Apply 4-fold bilinear up-sampling.
Purpose & Implementation
▪ The ASPP is poor at capturing fine details.
▪ The decoder is used to improve the
resolution of the image.
▪ The intermediate layer has 1x1
convolutions to reduce channel number.

29. ASPP: Atrous Spatial Pyramid Pooling

30. The ASPP Layer
▪ Encodes multi-scale contextual
information through multiple rates.
▪ Concatenate all extracted features and an
up-sampled global average pooling layer
channel-wise.
▪ Use Atrous Depth-wise separable
convolutions for multiple channels.
▪ Bad at capturing sharp object boundaries.

31. Modified Aligned Xception Network
▪ Xception: Extreme Inception Network.
▪ Backbone network for DeepLabV3+
▪ Uses residual blocks and separable
convolutions.

32. Explanation of Xception
▪ Takes the “Inception Hypothesis”, which states that cross-channel correlations
and spatial correlations are sufficiently decoupled that it is preferable not to
map them jointly, to the extreme.
▪ The extensive use of separable convolutions and atrous convolutions allows
the model to fit in GPU memory despite the huge number of layers.
▪ Originally applied point-wise convolution before depth-wise convolution.
▪ Invented by François Chollet.

33. Architecture Review

34. The End