This is the review of the paper "Memory-Efficient Implementation of DenseNets"

1. Memory-Efficient
Implementation of
DenseNets
Hyungjoo Cho

3. DenseNets
- ResNets
- Dense connectivity
- Composite function
- Pooling layers
- Growth rate
- Bottleneck layers
- Compression

6. Memory optimization
with Computation Graph

7. “Training Deep Nets with Sublinear Memory Cost” - Tianqi Chen et. al.

8. “Training Deep Nets with Sublinear Memory Cost” - Tianqi Chen et. al.

9. “Training Deep Nets with Sublinear Memory Cost” - Tianqi Chen et. al.

10. Memory Efficient DenseNets

11. Pre-activation batch normalization
- Apply batch-norm and non-linearities before
the conv-operation
- Given m layers, pre-activations generate up
to m copies of each layer
- Typically ( m - 1 ) ( m - 2 ) / 2
“Identity Mapping in Deep Residual Networks” - Kaiming He et. al.

12. Contiguous Concatenation
- Convolution is most efficient when input
lies in a contiguous block of memory
- To make a contiguous input, each layer
must copy all previous features
- Given m layers, each feature may be
copied up to m times
“Memory-Efficient Implementation of DenseNets” - Geoff Pleiss et. al.

13. Concatenation and normalization operation are
computationally extremely cheap

14. Copying to pre-allocated memory is significantly faster than
allocating new memory

15. “Memory-Efficient Implementation of DenseNets” - Geoff Pleiss et. al.

16. Shared storage for concatenation
- Rather than allocating memory for each concatenation operation, assign the
outputs to a memory allocation shared across all layers
- Shared Memory Storage 1 is used by all network layers, its data is not
permanent
- Need to be recomputed during back-propagation

17. Shared storage for batch normalization
- Assign the outputs of batch normalization to a shared memory allocation
- The data in Shared Memory Storage 2 is not permanent and will be
overwritten by the next layer
- Should recompute the batch normalization outputs during back-propagation

18. Implementation Details ( Pytorch )
- Pre allocate shared memories when DenseBlock is initialized

19. Implementation Details ( Pytorch )
- Define bottleneck layer function as an Autograd Function

20. Implementation Details ( Pytorch )
- Define custom functions ( BN, Cat, ReLU, Conv2d ) with backward as well

21. Appendix
- Pytorch C libraries ( i.e, ReLU )

22. Result

25. Thanks !