This document discusses different deep learning approaches for image segmentation. It covers using convolutional neural networks (CNNs), recurrent neural networks (RNNs), and generative adversarial networks (GANs). For CNNs, it describes fully convolutional networks, deconvolution networks, U-Net, and other models. For RNNs, it discusses multi-dimensional RNNs, scene labeling with LSTMs, turned and pyramid RNNs, and grid LSTMs. It also reviews Pix2Pix and other GAN-based models for image segmentation tasks.

1. Image Segmentation
DeepBio
Hyungjoo Cho

2. Image Segmentation

3. Image Segmentation

4. Pixel wise prediction(classification)

5. Image Segmentation
• CNNs
• RNNs
• GANs

6. Using CNNs

7. Fully Convolution Network
http://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Long_Fully_Convolutional_Networks_2015_CVPR_paper.pdf
• End-to-end, Pixel-to-pixel prediction
• Backwards convolution for up-sampling
• Per-pixel multinomial logistic loss

8. Limitations
• Fixed size receptive field
• Too simple structure to get detailed features

9. Deconvolution Network
http://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Noh_Learning_Deconvolution_Network_ICCV_2015_paper.pdf
• Combining unpooling, deconvolution(with crop), and Relu
• Reconstruction of the detailed structure of an object in finer resolution
• Batch-normalization

10. Limitations
• Difficult to learn
• Still lose spatial information

11. U-Net
https://arxiv.org/pdf/1505.04597.pdf
• Do not use unpooling(only up-convolution)
• Skip-connection(with concat)
• Do not have fully connected layer
• Elastic deformation

12. Limitations
• Didn’t use batch-norm
• VGG is not the best solution for feature extracting

13. Deep contextual networks
http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/11789
• Auxiliary connection, classifier
• Ensemble
• Lower memory consumption

14. Limitations
• Didn’t use batch-norm
• VGG is not the best solution for feature extracting

15. FusionNet
https://arxiv.org/pdf/1612.05360.pdf
• Skip-connection(with summation)
• Residual block(shortcut connection)
• Elastic deformation

16. Limitations
• Memory
• Memory
• Memory
• Memory
• Memory

17. Pyramid Scene Parsing Net
https://arxiv.org/pdf/1612.01105.pdf
• Pre-trained FCN with ResNet(1/8 sized feature map)
• Pyramid pooling & 1x1 cone
• Bilinear interpolation
• Avg pooling is better than Max pooling

18. Using RNNs

19. Multi-Dimensional RNNs
https://arxiv.org/pdf/0705.2011.pdf
• GOD GRAVES!!
• 1D RNNs(Bi-directional RNNs) couldn’t explain images well
• Need to access to the surrounding context in all directions
• N-dimensional data : At least 2^(N) hidden layers
• The input layer is size 3(RGB) or 1(Gray) or patch and the output layer(softmax) is size of classes

20. Assume that…
A00 A01 A02
A10 A11 A12
A20 A21 A22
• 3X3 IMAGE

21. A00 A01 A02 A10 A11 A12 A20 A21 A22
O00 O01 O02 O10 O11 O12 O20 O21 O22

22. Scene Labeling with LSTM
http://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Byeon_Scene_Labeling_With_2015_CVPR_paper.pdf
• Patch without overlapping
• Four separate 2D-LSTM block with summation
• The size of the layer corresponds
to the number of feature maps

23. Turned MD RNNs
https://arxiv.org/pdf/0705.2011.pdf
• Standard MD RNNs was not easy to parallelize
• Rotate 45 degrees
• Easy to parallelize!!
• This introduces context gaps

24. PyraMiD RNNs
https://arxiv.org/pdf/1506.07452.pdf
• Fill the blanks
• More spatial information
• For 3D-Image : PyraMiD needs only 6, while standard needs 8 cubes

25. Grid LSTM
https://arxiv.org/pdf/1506.07452.pdf
• GOD GRAVES
• Connections along depth dimension as well as temporal dimension
• 3D Grid LSTM = Multi-dimensional LSTM + memory connection

26. Using GANs

27. Pix2Pix
https://arxiv.org/pdf/1611.07004v1.pdf
• Pixel to pixel translation
• U-Net + Conditional Gan loss
• Also doing well segmentation tasks

28. Pix2Pix
https://arxiv.org/pdf/1611.07004v1.pdf

29. Adversarial Networks for the Detection of
Aggressive Prostate Cancer
https://arxiv.org/pdf/1702.08014.pdf
• Pix2pix structure
• Conditional Gan loss
• Instance norm in stead of batch norm

30. Thanks❤️