Presentation for the Berlin Computer Vision Group, December 2020 on deep learning methods for image segmentation: Instance segmentation, semantic segmentation, and panoptic segmentation.

1. Image Segmentation with
Deep Learning
Antonio Rueda-Toicen and Imran Kocabiyik
Berlin Computer Vision Group
December 2020

2. https://www.meetup.com/Berlin-Computer-Vision-Group/

3. Agenda
● Image segmentation
■ Semantic segmentation
● Fully convolutional networks, U-net
■ Instance segmentation
● Mask R-CNN
■ Panoptic segmentation
● Feature Pyramid Networks
○ Public datasets
■ COCO
■ Google Open Images
○ Implementations: Detectron2, Fast.ai

4. Classification, detection, and segmentation
Classification refers to image-wide labels
Detection refers to localization of bounding boxes with labels
Segmentation refers to pixel-wise localization of the labels

5. Goals of supervised image segmentation
Given an input image we wish to obtain:
1. A class label associated to each individual pixel in the image. This is also called pixel-wise
localization.
3. The probability score associated with each class label

6. Applications of image segmentation
Link

7. Applications of image segmentation
http://withoutbg.com/

8. Applications of image segmentation

9. Applications of image segmentation

10. Applications of image segmentation

11. Applications of image segmentation
https://www.segmentive.ai/

12. Segmentation as pixel-wise localization

13. Instance segmentation requires object detection

14. Panoptic segmentation
https://arxiv.org/pdf/1801.00868.pdf
Explore it in the detectron2 inference notebook

15. “Fully Convolutional” networks draw segmentation
masks
All layers in the network are convolutional, there is no fully connected (aka “dense”) layer like in most
classifiers, we use the local info of the pixel neighborhood

16. What is a convolution filter?
https://setosa.io/ev/image-kernels/

17. What is a convolution filter?
https://setosa.io/ev/image-kernels/

18. What is a convolution filter?
Convolution of 3x3 and stride = 1 without padding
Effect: the output loses one pixel on each dimension

19. What is a convolution filter?
Convolution of 3x3 and stride = 1 with zero padding
Effect: the output preserves original image size

20. What is a convolution filter?
Convolution of 3x3 and stride = 2 with zero padding
Effect: the output is downsampled to about half its size

21. “Fully Convolutional” networks draw segmentation
masks
All layers in the network are convolutional, there is no fully connected (aka “dense”) layer like in most
classifiers, we use the local info of the pixel neighborhood

22. U-net for semantic segmentation
All layers in the network are convolutional, there is no fully connected (aka “dense”) layer like in most
classifiers, we need this fully convolutional architecture to label images pixel by pixel preserving their
local info

23. U-net for semantic segmentation
All layers in the network are convolutional, there is no fully connected (aka “dense”) layer like in most
classifiers, we need this fully convolutional architecture to label images pixel by pixel preserving their
local info

24. Image pyramids

25. Image Pyramids in Feature Proposal Networks
(FPNs)

26. Convolutional networks implement “pyramids”
The deeper we go into the network, the more semantic value is compressed in lower x,y dimensions

27. Resnets

28. Nearest neighbor interpolation
Resnets in feature pyramid networks

29. 1x1 convolution
Resnets in feature pyramid networks

30. Feature Pyramid Networks

31. Image Pyramids in Feature Proposal Networks
(FPNs)

32. Mask R-CNN

33. The COCO dataset
http://cocodataset.org/#explore

34. The Google Open Images Dataset

35. https://storage.googleapis.com/openimages/web/index.html

37. https://storage.googleapis.com/openimages/web/visualizer/index.html?set=train&type=segmentation&r=false&c=%2Fm%2F03g8mr

38. https://storage.googleapis.com/openimages/web/visualizer/index.html?set=train&type=detection&c=%2Fm%2F04rmv

39. https://cocodataset.org/#explore

40. Detectron2
detectron2/MODEL_ZOO.md at master · facebookresearch/detectron2 · GitHub
Inference (Colab notebook)
Training (Colab notebook)
Generating validation set plots

41. Panoptic segmentation with feature pyramid network (FPN-50)

42. Detectron2 config files

43. https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md

44. Model output format
https://detectron2.readthedocs.io/tutorials/model
s.html#model-output-format

45. Objective
Example Case: Image Matting

46. ⊕
Using a Unet
Example Case: Image Matting

47. Matting algorithm:
Example Case: Image Matting

48. instance segment
Using trimap or instance segments?
Example Case: Image Matting
⊕ or ⊕ ?

49. Results
Example Case: Image Matting
Photo: Ayo Ogunseinde
https://unsplash.com/photos/THIs-cpyebg

50. Results
Example Case: Image Matting
Photo: Eugen Proskouriakov
https://unsplash.com/photos/C-gvAA8q3Tc

51. Results
Example Case: Image Matting
Photo: Mathieu Renier
https://unsplash.com/photos/4WBvCqeMaDE

52. Results
Example Case: Image Matting
Photo: Gulyás Bianka
https://unsplash.com/photos/3WOh54znPGU

53. For more examples:
withoutbg.com
Example Case: Image Matting

54. Which things should be kept in this picture?
Kid, ball, 2 dogs, 9 people?
Example Case: Image Matting
Photo: Treddy Chen
https://unsplash.com/photos/UdQWvefOXJk

55. Issue: When there is more than one person in the image...
Example Case: Image Matting

56. Review questions
- How do we compute the confusion matrix for a segmentation mask? How do we
compute it for a bounding box?
- Can we use the Intersection over Union equation to evaluate the quality of a
segmentation mask?
- What’s the recall of a classifier that only outputs ‘1’ (positive class)?
- What’s the precision of a classifier that outputs a single true positive, with all its
other predictions being equal to ‘0’ (negative class)?
- Why does precision go down when recall increases?
- Does the F1 measure weigh precision and recall equally?
- What’s the appeal of using Detectron2? Do we need to write a Pytorch model to
use it for inference or training?

57. Google Colab Notebooks
● Unet in FastAI 2
● Mask R-CNN and Panoptic Segmentation with Detectron 2

58. - How does panoptic segmentation combine instance and semantic
segmentation? Which method produces the ‘stuff’? Which method produces
the ‘things’?
- Is semantic segmentation more computationally costly than instance
segmentation? Why?
- Is panoptic segmentation more computationally costly than instance
segmentation? Why?
Review questions

59. References
● Stanford’s cs231n lecture on Object Detection and Segmentation
● PyImageSearch tutorial on Mask R-CNN