Mask R-CNN present a conceptually simple, flexible, and general framework for object instance segmentation. This approach efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small overhead to Faster R-CNN, running at 5 fps. Moreover, Mask R-CNN is easy to generalize to other tasks, e.g., allowing us to estimate human poses in the same framework. We show top results in all three tracks of the COCO suite of challenges, including instance segmentation, bounding-box object detection, and person keypoint detection. Without tricks, Mask R-CNN outperforms all existing, single-model entries on every task, including the COCO 2016 challenge winners. We hope our simple and effective approach will serve as a solid baseline and help ease future research in instance-level recognition. presentation: https://www.youtube.com/watch?v=FZePQKPEwoo (한국어) reference: He, Kaiming, et al. "Mask r-cnn." arXiv preprint arXiv:1703.06870 (2017).

1. Mask R-CNN
CM Seminar 2017.09.01
Jaehyun Jun
Biointelligence Laboratory
Interdisciplinary Program of Neuro Science, Seoul National Univertisy
http://bi.snu.ac.kr

2. Overview
 Task
 object detection: classify objects and localize using bounding box
 instance segmentation: classify each pixel into a fixed set of
categories
© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr 2

3. Main Idea
 Goal: develop a comparably enabling framework for
instance segmentation
 Extension of Faster R-CNN
 RoIPool -> RoIAlign
 decouple mask and class prediction
3© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

4. Related Work
 R-CNN (Region-based CNN)
 Bounding box: Selective Search -> AlexNet -> linear regression
 Classification: Selective Search -> AlexNet -> SVM
 Faster R-CNN
 Bounding box: Region Proposal Network (RPN)
 Bounding box + Classification
: extract feature using RoIPool
 RoIPool: large negative effect on
predicting pixel-accurate mask
 misalign problem
4© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

5. History
 R-CNN -> SPP-net
 SPP-net -> Fast R-CNN
5© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

6. History
 Fast R-CNN -> Faster R-CNN
6© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr
 Faster R-CNN -> Mask R-CNN

7. Related Work
 ResNeXt
 increasing cardinality is more effective than depth or width of
networks
 ResNeXt works better than ResNet having same number of
parameters
7© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

8. Mask R-CNN
 Two stage
1. Bounding box: Region Proposal Network (RPN)
2. Class and Box offset
 Output
 binary mask
 one for each classes
 Loss
 L = Lcls + Lbox + Lmask
 Lmask: average binary cross-entropy loss
 positive RoIs
8© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

9. Region Proposal Networks
1. Propose k reference boxes on sliding-window
2. Map sliding-window into lower-dimensional feature
3.1. feed on box-regression layer
3.2. feed on box-classification layer
9© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

10. RoIPooling
10© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

11. RoIPooling - Differentiable
11© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

12. RoIPooling - SGD step
12© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr
 Region-wise sampling to make mini-batches

13. RoIAlign
 RoIPool: misalignment problem
 RoIAlign: use decimal points as boundary size & apply
bilinear interpolation (e.g. [x/16] -> x/16)
13© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

14. Architecture
 Backbone: feature extraction
 ResNet-50 or ResNet-101
 ResNeXt-50 or ResNeXt-101
 Feature Pyramid Network (FPN)
 C4 or C5 (convolutional layer)
 RoIAlign: aligning the extracted features with the input
 quantization X -> bilinear interpolation
 [x/16] -> x/16
 Head: bounding-box recognition & mask prediction
 ResNet-C4 -> 9-layer ‘res5’
 FPN -> res5 + filter
14© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

15. Dataset
 MS COCO
 Object Instance Annotations
 Object Keypoint Annotations
15© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

16. Experiment
 Mask R-CNN vs. FCIS+++
 no artifacts
16© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

17. Experiment
 RoIPool vs. RoIWarp vs. RoIAlign
17© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

18. Experiment
 Mask R-CNN vs. Faster R-CNN
 Improve 3~6%
18© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

19. Experiment
 Human Pose Estimation (Keypoint Detection)
19© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

20. Experiment
 Cityscapes
 1st on Instance Level Semantic
Labeling Task
20© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr

21. Q & A