Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, Demetri Terzopoulos, Image Segmentation Using Deep Learning: A Survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 7, pp. 3523-3542, 1 July 2022, doi: 10.1109/TPAMI.2021.3059968. https://ieeexplore.ieee.org/document/9356353 https://arxiv.org/abs/2001.05566

1. Image Segmentation
Using Deep Learning:
A Survey
Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza,
Nasser Kehtarnavaz, and Demetri Terzopoulos
IEEE Transactions on Multimedia, 2021
,
2022/06/14

2. Image Segmentation
n
•
•
•
n
•
•
n パノプティックセグメンテーション
• オブジェクト単位で
全画素にラベルをつける

3. n
• 100
• 10
n
•
•
n2D 2.5D 3D
• 2.5D RGB+

4. nRNN LSTM [Hochreiter+, Neural Computation, 1997]
•
nCNN [LeCun+ ,Proc. of IEEE, 2009]
• 深層学習において最も成功したアーキテクチャの1つ
nGAN [Goodfellow+, NeurIPS2014]
•
nエンコーダデコーダ [Badrinarayanan+, TPAMI2017]
• 2段階のネットワークを介して学習

5. 1. FCN
2.
3.
4.
5. R-CNN
6.
7. RNN
8.
9.
10. CNN

6. 1. FCN

7. FCN
nFully Convolutional Networks (FCN) [Long+, CVPR2015]
•
• CNNの連結層を畳み込み層に置き換え
•
FCN

8. model was tested on PASCAL VOC, NYUDv2, and SIFT F
and achieved state-of-the-art segmentation performance
Fig. 8. Skip connections combine coarse, high-level information and
low-level information. From [31].
This work is considered a milestone in image segme
tion, demonstrating that deep networks can be trained
semantic segmentation in an end-to-end manner on varia
sized images. However, despite its popularity and effect
FCN
n
•
•
nFCN
•
• 3D
nParseNet [Liu+, ICLR2016]
•
•
FCN ParseNet

9. 2.

10. nConvolutional Random fields CRF [Zheng+, ICCV2015]
•
•
nCNN CRF
•
• CNN
CNN
Bi-linear Interpolation
CRF Refine
CNN+CRF [Chen+, CVPR2014]

11. 3.

12. エンコーダデコーダモデル
nエンコーダ
• 入力を潜在空間表現に圧縮
nデコーダ
• 潜在空間表現から出力を予測
nVGG16ベースモデル [Noh+ ,CVPR2015]
・エンコーダ
・畳み込み層を用いる
・入力は特徴ベクトル
•
•
•
VGG16

13. nSegNet [Badrinarayanan+, CVPR2016]
• VGG16
•
•
nU-net [Ronneberger+, MICCAI2015]
•
•
•
SegNet
U-net

14. 4.

15. nPyramid Scene Parsing Network (PSPN) [Zhao+, CVPR2017]
•
• ResNet
•
• 4
•

16. 5. R-CNN

17. R-CNN
n R-CNN [Girshick+, CVPR2014]
• オブジェクトらしき領域を抽出
• それぞれの領域に対して特徴を抽出
• 学習と予測に時間がかかる
nFaster R-CNN [Ren+, CVPR2015]
• 領域提案ネットワーク（RPN）を使用
• bboxを候補を提案
• bbox座標とオブジェクトのクラスを推論
• インスタンス分割問題などのタスクに対応
Faster R-CNN architecture

18. R-CNN
nMask R-CNN [He+, CVPR2017]
• 3
1. bbox
2.
3.
• 3
nMaskLab [Chen+, CVPR2017]
• 3
1.
2.
3.
• /
Mask R-CNN MaskLab

19. 6.

20. n
•
•
nDeepLab [Chen+, ECCV2018]
•
• conv
<latexit sha1_base64="G4TIL2lcMHRtACv4OVqQ+V9ieHA=">AAACjHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuoV8ZXZ9Yq2CrEFJfmxldn2xrWxlV711bEV0dnahdlx9aWA1lAKl5A2UDPAAwUMBmGUIYyAxQE5AssZ4hhSGHIZ0hmKGXIZUhlyGMoAbJzGBIZioEwmsGQwYChACgWy1ANFCsCsjLB8qkMtQxcQL2lQFWpQBWJQNFsIJkO5EVDRfOAfJCZxWDdyUBbcoC4CKhTgUHV4KrBSoPPBicMVhu8NPiD06xqsBkgt1QC6SSI3tSCeP4uieDvBHXlAukShgyELrxuLmFIY7AAuzUT6PYCsAjIF8kQ/WVV0z8HWwWpVqsZLDJ4DXT/QoObBoeBPsgr+5K8NDA1aDYDFzACDNGDG5MRZqRnaKZnEmii7OAEjQoOBmkGJQYNYHibMzgweDAEMIQC7e1l2Mywh2EvEx+TCZM1ky1EKRMjVI8wAwpgcgMAWdWZWA==</latexit>
yi =
K
X
k=1
x[i+rk]w[k]
r
DeepLabv3+
3×3 5×5
r = 2

21. 7. RNN

22. RNN
nRNN [Hochreiter+, Neural Computation, 1997]
• /
•
nCNN LSTM [Hu+, CVPR2016]
• CNN
• LSTM
“ woman sitting on the right bench”
CNN+LSTM
RNN

23. 8.

24. n重みつけ学習の注意機構を提案 [Chieh+, CVPR2016]
• マルチスケール特徴を柔らかく重み付け
• マルチスケール画像と注意モデルで共同学習
1.0
0.5

25. 9.

26. n
•
•
nGAN [Luc+, CVPR2016]
•
•
•
GAN

27. 10. CNN

28. ACM CNN
n ACM [Kass+ ,IJCV1988]
•
•
nACM
• [Chen+, CVPR2019]
nACM FCN
ACM

29. n2D
• PASCAL Visual Object Classes
[Everingham+, IJCV2015]
• PASCAL Context [Mottaghi+, CVPR2014]
• COCO [Lin+, ECCV2014]
• Cityscapes [Cordts+, CVPR2016]
• ADE20K dataset [Zhou+, CVPR2017]
• SiftFlow [Liu+, CVPR2019]
• Stanford background [Gould+, ICCV2009]
• BSD [Martin+, ICCV2001]
• Youtube-Objects [Prest+, CVPR2012]
• KITTI [Geiger+, IJRR2013]
n2.5D
• NYU-D V2 [Silberman+, ECCV2012]
• SUN-3D [Xiao+, ICCV2013]
• SUN RGB-D [Song+, CVPR2015]
• UW RGB-D Object [Lai+, ICRA2011]
n3D
• Stanford 2D-3D [Armeni+, arXiv2017]
• ShapeNet Core [Chang+, arXiv2015]
• Sydney Urban Object [Deuge+, MDPI2013]

30. nPixel accuracy (PA)
•
nMean Pixel Accuracy (MPA)
•
nIntersection over Union (IoU)
• A B
n Dice coefficient (Dice)
• 2
nPrecision Recall F1score
<latexit sha1_base64="B6F1b1wFp0Ow2XTJgUnkuUVhgh0=">AAACpnichVFNSxtRFD2OtrXpR9K6KXQzNKS4CjciVoRA1I1QkPiRRDDpdGZ80ZfMFzOTgA75A/4BF121IFL8GW7cuLTiTyhdWnDjojczAyWVtneY98497577znvP8CwZhETXY8r4xIOHjyYfZ548ffY8m3vxsh64Pd8UNdO1XH/L0ANhSUfUQhlaYsvzhW4blmgY3eXheqMv/EC6zma474mWre86si1NPWRKyy1WF9Wy2mz7uhk1g56tRU6ZBh/eq54Wyc5gkJIyIZOkM1Kh5fJUpDjU+6CUgjzSqLq5EzSxAxcmerAh4CBkbEFHwN82SiB4zLUQMeczkvG6wAAZ1va4SnCFzmyXx13OtlPW4XzYM4jVJu9i8e+zUkWBLukr3dA5ndJ3uvtrryjuMfSyz7ORaIWnZQ9fbdz+V2XzHGLvt+qfnkO0MR97lezdi5nhKcxE3z84utlYWC9Eb+kL/WD/n+mazvgETv+nebwm1j8hww9Q+vO674P6TLE0V5xdm81XltKnmMRrvME03/c7VLCCKmq87wku8A1XyrSyqtSURlKqjKWaKYyE8vEXFS6jYA==</latexit>
PA =
PK
n=0 pij
PK
i=0
PK
j=0 pij
<latexit sha1_base64="+2ri+INny+YT/97H3RZ1UP7J4EY=">AAACpnichVFNS9xQFD1GW3Vs61g3gpvgYBEKw41IlYKgdSOIMmrHERybJvGNfZovksyADfkD/gEXrhRExJ/hxo1LFX+CuLTQTRe9kwRKK21vyHvnnXPPffe9Z/q2DCOiuw6ls+vZ8+6e3kLfi5ev+osDr9dCrxlYomp5thesm0YobOmKaiQjW6z7gTAc0xY1c3eurddaIgil536M9nyx6RjbrmxIy4iY0ouzi5VZdVqtNwLDirUkXnirJfWw6eixO03Jp4VM8PVY7iRJnCk7qaLmpF4sUZnSUJ8CLQcl5FHxiqeoYwseLDThQMBFxNiGgZC/DWgg+MxtImYuYCRTXSBBgb1NzhKcYTC7y+M2rzZy1uV1u2aYui3exeY/YKeKUbqmM3qkSzqne/rx11pxWqPdyx7PZuYVvt6/P7T6/b8uh+cIX365/tlzhAam0l4l9+6nTPsUVuZvfT14XH2/Mhq/oWN64P6P6I4u+ARu65t1sixWDlHgB9D+vO6nYG28rL0rTyxPlGY+5E/Rg2GMYIzvexIzmEcFVd73FFe4wa0ypiwpVaWWpSoduWcQv4Xy+SeYSaOm</latexit>
MPA =
1
K + 1
K
X
n=0
pij
PK
j=0 pij
<latexit sha1_base64="L1qV22xzKXKIcYKMqVagyzFYZ+E=">AAACiHichVFNLwNBGH66vuuruEhcNhri1LwV8ZVIqAs3WkWiIrtjWhvb3c3utgnVH8AfcHAiEREHV84u/oCDnyCOJC4O3m43EQTvZGaeeeZ93nlmRndMw/OJHiNKQ2NTc0trW7S9o7OrO9bTu+rZJVfIrLBN213XNU+ahiWzvuGbct1xpVbUTbmm787X9tfK0vUM21rx9xy5WdQKlpE3hOYztRWLL9pZdUbN5V1NVA7m1JzQHDV1UK3jUoA5ixIUhPoTJEMQRxhLduwCOWzDhkAJRUhY8Bmb0OBx20ASBIe5TVSYcxkZwb5EFVHWljhLcobG7C6PBV5thKzF61pNL1ALPsXk7rJSxRA90CW90D1d0RO9/1qrEtSoednjWa9rpbPVfdSfeftXVeTZx86n6k/PPvKYDLwa7N0JmNotRF1f3j9+yUynhyrDdEbP7P+UHumOb2CVX8X5skyfIMofkPz+3D/B6mgiOZ4YWx6Lz6bCr2jFAAYxwu89gVksYAlZPvcQ17jBrRJVSJlQpuqpSiTU9OFLKKkPZuuWKw==</latexit>
IoU =
|A B|
|A [ B|
<latexit sha1_base64="K9yQUUgVvDPv15yU+LH9i9E/bVU=">AAACh3ichVHLSsNAFD3Gd31V3QhugkURhDqVUkUQrLpw6asqWCnJONXBNAlJWtDEvfgDLlwpiIjgVvdu/AEXfoK4VHDjwts0ICrqHWbmzJl77pyZ0W1Duh5jj3VKfUNjU3NLa6ytvaOzK97ds+paZYeLHLcMy1nXNVcY0hQ5T3qGWLcdoZV0Q6zpu7PV/bWKcFxpmSveni02S9q2KYuSax5RhfjAnORCnVLzRUfj/liQVfNcs9WZ4MAPssFIQKAQT7AkC0P9CVIRSCCKBSt+gTy2YIGjjBIETHiEDWhwqW0gBQabuE34xDmEZLgvcIAYacuUJShDI3aXxm1abUSsSetqTTdUczrFoO6QUsUge2CX7IXdsyv2xN5/reWHNape9mjWa1phF7qO+pbf/lWVaPaw86n607OHIiZCr5K82yFTvQWv6Sv7xy/Lk0uD/hA7Y8/k/5Q9sju6gVl55eeLYukEMfqA1Pfn/glWx5KpTDK9mE5Mz0Rf0YJ+DGCY3nsc05jHAnJ07iGucYNbpVUZVTLKRC1VqYs0vfgSSvYD54aV+g==</latexit>
Dice =
2|A B|
|A| + |B|

31. PASCAL VOC
COCO

32. PASCAL VOC
FCN 2014
NAS-FPN 2020
COCO

33. PASCAL VOC
COCO

34. n
•
• 3D
n
•
• FCN8 100ms
n3D
•
•

35. n
• 100 10
n
•
n2D 2.5D 3D
•