This document summarizes an approach to improving spatial codification in semantic segmentation. It proposes using figure-border-ground spatial pooling with object candidates as well as applying a contour-based spatial pyramid, including crown-based and cartesian-based variations. Experiments show these approaches improve accuracy over traditional spatial pooling, especially when using CPMC and MCG object candidates, achieving state-of-the-art performance on PASCAL VOC 2011 and 2012 datasets.

1. C. Ventura X. Giró-i-Nieto V. Vilaplana F. Marqués K. McGuinness N. O’Connor
Improving Spatial Codification in
Semantic Segmentation

2. Outline
● Introduction
● Related Work and Contributions
● Architecture
● Experiments
● Conclusions

3. Introduction
● Object segmentation vs Class segmentation
Image Object
segmentation
Class
segmentation
3

4. Outline
● Introduction
● Related Work and Contributions
● Architecture
● Experiments
● Conclusions

5. Related Work
● The Visual Extent of
an Object [1]
5
[1] Uijlings et al, The VIsual Extent of an Object. IJCV’12

6. 1st Contribution
● Using a Figure-Border-Ground spatial pooling with object
candidates
6
Figure-Ground
spatial pooling
Figure-Border-Ground
spatial pooling

7. Related Work
● Beyond bags of features:
Spatial pyramid matching
for recognizing natural
scene categories [1]
7
[1] Lazebnik et al, Beyond bags of features: Spatial pyramid matching for recognizing natural scenes. CVPR’06

8. Related Work
● Variations of SPM
○ Non-arbitrary division
■ Object-centric pooling [1]
■ Object confidence map
partition [2]
○ SPM over bounding boxes [3]
[4]
[1] Russakovky et al, Object-centric spatial pooling for image classification. ECCV’12
[2] Chen et al, Hierarchical Matching with Side Information for Image Classification. CVPR’12
[3] Arbeláez et at, Semantic segmentation using regions and parts. CVPR’12
[4] Gu et al, Multi-component models for object detection. ECCV’12
8

9. 2nd Contribution
● Applying a contour-based spatial pyramid (SP)
○ Crown-based SP
○ Cartesian-based SP
9
Crown-based
spatial pyramid
Cartesian-based
spatial pyramid

10. Outline
● Introduction
● Related Work and Contributions
● Architecture
● Experiments
● Conclusions

11. Architecture
● Architecture proposed and released in [1]
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12
Train Test
DataBase
Object
Candidates
Feature
Extraction
Test
Model
Prediction
Evaluation
AAC
Ground
Truth
Train
CPMC SIFT-based
features (O2P)
11

12. Outline
● Introduction
● Related Work and Contributions
● Architecture
● Experiments
● Conclusions

13. Experiments
● Experiments with ideal object
candidates
○ Train set: train11
○ Test set: val11
F[1] F-B F-G[1] F-B-G
eSIFT 63.9 66.2 66.4 68.6
eMSIFT 64.8 68.9 67.6 70.8
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 13

14. Experiments
● Experiments with ideal object
candidates
○ Train set: train11
○ Test set: val11
F F-B F-B-G
non SP 64.8 [1] 68.9 70.8
crown-based SP 68.7 71.1 71.7
Cartesian-based SP 67.7 71.6 72.7
Figure SP(Figure) Border Ground AAC
eSIFT+eMSIFT+eLBP eSIFT 72.98 [1]
eSIFT+eMSIFT eMSIFT+eSIFT eMSIFT+eSIFT 73.84
eSIFT+eMSIFT+eLBP eMSIFT eMSIFT+eSIFT eMSIFT+eSIFT 75.86
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12
14

15. Experiments
● Experiments with CPMC object candidates
○ Train set: train11
○ Test set: val11
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12
Figure SP(Figure) Border Ground AAC
eSIFT eSIFT 28.6 [1]
eSIFT eSIFT eSIFT 34.8
eSIFT+eMSIFT+eLBP eSIFT 37.2 [1]
eSIFT eSIFT eSIFT eSIFT 37.4
eSIFT+eMSIFT+eLBP eSIFT eSIFT eSIFT 39.6
15

16. Experiments
● Experiments with CPMC object
candidates in comp5 challenge
○ Train set: trainval11 /
trainval12
○ Test set: test11 / test12
F-G [1] F-B-G SP(F)-B-G
PASCAL
VOC11
38.8 43.8 40.3
PASCAL
VOC12
39.9 42.2 40.8
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12
16

17. Experiments
● Experiments with MCG object
candidates [1]
○ Train set: train11
○ Test set: val11
F-G[2] F-B-G SP(F)-B-G
CPMC 37.2 38.9 39.6
MCG 30.9 34.1 36.1
[1] Arbeláez et al, Multiscale Combinatorial Grouping. CVPR’14
[2] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 17

18. Experiments
● Qualitative results F-B-G spatial pooling with CPMC
18
F-G F-B-G F-G F-B-G
aeroplane
bicycle bicycle
birdcat
motorbike boat
bottle
bus
bus
motorbike
car
chair
cat
chair
chair
horse
cow
bird

19. Experiments
● Qualitative results F-B-G spatial pooling with CPMC
19
chair
dining table
cow
dog
person
horse
person motorbike
motorbike
motorbike
person
plotted plant bottle
sheep
sofa
dog
bus
train train
tvmonitor
F-G F-B-G F-G F-B-G

20. Outline
● Introduction
● Related Work and Contributions
● Architecture
● Experiments
● Conclusions

21. Conclusions
● 2 proposals beyond the classic Figure-Ground pooling
○ Figure-Border-Ground spatial pooling
■ Extended to realistic scenario with CPMC object
candidates
○ A novel contour-based spatial pyramid has been introduced
■ Cartesian-based spatial pyramid
■ Crown-based spatial pyramid
● Validation of both proposals also for MCG object candidates
21

23. Related Work
● The Visual
Extent of an
Object (Uijlings
et al, IJCV’12)
23