Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Improving Spatial Codification in Semantic Segmentation

564 views

Published on

https://imatge.upc.edu/web/publications/improving-spatial-codification-semantic-segmentation

Ventura C, Giró-i-Nieto X, Vilaplana V, McGuinness K, Marqués F, O'Connor N. Improving Spatial Codification in Semantic Segmentation. In: IEEE International Conference on Image Processing (ICIP), 2015. Quebec City: IEEE; 2015

This paper explores novel approaches for improving the spatial codification for the pooling of local descriptors to solve the semantic segmentation problem. We propose to partition the image into three regions for each object to be described: Figure, Border and Ground. This partition aims at minimizing the influence of the image context on the object description and vice versa by introducing an intermediate zone around the object contour. Furthermore, we also propose a richer visual descriptor of the object by applying a Spatial Pyramid over the Figure region. Two novel Spatial Pyramid configurations are explored: Cartesian-based and crown-based Spatial Pyramids. We test these approaches with state-of-the-art techniques and show that they improve the Figure-Ground based pooling in the Pascal VOC 2011 and 2012 semantic segmentation challenges.

Published in: Technology
  • Be the first to comment

Improving Spatial Codification in Semantic Segmentation

  1. 1. C. Ventura X. Giró-i-Nieto V. Vilaplana F. Marqués K. McGuinness N. O’Connor Improving Spatial Codification in Semantic Segmentation
  2. 2. Outline ● Introduction ● Related Work and Contributions ● Architecture ● Experiments ● Conclusions
  3. 3. Introduction ● Object segmentation vs Class segmentation Image Object segmentation Class segmentation 3
  4. 4. Outline ● Introduction ● Related Work and Contributions ● Architecture ● Experiments ● Conclusions
  5. 5. Related Work ● The Visual Extent of an Object [1] 5 [1] Uijlings et al, The VIsual Extent of an Object. IJCV’12
  6. 6. 1st Contribution ● Using a Figure-Border-Ground spatial pooling with object candidates 6 Figure-Ground spatial pooling Figure-Border-Ground spatial pooling
  7. 7. Related Work ● Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories [1] 7 [1] Lazebnik et al, Beyond bags of features: Spatial pyramid matching for recognizing natural scenes. CVPR’06
  8. 8. Related Work ● Variations of SPM ○ Non-arbitrary division ■ Object-centric pooling [1] ■ Object confidence map partition [2] ○ SPM over bounding boxes [3] [4] [1] Russakovky et al, Object-centric spatial pooling for image classification. ECCV’12 [2] Chen et al, Hierarchical Matching with Side Information for Image Classification. CVPR’12 [3] Arbeláez et at, Semantic segmentation using regions and parts. CVPR’12 [4] Gu et al, Multi-component models for object detection. ECCV’12 8
  9. 9. 2nd Contribution ● Applying a contour-based spatial pyramid (SP) ○ Crown-based SP ○ Cartesian-based SP 9 Crown-based spatial pyramid Cartesian-based spatial pyramid
  10. 10. Outline ● Introduction ● Related Work and Contributions ● Architecture ● Experiments ● Conclusions
  11. 11. Architecture ● Architecture proposed and released in [1] [1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC Ground Truth Train CPMC SIFT-based features (O2P) 11
  12. 12. Outline ● Introduction ● Related Work and Contributions ● Architecture ● Experiments ● Conclusions
  13. 13. Experiments ● Experiments with ideal object candidates ○ Train set: train11 ○ Test set: val11 F[1] F-B F-G[1] F-B-G eSIFT 63.9 66.2 66.4 68.6 eMSIFT 64.8 68.9 67.6 70.8 [1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 13
  14. 14. Experiments ● Experiments with ideal object candidates ○ Train set: train11 ○ Test set: val11 F F-B F-B-G non SP 64.8 [1] 68.9 70.8 crown-based SP 68.7 71.1 71.7 Cartesian-based SP 67.7 71.6 72.7 Figure SP(Figure) Border Ground AAC eSIFT+eMSIFT+eLBP eSIFT 72.98 [1] eSIFT+eMSIFT eMSIFT+eSIFT eMSIFT+eSIFT 73.84 eSIFT+eMSIFT+eLBP eMSIFT eMSIFT+eSIFT eMSIFT+eSIFT 75.86 [1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 14
  15. 15. Experiments ● Experiments with CPMC object candidates ○ Train set: train11 ○ Test set: val11 [1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 Figure SP(Figure) Border Ground AAC eSIFT eSIFT 28.6 [1] eSIFT eSIFT eSIFT 34.8 eSIFT+eMSIFT+eLBP eSIFT 37.2 [1] eSIFT eSIFT eSIFT eSIFT 37.4 eSIFT+eMSIFT+eLBP eSIFT eSIFT eSIFT 39.6 15
  16. 16. Experiments ● Experiments with CPMC object candidates in comp5 challenge ○ Train set: trainval11 / trainval12 ○ Test set: test11 / test12 F-G [1] F-B-G SP(F)-B-G PASCAL VOC11 38.8 43.8 40.3 PASCAL VOC12 39.9 42.2 40.8 [1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 16
  17. 17. Experiments ● Experiments with MCG object candidates [1] ○ Train set: train11 ○ Test set: val11 F-G[2] F-B-G SP(F)-B-G CPMC 37.2 38.9 39.6 MCG 30.9 34.1 36.1 [1] Arbeláez et al, Multiscale Combinatorial Grouping. CVPR’14 [2] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 17
  18. 18. Experiments ● Qualitative results F-B-G spatial pooling with CPMC 18 F-G F-B-G F-G F-B-G aeroplane bicycle bicycle birdcat motorbike boat bottle bus bus motorbike car chair cat chair chair horse cow bird
  19. 19. Experiments ● Qualitative results F-B-G spatial pooling with CPMC 19 chair dining table cow dog person horse person motorbike motorbike motorbike person plotted plant bottle sheep sofa dog bus train train tvmonitor F-G F-B-G F-G F-B-G
  20. 20. Outline ● Introduction ● Related Work and Contributions ● Architecture ● Experiments ● Conclusions
  21. 21. Conclusions ● 2 proposals beyond the classic Figure-Ground pooling ○ Figure-Border-Ground spatial pooling ■ Extended to realistic scenario with CPMC object candidates ○ A novel contour-based spatial pyramid has been introduced ■ Cartesian-based spatial pyramid ■ Crown-based spatial pyramid ● Validation of both proposals also for MCG object candidates 21
  22. 22. Related Work ● The Visual Extent of an Object (Uijlings et al, IJCV’12) 23

×