Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Efficient exploration of region hierarchies for semantic segmentation

https://imatge.upc.edu/web/publications/efficient-exploration-region-hierarchies-semantic-segmentation

The motivation of this work is the efficient exploration of hierarchical partitions for semantic segmentation as a method for locating objects in images. While many efforts have been focused on efficient image search in large-scale databases, few works have addressed the problem of locating and recognizing objects efficiently within a given image. My work considers as an input a hierarchical partition of an image that defines a set of regions as candidate locations to contain an object. This approach will be compared to other state of the art algorithms that extract object candidates for an image. The final goal of this work is to semantically segment images efficiently by exploiting the multiscale information provided by a hierarchical partition, maximizing the accuracy of the segmentation when only a very few regions of the partition are analysed.

  • Login to see the comments

  • Be the first to like this

Efficient exploration of region hierarchies for semantic segmentation

  1. 1. EFFICIENT EXPLORATION OF REGION HIERARCHIES FOR SEMANTIC SEGMENTATION Míriam Bellver Bueno Xavier Giró i Nieto Carles Ventura 1
  2. 2. Outline ● Motivation ● Related Work ● Methodology ● Results ● Conclusions and Future Work 2
  3. 3. Outline ● Motivation ● Related Work ● Methodology ● Results ● Conclusions and Future Work 3
  4. 4. Motivation Recognition Tasks Object Detection Content-based ImageRetrieval Medical Imaging 4
  5. 5. Motivation Recognition Tasks Object Detection Content-based ImageRetrieval Medical Imaging 5 Image Segmentation
  6. 6. Motivation Semantic Segmentation Segmentation Prediction Image Object Candidates Final segmentation 6
  7. 7. Motivation Semantic Segmentation Segmentation Prediction Image Object Candidates Final segmentation 7
  8. 8. Motivation Efficient Semantic Segmentation Segmentation Prediction Only a few regions The minimum computational time for the calculation of each region Image Object Candidates Final segmentation 8
  9. 9. Motivation Efficient Semantic Segmentation Segmentation Prediction Only a few regions The minimum computational time for the calculation of each region Image Object Candidates Final segmentation But what regions?? 9
  10. 10. Motivation Efficient Semantic Segmentation Segmentation Prediction Image Object Candidates Semantic Segmentation UCM hierarchical partitions CPMC object candidates MULTI-SCALE INFORMATION 10Carreira, J., & Sminchisescu, C. (2012). Cpmc: Automatic object segmentation using constrained parametric min-cuts.
  11. 11. Outline ● Motivation ● Related Work ● Methodology ● Results ● Conclusions and Future Work 11
  12. 12. Related Work 12 Object Detection and Recognition Sliding Windows Partition e.g. Viola Jones Hierarchical Flat e.g UCM e.g CPMC e.g Watershed Partition Object Proposals Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features.
  13. 13. Ultrametric Contour Map (UCM) 13 Original Image Ultrametric Contour Map Dendrogram Arbelaez, P. (2006, June). Boundary extraction in natural images using ultrametric contour maps root node leavescosts
  14. 14. Outline ● Motivation ● Related Work ● Methodology ● Results ● Conclusions and Future Work 14
  15. 15. Pipeline: Database Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC Ground Truth Train 15
  16. 16. Pipeline: Object Candidates Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC Ground Truth Train CPMC UCM 16
  17. 17. Pipeline: Features Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC Ground Truth Train SIFT-based features [O2P] [O2P] Carreira, Caseiro, Batista, Sminchisescu, “Semantic Segmentation with Second-Order Pooling” (ECCV 2012). 17
  18. 18. Pipeline: Assessment Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC Ground Truth Train 18
  19. 19. Assessment Average Accuracy per Category (AAC) / Intersection over Union (IoU) 19
  20. 20. Pipeline Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC Ground Truth Train CPMC UCM SIFT-based features [O2P] [O2P] Carreira, Caseiro, Batista, Sminchisescu, “Semantic Segmentation with Second-Order Pooling” (ECCV 2012) 20
  21. 21. Object candidates Class-agnostic exploration Class-dependent exploration Comparison Based on a ranked list (CPMC) Based on a partition tree (UCM) Contributions 21
  22. 22. Class-agnostic tree exploration 22 COSTS INDEXES 1 2
  23. 23. Class-agnostic tree exploration 23
  24. 24. Class-agnostic tree exploration: Indexes 24 521 525 easier to generate than its sibling… more homogeneous Merging Sequence 527
  25. 25. 3 7 6 11 Ranked List of Object Candidates 7 8 5 3 10 1 2 Max Queue 7 8 5 4 5 10 8 9 9 3 1 1 6 4 2 11 Class-agnostic tree exploration: Indexes 25
  26. 26. Class-agnostic tree exploration: Indexes 26 521 525 527
  27. 27. Class-agnostic tree exploration: Costs 27 521 525 527 511 495 MAX MIN SECOND DERIVATIVE 0.1 0.3 0.7
  28. 28. Class-agnostic tree exploration 28 Class-dependent tree exploration Class-agnostic tree exploration ... OBJECTS
  29. 29. Class-dependent tree exploration table chair plane sofa 0.15 0.05 0.60 0.05 table chair plane sofa 0.05 0.00 0.05 0.05 cow 0.01 ... ANALYSED IN QUEUE 29 confidence values confidence values cow 0.01 ...
  30. 30. Class-dependent tree exploration table chair plane sofa 0.05 0.05 0.70 0.00 table chair plane sofa 0.05 0.00 0.05 0.05 ANALYSED IN QUEUE 30 confidence values confidence values cow 0.01 ... cow 0.01 ...
  31. 31. Class-dependent tree exploration table chair plane sofa 0.05 0.05 0.65 0.00 table chair plane sofa 0.05 0.00 0.35 0.05 IN QUEUE IN QUEUE 31 confidence values confidence values cow 0.01 ... cow 0.01 ...
  32. 32. Outline ● Motivation ● Related Work ● Methodology ● Results ● Conclusions and Future Work 32
  33. 33. Results: SIFT-based features [O2P] 33 UCM Class-agnostic exploration UCM Class-dependent exploration CPMC
  34. 34. Pipeline Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC (IoU) Ground Truth Train CPMC UCM Deep learning features [SDS] SIFT-based features [O2P] [SDS] Hariharan, Arbeláze, Girshick, Malik, “Simulatenous Detection and Segmentation” (ECCV 2014) - More details on [Eduard Fontdevila BSc 2015]. 34
  35. 35. Results: Deep Learning feat. (SDS) 35 UCM Class-agnostic exploration UCM Class-dependent exploration CPMC Arbelaez, P., Pont-Tuset, J., Barron, J., Marques, F., & Malik, J. (2014, June). Multiscale combinatorial grouping Kuo, W., Hariharan, B., & Malik, J. (2015). DeepBox: Learning Objectness with Convolutional Networks CPMC ≈ x 8 UCMComputation Time
  36. 36. Outline ● Motivation ● Related Work ● Methodology ● Results ● Conclusions and Future Work 36
  37. 37. Conclusions 1. Better results when using a few regions of UCM compared to CPMC. 37 CPMCUCM
  38. 38. Conclusions 2. The class-dependent exploration of UCM regions is the best configuration for a budget of a few regions. 38 Class-dependent tree exploration ...
  39. 39. Conclusions 3. SDS descriptors extracted from a CNN obtain better results than O2P. 39 Deep learning features [SDS]hand-crafted features [O2P]
  40. 40. Future Work ● Class-dependent tree exploration using two classifiers ● Compare performance using different object candidates, such as MCG. 40 Is there a face on this node? Is this a face? 1 2 X. Giró, 2012, Part-based object retrieval with binary partition trees. Arbelaez, P., Pont-Tuset, J., Barron, J., Marques, F., & Malik, J. (2014, June). Multiscale combinatorial grouping
  41. 41. 41
  42. 42. 42
  43. 43. Related Work 43 Object Detection and Recognition Sliding Windows Segmentation e.g. Viola Jones Hierarchical Segmentation Flat segmentation e.g UCM Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. e.g CPMC
  44. 44. Related Work: Tree exploration Regions generated from a hierarchical partition taking advantage of its multi- scale information in order to guide an efficient exploration throughout the tree. X. Giró, 2012, Part- based object retrieval with binary partition trees. X. Giró, 2012, Part-based object retrieval with binary partition trees. 44
  45. 45. Constrained Parametric Min-Cuts (CPMC) Carreira, J., & Sminchisescu, C. (2012). Cpmc: Automatic object segmentation using constrained parametric min-cuts. 45
  46. 46. Motivation Local Feature Descriptors: SIFT HOG Learned Descriptors From hand-crafted to learned features ~1995 to ~2005 ~2005 to ~2010 ~2010 to ~2015 Feature visualization of convolutional net trained on ImageNet from [Zeiler & Fergus 2013] hand-crafted descriptors 46
  47. 47. Features: SDS features [SDS] Hariharan, Arbeláze, Girshick, Malik, “Simulatenous Detection and Segmentation” (ECCV 2014) - More details on [Eduard Fontdevila BSc 2015]. 47
  48. 48. Features: Second Order Pooling (O2P) Average Pooling Max Pooling 2nd order SIFT O2PSIFT Second Order SIFT Pooling Carreira, J., Caseiro, R., Batista, J., & Sminchisescu, C. (2012). Semantic segmentation with second-order pooling. 48
  49. 49. Results: Deep Learning feat. (O2P) 49
  50. 50. 50
  51. 51. Class-agnostic tree exploration: Costs Big difference of cost between node and its father Small difference of cost between node and one of its children 51 SECOND DERIVATIVE
  52. 52. Class-agnostic tree exploration Second derivatives costs associated to consecutive nodes yield good results 52
  53. 53. Class-agnostic tree exploration: Costs Big difference of cost between node and its father Small difference of cost between node and one of its children 53
  54. 54. 54
  55. 55. Class-agnostic tree exploration: Indexes Objects can be found in regions associated to indexes that differ from the indexes of their adjacent regions 150 149 147 cost easier to generate than its sibling… more homogeneous indexes 55 521
  56. 56. Database 56
  57. 57. Class-agnostic tree exploration Contours of UCM Merging Sequence INDEXES of the merging sequence COSTS values of the contours Input image Based on the structure of the UCM partition, defined by these two files: 1 2 57
  58. 58. 58
  59. 59. Class-dependent tree exploration Guide a top-down efficient exploration throughout the tree based on the classifier’s decision. X. Giró, 2012, Part-based object retrieval with binary partition trees. Motivation 59
  60. 60. 60
  61. 61. 61
  62. 62. 62
  63. 63. Related Work 63 Object Detection and Recognition Sliding Windows Segmentation e.g. Viola Jones Hierarchical Segmentation Flat segmentation e.g UCM Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. e.g CPMC
  64. 64. Related Work 64 Object Detection and Recognition Sliding Windows Segmentation e.g. Viola Jones Hierarchical Segmentation Flat segmentation e.g UCM Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. e.g CPMC
  65. 65. Pipeline: Features Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC Ground Truth Train Deep learning features [SDS] SIFT-based features [O2P] [O2P] Carreira, Caseiro, Batista, Sminchisescu, “Semantic Segmentation with Second-Order Pooling” (ECCV 2012) [SDS] Hariharan, Arbeláze, Girshick, Malik, “Simulatenous Detection and Segmentation” (ECCV 2014) - More details on [Eduard Fontdevila BSc 2015]. 65
  66. 66. Pipeline Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC (IoU) Ground Truth Train CPMC UCM Deep learning features [SDS] SIFT-based features [O2P] [O2P] Carreira, Caseiro, Batista, Sminchisescu, “Semantic Segmentation with Second-Order Pooling” (ECCV 2012) [SDS] Hariharan, Arbeláze, Girshick, Malik, “Simulatenous Detection and Segmentation” (ECCV 2014) - More details on [Eduard Fontdevila BSc 2015]. 66
  67. 67. 67
  68. 68. Motivation Recognition Tasks Object Detection Content-based ImageRetrieval Medical Imaging 68
  69. 69. Motivation Recognition Tasks Object Detection Content-based ImageRetrieval Medical Imaging 69 Image Segmentation
  70. 70. Motivation Goal: Guide a top-down exploration of a hierarchical partition by answering the following question: ● Does this region contain the object we are seeking? ● If so, does this region represent the object we are seeking? 70
  71. 71. Motivation 71
  72. 72. Motivation Recognition Tasks Object Detection 72
  73. 73. 73
  74. 74. Acknowledgments Technical support Albert Gil Josep Pujal 74

×