Efficient exploration of region hierarchies for semantic segmentation
•
0 likes•1,127 views
https://imatge.upc.edu/web/publications/efficient-exploration-region-hierarchies-semantic-segmentation The motivation of this work is the efficient exploration of hierarchical partitions for semantic segmentation as a method for locating objects in images. While many efforts have been focused on efficient image search in large-scale databases, few works have addressed the problem of locating and recognizing objects efficiently within a given image. My work considers as an input a hierarchical partition of an image that defines a set of regions as candidate locations to contain an object. This approach will be compared to other state of the art algorithms that extract object candidates for an image. The final goal of this work is to semantically segment images efficiently by exploiting the multiscale information provided by a hierarchical partition, maximizing the accuracy of the segmentation when only a very few regions of the partition are analysed.
Recommended
More Related Content
What's hot
What's hot (17)
Viewers also liked
Viewers also liked (20)
Similar to Efficient exploration of region hierarchies for semantic segmentation
Similar to Efficient exploration of region hierarchies for semantic segmentation (20)
More from Universitat Politècnica de Catalunya
More from Universitat Politècnica de Catalunya (20)
Recently uploaded
Recently uploaded (20)
Efficient exploration of region hierarchies for semantic segmentation
- 1. EFFICIENT EXPLORATION OF REGION HIERARCHIES FOR SEMANTIC SEGMENTATION Míriam Bellver Bueno Xavier Giró i Nieto Carles Ventura 1
- 2. Outline ● Motivation ● Related Work ● Methodology ● Results ● Conclusions and Future Work 2
- 3. Outline ● Motivation ● Related Work ● Methodology ● Results ● Conclusions and Future Work 3
- 4. Motivation Recognition Tasks Object Detection Content-based ImageRetrieval Medical Imaging 4
- 5. Motivation Recognition Tasks Object Detection Content-based ImageRetrieval Medical Imaging 5 Image Segmentation
- 6. Motivation Semantic Segmentation Segmentation Prediction Image Object Candidates Final segmentation 6
- 7. Motivation Semantic Segmentation Segmentation Prediction Image Object Candidates Final segmentation 7
- 8. Motivation Efficient Semantic Segmentation Segmentation Prediction Only a few regions The minimum computational time for the calculation of each region Image Object Candidates Final segmentation 8
- 9. Motivation Efficient Semantic Segmentation Segmentation Prediction Only a few regions The minimum computational time for the calculation of each region Image Object Candidates Final segmentation But what regions?? 9
- 10. Motivation Efficient Semantic Segmentation Segmentation Prediction Image Object Candidates Semantic Segmentation UCM hierarchical partitions CPMC object candidates MULTI-SCALE INFORMATION 10Carreira, J., & Sminchisescu, C. (2012). Cpmc: Automatic object segmentation using constrained parametric min-cuts.
- 11. Outline ● Motivation ● Related Work ● Methodology ● Results ● Conclusions and Future Work 11
- 12. Related Work 12 Object Detection and Recognition Sliding Windows Partition e.g. Viola Jones Hierarchical Flat e.g UCM e.g CPMC e.g Watershed Partition Object Proposals Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features.
- 13. Ultrametric Contour Map (UCM) 13 Original Image Ultrametric Contour Map Dendrogram Arbelaez, P. (2006, June). Boundary extraction in natural images using ultrametric contour maps root node leavescosts
- 14. Outline ● Motivation ● Related Work ● Methodology ● Results ● Conclusions and Future Work 14
- 16. Pipeline: Object Candidates Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC Ground Truth Train CPMC UCM 16
- 17. Pipeline: Features Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC Ground Truth Train SIFT-based features [O2P] [O2P] Carreira, Caseiro, Batista, Sminchisescu, “Semantic Segmentation with Second-Order Pooling” (ECCV 2012). 17
- 19. Assessment Average Accuracy per Category (AAC) / Intersection over Union (IoU) 19
- 20. Pipeline Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC Ground Truth Train CPMC UCM SIFT-based features [O2P] [O2P] Carreira, Caseiro, Batista, Sminchisescu, “Semantic Segmentation with Second-Order Pooling” (ECCV 2012) 20
- 21. Object candidates Class-agnostic exploration Class-dependent exploration Comparison Based on a ranked list (CPMC) Based on a partition tree (UCM) Contributions 21
- 24. Class-agnostic tree exploration: Indexes 24 521 525 easier to generate than its sibling… more homogeneous Merging Sequence 527
- 25. 3 7 6 11 Ranked List of Object Candidates 7 8 5 3 10 1 2 Max Queue 7 8 5 4 5 10 8 9 9 3 1 1 6 4 2 11 Class-agnostic tree exploration: Indexes 25
- 26. Class-agnostic tree exploration: Indexes 26 521 525 527
- 27. Class-agnostic tree exploration: Costs 27 521 525 527 511 495 MAX MIN SECOND DERIVATIVE 0.1 0.3 0.7
- 28. Class-agnostic tree exploration 28 Class-dependent tree exploration Class-agnostic tree exploration ... OBJECTS
- 29. Class-dependent tree exploration table chair plane sofa 0.15 0.05 0.60 0.05 table chair plane sofa 0.05 0.00 0.05 0.05 cow 0.01 ... ANALYSED IN QUEUE 29 confidence values confidence values cow 0.01 ...
- 30. Class-dependent tree exploration table chair plane sofa 0.05 0.05 0.70 0.00 table chair plane sofa 0.05 0.00 0.05 0.05 ANALYSED IN QUEUE 30 confidence values confidence values cow 0.01 ... cow 0.01 ...
- 31. Class-dependent tree exploration table chair plane sofa 0.05 0.05 0.65 0.00 table chair plane sofa 0.05 0.00 0.35 0.05 IN QUEUE IN QUEUE 31 confidence values confidence values cow 0.01 ... cow 0.01 ...
- 32. Outline ● Motivation ● Related Work ● Methodology ● Results ● Conclusions and Future Work 32
- 33. Results: SIFT-based features [O2P] 33 UCM Class-agnostic exploration UCM Class-dependent exploration CPMC
- 34. Pipeline Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC (IoU) Ground Truth Train CPMC UCM Deep learning features [SDS] SIFT-based features [O2P] [SDS] Hariharan, Arbeláze, Girshick, Malik, “Simulatenous Detection and Segmentation” (ECCV 2014) - More details on [Eduard Fontdevila BSc 2015]. 34
- 35. Results: Deep Learning feat. (SDS) 35 UCM Class-agnostic exploration UCM Class-dependent exploration CPMC Arbelaez, P., Pont-Tuset, J., Barron, J., Marques, F., & Malik, J. (2014, June). Multiscale combinatorial grouping Kuo, W., Hariharan, B., & Malik, J. (2015). DeepBox: Learning Objectness with Convolutional Networks CPMC ≈ x 8 UCMComputation Time
- 36. Outline ● Motivation ● Related Work ● Methodology ● Results ● Conclusions and Future Work 36
- 37. Conclusions 1. Better results when using a few regions of UCM compared to CPMC. 37 CPMCUCM
- 38. Conclusions 2. The class-dependent exploration of UCM regions is the best configuration for a budget of a few regions. 38 Class-dependent tree exploration ...
- 39. Conclusions 3. SDS descriptors extracted from a CNN obtain better results than O2P. 39 Deep learning features [SDS]hand-crafted features [O2P]
- 40. Future Work ● Class-dependent tree exploration using two classifiers ● Compare performance using different object candidates, such as MCG. 40 Is there a face on this node? Is this a face? 1 2 X. Giró, 2012, Part-based object retrieval with binary partition trees. Arbelaez, P., Pont-Tuset, J., Barron, J., Marques, F., & Malik, J. (2014, June). Multiscale combinatorial grouping
- 41. 41
- 42. 42
- 43. Related Work 43 Object Detection and Recognition Sliding Windows Segmentation e.g. Viola Jones Hierarchical Segmentation Flat segmentation e.g UCM Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. e.g CPMC
- 44. Related Work: Tree exploration Regions generated from a hierarchical partition taking advantage of its multi- scale information in order to guide an efficient exploration throughout the tree. X. Giró, 2012, Part- based object retrieval with binary partition trees. X. Giró, 2012, Part-based object retrieval with binary partition trees. 44
- 45. Constrained Parametric Min-Cuts (CPMC) Carreira, J., & Sminchisescu, C. (2012). Cpmc: Automatic object segmentation using constrained parametric min-cuts. 45
- 46. Motivation Local Feature Descriptors: SIFT HOG Learned Descriptors From hand-crafted to learned features ~1995 to ~2005 ~2005 to ~2010 ~2010 to ~2015 Feature visualization of convolutional net trained on ImageNet from [Zeiler & Fergus 2013] hand-crafted descriptors 46
- 47. Features: SDS features [SDS] Hariharan, Arbeláze, Girshick, Malik, “Simulatenous Detection and Segmentation” (ECCV 2014) - More details on [Eduard Fontdevila BSc 2015]. 47
- 48. Features: Second Order Pooling (O2P) Average Pooling Max Pooling 2nd order SIFT O2PSIFT Second Order SIFT Pooling Carreira, J., Caseiro, R., Batista, J., & Sminchisescu, C. (2012). Semantic segmentation with second-order pooling. 48
- 49. Results: Deep Learning feat. (O2P) 49
- 50. 50
- 51. Class-agnostic tree exploration: Costs Big difference of cost between node and its father Small difference of cost between node and one of its children 51 SECOND DERIVATIVE
- 52. Class-agnostic tree exploration Second derivatives costs associated to consecutive nodes yield good results 52
- 53. Class-agnostic tree exploration: Costs Big difference of cost between node and its father Small difference of cost between node and one of its children 53
- 54. 54
- 55. Class-agnostic tree exploration: Indexes Objects can be found in regions associated to indexes that differ from the indexes of their adjacent regions 150 149 147 cost easier to generate than its sibling… more homogeneous indexes 55 521
- 56. Database 56
- 57. Class-agnostic tree exploration Contours of UCM Merging Sequence INDEXES of the merging sequence COSTS values of the contours Input image Based on the structure of the UCM partition, defined by these two files: 1 2 57
- 58. 58
- 59. Class-dependent tree exploration Guide a top-down efficient exploration throughout the tree based on the classifier’s decision. X. Giró, 2012, Part-based object retrieval with binary partition trees. Motivation 59
- 60. 60
- 61. 61
- 62. 62
- 63. Related Work 63 Object Detection and Recognition Sliding Windows Segmentation e.g. Viola Jones Hierarchical Segmentation Flat segmentation e.g UCM Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. e.g CPMC
- 64. Related Work 64 Object Detection and Recognition Sliding Windows Segmentation e.g. Viola Jones Hierarchical Segmentation Flat segmentation e.g UCM Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. e.g CPMC
- 65. Pipeline: Features Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC Ground Truth Train Deep learning features [SDS] SIFT-based features [O2P] [O2P] Carreira, Caseiro, Batista, Sminchisescu, “Semantic Segmentation with Second-Order Pooling” (ECCV 2012) [SDS] Hariharan, Arbeláze, Girshick, Malik, “Simulatenous Detection and Segmentation” (ECCV 2014) - More details on [Eduard Fontdevila BSc 2015]. 65
- 66. Pipeline Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC (IoU) Ground Truth Train CPMC UCM Deep learning features [SDS] SIFT-based features [O2P] [O2P] Carreira, Caseiro, Batista, Sminchisescu, “Semantic Segmentation with Second-Order Pooling” (ECCV 2012) [SDS] Hariharan, Arbeláze, Girshick, Malik, “Simulatenous Detection and Segmentation” (ECCV 2014) - More details on [Eduard Fontdevila BSc 2015]. 66
- 67. 67
- 68. Motivation Recognition Tasks Object Detection Content-based ImageRetrieval Medical Imaging 68
- 69. Motivation Recognition Tasks Object Detection Content-based ImageRetrieval Medical Imaging 69 Image Segmentation
- 70. Motivation Goal: Guide a top-down exploration of a hierarchical partition by answering the following question: ● Does this region contain the object we are seeking? ● If so, does this region represent the object we are seeking? 70
- 71. Motivation 71
- 73. 73
- 74. Acknowledgments Technical support Albert Gil Josep Pujal 74