발표자: 이세호(고려대 박사과정) 발표일: 2017.8. 개요: 슈퍼픽셀 알고리즘은 입력 영상을 다수의 의미 있는 영역으로 과분할 하는 기법이다. 입력 영상을 픽셀 단위로 표현할 때와 비교하여, 슈퍼픽셀 단위의 표현은 입력 영상의 단위의 수를 크게 줄이는 장점이 있다. 각 슈퍼픽셀은 객체의 윤곽선을 넘어서는 영역을 포함하지 않는 동시에, 단일 객체만을 담아야 한다. 본 발표에서는 객체의 윤곽선 정보를 고려한 윤곽선 제약 슈퍼픽셀 기법(contour-constrained superpixel algorithm)을 제안한다.

1. Contour-Constrained Superpixels for Image and Video
Processing
Se-Ho Lee, Won-Dong Jang, and Chang-Su Kim
Korea University
CVPR 2017

2. Overview
• Introduction
• Contour-constrained superpixels (CCS)
• Superpixel labeling
• Contour constraint
• Hierarchical superpixel refinement
• Temporal superpixels
• Temporal superpixel labeling
• Merging, splitting, and relabeling
• Experimental results
• Conclusions

3. • Superpixel
• Over-segment input images into a set of meaningful regions
• Can reduce the number of image primitives or units greatly
• Superpixels should adhere to object contours
Introduction

4. Superpixel
• Applications
• Preprocessing technique for many vision applications
• ex) Saliency detection and image segmentation

5. Superpixel
• Conventional methods
• K-means based methods
• Assign each pixel to the nearest cluster and update the cluster
center iteratively
• SLIC (TPAMI’12), LSC (CVPR’15), and MSLIC (CVPR’16)
• Coarse-to-fine methods
• Change the superpixel label of boundary regions iteratively based
on the cost function
• From block-level to pixel-level
• SEEDS (IJCV’15) and Yao et al. (CVPR’15)

6. Contour-Constrained Superpixels
• Contributions
• Introduction of the contour constraint
• Extension of the proposed algorithm for video processing
• Remarkable performance achievement and improvement of
many computer vision algorithms

7. Contour-Constrained Superpixels
• Coarse-to-fine refinement
• From block-level to pixel-level
• Hierarchical block structure

8. Contour-Constrained Superpixels
• Superpixel labeling
• Hierarchical block structure
• From block to pixel levels
• Divide only inhomogeneous regions
• Can maintain relatively regular and compact shape

9. Contour-Constrained Superpixels
• Superpixel labeling
• Change the label of region (block or pixel) 𝑅𝑖 from 𝑙 𝑅𝑖 to
𝑙 𝑅𝑗 of neighboring region 𝑅𝑗 ∈ 𝒩𝑅 𝑖
• Change the label of region by minimizing the cost function
• 𝐸 𝑖, 𝑗 = 𝐸D 𝑖, 𝑗 + 𝛾𝐸L 𝑖, 𝑗 + 𝜂𝐸I 𝑖, 𝑗 × 𝐸C 𝑖, 𝑗
• To preserve the topology of superpixels
• Check the boundary region 𝑅𝑖 is simple point
Initial Level 1 Level 2 Level 3 Level 4

10. Superpixel Labeling
• Feature distance from superpixel centroid
• 𝐸D 𝑖, 𝑗 = 𝐜 𝑅𝑖 − 𝐜 𝑆𝑙 𝑅 𝑗
2
+ 𝐩 𝑅𝑖 − 𝐩 𝑆𝑙 𝑅 𝑗
2
• Color distance term makes the color of each superpixel to be regular
• Spatial distance term imposes the superpixels to be distributed
compactly
Color distance Spatial distance

11. Superpixel Labeling
• Boundary length cost
• 𝐸L 𝑖, 𝑗 = 𝜆 𝑅𝑖, 𝑙 𝑅𝑗 − 𝜆 𝑅𝑖, 𝑙 𝑅𝑖
• 𝜆 𝑅𝑖, 𝑘 : total number of boundary regions when the superpixel
label of 𝑅𝑖 is 𝑘
• Minimize the boundary length of superpixels
• Make the superpixels distributed compactly

12. Superpixel Labeling
• Inter-region color cost
• 𝐸I 𝑖, 𝑗 = max 0, 𝐜 𝑅𝑖 − 𝐜 𝑅𝑗
2
− 𝜅 𝑆𝑙 𝑅 𝑗
• Neighboring regions with dissimilar color information
→ Assign different superpixel labels
• Internal difference
• Measure the amount of texture information of superpixels
• 𝜅 𝑆𝑙 𝑅 𝑖
= max
𝑅 𝑚,𝑅 𝑛∈𝑆𝑙 𝑅 𝑖
,
𝑅 𝑛∈𝒩 𝑅 𝑚
𝐜 𝑅 𝑚 − 𝐜 𝑅 𝑛
2
• Maximum color difference between neighboring regions within 𝑆𝑙 𝑅 𝑖

13. Superpixel Labeling
• Contour constraint 𝐸C 𝑖, 𝑗
• Amplify the cost function when there is an object contour
between two regions
• Use holistically-nested edge detection (HED)*
• Deep learning based edge detection method
𝐸 𝑖, 𝑗 = 𝐸D 𝑖, 𝑗 + 𝛾𝐸L 𝑖, 𝑗 + 𝜂𝐸I 𝑖, 𝑗 × 𝐸C 𝑖, 𝑗
* S. Xie and Z. Tu, Holistically-nested edge detection. In ICCV, pages 1395-1403, 2015.

14. Contour Constraint
• Contour pattern set extraction
• Use 200 training images in the BSDS500 dataset
• Denote patch centered at each contour pixel as a contour
pattern
• Select top 1,000 frequently occurring patterns

15. Contour Constraint
• Contour pattern set
• Only consider the patterns which divide the patch into two regions
• 1,000 patterns cover 90.5% of the patches in the training dataset

16. Contour Constraint
• Contour pattern matching
• Get thin binary contour map
• Perform non-maximum suppression and then do thresholding
• Compute Hamming distances from the contour patterns and
select the best matching pattern

17. Contour Constraint
• Contour probability 𝜙 𝑢, 𝑣
• Difficult to estimate the existence of object contours only
considering the thin contour map
• Measure the proportion of patches whose matching patterns
separate 𝑢 from 𝑣
• Consider patches containing both pixels 𝑢 and 𝑣

18. Contour Constraint
• Determining contour constraint 𝐸C 𝑖, 𝑗
• Contour probability 𝜓 𝑅𝑖, 𝑅𝑗 between regions
• Find the maximum contour probability between the pixels in 𝑅𝑖
and 𝑅𝑗
• 𝜓 𝑅𝑖, 𝑅𝑗 = max
𝑢∈𝑅 𝑖,𝑣∈𝑅 𝑗
𝜙 𝑢, 𝑣
• 𝐸C 𝑖, 𝑗 = exp 𝛽 × 𝜓 𝑅𝑖, 𝑅𝑗

19. Hierarchical Superpixel Refinement
• Hierarchical block structure
• Divide only inhomogeneous regions into four blocks
• Dissimilarity function
• 𝜃 𝑅𝑖 = max
𝑢,𝑣∈𝑅 𝑖,𝑣∈𝒩𝑢
𝐜 𝑢 − 𝐜 𝑣 2 + exp 𝛽 × max
𝑢,𝑣∈𝑅 𝑖,𝑣∈𝒩𝑢
𝜙 𝑢, 𝑣
• Divide the region 𝑅𝑖 if 𝜃 𝑅𝑖 > 𝜏div

20. Contour-Constrained Superpixels
• Algorithm

21. Temporal Superpixels
• Superpixels for videos
• Get temporally consistent superpixels
Input SLIC CCS

22. Temporal Superpixels
• Initialization
• To make temporally consistent superpixels
• Estimate optical flows from 𝐼 𝑡−1
to 𝐼 𝑡
• Transfer the label of each superpixel by employing the average
optical flow of the superpixel
• Do not assign any labels to occluded or disoccluded pixels
• Occluded pixel: a pixel mapped from multiple superpixels
• Disoccluded pixel: a pixel mapped from no superpixel

23. Temporal Superpixels
• Temporal superpixel labeling
• Performed at the pixel level only
• Based on the energy function
• 𝐸 𝑖, 𝑗 = 𝐸D 𝑖, 𝑗 + 𝛾𝐸L 𝑖, 𝑗 + 𝜂𝐸I 𝑖, 𝑗 × 𝐸T 𝑖, 𝑗
• 𝐸T 𝑖, 𝑗 : temporal contour constraint

24. Temporal Superpixel Labeling
• Temporal Contour constraint
• Make superpixels temporally consistent
• Make superpixels compatible with image contours
• 𝐸T 𝑖, 𝑗, 𝑡 = 𝐸C 𝑖, 𝑗, 𝑡 × 𝜌 𝑖, 𝑗, 𝑡
• Relaxation factor 𝜌 𝑖, 𝑗, 𝑡
• 𝜌 𝑖, 𝑗, 𝑡 = ൞
1
1+exp −𝜁×ℎ 𝑅𝑖
𝑡
, if 𝑙 𝑅𝑗
𝑡
∈ ℒ 𝑖
𝑡
1, otherwise
• ℒ 𝑖
𝑡
is the set of labels that are mapped to 𝑅𝑖
𝑡
from 𝐼 𝑡−1
• Relax 𝐸T 𝑖, 𝑗, 𝑡 when 𝑅𝑖
𝑡
does not contain object contour

25. Merging, Splitting, and Relabeling
• Merging and splitting
• Prevent irregular superpixel size
• When Τ𝐴 𝑘
𝑡 ҧ𝐴 is larger than 𝜏spl → split
• Consider the biggest eigenvector of the spatial distribution
• When Τ𝐴 𝑘
𝑡 ҧ𝐴 is smaller than 𝜏mer → merge
• Merge to the nearest superpixel

26. Merging, Splitting, and Relabeling
• Relabeling
• Avoid incorrect labeling
• Because of occlusion or illumination variation
• Define color consistency 𝐶 𝑘
• 𝐶 𝑘 = 𝐜1:𝑡−1 𝑆 𝑘 − 𝐜 𝑡 𝑆 𝑘
2
• If 𝐶 𝑘 is larger than 𝜏rel → relabel

27. Temporal Superpixels
• Algorithm

28. Experimental Results
• Superpixel results
• On BSDS500 dataset
• CCS-wo-CC denotes proposed algorithm without the contour
constraint
ASA ↑ BR ↑ UE ↓

29. Experimental Results
• Temporal superpixel results
• On SegTrack dataset
SA2D ↑ BR2D ↑ UE2D ↓
SA3D ↑ BR3D ↑ UE3D ↓

30. Experimental Results
• Visual comparison of superpixel methods
Input SEEDS LSC Proposed

31. Experimental Results
• Visual comparison of temporal superpixel methods
TCS TSP Proposed

32. Experimental Results
• Run-times comparison
• Superpixel algorithms
• Temporal superpixel algorithms
Turbopixels SLIC ERS LSC SEEDS MSLIC CCS
Times(s) 8.09 0.26 1.52 0.34 0.06 0.36 0.97
sGBH SLIC TCS TSP CCS
Times(s) 5.71 0.08 7.83 2.39 1.70

33. Experimental Results
• Application to video object segmentation
• To superpixel-based video object segmentation method*
• Use CCS instead of SLIC as a preprocessing
• Intersection over union (IoU) is increased from 0.532 to 0.571
* W.-D. Jang and C.-S. Kim, Semi-supervised video object segmentation using multiple random walkers. In BMVC, 2016.

34. Experimental Results
• Application to video saliency detection
• Postprocessing
• Average the saliency values of the pixels in all frames, constituting
each superpixel
• Applied to HS* and DHSNet**
* Q. Yan, L. Xu, J. Shi, and J. Jia, Hierarchical saliency detection. In CVPR, 2013.
** N. Liu and J. Han, DHSNet: Deep hierarchical saliency network for salient object detection. In CVPR, 2016.

35. Experimental Results

36. Conclusions
• Contour-constrained superpixels (CCS)
• Perform hierarchical refinement from block to pixel levels
• Based on the contour constraint
• Temporal superpixels
• CCS algorithm for video processing
• Use optical flow to obtain temporally consistent superpixels
• Experimental results
• CCS outperforms the state-of-the-art superpixel methods
• Can be applied to object segmentation and saliency detection

37. Thank You
Q & A