발표자: 이준태 (고려대 박사과정) 발표일: 2017.9. 개요: Algorithm to semantic line detection and its applications will be presented. First, I will introduce the concept of semantic line. Second, semantic line detection method will be described. Then, two applications will be presented: composition enhancement and image simplification.

1. Semantic Line Detection and
Its Applications
ICCV 2017
Jun-Tae Lee
2017. 09. 13

2. Contents
• What is semantic line?
• Semantic line detection
• Semantic line network (SLNet)
• Primary semantic line
• Multiple semantic line
• Results
• Applications
• Composition enhancement
• Image simplification

3. Semantic Line
• Semantic lines characterize the layout of an image
Diagonal line
Vertical line
Horizontal line
Symmetric line

4. Applications of Semantic Line
• Composition enhancement

5. Applications of Semantic Line
• Image simplification

6. Semantic Line Network (SLNet)
• Network architecture
Image
VGG16 network
Semantic line
fc1
Cls
fc2
Reg
conv10 conv13
lp2
lp1
lp2
Candidate line
𝐥 = (𝑥s, 𝑦s, 𝑥e, 𝑦e)
Output of Reg: ∆𝐥 = (∆𝑥s, ∆𝑦s, ∆𝑥e, ∆𝑦e)
Output of Cls: 𝐩 = (𝑝, 𝑞)
ሚ𝐥 = 𝐥 + ∆𝐥

7. Semantic Line Network (SLNet)
• Line pooling layer
• Extract a regional representation of a candidate line from a
convolutional feature map
Line
Convolution activation Line pooling activation

8. Semantic Line Network (SLNet)
• Multi-task loss function
• Candidate line 𝐥 = (𝑥s, 𝑦s, 𝑥e, 𝑦e)
• Ground truth binary vector ഥ𝐩 = ( ҧ𝑝, ത𝑞)
𝐿cls 𝐩, ഥ𝐩 = −𝑝 log 𝑝 − 𝑞 log 𝑞
• Ground truth offset ∆ ҧ𝐥 = (∆ ҧ𝑙1, ∆ ҧ𝑙2, ∆ ҧ𝑙3, ∆ ҧ𝑙4)
𝐿reg ∆𝐥, ∆ ҧ𝐥 = ෍
𝑖=1
4
𝜂(∆𝑙𝑖 − ∆ ҧ𝑙𝑖)
𝜂 𝑥 = ቊ
0.5𝑥2, if |𝑥| < 1,
|𝑥| − 0.5, otherwise
• Multi-task loss function
𝐿 𝐩, ഥ𝐩, ∆𝐥, ∆ ҧ𝐥 = 𝐿cls(𝐩, ഥ𝐩) + 𝐿reg(∆𝐥, ∆ ҧ𝐥)

9. Semantic Line Network (SLNet)
• Network training
• Learning rate 𝜖 ← 0.1𝜖 after every two epochs (start: 0.001)
• Momentum 0.9, weight decay 0.0005
• Input image size: 400 x 400
• Candidate lines
Ground truth Uniform sampling Augment positives

10. Primary Semantic Line
• Primary semantic line characterizes the photographic
composition of an image
• For 1,308 candidate lines, the candidate 𝐥max with the
maximum soft-max probability is selected
ሚ𝐥max = 𝐥max + ∆𝐥max
Initial candidate
Regressed line

11. Multiple Semantic Line
• A candidate line is declared as a semantic line and
regressed, if its soft-max probability is higher than 0.5
Input image Multiple semantic lines

12. Multiple Semantic Line
• Non-maximum suppression(NMS)
• Prevents semantic lines from being too close to one another
• Edge density for each regressed line ሚ𝐥
𝜌 ሚ𝐥 =
𝑁edge
𝑁all
A semantic line example Edge map [1]
[1] S. Xie, and T. Zhuowen, "Holistically-nested edge detection," in Proc. of the IEEE ICCV. 2015.

13. Multiple Semantic Line
• Non-maximum suppression(NMS)
• Prevents semantic lines from being too close to one another
• Measure mean intersection over union (mIoU) between two
semantic lines
• If mIoU > 0.85, two semantic lines are regarded as duplicated, and
the line with a lower edge density is removed
𝑅
𝑄
𝑅′
𝑄′
mIoU =
1
2
(
𝑅 ∩ 𝑅′
𝑅 ∪ 𝑅′
+
𝑄 ∩ 𝑄′
𝑄 ∪ 𝑄′
)

14. Semantic Line Detection Results
• Primary semantic line detection
• 𝑁𝑐 : the number of the test images whose primary semantic lines
are correctly detected
Accuracy =
𝑁𝑐
𝑁
• Multiple semantic line detection
• 𝑁𝑠 : the number of correctly detected semantic lines
• 𝑁𝑒 : the number of false positives
• 𝑁 𝑚 : the number of false negatives
Precision =
𝑁𝑠
𝑁𝑠 + 𝑁𝑒
, Recall =
𝑁𝑠
𝑁𝑠 + 𝑁 𝑚

15. Semantic Line Detection Results
• The accuracy, precision, and recall curves
• Area under curve scores

16. Semantic Line Detection Results

17. Semantic Line Detection Results

18. Semantic Line Detection Results

19. Application Ⅰ. Composition Enhancement
• Horizon estimation + aesthetic cropping
Primary semantic line Rotated image Cropping result

20. Application Ⅰ. Composition Enhancement
• Horizon estimation + aesthetic cropping
Primary semantic line Rotated image &
Aesthetic region
Cropping result

21. Application Ⅰ. Composition Enhancement
• Horizon estimation + aesthetic cropping
Primary semantic line Rotated image &
Aesthetic region
Cropping result

22. Application Ⅱ. Image Simplification
• Image simplification
• Divide an image into polygonal regions along semantic lines
• Make the regions homogeneous, while retaining the spatial layout

23. Application Ⅱ. Image Simplification
• Image simplification
• To prevent over-segmentation, some line segments are removed
• Semantic score of a line segment
𝑆 = 𝑆sl + 𝑆sz + 𝑆ct
• 𝑆sl : the soft-max probability of the line segment
• 𝑆sz : 𝑅 ∪ 𝑄 /(the size of the test image)
• 𝑆ct : 𝜒2 𝐟 𝑅, 𝐟 𝑅∪𝑄 + 𝜒2 𝐟 𝑄, 𝐟 𝑅∪𝑄
𝑅
𝑄

24. Application Ⅱ. Image Simplification
• Extremely simplified images
• MOS 3.17
• 1 (unacceptable), 2 (bad), 3 (medium), 4 (good), 5 (superb)
• Compression ratio 8.05 × 104

25. Application Ⅱ. Image Simplification
• Simplified images remaining object regions
• MOS 3.92
• 1 (unacceptable), 2 (bad), 3 (medium), 4 (good), 5 (superb)
• Compression ratio 5.12 × 104

26. Conclusions
• Semantic line detector
• Multi-scale feature map for an image
• Line pooling layer to extract line descriptors
• Multi-task learning for exquisite semantic line detection
• Two applications
• Composition enhancement
• Image simplification

27. Thank You
• Question?