Review : DeepStrip: High Resolution Boundary Refinement - by Seunghyun Hwang (Yonsei University, Severance Hospital, Center for Clinical Data Science)

1. DeepStrip: High Resolution Boundary Refinement
Hwang seung hyun
Yonsei University Severance Hospital CCIDS
University of Maryland & Adobe Research
CVPR 2020
2020.05.17

2. Introduction Related Work Methods and
Experiments
01 02 03
Conclusion
04
Yonsei Unversity Severance Hospital CCIDS
Contents

3. DeepStrip
Introduction – Proposal
• Boundary detection is a well-studied problem and fundamental
for human recognition
• Current methods are usually computed on low resolution(LR)
images, but most photos taken these days are much larger and
high resolution(HR) images
• Most studies simply upsample LR prediction to reach HR
prediction.
• Deep Strip targets on refining the boundaries in high resolution
images given low resolution masks
Introduction / Related Work / Methods and Experiments / Conclusion

4. DeepStrip
Introduction – Contributions
• Propose an approach to predict the boundary in a strip image, which is
computationally and memory wise efficient.
• To improve performance, propose novel losses including boundary distance, matching
and C0 continuity loss.
• Create a high resolution dataset “PixaHR” for evaluation.
Introduction / Related Work / Methods and Experiments / Conclusion

5. Related Work
1. Boundary Refinement
Introduction / Related Work / Methods and Experiments / Conclusion
• Explore rich convolutional features or fuse both low and high level features to
detect edges
• “Conditional Random Fields(CRF)”, “Graph Cuts”
• These methods mainly explore edge detection in LR images, while DeepStrip
target HR boundary refinement.
2. Active Contours
• “Snakes” (Active contour model)
• “Deep active contour” predict boundary pixels in a patch. But, cannot
guarantee a continuous boundary prediction
• These methods process the entire image or perform patch-based training,
which requires heavy computation and memory overhead
3. High Resolution Up-sampling
• Conventional methods reach HR segmentation masks by applying upsampling
to LR mask.

6. Methods and Experiments
DeepStrip – Architecture
Introduction / Related Work / Methods and Experiments / Conclusion
• Predict on strip image that captures the potential boundary region rather than the
entire HR image.
• Refines the edges on the strip image using a network
• Reconstruct prediction in the original image from the strip boundary prediction.

7. Methods and Experiments
DeepStrip – Strip Image Creation
Introduction / Related Work / Methods and Experiments / Conclusion
• Extract pixels near the upsampled boundary to create a strip image
• Use B-spline method to represent contour in the LR mask
• HR region along the normal direction at each point on the curve of the contour is extracted
• For GT label, add labels at the border of strip if no boundary pixel is included in strip image.
• If the strip height is large and multiple boundary pixels are included in each column, filter out
the extraneous boundaries that are not connected to the current one.

8. Methods and Experiments
DeepStrip – Strip Boundary Prediction
Introduction / Related Work / Methods and Experiments / Conclusion
• Train U-Net to predict the corresponding boundaries within the strip domain.
• Use instance normalization to apply for different resolution of images
• Extract the last upsampling layer and apply sigmoid function to predict all potential boundaries.
• Selection layer pick up the target boundary from potential boundaries
s = final output, x = initial prediction, m = softmax output of the selection layer

9. Methods and Experiments
DeepStrip – Loss Function
Introduction / Related Work / Methods and Experiments / Conclusion
1. Basic Loss Function (l1, Dice)
2. Boundary Distance Loss
3. Matching Loss (l1)
4. C0 Continuity Regularization (calculate
marginal difference between columns and penalize
the discontinuous position)
5. Total Loss

10. Methods and Experiments
DeepStrip – Strip Reconstruction at Inference stage
Introduction / Related Work / Methods and Experiments / Conclusion
• Mapping between the predicted strip boundaries and the full HR mask is required at
inference
• For every strip image, coordinates in the HR image are recorded for reconstruction
• Use dynamic programming similar to “seam carving” to find the path.
• Enables different strip sizes (width of strip) for different images
• Fix the height of strip, assuming all target boundaries are involved
w
h

11. Methods and Experiments
Dataset
Introduction / Related Work / Methods and Experiments / Conclusion
- DAVIS 2016 (benchmark for video segmentation, consists of 50 classes with
precise annotations in both 480P and 1080P)
- Pixa HR (100 manually annotated images with average resolution 7K x 7K)
• Downsample HR mask to LR by 8x, 16x, 32x for evaluation and training.
• Boundary-based F score for evaluation metrics

12. Methods and Experiments
Main Results
Introduction / Related Work / Methods and Experiments / Conclusion
* Baseline Model: only trained with l1 loss, without selection layer

13. Methods and Experiments
Main Results
Introduction / Related Work / Methods and Experiments / Conclusion

14. Methods and Experiments
Main Results
Introduction / Related Work / Methods and Experiments / Conclusion

15. Methods and Experiments
Main Results
Introduction / Related Work / Methods and Experiments / Conclusion

16. Methods and Experiments
Main Results
Introduction / Related Work / Methods and Experiments / Conclusion

17. Methods and Experiments
Ablation Studies
Introduction / Related Work / Methods and Experiments / Conclusion

18. Methods and Experiments
Ablation Studies
Introduction / Related Work / Methods and Experiments / Conclusion
• Performance increased when dividing the whole contour into 2 segments
which allows variable height for different regions
• Showed effectiveness of having flexible height

19. Conclusion
Introduction / Related Work / Methods and Experiments / Conclusion
• This paper presented a novel strategy to handle HR boundary refinement
computationally and memory efficiently given LR precise masks.
• Proposed extracting boundary regions along the upsampled boundary
spline to form strip images and make prediction within them.
• Boundary distance, matching loss, and C0 continuity regularization have
been proposed
• Current approach still has difficulty predicting complicated topology and
soft boundary regions
• Smarter adaptive strip height adjustment for every pixel might be a
potential solution