Weakly supervised semantic segmentation using web crawled videos

1. Weakly Supervised
Semantic Segmentation
using Web-Crawled Videos
Author: Seunghoon Hong, Donghun Yeo,Suha Kwak,
Honglak Lee, Bohyung Han
Publish: CVPR2017

2. Goal
 Problem:
Model overly focuses on discriminative parts
rather than the entire object area
 Solution:
 Retrieving video and generating segmentation
labels from the retrieving video without human
intervention to simulate string supervised for
semantic segmentation

3. Video
 Estimate shape and extent of object in
video by motion
 Generating segmentation labels

4. Video
 Temporally ambiguous:
Some frames are not relevant to label
 Spatial ambiguity:
Some motion object not relevant to label

5. Weakly annotated images
 Image and clean class labels given manually
 Alleviate the ambiguities in video

6. Achitecture – two part
 Encoder for image classification and
discriminative localization
 Decoder for image segmentation

8. Section 3.1
: Based on Pre-trained VGG-16 network

9. Section 3.1
Remove full-connected layer
Place a new convolutional layer after the last
convolutional layer of VGG-16
For better ada-ptation to our task
Global average pooling followed by a fully-
connected layer at bottom

10. Global average pooling

11. Section 3.1

12. Attention - Class Activation
Mapping

13. Section 3.2 - Eliminating
irrelevent frames
< Threshold -> Eliminate

14. Section 3.2 - Eliminating
irrelevent frames
 If more than 5 consecutive frames are
chosen, we consider them as a single
relevant video

15. Section 3.2 - spatio-temporal
segmentation
Graph-based optimization problem
i-th superpixel of frame t

16. Section 3.2 - spatio-temporal
segmentation
Estimate a binary label for

17. Section 3.2 - spatio-temporal
segmentation
By GMM
By Inside outside map

18. Section 3.2 – Attention

19. Section 3.2 - spatio-temporal
segmentation

20. Section 3.3 – Learning with
video

21. Result

22. Reference
 Weakly Supervised Semantic Segmentation
using Web-Crawled Videos ,CVPR2017