이번 PR12 365번째 논문으로 소개드릴 내용은 조금 특이한 접근법입니다. 우리가 실생활에서 접하는 대부분의 비디오는 Compressed 된 형태의 Video인데요, 실제 Computer Vision Task에서 input이 Compressed Video라는 가정을 하게 되면 생각보다 큰 이점을 얻을 수 있습니다. 바로 Compressed Video에는 Motion Vector가 포함되어있다는 점입니다. 이를 이용하면 생각보다 많은 것들을 할 수 있게 됩니다. 그 예시로 Object Detection의 연산량을 크게 줄인 case를 하나 소개드려보고자 합니다. paper link: https://openaccess.thecvf.com/content_ICCV_2019/html/Wang_Fast_Object_Detection_in_Compressed_Video_ICCV_2019_paper.html video link: https://youtu.be/9n6OtHtJvJ0

1. Fast Object Detection in Compressed Video
Hyeongmin Lee
Yonsei University
PR-365
2022.1.9

3. H.264 Codec

4. H.264 Codec
 Codec? [PR-340]
0011010100111...

5. H.264 Codec
 Decoding Time Stamp (DTS)
Intra Coding: I-Frame compression
Inter Coding: P-Frame compression

6. H.264 Codec
 Inter Compression
𝑰𝑰𝟏𝟏, 𝑰𝑰𝟐𝟐, 𝑰𝑰𝟑𝟑, 𝑰𝑰𝟒𝟒, … , 𝑰𝑰𝑵𝑵
𝑰𝑰𝟏𝟏, 𝑹𝑹𝟐𝟐, 𝑹𝑹𝟑𝟑, 𝑹𝑹𝟒𝟒, … , 𝑹𝑹𝑵𝑵
𝑭𝑭𝟏𝟏𝟏𝟏, 𝑭𝑭𝟐𝟐𝟐𝟐, 𝑭𝑭𝟑𝟑𝟑𝟑, … , 𝑭𝑭𝑵𝑵−𝟏𝟏,𝑵𝑵

7. H.264 Codec
 Traditional Video Compression
• H.264
• H.265 (HEVC)
1
𝒙𝒙𝒕𝒕 �
𝒙𝒙𝒕𝒕−𝟏𝟏
Motion Estimation
𝒗𝒗𝒕𝒕
Entropy Coding
11000110101

8. H.264 Codec
 Traditional Video Compression
• H.264
• H.265 (HEVC)
2
�
𝒙𝒙𝒕𝒕−𝟏𝟏
Warping
�
𝒙𝒙𝒕𝒕
𝒗𝒗𝒕𝒕

9. H.264 Codec
 Traditional Video Compression
• H.264
• H.265 (HEVC)
3
𝒓𝒓𝒕𝒕 = 𝒙𝒙𝒕𝒕 − �
𝒙𝒙𝒕𝒕

10. H.264 Codec
 Traditional Video Compression
• H.264
• H.265 (HEVC)
4
𝒓𝒓𝒕𝒕
Transform
𝒚𝒚𝒕𝒕
Quantization
�
𝒚𝒚𝒕𝒕
Inverse
Transform
�
𝒓𝒓𝒕𝒕
Entropy Coding
11000110101
Buffer
�
𝒙𝒙𝒕𝒕 = �
𝒙𝒙𝒕𝒕 + �
𝒓𝒓𝒕𝒕

11. H.264 Codec
 Inter Compression
𝑰𝑰𝟏𝟏, 𝑰𝑰𝟐𝟐, 𝑰𝑰𝟑𝟑, 𝑰𝑰𝟒𝟒, … , 𝑰𝑰𝑵𝑵
𝑰𝑰𝟏𝟏, 𝑹𝑹𝟐𝟐, 𝑹𝑹𝟑𝟑, 𝑹𝑹𝟒𝟒, … , 𝑹𝑹𝑵𝑵
𝑭𝑭𝟏𝟏𝟏𝟏, 𝑭𝑭𝟐𝟐𝟐𝟐, 𝑭𝑭𝟑𝟑𝟑𝟑, … , 𝑭𝑭𝑵𝑵−𝟏𝟏,𝑵𝑵
𝑰𝑰𝟏𝟏, 𝑹𝑹𝟐𝟐, 𝑹𝑹𝟑𝟑, 𝑹𝑹𝟒𝟒, … , 𝑹𝑹𝑵𝑵
𝑭𝑭𝟏𝟏𝟏𝟏, 𝑭𝑭𝟐𝟐𝟐𝟐, 𝑭𝑭𝟑𝟑𝟑𝟑, … , 𝑭𝑭𝑵𝑵−𝟏𝟏,𝑵𝑵
𝑰𝑰𝟏𝟏, 𝑰𝑰𝟐𝟐, 𝑰𝑰𝟑𝟑, 𝑰𝑰𝟒𝟒, … , 𝑰𝑰𝑵𝑵
11000110101
Encoder
(Premiere Pro)
Decoder
(Viewer)
[Channel]

12. Fast Object Detection in
Compressed Video

13. Fast Object Detection in Compressed Video
 Overview

14. Fast Object Detection in Compressed Video
 Network Structure
Feature Extractor Memory Network Detection Network

15. Fast Object Detection in Compressed Video
 Pyramidal Feature Attention
ResNet-101

16. Fast Object Detection in Compressed Video
 Motion-aided LSTM
[Motion-Based Warping]
[LSTM]

17. Experiments

18. Experiments
 Ablation Study

19. Experiments
 Ablation Study – motion alignment

20. Experiments
 vs Flownet

21. Experiments
 Comparisons

22. Thank You!