Korean Paper Review of "Abnormal Event Detection in Videos using Generative Adversarial Nets" (Review date: 2021.05.17 @ Soongsil Univ. Cognitive Science Class)

1. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (1/26)
2021/05/17
ABNORMAL EVENT DETECTION
IN VIDEOS USING GENERATIVE
ADVERSARIAL NETS
GAN을 활용한 영상 내 비정상 이벤트 탐지 기법
Paper Review
Presented by 이명규

2. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (2/26)
I N D E X
01
02
03
Introduction
Paper Overview
Conclusion

3. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (3/26)
Introduction
Part 01
1. Paper Introduction
2. Related Works
3. Background

4. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (4/26)
Paper Introduction
1-1
• 발표: ICIP 2018
(IEEE International Conference on Image Processing)
• 저자: Mahdyar Ravanbakhsh et al. (University of Genova)
• 인용수: 195회
• 1저자 Google Scholar 주소: Mahdyar Ravanbakhsh
• 논문 개요: GAN 네트워크를 활용해 영상으로부터 비정상 이벤트 탐지

5. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (5/26)
Related Works
1-2
• Various Methods for Detect “Abnormality”
➢ 전통적인 Feature Extractor들을 활용해 Abnormality를 탐지 [1, 2, 3, 4, 5, 6, 7]
e.g) Optical-Flow, Tracklets etc…)
➢ CNN을 활용해 Abnormality를 탐지 [15, 16]
➢ Denoising AE와 같은 Generative Method를 통해 Abnormality를 탐지 [17]
• Why is Anomaly Detection Task Challenging?
➢ 지도학습 기반으로 학습하기에는 비정상 샘플이 지나치게 적음
➢ Abnormality를 명확하게 정의하기 어려움

6. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (6/26)
↳ What is Anomaly Detection?
Background
1-3
• Normal(정상) sample과 Abnormal(비정상) sample을 구별하는 Task
• 대부분의 데이터 샘플들과 크게 다른 차이를 보이는 이벤트를 탐지
• Bank Fraud Detection, Structural Defect Detection 등의 다양한 분야에서 활용
Improving Unsupervised Defect Segmentation by Applying Structural Similarity To Autoencoders

7. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (7/26)
↳ What is GAN?
https://www.notion.so/A-Brief-Introduction-To-GANs-397de071301f4e56b4907a65d93cef7b
Background
1-3

8. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (8/26)
↳ What is GAN?
https://www.notion.so/A-Brief-Introduction-To-GANs-397de071301f4e56b4907a65d93cef7b
Background
1-3

9. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (9/26)
↳ What is GAN?
https://www.notion.so/A-Brief-Introduction-To-GANs-397de071301f4e56b4907a65d93cef7b
Background
1-3
“Want to learn 𝑷𝒎𝒐𝒅𝒆𝒍(𝒙) similar to 𝑷𝒅𝒂𝒕𝒂(𝒙)”
• Discriminative Modeling:
• Focus on the decision boundary
• Only for supervised tasks
• 𝑺𝒂𝒎𝒑𝒍𝒆 𝒙가 주어졌을 때 𝒍𝒂𝒃𝒆𝒍 𝒚의 확률 𝑷(𝒚|𝒙)추정하는 문제
• Generative Modeling:
• Probabilistic model of each class
• 𝑺𝒂𝒎𝒑𝒍𝒆 𝒙의 𝑷(𝒙)를 추정하는 문제
(단 cGAN의 경우는 𝒄𝒐𝒏𝒅𝒊𝒕𝒊𝒐𝒏 𝒗𝒆𝒄𝒕𝒐𝒓 𝒚가 주어진 경우 𝑷(𝒙|𝒚)를 추정)

10. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (10/26)
↳ GAN Application:
Image to Image Translation
“Latent Vector Magic!”
Background
1-3

11. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (11/26)
↳
https://brstar96.github.io/paperreview/shoveling/3D-shape-reconstruction-from-sketches-via-multi-view-convolutional-networks/
“Sketch to 3D Mesh”
GAN Application:
Image to Image Translation
Background
1-3

12. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (12/26)
↳ What is Optical Flow?
Background
1-3
• 빛의 흐름을 2D Vector field representation으로 표현하는 기법
(본 논문에서는 Conditinal GAN의 Condition Vector로 활용하기 위해 사용)
• 모션을 활용한 다양한 어플리케이션에 활용
e.g. SfM(Structure from motion), Video Compression, Video Stabilization, …
https://bkshin.tistory.com/entry/OpenCV-31-%EA%B4%91%ED%95%99-%ED%9D%90%EB%A6%84Optical-Flow , https://en.wikipedia.org/wiki/Optical_flow
▲ Gunner Farneback Algorithm
(cv2.calcOpticalFlowFarneback)
▲ Lucas-Kanade Algorithm
(cv2. calcOpticalFlowPyrLK)

13. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (13/26)
↳ Optical Flow Application:
Structure from motion
Background
1-3

14. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (14/26)
↳ Optical Flow Application:
Video Stabilization
Background
1-3

15. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (15/26)
Paper Overview
Part 02
1. Datasets
2. Network Architecture

16. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (16/26)
↳
Datasets
2-1
Dataset Spec: UCSD Dataset
• 움직이는 다양한 물체 포함
• Ped1, Ped2로 분리해 제공
• Ped1: 34 Train, 16 Test Sequences
• Ped2: 16 Train, 12 Test Videos
• 238*158 Low Resolution *.tif files
• Test 이미지는 정답에 대한 바이너리
마스크 이미지 제공
▲ Test Image에 정답 마스크를 덧씌운 샘플

17. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (17/26)
↳
Datasets
2-1
Dataset Spec: UMN Dataset
• 11 Videos in 3 Different Scenes (7700 frames)
• “Abnormal Crowd Behavior Detection using Social Force Model” 논문에서 발표

18. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (18/26)
↳
෡
𝑶𝒕 = 𝑮𝑭→𝑶 𝑭𝒕, 𝒛𝟏
෡
𝑭𝒕 = 𝑮𝑶→𝑭 𝑶𝒕, 𝒛𝟐
𝐷 𝑇𝑎𝑘𝑒𝑠 𝑒𝑖𝑡ℎ𝑒𝑟 𝐹𝑡, ෠
𝑂𝑡 𝑜𝑟 𝑂𝑡, 𝐹𝑡 𝑎𝑠 𝑖𝑛𝑝𝑢𝑡.
𝑤ℎ𝑒𝑟𝑒 𝑧 𝑖𝑠 𝑟𝑎𝑛𝑑𝑜𝑚 𝑠𝑎𝑚𝑝𝑙𝑒𝑑 𝑛𝑜𝑖𝑠𝑒 𝑓𝑟𝑜𝑚 𝑍,
𝑶𝒕 𝑖𝑠 𝑐𝑜𝑚𝑝𝑢𝑡𝑒𝑑 𝑤𝑖𝑡ℎ 28 ′
𝑠 𝑚𝑒𝑡ℎ𝑜𝑑.
Network Architecture
2-1
Entire Architecture

19. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (19/26)
↳
Network Architecture
2-1
Loss Functions
𝑮𝑨𝑵 𝑳𝒐𝒔𝒔 = 𝐦𝐢𝐧
𝑮
𝒎𝒂𝒙
𝑫
𝑽 𝑫, 𝑮 = 𝔼𝒙~𝑷𝒅𝒂𝒕𝒂 𝒙 𝒍𝒐𝒈𝑫 𝒙 +𝔼𝒛~𝑷𝒛 𝒁 [𝐥𝐨𝐠(𝟏−𝑫(𝑮(𝒛))]
𝑪𝒐𝒏𝒅𝒊𝒕𝒊𝒐𝒏𝒂𝒍 𝑮𝑨𝑵 𝑳𝒐𝒔𝒔 = 𝔼𝒙~𝑷𝒅𝒂𝒕𝒂 𝒙 𝒍𝒐𝒈𝑫 𝒙 +𝔼𝒛~𝑷𝒛 𝒁 [𝐥𝐨𝐠(𝟏−𝑫(𝑮 𝒛,𝒚 ,𝒚)]
𝓛𝑳𝟏 𝒙, 𝒚 = 𝒚 − 𝑮 𝒙, 𝒛 𝟏
𝓛𝒄𝑮𝑨𝑵 𝑮, 𝑫 = 𝔼 𝒙,𝒚 ∈𝓧 𝒍𝒐𝒈𝑫 𝒙, 𝒚 + 𝔼 𝒙 ∈ 𝑭𝒕 , 𝒛∈𝒁[𝐥𝐨𝐠(𝟏 − 𝑫(𝒙, 𝑮(𝒙, 𝒛)))]
𝓧 = 𝑭𝒕, 𝑶𝒕 𝒘𝒉𝒆𝒏 𝓝𝑭→𝑶
, 𝒗𝒊𝒄𝒆 𝒗𝒆𝒓𝒔𝒂.

20. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (20/26)
↳
Network Architecture
2-1
Abnormality Detection(1/2) - 차이 맵 계산
• 비정상 픽셀 검출은 ∆𝑶(𝑶𝒕 − ෡
𝑶𝒕)와 ∆𝑺(𝒉 𝑭 − 𝒉(෡
𝑭))를 합성해 수행
➢ 𝑶𝒕 는 𝑻𝒆𝒔𝒕 𝑰𝒏𝒑𝒖𝒕 𝑭𝒓𝒂𝒎𝒆 𝑭𝒕와 𝑭𝒕−𝟏를 이용해 [28]의 방법으로 연산
➢ ෡
𝑶𝒕는 정상 이미지만으로 학습된 𝑮를 통해 연산
➢ 𝒉 는 Pre-trained AlexNet, ෡
𝑭는 정상 이미지만으로 학습된 𝑮를 통해 연산
• 단순히 𝑻𝒆𝒔𝒕 𝑰𝒏𝒑𝒖𝒕 𝑭𝒓𝒂𝒎𝒆 𝑭와 𝑮𝒆𝒏𝒆𝒓𝒂𝒕𝒆𝒅 𝒑𝑭의 차이를 구하는 것은
∆𝑶보다 정보량이 적은 문제 발생
➢ 따라서 𝒉 에 𝑭와 ෡
𝑭를 통과시킨 후 5th layer output으로 ∆𝑺 연산

21. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (21/26)
↳
Network Architecture
2-1
Abnormality Detection(2/2) - 차이 맵 합성
Upsample
∆𝑺
Same resolution as ∆𝑶
∆𝑶
∆𝑺
∆′
𝑺 MinMax
Normalize
With respect to corresponding
channel-value range
𝑵𝑶(𝒊, 𝒋) = 𝟏/𝒎𝑶∆𝑶(𝒊, 𝒋),
𝑵𝑺(𝒊, 𝒋) = 𝟏/𝒎𝑺∆′
𝑺(𝒊, 𝒋)
𝑵𝑺, 𝑵𝑶
Sum
𝑨 = 𝑵𝑺 + 𝝀𝑵𝑶,
𝑾𝒉𝒆𝒓𝒆 𝝀 = 𝟐
Final
Abnormality
Heatmap

22. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (22/26)
Conclusion
Part 03
1. Experiments
2. Conclusion

23. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (23/26)
↳
Experiments
3-1
Visual Results

24. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (24/26)
↳
Conclusion
3-2
Quantitative Results

25. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (25/26)
↳
Conclusion
3-2
Quantitative Results
https://angeloyeo.github.io/2020/08/05/ROC.html
TPR: “암에 걸린 환자를 암환자로 분류한 비율”
FPR: “암에 걸리지 않은 환자를 암환자로 분류한 비율”

26. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (26/26)
↳
Conclusion
3-2
Quantitative Results
▲ UCSD Testset Results ▲ UMN Testset Results

27. 이명규
ABNORMAL EVENT DETECTION IN VIDEOS USING GENERATIVE ADVERSARIAL NETS (27/26)
Thank you for Watching.
Myeong-Gyu LEE | Ph.D. Student @SSU
🧪 Computer Graphics Lab (Advised by Prof. KyoungSu Oh)
Department of Digital Media
💼 Espreso Media co., Application Tech. Development
Assistant Research Engineer
✉ brstar96@naver.com