Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to PR-214: FlowNet: Learning Optical Flow with Convolutional Networks
Similar to PR-214: FlowNet: Learning Optical Flow with Convolutional Networks (20)
More from Hyeongmin Lee
More from Hyeongmin Lee (20)
Recently uploaded
Recently uploaded (20)
PR-214: FlowNet: Learning Optical Flow with Convolutional Networks
- 1. FlowNet: Learning Optical Flow with Convolutional Networks Hyeongmin Lee Image and Video Pattern Recognition LAB Electrical and Electronic Engineering Dept, Yonsei University 4th Semester 2019.12.15
- 2. What is Optical Flow?
- 3. What is Optical Flow? ◆ Optical Flow 연속한 두 Frame 사이에서 각 Pixel의 Motion을 나타내는 Vector Map (Pixel Displacement)
- 4. What is Optical Flow? ◆ Visualizing Optical Flow <2-Dimensional Map> • Color: Direction • Saturation: Magnitude
- 6. Optical Flow Constraint ◆ The Optical Flow Constraint “Flow Vector로 이어진 두 Pixel 값은 동일하다.”
- 7. Optical Flow Constraint ◆ The Optical Flow Constraint “Flow Vector로 이어진 두 Pixel 값은 동일하다.” 𝐼(𝑥, 𝑦, 𝑡) (𝑢, 𝑣) Image Flow
- 8. Optical Flow Constraint ◆ The Optical Flow Constraint “Flow Vector로 이어진 두 Pixel 값은 동일하다.” 𝐼(𝑥, 𝑦, 𝑡) (𝑢, 𝑣) Image Flow 𝐼 𝑥, 𝑦, 𝑡 = 𝐼(𝑥 + 𝑢, 𝑦 + 𝑣, 𝑡 + 1)
- 9. Optical Flow Constraint ◆ The Optical Flow Constraint 𝐼 𝑥, 𝑦, 𝑡 = 𝐼 𝑥 + 𝑢, 𝑦 + 𝑣, 𝑡 + 1 = 𝐼 𝑥, 𝑦, 𝑡 + 1 + 𝜕𝐼 𝜕𝑥 𝑢 + 𝜕𝐼 𝜕𝑥 𝑣 𝐼 𝑥, 𝑦, 𝑡 + 1 − 𝐼 𝑥, 𝑦, 𝑡 + 𝜕𝐼 𝜕𝑥 𝑢 + 𝜕𝐼 𝜕𝑥 𝑣 = 0 𝐼𝑡 + 𝐼 𝑥 𝑢 + 𝐼 𝑦 𝑣 = 0 First-Order Taylor Approximation The Optical Flow Constraint
- 10. Optical Flow Constraint ◆ Aperture Problem 𝐼𝑡 + 𝐼 𝑥 𝑢 + 𝐼 𝑦 𝑣 = 0 The Optical Flow Constraint (Underdetermined) We need some additional constraints!!
- 11. Optical Flow Estimation by Optimization
- 12. Optical Flow Estimation by Optimization ◆ Lucas-Kanade Method 𝐼𝑡 (1) + 𝐼 𝑥 (1) 𝑢 + 𝐼 𝑦 (1) 𝑣 = 0 𝐼𝑡 (2) + 𝐼 𝑥 (2) 𝑢 + 𝐼 𝑦 (2) 𝑣 = 0 𝐼𝑡 (3) + 𝐼 𝑥 (3) 𝑢 + 𝐼 𝑦 (3) 𝑣 = 0 𝐼𝑡 (𝑛) + 𝐼 𝑥 (𝑛) 𝑢 + 𝐼 𝑦 (𝑛) 𝑣 = 0
- 13. Optical Flow Estimation by Optimization ◆ Variational Method Optical Flow ConstraintSmoothness Constraint (Total Variation Loss)
- 14. Optical Flow Estimation by Optimization ◆ Variational Method – Quadratic Relaxation For N Iterations
- 15. Optical Flow Estimation by Energy Minimization ◆ Large Displacement 𝐼 𝑥, 𝑦, 𝑡 = 𝐼 𝑥 + 𝑢, 𝑦 + 𝑣, 𝑡 + 1 = 𝐼 𝑥, 𝑦, 𝑡 + 1 + 𝜕𝐼 𝜕𝑥 𝑢 + 𝜕𝐼 𝜕𝑥 𝑣 First-Order Taylor Approximation Short Displacement가 전제됨! ➔ Large Displacement에 취약 ✓ Coarse-to-Fine Method Error Propagation
- 16. Optical Flow Estimation by Energy Minimization ◆ Large Displacement Optical Flow [TPAMI 2011] Descriptor Loss Descriptor Based Feature Matching을 Guide로 주어, Large Displacement에 대한 성능을 보완
- 17. Optical Flow Estimation by Energy Minimization ◆ Large Displacement Optical Flow [TPAMI 2011] ✓ Feature Matching using Descriptors Large Displacement에 강인. Sparse한 점에 대해서만 Matching 가능.
- 18. Optical Flow Estimation by Energy Minimization ◆ EpicFlow [CVPR 2015] Coarse-to-Fine Interpolation 시에 Edge를 고려하여 Sharp한 결과를 얻음
- 19. Optical Flow Estimation by Energy Minimization ◆ EpicFlow [CVPR 2015] Interpolation using Euclidean distance Euclidean edge far close Interpolation using Geodesic distance
- 21. FlowNet ◆ Why optimization based? ✓ Hardware와 Deep Learning의 기술적 Baseline 부족 ✓ Hard to get ground truth (Lack of Dataset)
- 22. FlowNet ◆ Why optimization based? ✓ Hardware와 Deep Learning의 기술적 Baseline 부족 ✓ Hard to get ground truth (Lack of Dataset) GPU & Parallel Processing Convolutional Neural Networks 만들자!
- 23. FlowNet ◆ Flying Chairs Dataset Flicker DB + 3d Chairs
- 25. FlowNet ◆ Correlation Layer 𝐱 𝟏 𝐱 𝟐 𝐷 𝐷 𝐾 𝐾 𝑊 𝐻 𝑊 𝐻 𝐷2
- 26. FlowNet ◆ Refinement Layer EPE loss EPE loss EPE loss EPE loss0.08 0.02 0.01 0.005 EPE(End Point Error) Loss: Estimated & GT간의 L2 Loss
- 30. FlowNet2.0 ◆ Change on Training Dataset & Scheduling 1. FlowNet에서 사용한 Flying Chair Dataset으로 pre-train 2. Mayer et al. 에서 제안한 Flying Things 3D Dataset으로 추가 학습
- 31. FlowNet2.0 ◆ Change on Network Architecture
- 32. FlowNet2.0 ◆ New Dataset & Architecture for Small Displacement 1. ChairsSDHom Dataset (like UCF101) 2. FlowNet-SD Block & Fusion Block
- 35. Thank You!