Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Final presentation optical flow estimation with DL

649 views

Published on

Summer Intern

Published in: Engineering
  • You can try to use this service ⇒ www.HelpWriting.net ⇐ I have used it several times in college and was absolutely satisfied with the result.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • ⇒ www.WritePaper.info ⇐ This service will write as best as they can. So you do not need to waste the time on rewritings.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • paid surveys and get paid for it? ★★★ https://t.cn/A6ybK3XL
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Final presentation optical flow estimation with DL

  1. 1. Optical flow estimation with Blueoil K-Inoue @ki42 & Oscar @wang Blueoil Division 1
  2. 2. What is optical flow estimation? ● Optical flow... represents the apparent motion of objects. ● Optical flow estimation... can predict the movement of objects in a video. Miloud, Hadj achour. (2017). Fragmentation de métal liquide dans l’eau. https://www.codeproject.com/Articles/1192205/Capturing-motion-from-video-using-the-Emgu-CV-libr 2
  3. 3. Optical flow estimation is important ● Widely used by insects and birds ● Practical usage ○ Analyze motion ○ Avoid collision ○ Assist in navigation ● Real-world Applications ○ Video/Motion classification ○ Navigation assistance ■ Self driving cars ■ Drones https://nanonets.com/blog/optical-flow/ 3
  4. 4. DL approaches are increasing 4 ● 2015. FlowNet S (Simple) ● 2015. FlowNet C (Correlated) ● 2016. FlowNet 2 ● 2018. LiteFlowNet ● ... Color coding https://www.youtube.com/watch?v=k_wkDLJ8lJE https://www.youtube.com/watch?v=pfQ0zFwv-hM
  5. 5. However... ● Existing DL approaches require GPU to execute 👎 High power consumption 👎 Low runtime speed on CPU environment ● We propose LmFlowNet S 👍 ○ Modification of FlowNet S [P. Fischer+ 2015] ○ Goals: ■ Edge Computing ■ Run on FPGA-based accelerator ■ Use quantization to reduce inference time while achieving good prediction performance 5
  6. 6. FlowNet S vs LmFlowNet S 6
  7. 7. Network of FlowNet S 7 [N, 384, 512, 6] [N, 384, 512, 2] Detailed ops inside each color block is shown in appendix
  8. 8. Network of FlowNet S 8 [N, 384, 512, 6] [N, 384, 512, 2] Encoder Decoder Detailed ops inside each color block is shown in appendix
  9. 9. Network of FlowNet S 9 [N, 384, 512, 6] [N, 384, 512, 2] Detailed ops inside each color block is shown in appendix Not supported by Blueoil DLK
  10. 10. 10 Network of LmFlowNet S (DLK supported) [N, 384, 512, 6] [N, 384, 512, 2] Detailed ops inside each color block is shown in appendix
  11. 11. 11 Network of LmFlowNet S [N, 384, 512, 6] [N, 384, 512, 2] Quantized Detailed quantization inside each color block is shown in appendix
  12. 12. Loss function: End Point Error (EPE) 12 (x1 ,y1 ) (x2 ,y2 ) EPEflow2 EPEflow3 EPEflow4 EPEflow5 EPEflow6 Weighted EPE = 0.32 * EPEflow2 + 0.08 * EPEflow3 + 0.02 * EPEflow4 + 0.01 * EPEflow5 + 0.005 * EPEflow6 Down- sampled Ground Truth Training hyper-parameters are shown in appendix 24 x 32 48 x 6496 x 128 192 x 256 12 x 16
  13. 13. Artificial dataset: Flying Chairs 13 ● Dataset Name Frame pairs Train validation ratio size Flying Chairs 22,872 9:1 30GB ● Data Augmentation ○ Crop, Rotate, Translate, FlipLeftRight, FlipTopBottom ○ Gaussian noise, Brightness, Contrast, Gamma, and Color Parameters used in data augmentation are shown in appendix https://arxiv.org/pdf/1504.06852.pdf
  14. 14. 14 Results
  15. 15. 15 Results - Avg. EPE & Inference time Method Avg. EPE (pixel) (Flying Chairs) Inference time per frame (ms) [1] CPU (dlk-convert) GPU (tensorflow) FlowNet S 2.94 - 11.65 LmFlowNet S 5.33 1360.49 13.81 LmFlowNet S Quantized 9.01 637.467 17.60 [1] CPU and GPU specs available in appedix FlowNet S Trained for 1.2M LmFlowNet S Trained for 400K LmFlowNet S Quantized Trained for 400K
  16. 16. 16 Live Demo
  17. 17. 17 Live demonstration ● Three demonstration ○ FlowNet S ○ LmFlowNet S ○ LmFlowNet S Quantized ● NOTE: Running on GPU (not on CPU / FPGA) ○ Failed to run on CPU/FPGA due to several problems 😢 ■ etc. segmentation fault, memory error... ○ Fixing and debugging them in the future 👊
  18. 18. 18
  19. 19. Challenges 19
  20. 20. Challenges ● Training takes a very long time ( > 2 weeks...😢) ○ Heavy data augmentation & pre-processing ■ Pre-processing on GPU is not supported now. ● Unique network structure, not compatible with Blueoil ○ Input is a stack of 2 images (6 channels) ○ Multiple and branched outputs ● DLK Limitation. No documentation. 🤯 ○ No support for kernel size 7x7, 5x5 ○ No support for Conv2dTranspose ○ Cannot concat quantized value and float together ○ Requires the depth of Space2Depth to be 32 * N 20
  21. 21. Thank you for your attention! our source code: https://github.com/ki-lm/blueoil/tree/lmflownets 21
  22. 22. 22
  23. 23. Appendix 23
  24. 24. 24
  25. 25. LmFlowNet S | Training & Hyper-params ● Optimizer: Adam ● Max steps: 1200k ● Fixed parameters in Adam: β1 =0.9 and β2 = 0.999. ● Batch size: 8 ● Learning rate ○ values: [1e-4 , 5e-5 , 2.5e-5 ,1.25e-5 ,6.25e-6 ] ○ boundaries: [400K, 600K, 800K, 1000K] ● Learning rate for quantization ○ values: [1.25e-5 , 1e-4 , 5e-5 , 2.5e-5 ,1.25e-5 ,6.25e-6 ] ○ boundaries: [50K, 400K, 600K, 800K, 1000K] 25 Tokunaga Scheduling 😆
  26. 26. LmFlowNet S | Data Augmentation ● Translation: [20%, 20%] of the image width for x and y ● Rotation: [17o , 17o ] ● Scaling: [0.9, 2.0] ● Gaussian noise: sigma uniformly sampled from [0, 0.04] ● Contrast: [0.8, 0.4] ● Multiplicative color changes to the RGB channels per image: [0.5, 2] ● Gamma values: [0.7, 1.5] ● Additive brightness changes: Gaussian with a sigma of 0.2 26
  27. 27. 27 CPU & GPU specs ● CPU ○ Intel(R) Core(TM) i7-5500U CPU @ 2.40GHz ○ 4 cores, 8 threads ○ Inference time (ms) ■ LmFlowNet S: 1358.39, 1358.39, 1293.26, 1307.28, 1485.11 ■ LmFlowNet S Quant: 640.541, 640.732, 635.354, 635.354, 635.354 ● GPU ○ NVIDIA Tesla V100 on DGX-1 ○ Inference time (ms) ■ FlowNet S: 0.0111, 0.0105, 0.0116, 0.0117, 0.0119, 0.0118, 0.0121, 0.0118, 0.0116, 0.0124 ■ LmFlowNet S: 0.0137, 0.0140, 0.0137, 0.0136, 0.0141, 0.0135, 0.0139, 0.0138, 0.0136, 0.0142 ■ LmFlowNet S Quant: 0.0168, 0.0161, 0.0165, 0.0177, 0.0173, 0.0166, 0.0186, 0.0195, 0.0183, 0.0187
  28. 28. Our FlowNet S versions 28 Version Architecture DLK support V1 (FlowNet S) Same as the paper X V2 7x7, 5x5 => 3x3 Striding 2 => SpaceToDepth X V3 Conv2dTranspose => ResizeNearestNeighbor + Conv2d ResizeBilinear => ResizeNearestNeighbor X V3 Quant. (LmFlowNet S) Quantize except first, last layer, and activation before last layer △ V4 Quant. Change all output depths from SpaceToDepth to 32 * N O
  29. 29. List of source code links ● FlowNet S/C, and 2 (TensorFlow): https://github.com/sampepose/flownet2-tf/ ● FlowNet S/C (Original paper, Caffe): https://lmb.informatik.uni-freiburg.de/Publications/2015/DF IB15/ ● FlowNet 2 (Original paper, Caffe): https://github.com/lmb-freiburg/flownet2 29

×