Successfully reported this slideshow.
Your SlideShare is downloading. ×

Video smart cropping web application

Video smart cropping web application

Download to read offline

Presentation of the paper titled "A Web Service for Video Smart-Cropping", by K. Apostolidis, V. Mezaris, delivered at the IEEE Int. Symposium on Multimedia (ISM), Dec. 2021. The corresponding software and dataset are available at https://github.com/bmezaris/RetargetVid.

Presentation of the paper titled "A Web Service for Video Smart-Cropping", by K. Apostolidis, V. Mezaris, delivered at the IEEE Int. Symposium on Multimedia (ISM), Dec. 2021. The corresponding software and dataset are available at https://github.com/bmezaris/RetargetVid.

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Video smart cropping web application

  1. 1. A Web Service for Video Smart-Cropping Konstantinos Apostolidis, Vasileios Mezaris CERTH-ITI Thessaloniki, Greece This work was supported by the H2020 project ReTV (grant agreement No 780656) 1 23rd IEEE International Symposium on Multimedia (ISM 2021)
  2. 2. Problem Statement ● Traditional TV and desktop computer monitors: landscape aspect ratios (16:9 or 4:3) ● Nowadays, mobile devices use different aspect ratios ● Video sharing platforms dictate the use of specific video aspect ratios ● Existing videos would have to be transformed to comply with their specifications 2
  3. 3. ● Straightforward approaches for transforming a video to a different aspect ratio: ○ Static cropping of content ○ Padding the frames with black borders Problem Statement ● The results of such simple approaches are often unsatisfactory ● Common video aspect ratio transformation methods of the literature: ○ Warping ○ Seam carving ➢Both introduce distortions and may alter the semantics of the video 3 ➢Significant loss of visual content that might even be in the center of attention ➢Shrinks the original video by introducing large borders in the output video
  4. 4. Problem Statement ● Several research works and some commercial software for video retargeting available ● No easy and free video retargeting tools! ● Motivated by this, we built a freely accessible Web application for video retargeting that consists of: ○ A REST service hosting the developed technologies for video cropping ○ An interactive user interface ● We used a modified version of the method in [1] [1] Apostolidis, Konstantinos, and Vasileios Mezaris. "A Fast Smart-Cropping Method and Dataset for Video Retargeting." In 2021 IEEE International Conference on Image Processing (ICIP), pp. 2618-2622. IEEE, 2021. 4
  5. 5. Let’s quickly remind how the method of [1] works! 5
  6. 6. Method of [1] ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing 6
  7. 7. ow/oh = 16/9 fw/fh = 4/5 fh = oh fw < ow Original frame Final frame Method of [1] ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing 7 fh ow oh fw
  8. 8. ow oh fh fw ow/oh = 4/5 fw/fh = 16/9 fh < oh fw = ow Original frame Final frame Method of [1] ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing 8
  9. 9. Method of [1] ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing 9
  10. 10. Method of [1] ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing 10
  11. 11. Method of [1] ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing 11
  12. 12. Method of [1] ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing 12
  13. 13. Method of [1] ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing yi xi 13
  14. 14. Method of [1] ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing t Time-series of center of mass displacement
  15. 15. Method of [1] ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing t Shot transitions Video frames Shot #1 Shot #2 Shot #3 15
  16. 16. Method of [1] ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing Smoothed time-series Inferred time-series t Smoothed time-series of center of mass displacement 16
  17. 17. ● Read the first out of every five videos frames ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing Proposed Method Our modifications: Optimized set of parameters 17
  18. 18. ● Read the first out of every five videos frames ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing Proposed Method Our modifications: Optimized set of parameters Spatial sub-sampling 18
  19. 19. ● Read the first out of every five videos frames ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing Proposed Method Spatial sub-sampling “Focus stability” mechanism Our modifications: Optimized set of parameters 19
  20. 20. ● Read the first out of every five videos frames ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing Proposed Method Replace LOESS with a Savitzky-Golay filter Spatial sub-sampling “Focus stability” mechanism Our modifications: Optimized set of parameters 20
  21. 21. Proposed Method Deployed a Web application that: 1. Retrieves a video file 2. Analyzes the video 3. Transforms the video frames to the target aspect ratio 4. Renders the transformed video 21
  22. 22. Proposed Method The REST service works through a 3-step process: 1. HTTP POST call to submit a video for analysis and the initiation of a relevant session in the REST service 2. HTTP GET call to query the status of the initialized session and the progress of the analysis 3. HTTP GET call to retrieve the results of a successfully completed session 22
  23. 23. User interface ● User can submit videos and transform their aspect ratio 23
  24. 24. User interface ● User can submit videos and transform their aspect ratio ● Predefined list of target aspect ratios 24
  25. 25. User interface ● User can submit videos and transform their aspect ratio ● Predefined list of target aspect ratios ● Videos can be either available on-line or locally stored 25
  26. 26. User interface ● User can submit videos and transform their aspect ratio ● Predefined list of target aspect ratios ● Videos can be either available on-line or locally stored ● The landing page includes 10 demo videos, and the ability to provide feedback 26
  27. 27. User interface ● User can submit videos and transform their aspect ratio ● Predefined list of target aspect ratios ● Videos can be either available on-line or locally stored ● The landing page includes 10 demo videos, and the ability to provide feedback 27
  28. 28. User interface ● User can submit videos and transform their aspect ratio ● Predefined list of target aspect ratios ● Videos can be either available on-line or locally stored ● The landing page includes 10 demo videos, and the ability to provide feedback ● Analysis procedure monitoring 28
  29. 29. User interface ● User can submit videos and transform their aspect ratio ● Predefined list of target aspect ratios ● Videos can be either available on-line or locally stored ● The landing page includes 10 demo videos, and the ability to provide feedback ● Analysis procedure monitoring ● On-line inspection of the results through the UI of our tool or download the video file 29
  30. 30. Results ● We utilize the RetargetVid dataset and the evaluation protocol of [1] to compare: ○ Method of [1] ○ Method of [1] + modifications 30
  31. 31. Method Worst (↑) Best (↑) Mean (↑) T (% ↓) Results for 1:3 target aspect ratio [1] 48.6 50.9 49.9 19 [1] + proposed modifications 51.7 53.8 52.9 13 Results for 3:1 target aspect ratio [1] 70.1 73.6 71.4 20 [1] + proposed modifications 74.4 77.0 75.3 14 Results (IoU; time as a percentage of the videos’ duration) (↑: the higher the better; ↓: the lower the better) 31
  32. 32. Links Instructional Video: https://youtu.be/_pdTDMWbIfs Web Application: http://multimedia2.iti.gr/videosmartcropping/service/start.html GitHub Repository: https://github.com/bmezaris/RetargetVid ● Source code of SmartVidCrop ● Source code of SmartVidCrop + our modifications ● Ground-truth annotations for the RetargetVid dataset 32
  33. 33. Try it yourself at: Contacts: Vasileios Mezaris, bmezaris@iti.gr Konstantinos Apostolidis, kapost@iti.gr This work was supported by the H2020 project ReTV (grant agreement No 780656) http://multimedia2.iti.gr/videosmartcropping/service/start.html Or ask us about the underlying algorithms and how these can be integrated in your system 33

×