Successfully reported this slideshow.
Your SlideShare is downloading. ×

A Fast Smart-Cropping Method and Dataset for Video Retargeting

Ad

A fast smart-cropping method and dataset for
video retargeting
Konstantinos Apostolidis, Vasileios Mezaris
Information Tec...

Ad

Problem Statement
● Traditional TV and desktop computer monitors: landscape aspect ratios (16:9 or 4:3)
● Nowadays, mobile...

Ad

● Straightforward approaches for transforming a video to a different aspect ratio:
○ static cropping of content
○ padding ...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Upcoming SlideShare
CA-SUM Video Summarization
CA-SUM Video Summarization
Loading in …3
×

Check these out next

1 of 19 Ad
1 of 19 Ad

A Fast Smart-Cropping Method and Dataset for Video Retargeting

Download to read offline

K. Apostolidis, V. Mezaris, "A Fast Smart-Cropping Method and Dataset for Video Retargeting", Proc. 28th IEEE Int. Conf. on Image Processing (ICIP), Anchorage, Alaska, US, Sept. 2021. DOI:10.1109/ICIP42928.2021.9506390. Software and dataset available at https://github.com/bmezaris/RetargetVid

In this paper a method that re-targets a video to a different aspect ratio using cropping is presented. We argue that cropping methods are more suitable for video aspect ratio transformation when the minimization of semantic distortions is a prerequisite. For our method, we utilize visual saliency to find the image regions of attention, and we employ a filtering-through clustering technique to select the main region of focus. We additionally introduce the first publicly available benchmark dataset for video cropping, annotated by 6 human subjects. Experimental evaluation on the introduced dataset shows the competitiveness of our method.

K. Apostolidis, V. Mezaris, "A Fast Smart-Cropping Method and Dataset for Video Retargeting", Proc. 28th IEEE Int. Conf. on Image Processing (ICIP), Anchorage, Alaska, US, Sept. 2021. DOI:10.1109/ICIP42928.2021.9506390. Software and dataset available at https://github.com/bmezaris/RetargetVid

In this paper a method that re-targets a video to a different aspect ratio using cropping is presented. We argue that cropping methods are more suitable for video aspect ratio transformation when the minimization of semantic distortions is a prerequisite. For our method, we utilize visual saliency to find the image regions of attention, and we employ a filtering-through clustering technique to select the main region of focus. We additionally introduce the first publicly available benchmark dataset for video cropping, annotated by 6 human subjects. Experimental evaluation on the introduced dataset shows the competitiveness of our method.

Advertisement
Advertisement

More Related Content

Advertisement

A Fast Smart-Cropping Method and Dataset for Video Retargeting

  1. 1. A fast smart-cropping method and dataset for video retargeting Konstantinos Apostolidis, Vasileios Mezaris Information Technologies Institute (ITI), CERTH Thessaloniki, Greece This work was supported by the H2020 project ReTV (grant agreement No 780656)
  2. 2. Problem Statement ● Traditional TV and desktop computer monitors: landscape aspect ratios (16:9 or 4:3) ● Nowadays, mobile devices use different aspect ratios ● Video sharing platforms dictate the use of specific video aspect ratios ● Existing videos would have to be transformed to comply with their specifications
  3. 3. ● Straightforward approaches for transforming a video to a different aspect ratio: ○ static cropping of content ○ padding the frames with black borders ➢significant loss of visual content that might even be in the center of attention ➢shrinks the original video by introducing large borders in the output video Problem Statement ● The results of such simple approaches are often unsatisfactory ● Common video aspect ratio transformation methods of the literature: ○ Warping ○ Seam carving ➢Both introduce distortions and may alter the semantics of the video
  4. 4. Problem Statement 1. SmartVidCrop: ○ New video cropping method that minimizes the loss of semantically important visual content ○ Selects the main focus out of the multiple possible salient regions of the video ○ Introduces a new filtering-through-clustering processing step 2. RetargetVid: ○ First publicly available benchmark dataset for video retargeting ○ Ground-truth video cropping results for 200 videos ○ Each video was annotated by 6 human annotators ○ Two target aspect ratios
  5. 5. Proposed Method ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing
  6. 6. ow oh fh fw ow/oh = 16/9 fw/fh = 4/5 fh = oh fw < ow fh ow oh fw Original frame Final frame Proposed Method ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing ow/oh = 4/5 fw/fh = 16/9 fh < oh fw = ow
  7. 7. Proposed Method ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing
  8. 8. Proposed Method ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing
  9. 9. Proposed Method ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing
  10. 10. Proposed Method ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing
  11. 11. Proposed Method ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing yi xi t Time-series of center of mass displacement
  12. 12. Proposed Method ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing t Shot transitions Video frames Shot #1 Shot #2 Shot #3
  13. 13. Proposed Method ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing Smoothed time-series Inferred time-series t Smoothed time-series of center of mass displacement
  14. 14. ● Read the first out of every five videos frames ● Remove borders ● Calculate crop window dimensions ● Saliency detection ● Thresholding ● Filtering-through-clustering procedure ● Center of mass ● Shot detection ● Time-series smoothing Proposed Method
  15. 15. RetargetVid dataset ● Motivation: Objective comparisons of video retargeting methods are difficult and scarce in the literature ● Construction of a dataset for the task of video aspect ratio transformation ● Publicly available annotations ● 200 videos from the publicly available videos of the DHF1k dataset ● Annotations by 6 human subjects ● 1:3 and 3:1 target aspect ratios ● Dataset presents a good balance between inter-annotator agreement and diversity
  16. 16. Results ● Google AutoFlip: a cropping method for video retargeting with available source code ● We utilize the constructed RetargetVid dataset to compare our method to AutoFlip
  17. 17. Method Worst Best Mean t Results for 1:3 target aspect ratio AutoFlip 50.0 52.1 50.8 40 Ours (w/o clustering step) 45.3 47.4 46.1 17 SmartCrop 48.6 50.9 49.9 19 Results for 3:1 target aspect ratio AutoFlip 70.9 74.7 72.2 41 Ours (w/o clustering step) 66.3 72.9 68.1 17 SmartCrop 70.1 73.6 71.4 20 Results
  18. 18. RetargetVid dataset and SmartVidCrop ● Source code of SmartVidCrop ● Ground-truth annotations for the RetargetVid dataset ● Updated results of our SmartVidCrop method, achieved by further optimizing it are publicly available at: https://github.com/bmezaris/RetargetVid
  19. 19. Try it yourself at: Contacts: Vasileios Mezaris, bmezaris@iti.gr Konstantinos Apostolidis, kapost@iti.gr This work was supported by the H2020 project ReTV (grant agreement No 780656) http://multimedia2.iti.gr/videosmartcropping/service/start.html Or ask us about the underlying algorithms and how these can be integrated in your system

×