Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Toward Building a Content based Video Recommendation System Based on Low-level Features

274 views

Published on

In this presentation, I briefly discuss the use of automatically extracted visual features of videos in the context of recommender systems that brings some novel contributions in the domain of video recommendations. The proposed content-based recommender system encompasses a technique to automatically analyze video contents and to extract a set of representative stylistic features (lighting, color, and motion) grounded on existing approaches of Applied Media Theory.
Proposed recommender can be used in combination with more traditional content-based recommendation techniques that exploit explicit content features associated to video files, in order to improve the accuracy of recommendations. Proposed recommender can also be used alone, to address the problem originated from video files that have no meta-data, a typical situation of popular movie-sharing websites (e.g., YouTube) where every day hundred millions of hours of videos are uploaded by users and may contain no associated information. As they lack explicit content, these items cannot be considered for recommendation purposes by conventional content-based techniques even when they could be relevant for the user.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Toward Building a Content based Video Recommendation System Based on Low-level Features

  1. 1. Toward Building a Content-based Video Recommendation System based on Low-level Features Yashar Deldjoo Mehdi Elahi Massimo Quadrana Paolo Cremonesi Corresponding journal ar8cle: Deldjoo, Yashar; Elahi, Mehdi; Cremonesi, Paolo; Garzo?o, Franca; Piazzolla, Pietro; Quadrana, Massimo; ",Content-Based Video Recommenda1on System Based on Stylis1c Visual Features, Journal on Data Seman.cs, 1-15, 2016, Springer
  2. 2. Outline •  Introduction •  New item problem •  LL Feature base Recommendation •  Evaluation and Results •  Future work
  3. 3. tools that support users decision making by suggesting products that can be interesting to them. Examples of Recommender Systems: Recommender Systems:
  4. 4. Is typical done by predicting unknown ratings, by exploiting the content of items or/and ratings given by users. Recommendation: 3 1 2 5 2 3 4
  5. 5. when a new item is added to the catalogue and we don’t have information about it, e.g., no rating is available. New Item Problem: New Item 3 1 ? 2 5 2 ? 3 4 ?
  6. 6. Extreme New Item Problem We have absolutely no information about an item. Example: An unknown video content is uploaded by a unknown user and there is no metadata available. ? ? ? ? ? ? ? ? ? ?
  7. 7. How to make recommendation?
  8. 8. Video Content § There exist 3 main modalities in a video. Visual Audio Text
  9. 9. Video Content § There exist 3 main modalities in a video. § There exists many fearures in each modality. Visual Audio Text Visual Feaures Audio Feaures Textual Features
  10. 10. Video Content § There exist 3 main modalities in a video. § There exists many fearures in each modality. § Our focus Visual features Visual Audio Text Visual Feaures Audio Feaures Textual Features
  11. 11. Visual Features Visual Audio Text Feaures Audio Feaures Textual Features Visual Structure Content
  12. 12. Video Structure Scene: A number of shots that form a semantic unit. Shot: All frames within a single camera action. Frame: One static image from a series of static images constituting a video. Figure: Hierarchical decomposition and representation of video content, http://www.scholarpedia.org/article/Video_Content_Structuring
  13. 13. Example A: SHOT1
  14. 14. Example A: SHOT2
  15. 15. Example A: SHOT2 2CameraShots– 1Scene
  16. 16. Example B: SHOT1
  17. 17. Example B: SHOT1 1CameraShot– 1Scene
  18. 18. Shot Detection Figure: A schematic illustration of shot detection http://www.scholarpedia.org/article/Video_Content_Structuring
  19. 19. Visual Features Average Shotlength Shot Motion Color Variance Lighening Key
  20. 20. Average Shot Length Idea : Slower paced film (e.g. drama) have larger average shot length whereas action movies appear to have shorter average shot length.
  21. 21. Comparing Average Shot Length 0 2 4 6 8 10 12 14 Drama Action shot 1 shot 2 shot 3 shot 4 shot 5 shot 6 shot 7 shot 8 shot 9 shot 10 shot 11 shot 12 shot 13 shot 14 shot 15 shot 16 shot 17 shot 18
  22. 22. Visual Features Average Shotlength Shot Motion Color Variance Lighening Key
  23. 23. Color Variance figure Color Var (a) 0 (b) 0 (c) 0 (d) 0 (e) 0.25 (f) 17.8 (g) 1.4e+9
  24. 24. Horror Comedy
  25. 25. Visual Features Average Shotlength Shot Motion Color Variance Lighening Key Opticalflow
  26. 26. Shot Motion Figure 3: Optical flow of a sample image shown
  27. 27. Visual Features Average Shotlength Shot Motion Color Variance Lighening Key
  28. 28. Video Classfication •  FeatureExtraction
  29. 29. Evaluation •  120 Videos •  4 Main Genres
  30. 30. DissimilarityMetrics
  31. 31. Rating Prediction
  32. 32. Evaluation
  33. 33. Results N 1 2 3 4 5 6 7 8 9 10 recall 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 LL HL - Genre LL+ HL - Genre Corresponding journal ar8cle: Deldjoo, Yashar; Elahi, Mehdi; Cremonesi, Paolo; Garzo?o, Franca; Piazzolla, Pietro; Quadrana, Massimo; ",Content-Based Video Recommenda1on System Based on Stylis1c Visual Features, Journal on Data Seman.cs, 1-15, 2016, Springer
  34. 34. Classification ClassificationAccuracy=73.33% a b c d Classified as 22 5 2 1 a = Action 1 27 2 0 b= Comedy 6 1 22 1 c= Drama 8 0 5 17 d=Horror
  35. 35. Conclusion •  we propose a method to remedy the (extreme) New Item problem in video recommendation domain •  we assume a more realistic scenario, i.e., an up- and-running video recommender with thousands of users •  Result of our experiments shown that we have achieved excellent performance in comparison with considered baselines
  36. 36. Future Work •  Further analysis with bigger datasets, in order to better understand the performance differences among the compared methods. •  Investigation of the impact of different recommendation algorithms, such as those based on Bayesian, or SVD, on the performance of our method. •  including additional sources of information, such as, audio features, in order to farther improve the quality of our content based recommendation method.
  37. 37. Thank you! Corresponding journal ar8cle: Deldjoo, Yashar; Elahi, Mehdi; Cremonesi, Paolo; Garzo?o, Franca; Piazzolla, Pietro; Quadrana, Massimo; ",Content-Based Video Recommenda1on System Based on Stylis1c Visual Features, Journal on Data Seman.cs, 1-15, 2016, Springer

×