Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Implementing artificial intelligence strategies for content annotation and publication online

326 views

Published on

a presentation by Vasileios Mezaris from CERTH-ITI and Johan Oomen from NISV at the FIAT / ITFA 2019 conference.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Implementing artificial intelligence strategies for content annotation and publication online

  1. 1. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Implementing artificial intelligence strategies for content annotation and publication online Vasileios Mezaris, CERTH-ITI Johan Oomen, NISV 1
  2. 2. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Archives’ needs Fundamental need: - Generate value out of your own AV content; nothing good comes out of just keeping the content locked in your digital basement Technology-wise, this requires: - Understanding the content / making it discoverable - Adapting / re-purposing the (discovered) content; generating video summaries This is where AI can step in! 2
  3. 3. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Understanding the content / making it discoverable Content fragmentation and annotation: - Identify the different temporal fragments of a video (subshots/shots/scenes) - Annotate fragments with concept labels that describe them (many thousand labels) - Generate descriptive captions for each fragment Research (and business) challenges: - Accuracy - Computational efficiency / compactness of the deep networks -> affects costs! (faster than real-time for a bundle of analysis methods that include fragmentation, concept detection, brand and logo detection, ad detection,...) 3
  4. 4. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Understanding the content / making it discoverable 4 Shot #15 Scene #4 Scene #5 Shot #11 Shot #12 Shot #13 Shot #14 Shot #16 Subshot #58 Subshot #59 Shot #17 Shot #18 Subshot #60 … … … ……
  5. 5. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Understanding the content / making it discoverable 5 Sample video frame Top detected concepts
  6. 6. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Understanding the content / making it discoverable Web application for video analysis and search (try it with your video!): http://multimedia2.iti.gr/onlinevideoanalysis/service/start.html Demo video: https://youtu.be/mO-NRpIJ9UU REST service available (for integration in different applications / CMSs) 6
  7. 7. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Understanding the content / making it discoverable Behind the scenes: - Frame-comparison-based methods for video fragmentation [1]; soon to be augmented with a deep-learning-based method - Elaborate deep-convolutional-neural-network architectures for concept-based annotation [2][3] (and for video captioning; not shown in the demo) [1] E. Apostolidis, V. Mezaris, "Fast Shot Segmentation Combining Global and Local Visual Descriptors", Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy, May 2014. Software available at https://mklab.iti.gr/results/video-shot-and-scene-segmentation/. [2] F. Markatopoulou, V. Mezaris, I. Patras, "Implicit and Explicit Concept Relations in Deep Neural Networks for Multi-Label Video/Image Annotation", IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 6, pp. 1631-1644, June 2019. DOI:10.1109/TCSVT.2018.2848458. Software available at https://github.com/markatopoulou/fvmtl-ccelc. [3] N. Gkalelis, V. Mezaris, "Subclass deep neural networks: re-enabling neglected classes in deep network training for multimedia classification", Proc. 26th Int. Conf. on Multimedia Modeling (MMM2020), Daejeon, Korea, Jan. 2020. 7
  8. 8. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Adapting / re-purposing the content Main requirements: - Target distribution platforms & devices have varying requirements (e.g. the optimal duration of a video differs from one platform to another) - Target audiences have different preferences / information needs Video summarization: - Create editions of the content that are adapted to different platforms and audiences - Post these versions on different platforms: generate value from your content; attract more audience to it! 8
  9. 9. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Adapting / re-purposing the content Example - Original video (1’38’’) - 14’’ summary - Fully automatic summary generation; but, editor-in-the-loop mode is also supported - REST service available (for integration in applications / CMSs) 9
  10. 10. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Adapting / re-purposing the content Behind the scenes: - Elaborate generative adversarial learning architectures (GANs) for unsupervised learning [4][5] - Can be trained differently for different content, e.g. separate trained models can be used for different shows; but, creating these models does not require manually-generated training data (it’s (almost) for free!) [4] E. Apostolidis, A. Metsai, E. Adamantidou, V. Mezaris, I. Patras, "A Stepwise, Label-based Approach for Improving the Adversarial Training in Unsupervised Video Summarization", Proc. 1st Int. Workshop on AI for Smart TV Content Production, Access and Delivery (AI4TV'19) at ACM Multimedia 2019, Nice, France, October 2019. [5] E. Apostolidis, E. Adamantidou, A. Metsai, V. Mezaris, I. Patras, "Unsupervised Video Summarization via Attention-Driven Adversarial Learning", Proc. 26th Int. Conf. on Multimedia Modeling (MMM2020), Daejeon, Korea, Jan. 2020. 10
  11. 11. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project ReTV: Audiovisual Content Adaptation, Repurposing and Publication across Digital Vectors 11 Professional use case: editorial workflow support Consumer use case: chat bot
  12. 12. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Editorial workflow for content publication 12 Topic Selection Content Adaptation Optimal Publication Engagement Monitoring
  13. 13. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Editorial workflow for content publication 13 Topic Selection Content Adaptation Optimal Publication Engagement Monitoring - real-time monitoring of trends in the media - prediction of trending topics related to your collection - suggestions for topics in the editorial calendar
  14. 14. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project example: trends at IFA 14
  15. 15. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 15
  16. 16. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Editorial workflow for content publication 16 Topic Selection Content Adaptation Optimal Publication Engagement Monitoring - automated video summarisation replacing manual video editing - adaptation for specific social media platforms - different length, cropping format
  17. 17. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 17
  18. 18. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Editorial workflow for content publication 18 Topic Selection Content Adaptation Optimal Publication Engagement Monitoring - publishing time tailored for each vector based audience behaviour - text suggestions for creating stories with impact
  19. 19. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 19
  20. 20. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Editorial workflow for content publication 20 Topic Selection Content Adaptation Optimal Publication Engagement Monitoring - improving future posts by monitoring audience engagement
  21. 21. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project ReTV Chatbot Bringing TV content via channels convenient to audiences Delivering content tailored for online consumption Creating engagement Content personalisation for each user via interaction with via chatbot 21
  22. 22. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 22
  23. 23. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Vasileios Mezaris, CERTH-ITI bmezaris@iti.gr Johan Oomen, NISV joomen@beeldengeluid.nl @johanoomen 23 This work was supported by the EUs Horizon 2020 research and innovation programme under grant agreement H2020-780656 ReTV

×