1. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Implementing artificial
intelligence strategies for
content annotation and
publication online Vasileios Mezaris, CERTH-ITI
Johan Oomen, NISV
1

2. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Archives’ needs
Fundamental need:
- Generate value out of your own AV content; nothing good comes out of just
keeping the content locked in your digital basement
Technology-wise, this requires:
- Understanding the content / making it discoverable
- Adapting / re-purposing the (discovered) content; generating video summaries
This is where AI can step in!
2

3. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Understanding the content / making it discoverable
Content fragmentation and annotation:
- Identify the different temporal fragments of a video (subshots/shots/scenes)
- Annotate fragments with concept labels that describe them (many thousand labels)
- Generate descriptive captions for each fragment
Research (and business) challenges:
- Accuracy
- Computational efficiency / compactness of the deep networks -> affects costs!
(faster than real-time for a bundle of analysis methods that include fragmentation,
concept detection, brand and logo detection, ad detection,...)
3

4. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Understanding the content / making it discoverable
4
Shot #15
Scene #4 Scene #5
Shot #11 Shot #12 Shot #13 Shot #14 Shot #16
Subshot #58 Subshot #59
Shot #17 Shot #18
Subshot #60
…
… …
……

5. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Understanding the content / making it discoverable
5
Sample video frame Top detected concepts

6. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Understanding the content / making it discoverable
Web application for video analysis and search (try it with your video!):
http://multimedia2.iti.gr/onlinevideoanalysis/service/start.html
Demo video:
https://youtu.be/mO-NRpIJ9UU
REST service available (for integration
in different applications / CMSs)
6

7. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Understanding the content / making it discoverable
Behind the scenes:
- Frame-comparison-based methods for video fragmentation [1]; soon to be
augmented with a deep-learning-based method
- Elaborate deep-convolutional-neural-network architectures for concept-based
annotation [2][3] (and for video captioning; not shown in the demo)
[1] E. Apostolidis, V. Mezaris, "Fast Shot Segmentation Combining Global and Local Visual Descriptors", Proc. IEEE Int. Conf. on Acoustics, Speech and Signal
Processing (ICASSP), Florence, Italy, May 2014. Software available at https://mklab.iti.gr/results/video-shot-and-scene-segmentation/.
[2] F. Markatopoulou, V. Mezaris, I. Patras, "Implicit and Explicit Concept Relations in Deep Neural Networks for Multi-Label Video/Image Annotation", IEEE
Transactions on Circuits and Systems for Video Technology, vol. 29, no. 6, pp. 1631-1644, June 2019. DOI:10.1109/TCSVT.2018.2848458. Software available at
https://github.com/markatopoulou/fvmtl-ccelc.
[3] N. Gkalelis, V. Mezaris, "Subclass deep neural networks: re-enabling neglected classes in deep network training for multimedia classification", Proc. 26th Int.
Conf. on Multimedia Modeling (MMM2020), Daejeon, Korea, Jan. 2020.
7

8. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Adapting / re-purposing the content
Main requirements:
- Target distribution platforms & devices have varying requirements (e.g. the
optimal duration of a video differs from one platform to another)
- Target audiences have different preferences / information needs
Video summarization:
- Create editions of the content that are adapted to different platforms and
audiences
- Post these versions on different platforms: generate value from your content;
attract more audience to it!
8

9. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Adapting / re-purposing the content
Example
- Original video (1’38’’)
- 14’’ summary
- Fully automatic summary generation;
but, editor-in-the-loop mode is also
supported
- REST service available (for
integration in applications / CMSs)
9

10. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Adapting / re-purposing the content
Behind the scenes:
- Elaborate generative adversarial learning architectures (GANs) for
unsupervised learning [4][5]
- Can be trained differently for different content, e.g. separate trained models
can be used for different shows; but, creating these models does not require
manually-generated training data (it’s (almost) for free!)
[4] E. Apostolidis, A. Metsai, E. Adamantidou, V. Mezaris, I. Patras, "A Stepwise, Label-based Approach for Improving the Adversarial Training in Unsupervised
Video Summarization", Proc. 1st Int. Workshop on AI for Smart TV Content Production, Access and Delivery (AI4TV'19) at ACM Multimedia 2019, Nice, France,
October 2019.
[5] E. Apostolidis, E. Adamantidou, A. Metsai, V. Mezaris, I. Patras, "Unsupervised Video Summarization via Attention-Driven Adversarial Learning", Proc. 26th Int.
Conf. on Multimedia Modeling (MMM2020), Daejeon, Korea, Jan. 2020.
10

11. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
ReTV: Audiovisual Content Adaptation,
Repurposing and Publication across Digital Vectors
11
Professional use case:
editorial workflow support
Consumer use case:
chat bot

12. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Editorial workflow for content publication
12
Topic Selection
Content Adaptation
Optimal Publication
Engagement Monitoring

13. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Editorial workflow for content publication
13
Topic Selection
Content Adaptation
Optimal Publication
Engagement Monitoring
- real-time monitoring of trends in the
media
- prediction of trending topics related to
your collection
- suggestions for topics in the editorial
calendar

14. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
example: trends at IFA
14

15. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 15

16. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Editorial workflow for content publication
16
Topic Selection
Content Adaptation
Optimal Publication
Engagement Monitoring
- automated video summarisation replacing
manual video editing
- adaptation for specific social media
platforms - different length, cropping
format

17. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 17

18. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Editorial workflow for content publication
18
Topic Selection
Content Adaptation
Optimal Publication
Engagement Monitoring
- publishing time tailored for each vector
based audience behaviour
- text suggestions for creating stories with
impact

19. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 19

20. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Editorial workflow for content publication
20
Topic Selection
Content Adaptation
Optimal Publication
Engagement Monitoring
- improving future posts by monitoring
audience engagement

21. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
ReTV Chatbot
Bringing TV content via channels convenient to
audiences
Delivering content tailored for online consumption
Creating engagement
Content personalisation for each user via
interaction with via chatbot
21

22. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 22

23. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Vasileios Mezaris, CERTH-ITI
bmezaris@iti.gr
Johan Oomen, NISV
joomen@beeldengeluid.nl
@johanoomen
23
This work was supported by the EUs Horizon 2020
research and innovation programme under grant
agreement H2020-780656 ReTV