Implementing artificial intelligence strategies for content annotation and publication online

ReTV project
ReTV projectReTV project
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Implementing artificial
intelligence strategies for
content annotation and
publication online Vasileios Mezaris, CERTH-ITI
Johan Oomen, NISV
1
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Archives’ needs
Fundamental need:
- Generate value out of your own AV content; nothing good comes out of just
keeping the content locked in your digital basement
Technology-wise, this requires:
- Understanding the content / making it discoverable
- Adapting / re-purposing the (discovered) content; generating video summaries
This is where AI can step in!
2
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Understanding the content / making it discoverable
Content fragmentation and annotation:
- Identify the different temporal fragments of a video (subshots/shots/scenes)
- Annotate fragments with concept labels that describe them (many thousand labels)
- Generate descriptive captions for each fragment
Research (and business) challenges:
- Accuracy
- Computational efficiency / compactness of the deep networks -> affects costs!
(faster than real-time for a bundle of analysis methods that include fragmentation,
concept detection, brand and logo detection, ad detection,...)
3
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Understanding the content / making it discoverable
4
Shot #15
Scene #4 Scene #5
Shot #11 Shot #12 Shot #13 Shot #14 Shot #16
Subshot #58 Subshot #59
Shot #17 Shot #18
Subshot #60
…
… …
……
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Understanding the content / making it discoverable
5
Sample video frame Top detected concepts
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Understanding the content / making it discoverable
Web application for video analysis and search (try it with your video!):
http://multimedia2.iti.gr/onlinevideoanalysis/service/start.html
Demo video:
https://youtu.be/mO-NRpIJ9UU
REST service available (for integration
in different applications / CMSs)
6
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Understanding the content / making it discoverable
Behind the scenes:
- Frame-comparison-based methods for video fragmentation [1]; soon to be
augmented with a deep-learning-based method
- Elaborate deep-convolutional-neural-network architectures for concept-based
annotation [2][3] (and for video captioning; not shown in the demo)
[1] E. Apostolidis, V. Mezaris, "Fast Shot Segmentation Combining Global and Local Visual Descriptors", Proc. IEEE Int. Conf. on Acoustics, Speech and Signal
Processing (ICASSP), Florence, Italy, May 2014. Software available at https://mklab.iti.gr/results/video-shot-and-scene-segmentation/.
[2] F. Markatopoulou, V. Mezaris, I. Patras, "Implicit and Explicit Concept Relations in Deep Neural Networks for Multi-Label Video/Image Annotation", IEEE
Transactions on Circuits and Systems for Video Technology, vol. 29, no. 6, pp. 1631-1644, June 2019. DOI:10.1109/TCSVT.2018.2848458. Software available at
https://github.com/markatopoulou/fvmtl-ccelc.
[3] N. Gkalelis, V. Mezaris, "Subclass deep neural networks: re-enabling neglected classes in deep network training for multimedia classification", Proc. 26th Int.
Conf. on Multimedia Modeling (MMM2020), Daejeon, Korea, Jan. 2020.
7
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Adapting / re-purposing the content
Main requirements:
- Target distribution platforms & devices have varying requirements (e.g. the
optimal duration of a video differs from one platform to another)
- Target audiences have different preferences / information needs
Video summarization:
- Create editions of the content that are adapted to different platforms and
audiences
- Post these versions on different platforms: generate value from your content;
attract more audience to it!
8
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Adapting / re-purposing the content
Example
- Original video (1’38’’)
- 14’’ summary
- Fully automatic summary generation;
but, editor-in-the-loop mode is also
supported
- REST service available (for
integration in applications / CMSs)
9
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Adapting / re-purposing the content
Behind the scenes:
- Elaborate generative adversarial learning architectures (GANs) for
unsupervised learning [4][5]
- Can be trained differently for different content, e.g. separate trained models
can be used for different shows; but, creating these models does not require
manually-generated training data (it’s (almost) for free!)
[4] E. Apostolidis, A. Metsai, E. Adamantidou, V. Mezaris, I. Patras, "A Stepwise, Label-based Approach for Improving the Adversarial Training in Unsupervised
Video Summarization", Proc. 1st Int. Workshop on AI for Smart TV Content Production, Access and Delivery (AI4TV'19) at ACM Multimedia 2019, Nice, France,
October 2019.
[5] E. Apostolidis, E. Adamantidou, A. Metsai, V. Mezaris, I. Patras, "Unsupervised Video Summarization via Attention-Driven Adversarial Learning", Proc. 26th Int.
Conf. on Multimedia Modeling (MMM2020), Daejeon, Korea, Jan. 2020.
10
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
ReTV: Audiovisual Content Adaptation,
Repurposing and Publication across Digital Vectors
11
Professional use case:
editorial workflow support
Consumer use case:
chat bot
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Editorial workflow for content publication
12
Topic Selection
Content Adaptation
Optimal Publication
Engagement Monitoring
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Editorial workflow for content publication
13
Topic Selection
Content Adaptation
Optimal Publication
Engagement Monitoring
- real-time monitoring of trends in the
media
- prediction of trending topics related to
your collection
- suggestions for topics in the editorial
calendar
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
example: trends at IFA
14
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 15
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Editorial workflow for content publication
16
Topic Selection
Content Adaptation
Optimal Publication
Engagement Monitoring
- automated video summarisation replacing
manual video editing
- adaptation for specific social media
platforms - different length, cropping
format
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 17
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Editorial workflow for content publication
18
Topic Selection
Content Adaptation
Optimal Publication
Engagement Monitoring
- publishing time tailored for each vector
based audience behaviour
- text suggestions for creating stories with
impact
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 19
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Editorial workflow for content publication
20
Topic Selection
Content Adaptation
Optimal Publication
Engagement Monitoring
- improving future posts by monitoring
audience engagement
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
ReTV Chatbot
Bringing TV content via channels convenient to
audiences
Delivering content tailored for online consumption
Creating engagement
Content personalisation for each user via
interaction with via chatbot
21
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 22
retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project
Vasileios Mezaris, CERTH-ITI
bmezaris@iti.gr
Johan Oomen, NISV
joomen@beeldengeluid.nl
@johanoomen
23
This work was supported by the EUs Horizon 2020
research and innovation programme under grant
agreement H2020-780656 ReTV
1 of 23

Recommended

ICME 2020 Tutorial Part II: Video summary (re-)use and recommendation by
ICME 2020 Tutorial Part II: Video summary (re-)use and recommendationICME 2020 Tutorial Part II: Video summary (re-)use and recommendation
ICME 2020 Tutorial Part II: Video summary (re-)use and recommendationReTV project
803 views39 slides
Icme2020 tutorial video_summarization_part1 by
Icme2020 tutorial video_summarization_part1Icme2020 tutorial video_summarization_part1
Icme2020 tutorial video_summarization_part1VasileiosMezaris
1K views112 slides
ReTV at EBU MDN Workshop 2020 by
ReTV at EBU MDN Workshop 2020ReTV at EBU MDN Workshop 2020
ReTV at EBU MDN Workshop 2020ReTV project
733 views20 slides
ReTV AI4TV Summarization by
ReTV AI4TV SummarizationReTV AI4TV Summarization
ReTV AI4TV SummarizationReTV project
1.8K views36 slides
Unsupervised Video Summarization via Attention-Driven Adversarial Learning by
Unsupervised Video Summarization via Attention-Driven Adversarial LearningUnsupervised Video Summarization via Attention-Driven Adversarial Learning
Unsupervised Video Summarization via Attention-Driven Adversarial LearningVasileiosMezaris
1.3K views50 slides
ReTV: Come Aboard the Voyage of Audiovisual Content Reuse in the Digital Age by
ReTV: Come Aboard the Voyage of Audiovisual Content Reuse in the Digital AgeReTV: Come Aboard the Voyage of Audiovisual Content Reuse in the Digital Age
ReTV: Come Aboard the Voyage of Audiovisual Content Reuse in the Digital AgeReTV project
323 views16 slides

More Related Content

What's hot

Using TV Metadata to optimise the repurposing and republication of TV Content... by
Using TV Metadata to optimise the repurposing and republication of TV Content...Using TV Metadata to optimise the repurposing and republication of TV Content...
Using TV Metadata to optimise the repurposing and republication of TV Content...ReTV project
3.8K views24 slides
GAN-based video summarization by
GAN-based video summarizationGAN-based video summarization
GAN-based video summarizationVasileiosMezaris
367 views38 slides
PoR_evaluation_measure_acm_mm_2020 by
PoR_evaluation_measure_acm_mm_2020PoR_evaluation_measure_acm_mm_2020
PoR_evaluation_measure_acm_mm_2020VasileiosMezaris
110 views55 slides
ReTV AI4TV 2020 by
ReTV AI4TV 2020ReTV AI4TV 2020
ReTV AI4TV 2020ReTV project
472 views14 slides
ReTV @ cross media cafe 2018 by
ReTV @ cross media cafe 2018ReTV @ cross media cafe 2018
ReTV @ cross media cafe 2018ReTV project
2.4K views7 slides
Hard-Negatives Selection Strategy for Cross-Modal Retrieval by
Hard-Negatives Selection Strategy for Cross-Modal RetrievalHard-Negatives Selection Strategy for Cross-Modal Retrieval
Hard-Negatives Selection Strategy for Cross-Modal RetrievalVasileiosMezaris
80 views15 slides

What's hot(14)

Using TV Metadata to optimise the repurposing and republication of TV Content... by ReTV project
Using TV Metadata to optimise the repurposing and republication of TV Content...Using TV Metadata to optimise the repurposing and republication of TV Content...
Using TV Metadata to optimise the repurposing and republication of TV Content...
ReTV project3.8K views
ReTV @ cross media cafe 2018 by ReTV project
ReTV @ cross media cafe 2018ReTV @ cross media cafe 2018
ReTV @ cross media cafe 2018
ReTV project2.4K views
Hard-Negatives Selection Strategy for Cross-Modal Retrieval by VasileiosMezaris
Hard-Negatives Selection Strategy for Cross-Modal RetrievalHard-Negatives Selection Strategy for Cross-Modal Retrieval
Hard-Negatives Selection Strategy for Cross-Modal Retrieval
VasileiosMezaris80 views
From TV to ReTV, Keynote by Lyndon Nixon at TVX 2019 @datatv by ReTV project
 From TV to ReTV, Keynote by Lyndon Nixon at TVX 2019 @datatv  From TV to ReTV, Keynote by Lyndon Nixon at TVX 2019 @datatv
From TV to ReTV, Keynote by Lyndon Nixon at TVX 2019 @datatv
ReTV project1.4K views
HbbTV 2.0 for LinkedTV: specification and gaps by LinkedTV
HbbTV 2.0 for LinkedTV: specification and gapsHbbTV 2.0 for LinkedTV: specification and gaps
HbbTV 2.0 for LinkedTV: specification and gaps
LinkedTV 3.7K views
Requirements document for LinkedTV user interfaces by LinkedTV
Requirements document for LinkedTV user interfacesRequirements document for LinkedTV user interfaces
Requirements document for LinkedTV user interfaces
LinkedTV 1.9K views
LinkedTV Deliverable 9.3 Final LinkedTV Project Report by LinkedTV
LinkedTV Deliverable 9.3 Final LinkedTV Project ReportLinkedTV Deliverable 9.3 Final LinkedTV Project Report
LinkedTV Deliverable 9.3 Final LinkedTV Project Report
LinkedTV 2.4K views
About IRT Nanoelec by IRTNanoelec
About IRT NanoelecAbout IRT Nanoelec
About IRT Nanoelec
IRTNanoelec444 views
Ecpg recommendations tenderix_bendo_092010 by Zoltán Bendó
Ecpg recommendations tenderix_bendo_092010Ecpg recommendations tenderix_bendo_092010
Ecpg recommendations tenderix_bendo_092010
Zoltán Bendó187 views
LinkedTV Deliverable D2.6 LinkedTV Framework for Generating Video Enrichments... by LinkedTV
LinkedTV Deliverable D2.6 LinkedTV Framework for Generating Video Enrichments...LinkedTV Deliverable D2.6 LinkedTV Framework for Generating Video Enrichments...
LinkedTV Deliverable D2.6 LinkedTV Framework for Generating Video Enrichments...
LinkedTV 1.4K views

Similar to Implementing artificial intelligence strategies for content annotation and publication online

Content Adaptation, Personalisation and Fine-Grained Retrieval: Applying AI ... by
Content Adaptation, Personalisation and Fine-Grained Retrieval:  Applying AI ...Content Adaptation, Personalisation and Fine-Grained Retrieval:  Applying AI ...
Content Adaptation, Personalisation and Fine-Grained Retrieval: Applying AI ...ReTV project
34 views25 slides
Arneb by
ArnebArneb
ArnebAndrea Ferracani
287 views2 slides
Matteo Valoriani, Antimo Musone - The Future of Factory - Codemotion Rome 2019 by
Matteo Valoriani, Antimo Musone - The Future of Factory - Codemotion Rome 2019Matteo Valoriani, Antimo Musone - The Future of Factory - Codemotion Rome 2019
Matteo Valoriani, Antimo Musone - The Future of Factory - Codemotion Rome 2019Codemotion
396 views35 slides
Shanling_resume_1019 by
Shanling_resume_1019Shanling_resume_1019
Shanling_resume_1019lucifer1986
311 views4 slides
MICO — Towards Contextual Media Analysis by
MICO — Towards Contextual Media AnalysisMICO — Towards Contextual Media Analysis
MICO — Towards Contextual Media AnalysisThomas Kurz
2.6K views15 slides
Shanling_resume by
Shanling_resumeShanling_resume
Shanling_resumelucifer1986
259 views4 slides

Similar to Implementing artificial intelligence strategies for content annotation and publication online(20)

Content Adaptation, Personalisation and Fine-Grained Retrieval: Applying AI ... by ReTV project
Content Adaptation, Personalisation and Fine-Grained Retrieval:  Applying AI ...Content Adaptation, Personalisation and Fine-Grained Retrieval:  Applying AI ...
Content Adaptation, Personalisation and Fine-Grained Retrieval: Applying AI ...
ReTV project34 views
Matteo Valoriani, Antimo Musone - The Future of Factory - Codemotion Rome 2019 by Codemotion
Matteo Valoriani, Antimo Musone - The Future of Factory - Codemotion Rome 2019Matteo Valoriani, Antimo Musone - The Future of Factory - Codemotion Rome 2019
Matteo Valoriani, Antimo Musone - The Future of Factory - Codemotion Rome 2019
Codemotion396 views
Shanling_resume_1019 by lucifer1986
Shanling_resume_1019Shanling_resume_1019
Shanling_resume_1019
lucifer1986311 views
MICO — Towards Contextual Media Analysis by Thomas Kurz
MICO — Towards Contextual Media AnalysisMICO — Towards Contextual Media Analysis
MICO — Towards Contextual Media Analysis
Thomas Kurz2.6K views
Advene As A Tailorable Hypervideo Authoring Tool A Case Study by Laurie Smith
Advene As A Tailorable Hypervideo Authoring Tool  A Case StudyAdvene As A Tailorable Hypervideo Authoring Tool  A Case Study
Advene As A Tailorable Hypervideo Authoring Tool A Case Study
Laurie Smith2 views
[Webinar] Building a Front-end for the Nuxeo Platform with AngularJS by Nuxeo
[Webinar] Building a Front-end for the Nuxeo Platform with AngularJS[Webinar] Building a Front-end for the Nuxeo Platform with AngularJS
[Webinar] Building a Front-end for the Nuxeo Platform with AngularJS
Nuxeo5.2K views
"Platform Engineering in practice — Why and How to start", Serg Hospodarets by Fwdays
"Platform Engineering in practice — Why and How to start", Serg Hospodarets "Platform Engineering in practice — Why and How to start", Serg Hospodarets
"Platform Engineering in practice — Why and How to start", Serg Hospodarets
Fwdays103 views
SUMMARY GENERATION FOR LECTURING VIDEOS by IRJET Journal
SUMMARY GENERATION FOR LECTURING VIDEOSSUMMARY GENERATION FOR LECTURING VIDEOS
SUMMARY GENERATION FOR LECTURING VIDEOS
IRJET Journal3 views
Learning with (re)Purpose: How to Turn Any Event into Durable Online Video Le... by Michael Kolowich
Learning with (re)Purpose: How to Turn Any Event into Durable Online Video Le...Learning with (re)Purpose: How to Turn Any Event into Durable Online Video Le...
Learning with (re)Purpose: How to Turn Any Event into Durable Online Video Le...
Michael Kolowich990 views
Building a design system with (p)react by Bart Waardenburg
Building a design system with (p)reactBuilding a design system with (p)react
Building a design system with (p)react
Bart Waardenburg187 views
Rosinski ibm ai overview with several examples of projects in the media and l... by FIAT/IFTA
Rosinski ibm ai overview with several examples of projects in the media and l...Rosinski ibm ai overview with several examples of projects in the media and l...
Rosinski ibm ai overview with several examples of projects in the media and l...
FIAT/IFTA129 views
Leveraging OSGi-based Architecture, GWT, and Eclipse to build a large ajax-ba... by Nuxeo
Leveraging OSGi-based Architecture, GWT, and Eclipse to build a large ajax-ba...Leveraging OSGi-based Architecture, GWT, and Eclipse to build a large ajax-ba...
Leveraging OSGi-based Architecture, GWT, and Eclipse to build a large ajax-ba...
Nuxeo972 views
Review on content based video lecture retrieval by eSAT Journals
Review on content based video lecture retrievalReview on content based video lecture retrieval
Review on content based video lecture retrieval
eSAT Journals178 views
RAI Archives: Looking to the future. Alberto Messina, Laurent Boch, RAI. by FIAT/IFTA
RAI Archives: Looking to the future. Alberto Messina, Laurent Boch, RAI.RAI Archives: Looking to the future. Alberto Messina, Laurent Boch, RAI.
RAI Archives: Looking to the future. Alberto Messina, Laurent Boch, RAI.
FIAT/IFTA719 views
Freddie Mac Internship Overview by Charles Stolze
Freddie Mac Internship OverviewFreddie Mac Internship Overview
Freddie Mac Internship Overview
Charles Stolze85 views
SensorStudio introduction (IDC 2016) by Herve Blanc
SensorStudio introduction (IDC 2016)SensorStudio introduction (IDC 2016)
SensorStudio introduction (IDC 2016)
Herve Blanc227 views
Knowledge base Design for Project Based Consulting Orgs by SHAHZAD M. SALEEM
Knowledge base Design for Project Based Consulting OrgsKnowledge base Design for Project Based Consulting Orgs
Knowledge base Design for Project Based Consulting Orgs
SHAHZAD M. SALEEM195 views

Recently uploaded

.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra... by
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra....NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra...
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra...Marc Müller
38 views62 slides
DSD-INT 2023 Exploring flash flood hazard reduction in arid regions using a h... by
DSD-INT 2023 Exploring flash flood hazard reduction in arid regions using a h...DSD-INT 2023 Exploring flash flood hazard reduction in arid regions using a h...
DSD-INT 2023 Exploring flash flood hazard reduction in arid regions using a h...Deltares
5 views31 slides
Fleet Management Software in India by
Fleet Management Software in India Fleet Management Software in India
Fleet Management Software in India Fleetable
11 views1 slide
WebAssembly by
WebAssemblyWebAssembly
WebAssemblyJens Siebert
48 views18 slides
Software evolution understanding: Automatic extraction of software identifier... by
Software evolution understanding: Automatic extraction of software identifier...Software evolution understanding: Automatic extraction of software identifier...
Software evolution understanding: Automatic extraction of software identifier...Ra'Fat Al-Msie'deen
9 views33 slides
DSD-INT 2023 The Danube Hazardous Substances Model - Kovacs by
DSD-INT 2023 The Danube Hazardous Substances Model - KovacsDSD-INT 2023 The Danube Hazardous Substances Model - Kovacs
DSD-INT 2023 The Danube Hazardous Substances Model - KovacsDeltares
8 views17 slides

Recently uploaded(20)

.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra... by Marc Müller
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra....NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra...
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra...
Marc Müller38 views
DSD-INT 2023 Exploring flash flood hazard reduction in arid regions using a h... by Deltares
DSD-INT 2023 Exploring flash flood hazard reduction in arid regions using a h...DSD-INT 2023 Exploring flash flood hazard reduction in arid regions using a h...
DSD-INT 2023 Exploring flash flood hazard reduction in arid regions using a h...
Deltares5 views
Fleet Management Software in India by Fleetable
Fleet Management Software in India Fleet Management Software in India
Fleet Management Software in India
Fleetable11 views
Software evolution understanding: Automatic extraction of software identifier... by Ra'Fat Al-Msie'deen
Software evolution understanding: Automatic extraction of software identifier...Software evolution understanding: Automatic extraction of software identifier...
Software evolution understanding: Automatic extraction of software identifier...
DSD-INT 2023 The Danube Hazardous Substances Model - Kovacs by Deltares
DSD-INT 2023 The Danube Hazardous Substances Model - KovacsDSD-INT 2023 The Danube Hazardous Substances Model - Kovacs
DSD-INT 2023 The Danube Hazardous Substances Model - Kovacs
Deltares8 views
AI and Ml presentation .pptx by FayazAli87
AI and Ml presentation .pptxAI and Ml presentation .pptx
AI and Ml presentation .pptx
FayazAli8711 views
Advanced API Mocking Techniques by Dimpy Adhikary
Advanced API Mocking TechniquesAdvanced API Mocking Techniques
Advanced API Mocking Techniques
Dimpy Adhikary19 views
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI... by Marc Müller
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...
Marc Müller37 views
Quality Engineer: A Day in the Life by John Valentino
Quality Engineer: A Day in the LifeQuality Engineer: A Day in the Life
Quality Engineer: A Day in the Life
John Valentino6 views
SUGCON ANZ Presentation V2.1 Final.pptx by Jack Spektor
SUGCON ANZ Presentation V2.1 Final.pptxSUGCON ANZ Presentation V2.1 Final.pptx
SUGCON ANZ Presentation V2.1 Final.pptx
Jack Spektor22 views
Navigating container technology for enhanced security by Niklas Saari by Metosin Oy
Navigating container technology for enhanced security by Niklas SaariNavigating container technology for enhanced security by Niklas Saari
Navigating container technology for enhanced security by Niklas Saari
Metosin Oy14 views
DSD-INT 2023 European Digital Twin Ocean and Delft3D FM - Dols by Deltares
DSD-INT 2023 European Digital Twin Ocean and Delft3D FM - DolsDSD-INT 2023 European Digital Twin Ocean and Delft3D FM - Dols
DSD-INT 2023 European Digital Twin Ocean and Delft3D FM - Dols
Deltares7 views
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ... by Deltares
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
Deltares11 views
Copilot Prompting Toolkit_All Resources.pdf by Riccardo Zamana
Copilot Prompting Toolkit_All Resources.pdfCopilot Prompting Toolkit_All Resources.pdf
Copilot Prompting Toolkit_All Resources.pdf
Riccardo Zamana8 views
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports by Ra'Fat Al-Msie'deen
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug ReportsBushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t... by Deltares
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...
Deltares9 views

Implementing artificial intelligence strategies for content annotation and publication online

  • 1. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Implementing artificial intelligence strategies for content annotation and publication online Vasileios Mezaris, CERTH-ITI Johan Oomen, NISV 1
  • 2. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Archives’ needs Fundamental need: - Generate value out of your own AV content; nothing good comes out of just keeping the content locked in your digital basement Technology-wise, this requires: - Understanding the content / making it discoverable - Adapting / re-purposing the (discovered) content; generating video summaries This is where AI can step in! 2
  • 3. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Understanding the content / making it discoverable Content fragmentation and annotation: - Identify the different temporal fragments of a video (subshots/shots/scenes) - Annotate fragments with concept labels that describe them (many thousand labels) - Generate descriptive captions for each fragment Research (and business) challenges: - Accuracy - Computational efficiency / compactness of the deep networks -> affects costs! (faster than real-time for a bundle of analysis methods that include fragmentation, concept detection, brand and logo detection, ad detection,...) 3
  • 4. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Understanding the content / making it discoverable 4 Shot #15 Scene #4 Scene #5 Shot #11 Shot #12 Shot #13 Shot #14 Shot #16 Subshot #58 Subshot #59 Shot #17 Shot #18 Subshot #60 … … … ……
  • 5. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Understanding the content / making it discoverable 5 Sample video frame Top detected concepts
  • 6. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Understanding the content / making it discoverable Web application for video analysis and search (try it with your video!): http://multimedia2.iti.gr/onlinevideoanalysis/service/start.html Demo video: https://youtu.be/mO-NRpIJ9UU REST service available (for integration in different applications / CMSs) 6
  • 7. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Understanding the content / making it discoverable Behind the scenes: - Frame-comparison-based methods for video fragmentation [1]; soon to be augmented with a deep-learning-based method - Elaborate deep-convolutional-neural-network architectures for concept-based annotation [2][3] (and for video captioning; not shown in the demo) [1] E. Apostolidis, V. Mezaris, "Fast Shot Segmentation Combining Global and Local Visual Descriptors", Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy, May 2014. Software available at https://mklab.iti.gr/results/video-shot-and-scene-segmentation/. [2] F. Markatopoulou, V. Mezaris, I. Patras, "Implicit and Explicit Concept Relations in Deep Neural Networks for Multi-Label Video/Image Annotation", IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 6, pp. 1631-1644, June 2019. DOI:10.1109/TCSVT.2018.2848458. Software available at https://github.com/markatopoulou/fvmtl-ccelc. [3] N. Gkalelis, V. Mezaris, "Subclass deep neural networks: re-enabling neglected classes in deep network training for multimedia classification", Proc. 26th Int. Conf. on Multimedia Modeling (MMM2020), Daejeon, Korea, Jan. 2020. 7
  • 8. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Adapting / re-purposing the content Main requirements: - Target distribution platforms & devices have varying requirements (e.g. the optimal duration of a video differs from one platform to another) - Target audiences have different preferences / information needs Video summarization: - Create editions of the content that are adapted to different platforms and audiences - Post these versions on different platforms: generate value from your content; attract more audience to it! 8
  • 9. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Adapting / re-purposing the content Example - Original video (1’38’’) - 14’’ summary - Fully automatic summary generation; but, editor-in-the-loop mode is also supported - REST service available (for integration in applications / CMSs) 9
  • 10. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Adapting / re-purposing the content Behind the scenes: - Elaborate generative adversarial learning architectures (GANs) for unsupervised learning [4][5] - Can be trained differently for different content, e.g. separate trained models can be used for different shows; but, creating these models does not require manually-generated training data (it’s (almost) for free!) [4] E. Apostolidis, A. Metsai, E. Adamantidou, V. Mezaris, I. Patras, "A Stepwise, Label-based Approach for Improving the Adversarial Training in Unsupervised Video Summarization", Proc. 1st Int. Workshop on AI for Smart TV Content Production, Access and Delivery (AI4TV'19) at ACM Multimedia 2019, Nice, France, October 2019. [5] E. Apostolidis, E. Adamantidou, A. Metsai, V. Mezaris, I. Patras, "Unsupervised Video Summarization via Attention-Driven Adversarial Learning", Proc. 26th Int. Conf. on Multimedia Modeling (MMM2020), Daejeon, Korea, Jan. 2020. 10
  • 11. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project ReTV: Audiovisual Content Adaptation, Repurposing and Publication across Digital Vectors 11 Professional use case: editorial workflow support Consumer use case: chat bot
  • 12. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Editorial workflow for content publication 12 Topic Selection Content Adaptation Optimal Publication Engagement Monitoring
  • 13. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Editorial workflow for content publication 13 Topic Selection Content Adaptation Optimal Publication Engagement Monitoring - real-time monitoring of trends in the media - prediction of trending topics related to your collection - suggestions for topics in the editorial calendar
  • 14. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project example: trends at IFA 14
  • 15. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 15
  • 16. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Editorial workflow for content publication 16 Topic Selection Content Adaptation Optimal Publication Engagement Monitoring - automated video summarisation replacing manual video editing - adaptation for specific social media platforms - different length, cropping format
  • 17. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 17
  • 18. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Editorial workflow for content publication 18 Topic Selection Content Adaptation Optimal Publication Engagement Monitoring - publishing time tailored for each vector based audience behaviour - text suggestions for creating stories with impact
  • 19. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 19
  • 20. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Editorial workflow for content publication 20 Topic Selection Content Adaptation Optimal Publication Engagement Monitoring - improving future posts by monitoring audience engagement
  • 21. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project ReTV Chatbot Bringing TV content via channels convenient to audiences Delivering content tailored for online consumption Creating engagement Content personalisation for each user via interaction with via chatbot 21
  • 22. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project 22
  • 23. retv-project.eu @ReTV_EU @ReTVproject retv-project retv_project Vasileios Mezaris, CERTH-ITI bmezaris@iti.gr Johan Oomen, NISV joomen@beeldengeluid.nl @johanoomen 23 This work was supported by the EUs Horizon 2020 research and innovation programme under grant agreement H2020-780656 ReTV

Editor's Notes

  1. https://www.storypact.com/