Successfully reported this slideshow.
AUTOMATIC KEYFRAME SELECTION
BASED ON

MUTUAL REINFORCEMENT ALGORITHM

C. Ventura, X. Giró-i-Nieto, V. Vilaplana et al
1. Introducing the problem
2



Automatic selection of the representative
keyframe
1.1. What is the application?
3
1.2. Current implementation
4

ARBITRARY SAMPLING

MANUAL SELECTION
BY
PROFESSIONAL
2. Designing the system
5



2 scenarios:
 Intra-clip

mode

 Inter-clip

mode

Textual search
to retrieve
related vide...
2.1. General scheme
6

Frame
extraction

3
2
1

visual
features

Reranking

Textual
search

3
2
1

Inter-clip
mode

Simila...
2.2. Intra-clip mode
7



Mutual Reinforcement Algorithm (Joshi04)

 Gets

the frame with maximum coverage

(Joshi04) D....
2.3. Inter-clip mode
8



Reranking (based on Liu11)

INTRA

INPUT FRAMES

INTER

REPRESENTATIVE KEYFRAMES
FROM TEXTUAL S...
2.4. Post-processing block
9

Text filtering
 Goal:

To avoid representative keyframes with
textual captions
FRAME

HAAR...
3. Experiments
10




Qualitative evaluation
Quantitative evaluation
 MOS

Test
 Experimental dataset
3.1. Qualitative evaluation
11



Intra-clip mode
3.1. Qualitative evaluation
12



Inter-clip mode

Ranking scores
after mutual
reinforcement
(INTRA-CLIP
MODE)

Represent...
3.1. Qualitative evaluation
13



Inter-clip mode
 Final

ranking scores after reranking:
3.2. Quantitative evaluation
14



MOS (Mean Opinion Score) test
 Performed

by TVC professionals

 Scores
BAD

NON
ACC...
3.2. Quantitative evaluation
15



Database
NEWS DOMAIN

MORNING SHOW DOMAIN

ECONOMY

INTERVIEW
INTERNATIONAL

POLITICS
...
3.2. Quantitative evaluation
16



MOS test
4

different approaches

 Intra-clip
 Inter-clip
 Random

 Current
3.2. Quantitative evaluation
17

NEWS

MORNING
SHOW
3.2. Quantitative evaluation
18

NEWS

MORNING
SHOW
3. Experiments
19



Database and results are available on:
 imatge.upc.edu
4. Conclusions
20



Keyframe selection based on mutual
reinforcement algorithm




To get the frame with maximum cover...
21
2.3. Inter-clip mode
22



Textual search
2

modalities:

 Textual

searcher binary
 TF-IDF descriptors
Upcoming SlideShare
Loading in …5
×

Automatic Keyframe Selection based on Mutual Reinforcement Algorithm

787 views

Published on

Ventura, C.; Giro-i-Nieto, X.; Vilaplana, V.; Giribet, D.; Carasusan, E., "Automatic keyframe selection based on mutual reinforcement algorithm," Content-Based Multimedia Indexing (CBMI), 2013 11th International Workshop on , vol., no., pp.29,34, 17-19 June 2013
doi: 10.1109/CBMI.2013.6576548

This paper addresses the problem of video summarization through an automatic selection of a single representative keyframe. The proposed solution is based on the mutual reinforcement paradigm, where a keyframe is selected thanks to its highest and most frequent similarity to the rest of considered frames. Two variations of the algorithm are explored: a first one where only frames within the same video are used (intraclip mode) and a second one where the decision also depends on the previously selected keyframes of related videos (interclip mode). These two algorithms were evaluated by a set of
professional documentalists from a broadcaster’s archive, and results concluded that the proposed techniques outperform the semi-manual solution adopted so far in the company.

More details:
https://imatge.upc.edu/web/publications/automatic-keyframe-selection-based-mutual-reinforcement-algorithm

Published in: Technology, Art & Photos
  • Be the first to comment

  • Be the first to like this

Automatic Keyframe Selection based on Mutual Reinforcement Algorithm

  1. 1. AUTOMATIC KEYFRAME SELECTION BASED ON MUTUAL REINFORCEMENT ALGORITHM C. Ventura, X. Giró-i-Nieto, V. Vilaplana et al
  2. 2. 1. Introducing the problem 2  Automatic selection of the representative keyframe
  3. 3. 1.1. What is the application? 3
  4. 4. 1.2. Current implementation 4 ARBITRARY SAMPLING MANUAL SELECTION BY PROFESSIONAL
  5. 5. 2. Designing the system 5  2 scenarios:  Intra-clip mode  Inter-clip mode Textual search to retrieve related videos Database
  6. 6. 2.1. General scheme 6 Frame extraction 3 2 1 visual features Reranking Textual search 3 2 1 Inter-clip mode Similarity graph Mutual Reinforcement
  7. 7. 2.2. Intra-clip mode 7  Mutual Reinforcement Algorithm (Joshi04)  Gets the frame with maximum coverage (Joshi04) D. Joshi et al. The story picturing engine: finding elite images to illustrate a story using mutual reinforcement. In MIR ‘04
  8. 8. 2.3. Inter-clip mode 8  Reranking (based on Liu11) INTRA INPUT FRAMES INTER REPRESENTATIVE KEYFRAMES FROM TEXTUAL SEARCHER (Liu11) C. Liu et al. Query sensitive dynamic web video thumbnail generation. In ICIP ‘11
  9. 9. 2.4. Post-processing block 9 Text filtering  Goal: To avoid representative keyframes with textual captions FRAME HAAR WAVELET MORPHOLOGY
  10. 10. 3. Experiments 10   Qualitative evaluation Quantitative evaluation  MOS Test  Experimental dataset
  11. 11. 3.1. Qualitative evaluation 11  Intra-clip mode
  12. 12. 3.1. Qualitative evaluation 12  Inter-clip mode Ranking scores after mutual reinforcement (INTRA-CLIP MODE) Representative keyframes of the retrieved videos
  13. 13. 3.1. Qualitative evaluation 13  Inter-clip mode  Final ranking scores after reranking:
  14. 14. 3.2. Quantitative evaluation 14  MOS (Mean Opinion Score) test  Performed by TVC professionals  Scores BAD NON ACCEPTABLE GOOD ACCEPTABLE EXCELLENT
  15. 15. 3.2. Quantitative evaluation 15  Database NEWS DOMAIN MORNING SHOW DOMAIN ECONOMY INTERVIEW INTERNATIONAL POLITICS DISCUSSION
  16. 16. 3.2. Quantitative evaluation 16  MOS test 4 different approaches  Intra-clip  Inter-clip  Random  Current
  17. 17. 3.2. Quantitative evaluation 17 NEWS MORNING SHOW
  18. 18. 3.2. Quantitative evaluation 18 NEWS MORNING SHOW
  19. 19. 3. Experiments 19  Database and results are available on:  imatge.upc.edu
  20. 20. 4. Conclusions 20  Keyframe selection based on mutual reinforcement algorithm   To get the frame with maximum coverage within the video in the intra-clip approach Inter-clip approach Textual similarity to retrieve related videos  Linear fusion to get the new ranking scores   MOS test The semi-manual system (from TVC) can be replaced by the automatic approach.  Inter-clip approach outperforms intra-clip in controlled environments. 
  21. 21. 21
  22. 22. 2.3. Inter-clip mode 22  Textual search 2 modalities:  Textual searcher binary  TF-IDF descriptors

×