2. Challenge
• A t
Automatically insert Advertising in a video
ti ll i t Ad ti i i id
(PIP) at the right time and in the right corner
Low interest in this corner
…
Best moment
B
26‐27 March 2012 NoTube 3rd review 2
3. Where were we last year ?
Where were we last year ?
• Preliminary tests showed a mismatch between the automatic ranking and
Preliminary tests showed a mismatch between the automatic ranking and
the opinion of field testers
• Some improvements were identified:
• Improve the algorithm based on corner saliency
• Use audio
di
• Use scene cuts analysis
• Use global saliency maps analysis (not only corners)
• Adjust the Ad visibility
j y
• Open questions:
• How letterbox/pillarbox can be used for Ad insertion when present ?
• Audio processing during the Ad: Several options
A di i d i h Ad S l i
• Mix film sound track with Ad audio
• Replace film sound track by Ad Audio
• Keep film sound track
26‐27 March 2012 NoTube 3rd review 3
4. Where were we last year ?
Where were we last year ?
• Suggestions from the last review
– F
From the operator's point of view, it would be useful to be able to
th t ' i t f i it ld b f l t b bl t
insert a specific number of multiple ads in an entire program (e.g. a
movie)
• The algorithm allows to find a given number of sequences where the Ad can
be inserted. This number is a parameter of the algorithm that the operator
b i t d Thi b i t f th l ith th t th t
can use
– Sound is not currently used in this placement, and it definitely should
• Sound processing was taken into account for the improvement of the
algorithm and was assessed in the survey
l ith d d i th
– Ad choice relating to content is also very important, e.g. to avoid
alcohol advertisements on a driving scene
• The metadata describing scenes of the film and the Ad (EgtaMETA) could be
used but was not implemented (out of the scope of the project)
db l d( f h f h )
– Furthermore additional user testing mocking‐up a ‘real’ movie
consumption situation needs to look into general acceptance of the
ad insertion concept
• General acceptance was evaluated in the survey
26‐27 March 2012 NoTube 3rd review 4
5. Methodology used to improve the
Automatic Ad insertion technology
i di i h l
• M difi ti
Modification of the algorithm to have a
f th l ith t h
ranking with different criteria
• Survey to assess the automatic Ad insertion
technology
• Feedback from this survey was used to try and
find a way to avoid proposing sequences
where the Ad disturbs the viewer
26‐27 March 2012 NoTube 3rd review 5
6. Improvements in the algorithms
Improvements in the algorithms
– Corners picture saliency analysis: This is the basis of
Corners picture saliency analysis: This is the basis of
the algorithm to isolate N sequences per film
• On each picture, the saliency of each corner is calculated
• The algorithm looks after sequences minimizing the
integration of corner saliency throughout the duration of the
Ad
– 3 other values are calculated for the sequences found
by the previous algorithm
• Global picture saliency
Global picture saliency
• Number of scene cuts
• Sound level analysis
26‐27 March 2012 NoTube 3rd review 6
7. Ad insertion technology
Ad insertion technology
• The Ad insertion algorithms are based on the succession of
The Ad insertion algorithms are based on the succession of
two workflows
The first workflow analyses the movie in order to extract
y
the metadata describing the n “best sequences” available
to insert the Ad and writes these metadata in an XML file.
The second workflow inserts the Ad in the video thanks to
The second workflow inserts the Ad in the video thanks to
the metadata produced by the first workflow.
26‐27 March 2012 NoTube 3rd review 7
8. First workflow : Analysis
First workflow : Analysis
Input modules to get
the movie Video Analysis
Video decoding to
Video decoding to
TS input file to produce the
uncompress the data
and extract the metadata
compressed data
Result of this workflow
Result of this workflow
XML file describing the n « best sequences »
Demo
26‐27 March 2012 NoTube 3rd review 8
9. Second workflow :
Insertion of the Ad
The initial movie PiP Insertion
Input modules
to get the Video Generation of H264
movie TS
movie TS decoding to
decoding to
input file and uncompress video output file
extract the the movie
compressed data Video
data Processing Output
Video
to insert
to insert modules to
mod les to
H264
Picture in create a TS
Input modules encoding
Picture the output file
to get the Video Ad
Video Offset to
movie TS decoding to
Processing delay
input file and
input file and uncompress
uncompress
to resize the Ad
extract the the movie
the Ad insertion
compressed data
data
The ad resized and delayed
Demo
26‐27 March 2012 NoTube 3rd review 9
10. Survey to evaluate the algorithms
Survey to evaluate the algorithms
• 30
30 sequences were prepared for evaluation
df l ti
– 6 films
– 5 sequences per film
• They were uploaded on YouTube
• Questionnaires (Google docs) were prepared
p
and sent to partners
26‐27 March 2012 NoTube 3rd review 10
14. Results of the survey (1/5)
Results of the survey (1/5)
• F
From 20 to 26 answers
20 t 26
• Few answers, but results are consistent
– Standard deviation of sequence ranking is
between 0.7 and 1.3 (ranking is between 1 and 5)
Sequences ranking
Stand Dev Seq1 Seq2 Seq3 Seq4 Seq5
300 1,1 1,3 1,0 1,1 1,0
Doc 0,7 0,7 0,7 0,7 1,0
Drama 0,9
, 0,8
, 1,1
, 1,1
, 1,1
,
Sherlock 0,7 0,7 0,7 0,9 0,9
StayIn Alive 1,1 0,9 0,9 1,0 1,0
7years 1,1 0,7 0,9 1,0 1,2
26‐27 March 2012 NoTube 3rd review 14
15. Results of the survey (2/5)
Results of the survey (2/5)
• Good results
Good results
– Average mark: 3,6 (3 = acceptable, 4 = good)
– Only 3 sequences among the 30 sequences (10%) were considered as
“unacceptable” (mark ≤ 2.5)
– 83% of the sequences were, at least, “acceptable” (mark ≥ 3)
– 50 % of the sequences were judged “good” to “very good”
(mark ≥ 3.9)
Sequences ranking
Average Seq1 Seq2 Seq3 Seq4 Seq5
300 2,4 2,4 3,5 3,7 3,3
Doc 4,6 4,4 4,2 4,5 3,8
Drama 3,1 4,0 3,1 3,1 3,9
Sherlock 4,3 3,9 4,1 2,9 3,5
StayIn Alive 2,8 3,9 4,0 4,2 3,7
7years 4,0 4,0 3,9 3,8 2,5
26‐27 March 2012 NoTube 3rd review 15
16. Results of the survey (3/5)
Results of the survey (3/5)
• Very dependant on the film
Very dependant on the film
Does the way the Ad is inserted disturb you to Does the way the Ad is inserted disturb you to
understand what is happening ? understand what is happening ?
100% 100%
90% 90% 6
80% 80% 12
70% 70%
22 20 22
60% 60%
20 20 20 19 19 No No
50% 50%
40% Yes 40% 20 Yes
30% 30% 14
20% 20%
4 6 4
10% 10%
0 0 0 1 1
0% 0%
"Doc", "Doc", "Doc", "Doc", "Doc", "300", "300", "300", "300", "300",
Seq 1 Seq 2 Seq 3 Seq 4 Seq 5 Seq 1 Seq 2 Seq 3 Seq 4 Seq 5
• Most people are disturbed because, sometimes, the Ad hides
part of the face of people (hairs)
300 Doc Drama Sherlock Staying 7 years All films
Bottom left 23% 3% 8% 19% 2% 4% 39%
Bottom Right 15% 6% 10% 1% 5% 8% 29%
Top left 6% 1% 2% 2% 0% 3% 10%
Top Right 5% 0% 1% 2% 22% 6% 22%
26‐27 March 2012 NoTube 3rd review 16
17. Results of the survey (4/5)
Results of the survey (4/5)
• A
Are people ready to accept this technology ?
l d t t thi t h l ?
Yes 52%
No 33%
Don't know 14%
26‐27 March 2012 NoTube 3rd review 17
18. Results of the survey (5/5)
Results of the survey (5/5)
• R ki
Ranking given by the original algorithm
i b th i i l l ith
(corner saliency only) is not always enough to
get “good” sequences
t “ d”
Sequences ranking
Average Seq1 Seq2 Seq3 Seq4 Seq5
300 2,4
24 2,4
24 3,5
35 3,7
37 3,3
33
Doc 4,6 4,4 4,2 4,5 3,8
Drama 3,1 4,0 3,1 3,1 3,9
Sherlock 4,3 3,9 4,1 2,9 3,5
StayIn Alive 2,8
28 3,9
39 4,0
40 4,2
42 3,7
37
7years 4,0 4,0 3,9 3,8 2,5
• Other criteria were studied to improve the
results
l
26‐27 March 2012 NoTube 3rd review 18
19. Use of Global Saliency
Use of Global Saliency
• The saliency is calculated on the full picture and
The saliency is calculated on the full picture and
integrated throughout the Ad duration
Global sail. r
300 39 21 17 13 11 -0,7
Doc 40 49 46 27 29 0,3
Drama 31 32 49 31 21 -0,5
Sherlock 14 10 11 11 17 0,0
StayIn Alive 21 20 21 39 39 0,4
7years 27 24 28 30 30 -0,6
• The correlation coefficient (Global saliency versus
The correlation coefficient (Global saliency versus
survey ranking) varies too much (+/‐) to use the
Global Saliency to improve the results
y p
26‐27 March 2012 NoTube 3rd review 19
20. Use of Scene Cuts
Use of Scene Cuts
• The number of scene cuts are calculated
The number of scene cuts are calculated
throughout the Ad duration
Scene cuts
300 5 11 2 1 6 -0,8
-0 8
Doc 6 2 13 7 6 -0,1
Drama 6 5 12 5 2 -0,6
Sherlock 2 6 3 12 5 -0,9
StayIn Alive 5 4 5 2 2 -0,5
05
7years 1 9 5 3 7 -0,3
• The correlation coefficient (Nb of scene cuts
versus survey ranking) is always negative and the
ki ) i l ti d th
number of scene cuts could be used to improve
the results
26‐27 March 2012 NoTube 3rd review 20
21. Use of Sound level
Use of Sound level
• The LUFS is calculated throughout the Ad
The LUFS is calculated throughout the Ad
duration
LUFS
300 7,79E-04 1,26E-04 5,10E-04 1,44E-04 1,14E-04 -0,3
Doc 4,94E-03 4,76E-03 5,08E-03 5,67E-03 4,71E-03 0,5
Drama 9,48E-03 2,15E-03 4,50E-02 8,91E-02 3,56E-03 -0,7
Sherlock 1,05E-02 7,72E-03 8,23E-03 3,21E-03 1,20E-03 0,8
7years 2,33E-04 2,56E-03 5,32E-04 4,75E-04 9,42E-04
2 33E 04 2 56E 03 5 32E 04 4 75E 04 9 42E 04 0,1
01
• The correlation coefficient (LUFS versus survey
The correlation coefficient (LUFS versus survey
ranking) varies too much (+/‐) to use the LUFS to
improve the results
p
26‐27 March 2012 NoTube 3rd review 21
22. Conclusions
• Conclusions
Th
The use of corner saliency gives good results
f li i d lt
Global saliency and sound level analysis doesn’t give results which
could improve the algorithm
The number of scene cuts may be used to get slightly better results
e u be o sce e cu s ay be used o ge s g y be e esu s
but we are lacking enough experimental data to tune the algorithm
The main remaining problem is that the algorithm isn’t able to detect
faces
• P
Proposals for future studies
l f f t t di
– Add a face detection algorithm to discard sequences where the Ad
would hide faces
– Take into account the number of scene cuts
Take into account the number of scene cuts
– Carry additional field tests to fine tune the algorithm
• More data (more sequences/films)
• Higher dynamic (good and bad sequences)
• M
More people
l
26‐27 March 2012 NoTube 3rd review 22
