Towards Using Semantic Features for Near-Duplicate Video Detection

Towards using Semantic Features for Near-duplicate Video Detection Workshop on Visual Content Identification and Search Singapore – July 23, 2010 Hyun-seok Min , Wesley De Neve, Yong Man Ro Image and Video Systems Lab Department of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) email: hsmin@kaist.ac.kr

Outline semantic video signature 3. Experiments semantic concepts discussion methodology 4. Conclusions and future work background challenges 1. Introduction 2. Towards semantic-based NDVC detection /22

Background
Increasing amount of online digital video content
easy-to-use multimedia devices
cheap storage and bandwidth
Increasing number of near-duplicate video clips
digital video content can be easily edited and redistributed
Near-duplicate video clips or NDVCs may infringe copyright or clutter search results
need for identifying NDVCs
/22 Search results on YouTube for the query “I will survive Jesus” A significant number of search results are near-duplicates

NDVC Definition
Basic definition
identical or approximately identical videos
were the subject of at least one transformation
transformations preserve semantic information
addresses copyright infringement
Extended definition
includes a user-centric component
not considered as NDVCs by users: identical video clips that contain relevant complementary information
considered as NDVCs by users: video clips that are not alike, but that are visually similar and semantically related
addresses cluttered search results
/22

NDVC Examples /22 NDVC transformation (cam cording, subtitles) transformation (blur) original videos

Video Signature
What is a video signature ?
represents a video clip with a unique set of features
Conventional video signatures
often created by extracting low-level visual features from video frames
/22 video clip … video signature feature extraction

Video Signature based on Low-level Visual Features
Problem
near-duplicates are often not visually similar
/22 original video NDVC transformation (cam cording, subtitles) … … video signature video signature Visual match? No!

Video Signature based on Semantic Features
Observation
near-duplicates often contain similar semantics
10/22 original video NDVC transformation (cam cording, subtitles) Semantic concepts: Semantic concepts: Semantic match? Yes! indoor, man, face, … indoor, man, face, …

Semantic Video Signature
Represents the temporal variation of several semantic concepts in a video clip
allows dealing with a limited concept vocabulary
semantic video signature (binary-valued matrix) 1 1 0 1 1 sky ... 0 0 1 0 0 indoor ... keyframes (one for each shot) ... 0 1 0 0 1 architecture ... ...

Methodology (1/2)
Concept detection uses Support Vector Machines
3 criteria were used to select semantic concepts
represent visual concepts
need to be highly popular/common
detection is easy and reliable
Selected semantic concepts
‘ gravel’, ‘park’, ‘pavement’, ‘road’, ‘rock’, ‘sand’, ‘sidewalk’, ‘face’, ‘people’, ‘indoor-light’, ‘field’, ‘peak’, ‘wood’, ‘night’, ‘street-light’, ‘flowers’, ‘leaves’, ‘trees’, ‘cloudy’, ‘sunny’, ‘sunset’, ‘brick’, ‘arch’, ‘buildings’, ‘wall’, ‘windows’, ‘beach’, ‘high-wave’, ‘low-wave’, ‘still water’, ‘mirrored water’, and ‘snow’
/22

Methodology (2/2)
Database
reference video and query video set
MUSCLE-VCD-2007 set
size
reference video set: 101 videos
query video set: 15 videos
Low-level visual features
color
color structure (CS), color layout (CL), and scalable color (SC)
texture
homogeneous texture (HT), edge histogram (EH)
14/22

Robustness Against Transformations (1/2)
Requirement
NDVCs should have a highly similar semantic signature
Transformations applied
blur, picture-in-picture, brightness
Example transformation
/22 Original frame Blurred frames using various strengths (filter size: 5, 9, 13, 17, 21)

Robustness Against Transformations (2/2)
Observation
semantic video signature is robust against the content transformations applied
/22 similarity rates computed for our semantic video signature are about 90% for all transformations applied

Robustness Against Key Frame Selection
Requirement
video signatures should be robust against different key frame selection strategies
key frames from the original video and an NDVC may differ
Feature variation for varying key frames
Observation
semantic information does not change significantly throughout a shot, whereas low-level visual features do
/22 CS CL SC EH HT Semantic features Variation 8.31 1.69 7.11 4.80 4.05 0.90

Uniqueness (1/2)
Requirement
two semantically different videos should have a different semantic video signature
two semantically identical videos should have the same semantic video signature
Uniqueness was investigated by varying
the number of semantic concepts used
the number of shots in a query video clip
/22

U niqueness (2/2)
Observation
NDVC detection performance increases when
the number of semantic concepts describing a shot increases
the number of shots in a query video clip increases
/22 NDVC detection performance increases as the number of shots in a query video sequence increases, even when a limited semantic concept vocabulary is in use N correct / N queries

Conclusions and Future Work
Semantic video signatures for NDVC detection
exploit the temporal variation of semantic concepts
show a high level of robustness against
transformations of the video content
different key frame selection strategies
show a high degree of uniqueness
even when a limited semantic concept vocabulary is in use
Future research
scalability of semantic concept detection
comparison with the use of local features
re-ranking of video search results
/22

Thank you! Any questions? /22

Towards Using Semantic Features for Near-Duplicate Video Detection

by Wesley De Neve on Sep 08, 2010

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Towards Using Semantic Features for Near-Duplicate Video Detection Presentation Transcript