Towards using Semantic Features for Near-duplicate Video Detection Workshop on Visual Content Identification and Search Si...
Outline semantic video signature 3. Experiments semantic concepts discussion methodology 4. Conclusions and future work ba...
Outline semantic video signature 3. Experiments semantic concepts discussion methodology 4. Conclusions and future work ba...
Background <ul><li>Increasing amount of online digital video content </li></ul><ul><ul><li>easy-to-use multimedia devices ...
NDVC Definition <ul><li>Basic definition </li></ul><ul><ul><li>identical or approximately identical videos </li></ul></ul>...
NDVC Examples /22 NDVC transformation (cam cording, subtitles) transformation (blur) original videos
Video Signature <ul><li>What is a  video signature ? </li></ul><ul><ul><li>represents a video clip with a  unique set of f...
Video Signature based on Low-level Visual Features <ul><li>Problem </li></ul><ul><ul><li>near-duplicates are often  not vi...
Outline semantic video signature 3. Experiments semantic concepts discussion methodology 4. Conclusions and future work ba...
Video Signature based on Semantic Features <ul><li>Observation </li></ul><ul><ul><li>near-duplicates often contain  simila...
Semantic Video Signature  <ul><li>Represents the  temporal variation of several semantic concepts  in a video clip </li></...
Outline semantic video signature 3. Experiments semantic concepts discussion methodology 4. Conclusions and future work ba...
Methodology (1/2) <ul><li>Concept detection uses Support Vector Machines </li></ul><ul><li>3 criteria were used to select ...
Methodology (2/2) <ul><li>Database </li></ul><ul><ul><li>reference video and query video set  </li></ul></ul><ul><ul><ul><...
Robustness Against Transformations (1/2) <ul><li>Requirement </li></ul><ul><ul><li>NDVCs should have a highly similar sema...
Robustness Against Transformations (2/2) <ul><li>Observation </li></ul><ul><ul><li>semantic video signature is robust agai...
Robustness Against Key Frame Selection <ul><li>Requirement </li></ul><ul><ul><li>video signatures should be robust against...
Uniqueness (1/2) <ul><li>Requirement </li></ul><ul><ul><li>two semantically different videos should have a different seman...
U niqueness (2/2) <ul><li>Observation </li></ul><ul><ul><li>NDVC detection performance increases when </li></ul></ul><ul><...
Outline semantic video signature 3. Experiments semantic concepts discussion methodology 4. Conclusions and future work ba...
Conclusions and Future Work <ul><li>Semantic video signatures for NDVC detection </li></ul><ul><ul><li>exploit the tempora...
Thank you! Any questions? /22
Upcoming SlideShare
Loading in …5
×

Towards Using Semantic Features for Near-Duplicate Video Detection

1,164 views

Published on

Towards Using Semantic Features for Near-Duplicate Video Detection.

Paper presented at the ICME 2010 Workshop on Visual Content Identification and Search in Singapore.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,164
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Towards Using Semantic Features for Near-Duplicate Video Detection

  1. 1. Towards using Semantic Features for Near-duplicate Video Detection Workshop on Visual Content Identification and Search Singapore – July 23, 2010 Hyun-seok Min , Wesley De Neve, Yong Man Ro Image and Video Systems Lab Department of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) email: hsmin@kaist.ac.kr
  2. 2. Outline semantic video signature 3. Experiments semantic concepts discussion methodology 4. Conclusions and future work background challenges 1. Introduction 2. Towards semantic-based NDVC detection /22
  3. 3. Outline semantic video signature 3. Experiments semantic concepts discussion methodology 4. Conclusions and future work background challenges 1. Introduction 2. Towards semantic-based NDVC detection /22
  4. 4. Background <ul><li>Increasing amount of online digital video content </li></ul><ul><ul><li>easy-to-use multimedia devices </li></ul></ul><ul><ul><li>cheap storage and bandwidth </li></ul></ul><ul><li>Increasing number of near-duplicate video clips </li></ul><ul><ul><li>digital video content can be easily edited and redistributed </li></ul></ul><ul><li>Near-duplicate video clips or NDVCs may infringe copyright or clutter search results </li></ul><ul><ul><li>need for identifying NDVCs </li></ul></ul>/22 Search results on YouTube for the query “I will survive Jesus” A significant number of search results are near-duplicates
  5. 5. NDVC Definition <ul><li>Basic definition </li></ul><ul><ul><li>identical or approximately identical videos </li></ul></ul><ul><ul><ul><li>were the subject of at least one transformation </li></ul></ul></ul><ul><ul><ul><li>transformations preserve semantic information </li></ul></ul></ul><ul><ul><li>addresses copyright infringement </li></ul></ul><ul><li>Extended definition </li></ul><ul><ul><li>includes a user-centric component </li></ul></ul><ul><ul><ul><li>not considered as NDVCs by users: identical video clips that contain relevant complementary information </li></ul></ul></ul><ul><ul><ul><li>considered as NDVCs by users: video clips that are not alike, but that are visually similar and semantically related </li></ul></ul></ul><ul><ul><li>addresses cluttered search results </li></ul></ul>/22
  6. 6. NDVC Examples /22 NDVC transformation (cam cording, subtitles) transformation (blur) original videos
  7. 7. Video Signature <ul><li>What is a video signature ? </li></ul><ul><ul><li>represents a video clip with a unique set of features </li></ul></ul><ul><li>Conventional video signatures </li></ul><ul><ul><li>often created by extracting low-level visual features from video frames </li></ul></ul>/22 video clip … video signature feature extraction
  8. 8. Video Signature based on Low-level Visual Features <ul><li>Problem </li></ul><ul><ul><li>near-duplicates are often not visually similar </li></ul></ul>/22 original video NDVC transformation (cam cording, subtitles) … … video signature video signature Visual match? No!
  9. 9. Outline semantic video signature 3. Experiments semantic concepts discussion methodology 4. Conclusions and future work background challenges 1. Introduction 2. Towards semantic-based NDVC detection /22
  10. 10. Video Signature based on Semantic Features <ul><li>Observation </li></ul><ul><ul><li>near-duplicates often contain similar semantics </li></ul></ul>10/22 original video NDVC transformation (cam cording, subtitles) Semantic concepts: Semantic concepts: Semantic match? Yes! indoor, man, face, … indoor, man, face, …
  11. 11. Semantic Video Signature <ul><li>Represents the temporal variation of several semantic concepts in a video clip </li></ul><ul><ul><li>allows dealing with a limited concept vocabulary </li></ul></ul>semantic video signature (binary-valued matrix) 1 1 0 1 1 sky ... 0 0 1 0 0 indoor ... keyframes (one for each shot) ... 0 1 0 0 1 architecture ... ...
  12. 12. Outline semantic video signature 3. Experiments semantic concepts discussion methodology 4. Conclusions and future work background challenges 1. Introduction 2. Towards semantic-based NDVC detection /22
  13. 13. Methodology (1/2) <ul><li>Concept detection uses Support Vector Machines </li></ul><ul><li>3 criteria were used to select semantic concepts </li></ul><ul><ul><li>represent visual concepts </li></ul></ul><ul><ul><li>need to be highly popular/common </li></ul></ul><ul><ul><li>detection is easy and reliable </li></ul></ul><ul><li>Selected semantic concepts </li></ul><ul><ul><li>‘ gravel’, ‘park’, ‘pavement’, ‘road’, ‘rock’, ‘sand’, ‘sidewalk’, ‘face’, ‘people’, ‘indoor-light’, ‘field’, ‘peak’, ‘wood’, ‘night’, ‘street-light’, ‘flowers’, ‘leaves’, ‘trees’, ‘cloudy’, ‘sunny’, ‘sunset’, ‘brick’, ‘arch’, ‘buildings’, ‘wall’, ‘windows’, ‘beach’, ‘high-wave’, ‘low-wave’, ‘still water’, ‘mirrored water’, and ‘snow’ </li></ul></ul>/22
  14. 14. Methodology (2/2) <ul><li>Database </li></ul><ul><ul><li>reference video and query video set </li></ul></ul><ul><ul><ul><li>MUSCLE-VCD-2007 set </li></ul></ul></ul><ul><ul><li>size </li></ul></ul><ul><ul><ul><li>reference video set: 101 videos </li></ul></ul></ul><ul><ul><ul><li>query video set: 15 videos </li></ul></ul></ul><ul><li>Low-level visual features </li></ul><ul><ul><li>color </li></ul></ul><ul><ul><ul><li>color structure (CS), color layout (CL), and scalable color (SC) </li></ul></ul></ul><ul><ul><li>texture </li></ul></ul><ul><ul><ul><li>homogeneous texture (HT), edge histogram (EH) </li></ul></ul></ul>14/22
  15. 15. Robustness Against Transformations (1/2) <ul><li>Requirement </li></ul><ul><ul><li>NDVCs should have a highly similar semantic signature </li></ul></ul><ul><li>Transformations applied </li></ul><ul><ul><li>blur, picture-in-picture, brightness </li></ul></ul><ul><li>Example transformation </li></ul>/22 Original frame Blurred frames using various strengths (filter size: 5, 9, 13, 17, 21)
  16. 16. Robustness Against Transformations (2/2) <ul><li>Observation </li></ul><ul><ul><li>semantic video signature is robust against the content transformations applied </li></ul></ul>/22 similarity rates computed for our semantic video signature are about 90% for all transformations applied
  17. 17. Robustness Against Key Frame Selection <ul><li>Requirement </li></ul><ul><ul><li>video signatures should be robust against different key frame selection strategies </li></ul></ul><ul><ul><ul><li>key frames from the original video and an NDVC may differ </li></ul></ul></ul><ul><li>Feature variation for varying key frames </li></ul><ul><li>Observation </li></ul><ul><ul><li>semantic information does not change significantly throughout a shot, whereas low-level visual features do </li></ul></ul>/22 CS CL SC EH HT Semantic features Variation 8.31 1.69 7.11 4.80 4.05 0.90
  18. 18. Uniqueness (1/2) <ul><li>Requirement </li></ul><ul><ul><li>two semantically different videos should have a different semantic video signature </li></ul></ul><ul><ul><li>two semantically identical videos should have the same semantic video signature </li></ul></ul><ul><li>Uniqueness was investigated by varying </li></ul><ul><ul><li>the number of semantic concepts used </li></ul></ul><ul><ul><li>the number of shots in a query video clip </li></ul></ul>/22
  19. 19. U niqueness (2/2) <ul><li>Observation </li></ul><ul><ul><li>NDVC detection performance increases when </li></ul></ul><ul><ul><ul><li>the number of semantic concepts describing a shot increases </li></ul></ul></ul><ul><ul><ul><li>the number of shots in a query video clip increases </li></ul></ul></ul>/22 NDVC detection performance increases as the number of shots in a query video sequence increases, even when a limited semantic concept vocabulary is in use N correct / N queries
  20. 20. Outline semantic video signature 3. Experiments semantic concepts discussion methodology 4. Conclusions and future work background challenges 1. Introduction 2. Towards semantic-based NDVC detection /22
  21. 21. Conclusions and Future Work <ul><li>Semantic video signatures for NDVC detection </li></ul><ul><ul><li>exploit the temporal variation of semantic concepts </li></ul></ul><ul><ul><li>show a high level of robustness against </li></ul></ul><ul><ul><ul><li>transformations of the video content </li></ul></ul></ul><ul><ul><ul><li>different key frame selection strategies </li></ul></ul></ul><ul><ul><li>show a high degree of uniqueness </li></ul></ul><ul><ul><ul><li>even when a limited semantic concept vocabulary is in use </li></ul></ul></ul><ul><li>Future research </li></ul><ul><ul><li>scalability of semantic concept detection </li></ul></ul><ul><ul><li>comparison with the use of local features </li></ul></ul><ul><ul><li>re-ranking of video search results </li></ul></ul>/22
  22. 22. Thank you! Any questions? /22

×