Your SlideShare is downloading. ×
Video Fingerprinting and Applications: A Review
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Video Fingerprinting and Applications: A Review


Published on

This presentation reviews the development in video fingerprinting technology in the past decade and its applications in content identification.

This presentation reviews the development in video fingerprinting technology in the past decade and its applications in content identification.

Published in: Technology

1 Comment
  • Hi,

    Can anyone tell me how much can it cost to embed video fingerprinting technology to my software?

    I don't need any rocket science from market leaders. Just smth. workable.

    I've found a couple of small companies like this
    and, but the smallest price is starting from $5000, which is too big for me.

    If anyone know opensource or free video fingerprinting libraries, I would very appreciate if you could share this with me.

    - Albert
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Video Fingerprinting and Applications: a review Jian Lu Vobile, Inc. Media Forensics & Security Conference EI’09, San Jose, CA
  • 2. From Research to Applications 1999 2008 1999
  • 3. What’s Video Fingerprinting •  A video fingerprint is a unique identifier extracted from video content –  Video fingerprints are often just string of bits, representing some “signatures” of the video content, and usually not in fixed length. –  Video fingerprinting refers to the process of extracting fingerprints from the video content. –  Comparing to watermarking, fingerprinting does not add to or alter video content. –  Also known as “robust hashing”, “perceptual hashing”, “content-based copy detection (CBCD)” in research literature.
  • 4. Human vs. Video Fingerprint Human Fingerprint Video Fingerprint Uniquely identify human Uniquely identify video Physical form Digital form Pictorial Time-based binary
  • 5. Identification by Fingerprint Human identification Video identification
  • 6. Video Fingerprinting Algorithms
  • 7. Desired Properties •  Robust –  Largely invariant for the same content under various types of processing, conversion, and manipulation. •  Discriminating –  Distinctly different for different content. •  Compact –  Low data rate •  Low complexity –  Fast fingerprint generation and matching
  • 8. Type of Video Signatures Spatial Temporal Color Transform-D Signatures Signatures Signatures Signatures Granularity Group of Bins of 3D transforms Whole frame frames histograms on GOP Blocks or Down- Frame other types of sampled transforms subdivision frames Points of Key frames interest Every frame
  • 9. Variants of Spatial Signatures •  Block-based –  Quantized mean block intensity –  Luminance block patterns ✪ •  ordinal ranking of average block intensity –  Differential luminance block patterns ✪ •  Centroid of gradient orientations •  Dominant edge orientation •  Points-of-interest –  Corner features (Harris points) –  Scale-space features
  • 10. An Example of Spatial Signature
  • 11. Variants of Temporal Signatures •  Temporal luminance patterns –  Ordinal ranking of average frame or block intensity in a group of frames •  Temporal differential luminance patterns ✪ –  Sum of absolute pixel or block difference – quantized and thresholded –  Block motion vectors – histogram of quantized directions •  Shot duration sequence
  • 12. Color Signatures •  Histogram-based –  Level-quantized histogram, e.g., (32, 16, 16) for Y, U, V, followed by magnitude quantization on each bin ✪ –  Level-quantized histogram, followed by ordinal ranking of histogram bins by magnitude
  • 13. Transform-Domain Signatures •  Affine transformation resilient –  Polar Fourier transform –  Radon transform ✪ –  Singular Value Decomposition •  Energy compaction –  3D DCT –  3D Wavelet transform
  • 14. Which One to Use? •  Spatial signatures, particularly block-based, are the overall category winner, and most widely used. •  Temporal and color signatures are less robust, but can be used along with spatial signatures to enhance discriminability. •  Transform-domain signatures are computationally expensive and not widely used in practice. •  The weakness of block-based spatial signatures is their lack of resilience against excessive geometric distortion, e.g., rotation and cropping.
  • 15. Challenges of Geometric Distortions Original Rotation by 10 degrees Rotation + Cropping
  • 16. Fingerprinting performance •  Video fingerprint using block-based spatial signatures –  Data size: a few hundreds bits per frame or <10 Kbps –  Speed: 1/10 playback time (10x RT) or faster for standard-def video.
  • 17. Fingerprint Matching and Search
  • 18. Similarity Measures •  Distance-based ✪ –  L1 (Manhattan) or L2 (Euclidean) distance •  For non-binary signatures •  Weights can be assigned when multiple signatures are used –  Hamming Distance •  For binary signatures •  Probability-based –  Probabilistic models for common distortion vectors
  • 19. Complexity of Fingerprint Search •  Exhaustive search has linear complexity, or O(K*N) –  N is the size of reference fingerprint DB, in minutes or hours. –  K is length of the query video. –  N can be further decomposed into M*L •  M is number of reference video fingerprints in DB •  L is the average length of video fingerprints in DB •  The curse is on N or M, the DB size.
  • 20. Strategies for Fast Search Strategies Fingerprint Search Motion Vector Search Reduce search space ✪ LSH Greedy search Sequential alignment Hierarchical search Early exit Hamming distance > T SAD > T Approximation in Frame down-sampling Block down-sampling distance calculation
  • 21. Locality Sensitive Hashing (LSH) •  Consider ε-NNS problem, –  For a query point q, find an approximate point p such that d(q,p) < (1+ε) d(q,P) –  LSH guarantees p can be found, with high probability, in O(N1/(1+ε)) •  Geometric reasoning: –  Close points in space are likely to be close after hashing (e.g., a projection onto a lower dimensional space) –  By using multiple hash functions, the probability of close points falling close is increased
  • 22. Other Approximation Techniques •  Multi-resolution coarse-to-fine search –  Fine-level search can be terminated (early exit) if coarse-level search is far off. –  Rank candidates by coarse-level search scores and take only top N candidates for fine-level search. •  Adaptive hashing – “learning to hashing” –  Hashing is non-deterministic; system is trained to adapt to identification task and data. –  A substantial reduction in search space.
  • 23. Applications
  • 24. UGC & P2P – copyright concerns? P2P UGC •  UGC Traffic in 07/2007 (Source: comScore, November 30, 2007) –  70 million people viewed 2.5 billion videos on (39.4% of total UGC audience) –  38 million people viewed 360 million videos on (22.6% of total UGC audience) •  P2P Traffic 2007 (Source: iPoque, November 28, 2007) –  Average 50-60% total Internet traffic: 49% in Middle East; 83% in Eastern Europe. –  BitTorrent 66.7%, eDonkey 28.6% of total P2P traffic
  • 25. Video Content Registration •  A reference video fingerprint database is pre- populated. •  Two types of information are stored with video fingerprint data in the reference database –  Metadata, e.g., title, owner, release date, etc. –  Business rules, e.g., allow, filter, or advertise, possibly based on certain conditions •  MovieLabs’ Content Recognition Rules (CRR) is an industry standard interface for expressing and exchanging rules.
  • 26. Video Content Filtering
  • 27. Video Content Tracking
  • 28. Example: Video Content Tracking
  • 29. Tracking Olympic Video Distribution
  • 30. Other Applications •  Broadcast monitoring –  Audit TV program and commercial airings •  Contextual Ads (monetization) –  Pair ads with identified content like Google AdSense •  Video asset management –  Content-based IDs identify linkage between edits and sources •  Content-based video search –  Query by video clip
  • 31. Summary •  Research in video fingerprinting began a decade ago; it had developed into a technology and been adopted by the industry. •  Different types of signatures are used to form a video fingerprint, including spatial, temporal, color, and transform-domain signatures. •  Spatial signatures are overall winner judged by multiple criteria, and widely adopted as primary signatures; temporal and color signatures can be used as secondary signatures to enhance discriminability. •  Brute-force, exhaustive fingerprint search is an O(K*N) problem. •  Fast approximate algorithms make fingerprint search tractable and scalable for practical applications. •  Current applications focus on copyright enforcement, other applications being developed and experimented include contextual advertising, asset management, and content-based video search.