ENHANCED MULTIMEDIA SEARCH Lakshman Prasad 4MC02EC032 VIII semester E&C

Multimedia Search Types: Multimedia Search Content based Indexing, classifying ‘ Widely used today- Google Human’ factor Efficient when accurately indexed Similarity based Correlation of data / feature vectors Infallible Scope of usage very high Efficient for multimedia search

Content based

Indexing, classifying

‘ Widely used today- Google

Human’ factor

Efficient when accurately indexed

Similarity based

Correlation of data / feature vectors

Infallible

Scope of usage very high

Efficient for multimedia search

Similarity Based Search Search  Similarity based Use of ‘raw image data’ correlation Direct correlation of binary data Efficient for exact matching Use of extracted ‘feature vectors’ Extraction of ‘feature vectors’ Efficient when approximate matching suffices

Use of ‘raw image data’ correlation

Direct correlation of binary data

Efficient for exact matching

Use of extracted ‘feature vectors’

Extraction of ‘feature vectors’

Efficient when approximate matching suffices

‘ Feature Vectors’ Search  Similarity based  Feature vectors Full histogram comparison Direct comparison of histograms Simpler implementation Inefficient for large data. Use of ‘histogram pruning’ Comparison of parts of histogram Complex implementation Efficient search for large data

Full histogram comparison

Direct comparison of histograms

Simpler implementation

Inefficient for large data.

Use of ‘histogram pruning’

Comparison of parts of histogram

Complex implementation

Efficient search for large data

‘ Enhanced Multimedia Search =’ Search  Similarity based  Feature vectors  Histogram Pruning Search Similarity based search Feature vectors extracted similarity based search “ Histogram pruning based feature vector extracted similarity based search” “ Time series Active Search Algorithm”

Search

Similarity based search

Feature vectors extracted similarity based search

“ Histogram pruning based feature vector extracted similarity based search”

“ Time series Active Search Algorithm”

Algorithm Apply windows of query signal length Extract feature vectors from query signal Extract feature vectors from stored signal Classify feature vectors Obtain histograms Determine threshold of comparison Compare histograms Notify when similar / accurate Otherwise, shift forward the window, prun Obtain next Histogram, repeat

Apply windows of query signal length

Extract feature vectors from query signal

Extract feature vectors from stored signal

Classify feature vectors

Obtain histograms

Determine threshold of comparison

Compare histograms

Notify when similar / accurate

Otherwise, shift forward the window, prun

Obtain next Histogram, repeat

Time Series Active Search

Feature vector used Audio- Short term f spectrum thro’ BPF where k is the sampled time, f j (k) is the normalized short time power spectrum by, where y j (t) is output WF of BPF j at t. M is the time support of feature vector N is the # of frequency channels Video- Color distribution

Audio- Short term f spectrum thro’ BPF

where k is the sampled time,

f j (k) is the normalized short time power spectrum by,

where y j (t) is output WF of BPF j at t.

M is the time support of feature vector

N is the # of frequency channels

Video- Color distribution

Implementation details Histogram Similarity Pruning Detection criterion

Histogram

Similarity

Pruning

Detection criterion

Histogram Pruning Analogy EXAMPLE H ERE IS A SIMPLE EXAMPLE E XAMPLE H E RE IS A SIMPLE EXAMPLE E XAMPLE H E R E IS A SIMPLE EXAMPLE E XAMPLE HER E IS A SIMPLE EXAMPLE E XAMPLE HER E IS A SIMPLE EXAMPLE E XAMPLE HERE I S A SIMPLE EXAMPLE … ...... E XAMPLE HERE IS A SIMPLE E XAMPLE EX AMPLE HERE IS A SIMPLE E X AMPLE ……… EXAMPLE HERE IS A SIMPLE EXAMPLE

EXAMPLE

H ERE IS A SIMPLE EXAMPLE

E XAMPLE

H E RE IS A SIMPLE EXAMPLE

E XAMPLE

H E R E IS A SIMPLE EXAMPLE

E XAMPLE

HER E IS A SIMPLE EXAMPLE

E XAMPLE

HER E IS A SIMPLE EXAMPLE

E XAMPLE

HERE I S A SIMPLE EXAMPLE

… ......

E XAMPLE

HERE IS A SIMPLE E XAMPLE

EX AMPLE

HERE IS A SIMPLE E X AMPLE

………

EXAMPLE

HERE IS A SIMPLE EXAMPLE

‘ AND’ search

Experimental results: Speed 48h stored, 15s query PIII 933 MHz PC, Linux Audio: VHS HI-Fi, 11kHz sampling f, 16 bit Q, 7ch II order IIR BPF, M=110 Video: NTSC, 29.97 fps, 160x120 pixels/image, W=16

48h stored, 15s query

PIII 933 MHz PC, Linux

Audio: VHS HI-Fi, 11kHz sampling f, 16 bit Q, 7ch II order IIR BPF, M=110

Video: NTSC, 29.97 fps, 160x120 pixels/image, W=16

Experimental results: Accuracy 60m TV broadcast used as query & stored signal Random choice 100 times repeated Gaussian noise for audio MPEG1 compression for video

60m TV broadcast used as query & stored signal

Random choice

100 times repeated

Gaussian noise for audio

MPEG1 compression for video

Experimental results: Summary Matches reduced to 1/291 (video) and , 1/501 (audio), very fast Accurate and reasonable robust with AWGN and MPEG1 compression

Matches reduced to 1/291 (video) and , 1/501 (audio), very fast

Accurate and reasonable robust with AWGN and MPEG1 compression

Extensions to Algorithm ‘ OR’ model of parallel search Feature Distortion Absorption model Dynamic Threshold value determination

‘ OR’ model of parallel search

Feature Distortion Absorption model

Dynamic Threshold value determination

Applications Statistical analysis of broadcasted music, TV / radio commercials Copyright management on Internet ‘ Truly similar search’- similar music Fastening of biometric security algorithms

Statistical analysis of broadcasted music, TV / radio commercials

Copyright management on Internet

‘ Truly similar search’- similar music

Fastening of biometric security algorithms

Conclusion Discussed algorithm enables a faster and more efficient search of audio and video queries from long running audio/video More efficient ‘AND’ method discussed, other extensions overviewed Web search engines based on audio and video queries anticipated

Discussed algorithm enables a faster and more efficient search of audio and video queries from long running audio/video

More efficient ‘AND’ method discussed, other extensions overviewed

Web search engines based on audio and video queries anticipated

References “ A quick search method for Audio and Video signals based on histogram pruning” IEEE transactions on multimedia, Vol.5, #3, Sept 2003 Other IEEE transactions on Multimedia “ Digital Image Processing” Rafael C Gonzalez, Richard E woods “ Digital Signal Processing” Proakis and Manolakis http://www-igm.univ-mlv.fr/~lecroq/string/node14.html http://ma.i2r.a-star.edu.sg/Staff/~lingyu/FastVSearch.pdf http://www.cs.utexas.edu/users/moore/best-ideas/string-searching/kpm-example.html http://search.aol.co.uk/av_idx?help=av_adv Several links from http://www.google.com/search?hl=en&q=audio+and+video+search&btnG=Google+Search

“ A quick search method for Audio and Video signals based on histogram pruning” IEEE transactions on multimedia, Vol.5, #3, Sept 2003

Other IEEE transactions on Multimedia

“ Digital Image Processing” Rafael C Gonzalez, Richard E woods

“ Digital Signal Processing” Proakis and Manolakis

http://www-igm.univ-mlv.fr/~lecroq/string/node14.html

http://ma.i2r.a-star.edu.sg/Staff/~lingyu/FastVSearch.pdf

http://www.cs.utexas.edu/users/moore/best-ideas/string-searching/kpm-example.html

http://search.aol.co.uk/av_idx?help=av_adv

Several links from

http://www.google.com/search?hl=en&q=audio+and+video+search&btnG=Google+Search