The Real Problem of Bridging the Multimedia “Semantic Gap”

The Real Problem of Bridging the Multimedia “Semantic Gap” WWW 2007 John R. Smith Senior Manager, Intelligent Information Management IBM T. J. Watson Research Center Contact: jrsmith@watson.ibm.com

Multimedia and the Web

Multimedia on the Web has arrived in a big way:

Video is estimated to be 60% of web traffic by volume

70,000TB (or 101M hours) original TV & radio content produced annually; much is finding its way to Web

130B video streams will be served by 2010

Standards can have a role, but what exactly is the problem?

Playback – not today

Search?

Filtering?

Piracy prevention?

Advertising?

Let’s address search

Multimedia Standards and the Web No shortage of standards, metadata schemas, languages Too little understood about user needs and meanings related to the content Digital Content Chaos Metadata Chaos Semantic Chaos Formats Meanings

Need for Machine Tagging … Professional Cataloging and Social Tagging Not Enough!! Manual Cataloging – By Professionals Social Tagging – By Users Automated Tagging – By Machine Popularity Digital item Personal content Web, Deep archives, raw footage High-value content, hit-TV shows, movies “ Long tail” “ Bridging Semantic Gap”

Costly

Human resource intensive

Cannot keep up

Controlled vocabularies & standard taxonomies

Higher quality

Example: Fox, CNN, BBC, Broadcast TV

Cons Pros

Ambiguity

Uncontrolled vocabulary

Synonyms

User driven

Emergent folksonomies

Serpendipitous browsing

Examples: Del.icio.us and Flickr

Cons Pros

Requires training of models

Lower quality than manual tagging

Lower human cost

Domain & data driven approach to semantics

Example: Marvel, Informedia, TRECVID concept detection

Cons Pros

The Challenge

The “semantic gap” is a well-know problem in multimedia research (#1 on top-10 list)

Or so we thought …

The problem has been posed as classifying and searching multimedia content from low-level audio-visual features

But, most work to-date has been on the wrong side of the problem

Current State

Yes, machine learning technologies are an important tool

But, not being applied effectively to improve video search

Problem is lack of understanding of required semantics

Indoors/Outdoors ? Sunsets ? Cityscapes ?

Today’s approaches are ad hoc:

Researchers don’t have domain insights

Content/search providers don’t know full capability of today’s multimedia analysis technologies

Nobody has training data at required scale to effectively bridge semantic gap across full breadth and depth of multimedia semantics on Web

So, What do we do? Leverage Digital Masses to Create the Key Missing Resource – Media Net

Since video search is visual, the semantic spaces should be defined visually as well

Create large multimedia knowledge-base with exemplar content representing all semantic concepts relevant for search

Allow semantics space to evolve from end-user perspectives (across sports, entertainment, news)

Allow technology to focus on extracting the relevant semantics – truly providing the needed data-driven approach for bridging the multimedia semantic gap

… … … Sports Court Scene Players Game Scene Basketball News Crowd Scene