Multimedia Semantics:Metadata, Analysis and Interaction

Multimedia Semantics: Metadata, Analysis and Interaction Raphael Troncy < [email_address] > Multimedia Semantics, EURECOM

Some BIG numbers

User Generated Content (Jul'09)

3.7+ billion photos

10+ billion photos

110+ million videos

20 hours uploaded / min ≈ 75 000 full length movies / week

Archived TV content

1.5 million hours ≈ 120 km of shelves

300000 hours | 1 petabyte / year

News content

Content difficult to search and reuse

Barely invisible for the search engines

Image/Video indexing

Techniques used by mainstream search engines

search term occurs in the filename or in the caption or in user tags

no semantics

Image indexing: main problem

an image is not alphabetic: there is no countable discrete units, that, in combination will provide the meaning of the image

image descriptors are not given with the image: one needs to extract or interpret them

Video indexing: additional problem

a video has additionally a temporal dimension to take into account

a video has a priori no discrete units neither (i.e. frames, shots, sequences cannot be absolutely defined)

Why is it so difficult to find appropriate multimedia content, to reuse and repurpose content previously published and to present this content in interfaces that vary with user needs?

Sounds Familiar?

[Arnold Smeulders, PAMI, 2000 ] The semantic gap is the lack of coincidence between the information that one can extract from the sensory data and the interpretation that the same data has for a user in a given situation

long way

a little drop of semantics goes a

Jim Hendler [ 1997 ]

Agenda

Semantics in multimedia analysis

Detecting concepts for video indexing

Evaluating interactive search tasks

Semantics in metadata

Multimedia metadata interoperability

Expose your data following 4 basic principles

Re-use a growing amount of publicly open datasets

Semantics in user interfaces

Provide meaningful presentation of underlying data

Explore large knowledge bases powered by linked data

The science of labeling

Automatically detecting the presence of a concept in a video stream

Naming visual information

airplane

The Computer Vision Approach

Building detectors one-at-the-time

a face detector for frontal faces

a face detector for non -frontal faces

3 years later One (or more) PhD for every new concept

So how about these?

A Simple Concept Detector

K-nearest neighbor

Linear Classification

Support Vector Machine

Supervised Learner

NIST TRECVID Evaluation

Until 2001, everybody defined his own concepts

Using specific and small data sets

Hard to compare methodologies

Since 2001, worldwide evaluation by NIST

Promote progress in video retrieval search

Provide common datasets (shots, ASR, key frames)

Use open, metrics-based evaluation

Large-Scale Concept Ontology for Multimedia

Success and Criticism

More and more concept detectors available:

TRECVID 2005: 101 concept lexicon

TRECVID 2006: 491 concept lexicon

MediaMill Challenge 2007: 572 concept lexicon

... but focus is on the final result

relative merit of indexing methods: ignore intermediary steps while systems become more complex (several features and learning methods)

... but concept detectors developed mismatch user information needs

TRECVID Interactive Video Search Task

Query selection:

by keyword,

by concept,

by example

Topics unknown

Test set

English (2004)

Chinese (2005-6)

Dutch (2007-8-9)

VideOlympics

Benchmark performance cannot be sole criterion

Experience of searcher counts

Usability of systems matters

VideoOlympics: live interactive search task

Simultaneous exposure of video retrieval systems

Showcase that goes beyond a regular demo session

Fun to do (participants) & Fun to watch (audience)

VideOlympics Setup

One display

TRECVID like queries

Results pushed by searchers

Agenda

Multimedia: Description methods MPEG-1 MPEG-2 MPEG-4 MPEG-7 MPEG-21 ISO W3C

MPEG-7: a multimedia description language?

ISO standard since December of 2001

Main components :

Descriptors (Ds) and Description Schemes (DSs)

DDL (XML Schema + extensions)

Concern all types of media

Part 5 – MDS Multimedia Description Schemes

MPEG-7 and the Semantic Web

MDS Upper Layer represented in RDFS

2001: Hunter

Later on: link to the ABC upper ontology

MDS fully represented in OWL-DL

2004: Tsinaraki et al., DS-MIRF model

MPEG-7 fully represented in OWL-DL

2005: Garcia and Celma, Rhizomik model

Fully automatic translation of the whole standard

MDS and Visual parts represented in OWL-DL

2007: Arndt et al., COMM model

Re-engineering MPEG-7 using DOLCE design patterns

Example 1: Region Annotation

Example 2: Sequence Annotation

Image Annotation with Linked Data The " Big Three " at the Yalta Conference (Wikipedia)

Localize a region (bounding box)

Annotate the content (interpretation)

Tag: Winston Churchill, UK Prime Minister, Allied Forces, WWII

Link to knowledge on the Web

Reg1 :Reg1 foaf:depicts dbpedia:Winston_Churchill ---------------------------------------------- dbpedia:Winston_Churchill dbpedia:spouse dbpedia:Clementine_Churchill dbpedia:Winston_Churchill owl:sameAs fbase:Winston_Churchill

Video Annotation with Linked Data A history of G8 violence ( video ) (© Reuters)

Localize a region

Annotate the content

Tag: G8 Summit, Heiligendamn, 2007

Link to knowledge on the Web

EU Summit, Gothenburg, 2001 Seq1 Seq4 :Seq1 foaf:depicts dbpedia:34th_G8_Summit ---------------------------------------------- dbpedia:33rd_G8_Summit foaf:based_near geo:Heilegendamn geo:Heilegendamn skos:broader geo:Germany

What is linked data?

URIs, possibly identifying media fragments

+ annotations (tags)

+ links among fragments & annotations

dbpedia:Zidane foaf:depicts nar:location geonames:2950159 nar:subject nc:15054000 events:id wp:2006_FIFA_World_Cup#Final

Linked Data Principles

Tim Berners Lee [2006] ( Design Issues )

Use URIs to identify things (anything, not just documents);

Use HTTP URIs – globally unique names, distributed ownership – so that people can look up those names;

Provide useful information in RDF – when someone looks up a URI;

Include RDF links to other URIs – to enable discovery of related information

An Example: DBpedia

DBpedia is a community effort to:

extract structured "infobox" information from Wikipedia

interlink DBpedia with other datasets on the Web

Scrapping infobox data http:// dbpedia.org /resource/Bogotá

Automatic Links Among Open Datasets <http://dbpedia.org/resource/Bogotá> owl:sameAs <http://sws.geonames.org/3688689/> owl:sameAs <http://rdf.freebase.com/ns/guid.9202a8c04000641f8000000000167bab> dbpedia:population "6776009" ... <http://sws.geonames.org/3688689/> owl:sameAs <http://dbpedia.org/resource/Bogotá> wgs84_pos:lat "4.6" wgs84_pos:long "-74.0833333" geo:population "7102602" ... Geonames DBpedia

sameAs.org

Bogotá on Freebase

Bogotá on Geonames

How Much Linked Data is there ?

Linked Data Cloud – August 2007

Linked Data Cloud – March 2008

Linked Data Cloud – September 2008

Linked Data Cloud – March 2009

The Web of Data

Expose open datasets in RDF

Set RDF links among the data items for different datasets

Over 4.5 billion triples, 5 millions links (March 2009)

... still counting

Who are the users?

Why would they use the cloud?

What tasks can be supported?

How will the semantics help?

Agenda

Provide meaningful presentation of data

... and behind the scene

... link an artist to more data

... myspace

... last.fm

... IMDb

Going through the Walled Gardens David Simonds: Everywhere and nowhere. 19 May 2008, The Economist .

How can semantics help?

Query construction

disambiguate input (auto-completion)

selection of available terms (grouping and ranking algorithms)

(Semantic) search algorithm

graph traversal

query expansion

RDFS/OWL reasoning

Presentation of search results

grouping by property

visualization on timeline, map, etc.

News Workflow Interoperability

No integration of media (stories, photo, animation, video)

Little (or no) context in the news presentation

Lack of interoperability in the current workflow

NAR Schema Controlled Vocabularies Broadcaster Schema NewsCodes User Vocabulary

Exploratory Search

(Ultimate) Goal:

Provide an environment for searching and browsing contextualized multimedia news information

Required integration:

Data: various media, different forms, various sources

Metadata: schema integration, semantic models

Influence and implications of UI:

How to represent semantic multimedia metadata to facilitate presenting information?

in other words ... What constraints do end-user interfaces put on the modeling of the metadata?

News and Multimedia Formats (NAR) News Architecture NewsML G2 EventsML G2 SportsML G2

Modeling the News + Media Ontology foaf:Person ≈ nar:Person dc:Subject ≈ nar:Subject sioc:Item ≈ nar:Item geo:lat geo:long +

Enriching the News Metadata

Concepts/Entities that are subject of news

Thematic categories

People

Organizations

Geopolitical Areas

Points of Interest

Events

Products or artefacts

Enriching the News Metadata NAR Ontology NewsCodes Thesaurus Named Entity Recognition Domain Ontologies

Enriching the News Metadata NAR Ontology NewsCodes Thesaurus Domain Ontologies Concept Detectors

Presenting News Information

Dimensions used for searching news items

When time 10/07/2006

Where location Paris

What is depicted J. Chirac, Z. Zidane

Why event WC 2006

Who photographer Bertrand Guay, AFP

Metadata

Semantic Search of Multimedia News Powered by ClioPatria 1.0 alpha 3 1,067,754 Total 1,932 INA Broadcast Video (June/July 2006) 61,311 AFP Photos (June/July 2006) 804,446 AFP News Feed (June/July 2006) 53,468 DBpedia, Geonames 34,903 Thesauri: newscodes 104,358 Domain Specific Ontologies: football 7,336 General Ontologies: NAR, DC, FOAF Number of RDF Triples Description

Provide New Dimensions for Exploring

Take Home Message