Multimedia Data Navigation and the Semantic Web   Valery A. Petrushin   and   Bradley P. Allen
Outline <ul><li>About the authors </li></ul><ul><li>Faceted Navigation </li></ul><ul><li>Semantic Web Techniques </li></ul...
About the Authors <ul><li>Valery A. Petrushin, Ph.D. </li></ul><ul><ul><ul><ul><li>Sr. Researcher, Accenture Technology La...
Faceted navigation <ul><li>Facets  are metadata properties whose ranges form a near-orthogonal set of controlled vocabular...
Faceted Navigation Built Using Semantic Web Standards <ul><li>Define/reuse ontologies expressed in RDF(S)/OWL </li></ul><u...
Building Faceted Navigation Applications …  then represented as instances of concepts in ontologies and tagged using contr...
Semantic Web Technology <ul><li>RDF(S) – Resource Description Framework (Schema)  </li></ul><ul><li>Dublin Core </li></ul>...
RDF (S) <ul><li>RDF (S) - Resource Description Framework (Schema)  </li></ul><ul><ul><ul><li>http://www.w3.org/RDF/ </li><...
Dublin Core (DC) <ul><li>Dublin Core </li></ul><ul><ul><ul><li>http://dublincore.org/documents/   </li></ul></ul></ul><ul>...
SKOS <ul><li>SKOS – Simple Knowledge Organization System </li></ul><ul><ul><ul><li>http://www.w3.org/2004/02/skos/   </li>...
TGM – I & II <ul><li>TGM – Thesaurus for Graphic Materials (The Library of Congress) </li></ul><ul><ul><ul><li>TGM-I – Sub...
LSCOM, SMIL & MPEG-7 <ul><li>LSCOM – Large Scale Concept Ontology for Multimedia </li></ul><ul><ul><ul><li>http://www.acem...
Case Study: BBC Rushes <ul><li>Rushes are raw footage … </li></ul><ul><ul><ul><li>with a promise to turn into golden nugge...
BBC Rushes: Data Statistics - 1 <ul><li>Statistics: clip level </li></ul><ul><ul><ul><li>615 clips (308 development + 307 ...
BBC Rushes: Data Statistics - 2 <ul><li>Statistics: shot level </li></ul><ul><ul><ul><li>Number of shots 10,064 </li></ul>...
BBC Rushes: representation <ul><li>Ontologies </li></ul><ul><ul><li>RDFS, Dublin Core, SKOS </li></ul></ul><ul><li>Control...
Ontology Schema dcterms: partOf dc: title dc: creator dc: subject dc: subject dc: created skos: broader skos: broader skos...
BBC Rushes: visual facets <ul><li>Facets: color, texture, [shape] + combinations </li></ul><ul><ul><ul><li>Color, texture,...
Self-organizing Maps <ul><li>SOM = Kohonen NN = Topology-preserving map </li></ul><ul><li>Unsupervised learning (Clusterin...
BBC Rushes: RDF subgraph Chilli_peppers v159_001.wmv v159.mpg “ michelle jones” 2000-03-01 dc:subject dc:creator dcterms:p...
BBC Rushes: RDF/XML serialization <ul><li><trecvid:Clip rdf:about=&quot;http://swvideo.techlabs.accenture.com/v159.mpg&quo...
BBC Rushes Navigator: Architecture AJAX client in Firefox XRBR query XRBR response BBC Rushes RDF http://www.siderean.com/...
Lessons Learned <ul><li>Data preparation </li></ul><ul><ul><ul><li>Robust shot boundary detection </li></ul></ul></ul><ul>...
Summary <ul><li>Methodology of Multimedia Data Representation </li></ul><ul><ul><li>Semantic Web Technology </li></ul></ul...
Future work <ul><li>More facets </li></ul><ul><ul><ul><li>Shape + combinations </li></ul></ul></ul><ul><ul><ul><li>Geograp...
BBC Rushes Navigator: Navigation with LSCOM
BBC Rushes Navigator: Hierarchical Drill-down on People Facet
BBC Rushes Navigator: Faceted View of All Shots
BBC Rushes Navigator: Searching by Subject
BBC Rushes Navigator:  Searching by Color, Playlist composition
BBC Rushes Navigator:  Drill-down using Subject and Color
Contact Information <ul><li>Valery A. Petrushin </li></ul><ul><ul><li>[email_address]   </li></ul></ul><ul><li>Bradley P. ...
Upcoming SlideShare
Loading in...5
×

Multimedia Data Navigation and the Semantic Web (SemTech 2006)

548

Published on

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
548
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Multimedia Data Navigation and the Semantic Web (SemTech 2006)

  1. 1. Multimedia Data Navigation and the Semantic Web Valery A. Petrushin and Bradley P. Allen
  2. 2. Outline <ul><li>About the authors </li></ul><ul><li>Faceted Navigation </li></ul><ul><li>Semantic Web Techniques </li></ul><ul><ul><li>RDF(S) </li></ul></ul><ul><ul><li>Dublin Core </li></ul></ul><ul><ul><li>SKOS </li></ul></ul><ul><ul><li>TGM </li></ul></ul><ul><ul><li>LSCOM, SMIL & MPEG-7 </li></ul></ul><ul><li>Case Study: BBC Rushes </li></ul><ul><li>Implementation </li></ul><ul><ul><li>BBC Rushes Navigator </li></ul></ul><ul><ul><ul><li>Metadata representation </li></ul></ul></ul><ul><ul><ul><li>Architecture </li></ul></ul></ul><ul><ul><ul><li>User interface </li></ul></ul></ul><ul><li>Future work </li></ul><ul><li>Contact Information </li></ul><ul><li>Demo </li></ul>
  3. 3. About the Authors <ul><li>Valery A. Petrushin, Ph.D. </li></ul><ul><ul><ul><ul><li>Sr. Researcher, Accenture Technology Labs </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Semantics of programming languages </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Multimedia data mining, analysis, annotation and retrieval </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Georgia Tech, Glushkov Institute for Cybernetics </li></ul></ul></ul></ul><ul><li>Bradley P. Allen </li></ul><ul><ul><ul><ul><li>Founder and CTO Siderean Software, Inc. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Semantic-based navigation, Web personalization services, case-based reasoning </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Former founder and CTO of Limbex Corp. and TriVida Corp. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Carnegie-Mellon University </li></ul></ul></ul></ul>
  4. 4. Faceted navigation <ul><li>Facets are metadata properties whose ranges form a near-orthogonal set of controlled vocabularies </li></ul><ul><ul><ul><li>Creator: “Dickens, Charles” </li></ul></ul></ul><ul><ul><ul><li>Subject: Arsenic, Antimony </li></ul></ul></ul><ul><ul><ul><li>Location: World > U.S. > California > Venice </li></ul></ul></ul><ul><li>Facets form a frame of reference for information overview, access and discovery </li></ul><ul><ul><ul><li>Other properties serve as landmarks and cues </li></ul></ul></ul><ul><li>Faceted navigation uses facets to provide end user access and discovery in the context of large collections of semi-structured information </li></ul>
  5. 5. Faceted Navigation Built Using Semantic Web Standards <ul><li>Define/reuse ontologies expressed in RDF(S)/OWL </li></ul><ul><ul><ul><li>Classes for defining instances and controlled vocabularies </li></ul></ul></ul><ul><ul><ul><li>Properties for facets and additional asset metadata attributes </li></ul></ul></ul><ul><li>Import/transform aggregated instance metadata into an RDF representation </li></ul><ul><ul><ul><li>Resources referred to via URIs </li></ul></ul></ul><ul><ul><ul><li>Content and controlled vocabularies </li></ul></ul></ul><ul><li>Write application profiles in terms of RDF </li></ul>
  6. 6. Building Faceted Navigation Applications … then represented as instances of concepts in ontologies and tagged using controlled vocabularies… … then application profiles are created… … that define navigation services for user applications Metadata is aggregated… Term Event Person Place Text Application Profiles
  7. 7. Semantic Web Technology <ul><li>RDF(S) – Resource Description Framework (Schema) </li></ul><ul><li>Dublin Core </li></ul><ul><li>SKOS – Simple Knowledge Organization System </li></ul><ul><li>TGM-I & II – Thesaurus for Graphic Materials </li></ul><ul><li>LSCOM – Large Scale Concept Ontology for Multimedia </li></ul><ul><li>SMIL – Synchronized Multimedia Integration Language </li></ul><ul><li>MPEG-7 – Multimedia Content Description Interface </li></ul>
  8. 8. RDF (S) <ul><li>RDF (S) - Resource Description Framework (Schema) </li></ul><ul><ul><ul><li>http://www.w3.org/RDF/ </li></ul></ul></ul><ul><ul><ul><li>http://www.w3.org/TR/rdf-schema/ </li></ul></ul></ul><ul><ul><ul><li>language for representing metadata about Web resources </li></ul></ul></ul><ul><ul><ul><li>Triple : subject – predicate -- > object </li></ul></ul></ul><ul><ul><ul><li>Example: </li></ul></ul></ul><?xml version=&quot;1.0&quot;?> <rdf:RDF xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot; xmlns:contact=&quot;http://www.w3.org/2000/10/swap/pim/contact#&quot;> <contact:Person rdf:about=&quot;http://www.accenture.com/techlabs/VAP/contact#me&quot;> <contact:fullName>Valery A. Petrushin</contact:fullName> <contact:mailbox rdf:resource=&quot;mailto:valery.a.petrushin@accenture.com&quot;/> </contact:Person> </rdf:RDF>
  9. 9. Dublin Core (DC) <ul><li>Dublin Core </li></ul><ul><ul><ul><li>http://dublincore.org/documents/ </li></ul></ul></ul><ul><ul><ul><li>vocabulary for describing documents (title, creator, subject, description, publisher, contributor, date, type, format, identifier, source, language, relation, coverage, rights) </li></ul></ul></ul><ul><ul><ul><li>Example: </li></ul></ul></ul><?xml version=&quot;1.0&quot;?> <!DOCTYPE rdf:RDF PUBLIC &quot;-//DUBLIN CORE//DCMES DTD 2002/07/31//EN&quot; &quot;http://dublincore.org/documents/2002/07/31/dcmes-xml/dcmes-xml-dtd.dtd&quot;> <rdf:RDF xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot; xmlns:dc=&quot;http://purl.org/dc/elements/1.1/&quot;> <rdf:Description rdf:about=&quot;http://www.accenture/techlabs/Petrushin&quot;> <dc:title> Multimedia Data Mining and Knowledge Discovery</dc:title> <dc:creator> Valery A. Petrushin </dc:creator > <dc:publisher>Springer Verlag</dc:publisher> </rdf:Description> </rdf:RDF>
  10. 10. SKOS <ul><li>SKOS – Simple Knowledge Organization System </li></ul><ul><ul><ul><li>http://www.w3.org/2004/02/skos/ </li></ul></ul></ul><ul><ul><ul><li>model for expressing structure and content of concept schemes (thesauri, taxonomies, etc.) </li></ul></ul></ul><ul><ul><ul><li>Specifies concepts, collections of concepts and relations between concepts (broader, narrower, related) </li></ul></ul></ul><ul><ul><ul><li>Example: </li></ul></ul></ul><rdf:RDF xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot; xmlns:skos=&quot;http://www.w3.org/2004/02/skos/core#&quot;> <rdf:Description rdf:about=&quot;http://www.example.com/concepts#people&quot;> <skos:broader rdf:resource=&quot;http://www.example.com/concepts#mammals&quot;/> <skos:narrower rdf:resource=&quot;http://www.example.com/concepts#children&quot;/> <skos:narrower rdf:resource=&quot;http://www.example.com/concepts#adults&quot;/> </rdf:Description> </rdf:RDF>
  11. 11. TGM – I & II <ul><li>TGM – Thesaurus for Graphic Materials (The Library of Congress) </li></ul><ul><ul><ul><li>TGM-I – Subject Terms (6,300) </li></ul></ul></ul><ul><ul><ul><ul><li>http://www.loc.gov/rr/print/tgm1/toc.html </li></ul></ul></ul></ul><ul><ul><ul><li>TGM-II – Genre and Physical Characteristic Headings (600) </li></ul></ul></ul><ul><ul><ul><ul><li>http://www.loc.gov/rr/print/tgm2/ </li></ul></ul></ul></ul><ul><ul><ul><li>Example: </li></ul></ul></ul>TGM-I: Term: Sand Narrower Term: Quicksand Related Term: Dunes, Sand sculpture, Sandpaintings TGM-II: Term: Aerial views Public Note: Views from a high vantage point. Used For: Air views, Balloon views, Views, Aerial Broader Term: Views Narrower Term: Aerial photographs Related Term: Bird's-eye views, Panoramic views
  12. 12. LSCOM, SMIL & MPEG-7 <ul><li>LSCOM – Large Scale Concept Ontology for Multimedia </li></ul><ul><ul><ul><li>http://www.acemedia.org/aceMedia/files/multimedia_ontology/presentations_1st_meeting/arda.pdf </li></ul></ul></ul><ul><li>SMIL – Synchronized Multimedia Integration Language </li></ul><ul><ul><ul><li>http://www.w3.org/TR/REC-smil/ </li></ul></ul></ul><ul><ul><ul><li>Simple language for representing multiple synchronized media streams </li></ul></ul></ul><ul><li>MPEG-7 – Multimedia Content Description Interface </li></ul><ul><ul><ul><li>http://www.chiariglione.org/mpeg/standards/mpeg-7/mpeg-7.htm </li></ul></ul></ul><ul><ul><ul><li>Advanced language for representing multimedia content </li></ul></ul></ul><ul><ul><ul><li>ISO Standard </li></ul></ul></ul>
  13. 13. Case Study: BBC Rushes <ul><li>Rushes are raw footage … </li></ul><ul><ul><ul><li>with a promise to turn into golden nuggets of stockshots </li></ul></ul></ul><ul><li>TRECVID 2005 </li></ul><ul><ul><ul><li>Video Retrieval Competition at NIST </li></ul></ul></ul><ul><ul><ul><li>http://www-nlpir.nist.gov/projects/trecvid/ </li></ul></ul></ul><ul><li>Problem: </li></ul><ul><ul><ul><li>create a system that helps a TV program maker compose a video using current clips and rushes </li></ul></ul></ul><ul><li>Data Statistics: </li></ul><ul><ul><li>Duration: 49.3 hours </li></ul></ul><ul><ul><li>Content: </li></ul></ul><ul><ul><ul><ul><li>Clips about vacation and travel </li></ul></ul></ul></ul><ul><ul><ul><ul><li>4 issues of “Summer Holiday” (~ 2 hours) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>BBC One News (30’) + fragment (~3’) </li></ul></ul></ul></ul>
  14. 14. BBC Rushes: Data Statistics - 1 <ul><li>Statistics: clip level </li></ul><ul><ul><ul><li>615 clips (308 development + 307 test sets) </li></ul></ul></ul><ul><ul><ul><li>Duration (mm:ss) : </li></ul></ul></ul><ul><ul><ul><ul><li>Minimal / Maximal - 00:03.48 / 47:11 </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Mean / Median – 04:49 / 02:25 </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Std - 06:02.73 </li></ul></ul></ul></ul><ul><ul><ul><li>Keywords: </li></ul></ul></ul><ul><ul><ul><ul><li>Different keywords / Occurrences – 1036 / 4908 </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Mean / Median – 7.98 / 7 </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Minimal / Maximal – 0 / 34 </li></ul></ul></ul></ul>
  15. 15. BBC Rushes: Data Statistics - 2 <ul><li>Statistics: shot level </li></ul><ul><ul><ul><li>Number of shots 10,064 </li></ul></ul></ul><ul><ul><ul><li>Shot duration (mm:ss) </li></ul></ul></ul><ul><ul><ul><ul><ul><li>Minimal - 0:00.04 </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Maximal – 22:45.16 </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Mean – 0:17.51 </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Median – 0:09.74 </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Std - 0:33.97 </li></ul></ul></ul></ul></ul><ul><ul><ul><li>Number of key frames </li></ul></ul></ul><ul><ul><ul><ul><ul><li>Total: 39,132 </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Median per shot: 2 </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Mean per shot: 3.8 </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Maximal: 377 </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Minimal: 1 </li></ul></ul></ul></ul></ul>
  16. 16. BBC Rushes: representation <ul><li>Ontologies </li></ul><ul><ul><li>RDFS, Dublin Core, SKOS </li></ul></ul><ul><li>Controlled vocabularies </li></ul><ul><ul><li>TGM-1 (reflecting Light Scale Concept Ontology for Multimedia), ISO8601 (temporal hierarchy of dates), MPEG-7 (visual features) </li></ul></ul><ul><li>Instances </li></ul><ul><ul><li>trecvid:Shot, trecvid:Clip </li></ul></ul><ul><li>Application profile </li></ul><ul><ul><li>Retrieve instances of type trecvid:Clip </li></ul></ul><ul><ul><ul><li>Textual facets: dc:title (clip title), dc:subject (keywords), dc:creator (director), dcterms:created (production date), dcterms:issued (show date), dc:extent (duration) </li></ul></ul></ul><ul><ul><li>Retrieve instances of type trecvid:Shot </li></ul></ul><ul><ul><ul><li>Visual facets: dc:subject with values skos:narrower than trecvid:color, trecvid:texture and trecvid:colorplustexture </li></ul></ul></ul><ul><ul><ul><li>Textual facets through reference to containing clip </li></ul></ul></ul>
  17. 17. Ontology Schema dcterms: partOf dc: title dc: creator dc: subject dc: subject dc: created skos: broader skos: broader skos: broader skos: broader skos: broader skos: broader skos: broader skos: broader VISUAL TEXTUAL Clip Shot KeyFrame Color Texture Color+Texture Title Creator Subject Date
  18. 18. BBC Rushes: visual facets <ul><li>Facets: color, texture, [shape] + combinations </li></ul><ul><ul><ul><li>Color, texture, color+texture </li></ul></ul></ul><ul><li>To build facets </li></ul><ul><ul><ul><li>Extract features (MPEG-7): </li></ul></ul></ul><ul><ul><ul><ul><li>Color: dominantColor(24), colorStructure (256) , colorLayout (12) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Texture: edgeHistogram (80), homogenousTexture (60) </li></ul></ul></ul></ul><ul><ul><ul><li>SOM Clustering of keyframes </li></ul></ul></ul><ul><ul><ul><ul><li>Select as a visual “word” the closest keyframe to node centroid </li></ul></ul></ul></ul><ul><ul><ul><li>Represent keyframes as SKOS concepts, centroids as skos:broader of cluster members </li></ul></ul></ul><ul><li>Example: </li></ul><ul><ul><li>SOM for color 35x28 (=980 nodes) </li></ul></ul>
  19. 19. Self-organizing Maps <ul><li>SOM = Kohonen NN = Topology-preserving map </li></ul><ul><li>Unsupervised learning (Clustering + Visualization) </li></ul><ul><li>X = { x i } , x i  R d - input data </li></ul><ul><li>M = { m k } , m k  R d - prototype vectors (codebook) = neurons on 1D or 2D grid </li></ul><ul><li>Training: </li></ul><ul><ul><ul><li>1. Start with random m k </li></ul></ul></ul><ul><ul><ul><li>2. For x i find best-matching unit (BMU) m c </li></ul></ul></ul><ul><ul><ul><li>3. Update prototype vectors in neighborhood </li></ul></ul></ul><ul><ul><ul><li>where is the neighborhood kernel </li></ul></ul></ul><ul><ul><ul><li>is radius at time t </li></ul></ul></ul><ul><li>Two phases: rough and fine tuning </li></ul>
  20. 20. BBC Rushes: RDF subgraph Chilli_peppers v159_001.wmv v159.mpg “ michelle jones” 2000-03-01 dc:subject dc:creator dcterms:partOf dc:created dc:subject color#26547 f000000000.jpg skos:broader skos:broader 2000 2000-03 Hot_peppers Peppers Year skos:broader skos:broader skos:broader skos:broader “ thailand, chiang mai/chillis” dc:title Color skos:broader
  21. 21. BBC Rushes: RDF/XML serialization <ul><li><trecvid:Clip rdf:about=&quot;http://swvideo.techlabs.accenture.com/v159.mpg&quot;> </li></ul><ul><li><rdf:type rdf:resource=&quot;&dctype;MovingImage&quot; /> </li></ul><ul><li><dc:title>thailand, chiang mai/chillis</dc:title> </li></ul><ul><li><dcterms:extent>202200</dcterms:extent> </li></ul><ul><li><dc:creator>michelle jones</dc:creator> </li></ul><ul><li><dc:identifier>mrs320354</dc:identifier> </li></ul><ul><li><dcterms:created rdf:resource=&quot;tag:siderean.com,1752-09-14:2000-03-01&quot; /> </li></ul><ul><li><dcterms:issued rdf:resource=&quot;tag:siderean.com,1752-09-14:2000-07-18&quot; /> </li></ul><ul><li><dc:subject rdf:resource=&quot;&trecvid;thailand&quot; /> </li></ul><ul><li><dc:subject rdf:resource=&quot;&trecvid;chiang_mai&quot; /> </li></ul><ul><li><dc:subject rdf:resource=&quot;&trecvid;chillis&quot; /> </li></ul><ul><li><dc:subject rdf:resource=&quot;&trecvid;peppers&quot; /> </li></ul><ul><li><dc:subject rdf:resource=&quot;&trecvid;chilli_peppers&quot; /> </li></ul><ul><li><dc:subject rdf:resource=&quot;&trecvid;vegetables&quot; /> </li></ul><ul><li><dc:subject rdf:resource=&quot;&trecvid;markets&quot; /> </li></ul><ul><li><dc:subject rdf:resource=&quot;&trecvid;street_markets&quot; /> </li></ul><ul><li><dc:subject rdf:resource=&quot;&trecvid;food_markets&quot; /> </li></ul><ul><li><dc:subject rdf:resource=&quot;&trecvid;food&quot; /> </li></ul><ul><li><dc:subject rdf:resource=&quot;&trecvid;herbs&quot; /> </li></ul><ul><li><dc:relation>http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg </li></ul><ul><li></dc:relation> </li></ul><ul><li></trecvid:Clip> </li></ul><ul><li><skos:Concept rdf:about=&quot;&trecvid;chilli_peppers&quot;> </li></ul><ul><li><skos:broader rdf:resource=&quot;&tgm1;Hot_peppers&quot;/> </li></ul><ul><li><skos:prefLabel>chilli peppers</skos:prefLabel> </li></ul><ul><li></skos:Concept> </li></ul><ul><li><skos:Concept rdf:about='tag:siderean.com,1752-09-14:2000-03-01'> </li></ul><ul><li><skos:prefLabel>2000-03-01</skos:prefLabel> </li></ul><ul><li><skos:broader rdf:resource='tag:siderean.com,1752-09-14:2000-03'/> </li></ul><ul><li></skos:Concept> </li></ul><ul><li><trecvid:Shot rdf:about=&quot;http://swvideo.techlabs.accenture.com/shotsWMV/v159_001.wmv&quot;> </li></ul><ul><li><rdf:type rdf:resource=&quot;&dctype;MovingImage&quot; /> </li></ul><ul><li><dcterms:isPartOf rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.mpg&quot; /> </li></ul><ul><li><dcterms:extent>21000</dcterms:extent> </li></ul><ul><li><dc:relation>http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg</dc:relation> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000240.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000280.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000001440.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000003120.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000005440.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000009680.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000011520.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000012040.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000013800.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000014800.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000015120.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000016760.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000018280.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000019360.jpg&quot;/> </li></ul><ul><li><dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000021000.jpg&quot;/> </li></ul><ul><li></trecvid:Shot> </li></ul><ul><li><skos:Concept rdf:about=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg&quot;> </li></ul><ul><li><skos:broader rdf:resource=&quot;http://swvideo.techlabs.accenture.com/color#26547&quot; /> </li></ul><ul><li><skos:prefSymbol rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg&quot; /> </li></ul><ul><li></skos:Concept> </li></ul><ul><li><skos:Concept rdf:about=&quot;http://swvideo.techlabs.accenture.com/color#26547&quot;> </li></ul><ul><li><skos:broader rdf:resource=&quot;&trecvid;color&quot; /> </li></ul><ul><li><skos:prefSymbol rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v289.fset/f000048880.jpg&quot; /> </li></ul><ul><li></skos:Concept> </li></ul>
  22. 22. BBC Rushes Navigator: Architecture AJAX client in Firefox XRBR query XRBR response BBC Rushes RDF http://www.siderean.com/bbcrush/bbcrush.jsp (with Firefox 1.5) Metadata Aggregator Metadata Store Navigation Web Services
  23. 23. Lessons Learned <ul><li>Data preparation </li></ul><ul><ul><ul><li>Robust shot boundary detection </li></ul></ul></ul><ul><ul><ul><li>Careful selection of keyframes </li></ul></ul></ul><ul><ul><ul><ul><li>Motion based </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Salient object based </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Filtering redundant keyframes </li></ul></ul></ul></ul><ul><ul><ul><li>Using group-of-frames (GOF) features </li></ul></ul></ul><ul><li>Concept recognition/propagation </li></ul><ul><ul><ul><li>Propagate keywords from clip to shots </li></ul></ul></ul><ul><ul><ul><li>Recognize concepts from visual data </li></ul></ul></ul><ul><ul><ul><li>Probabilistic reasoning </li></ul></ul></ul><ul><ul><ul><li>Derive concepts from data (data mining) + labeling </li></ul></ul></ul>
  24. 24. Summary <ul><li>Methodology of Multimedia Data Representation </li></ul><ul><ul><li>Semantic Web Technology </li></ul></ul><ul><ul><li>Multimedia Data Mining </li></ul></ul><ul><li>Prototype of Multimedia Retrieval System </li></ul><ul><ul><li>BBC Rushes </li></ul></ul><ul><ul><li>Web-based Interface using AJAX </li></ul></ul>
  25. 25. Future work <ul><li>More facets </li></ul><ul><ul><ul><li>Shape + combinations </li></ul></ul></ul><ul><ul><ul><li>Geographical location </li></ul></ul></ul><ul><li>More Interfaces </li></ul><ul><ul><ul><li>Map of the world for browsing places </li></ul></ul></ul><ul><ul><ul><li>Hierarchy of SOM for browsing clips and shots </li></ul></ul></ul><ul><li>More Tools </li></ul><ul><ul><ul><li>Tagging tool for creating and managing metadata </li></ul></ul></ul><ul><ul><ul><li>Tools for creating video databases (shot extraction, feature extraction, clustering, classification of events, etc.) </li></ul></ul></ul><ul><ul><ul><li>Tools for creating audio-video compositions (TV programs, commercials, etc.) </li></ul></ul></ul>
  26. 26. BBC Rushes Navigator: Navigation with LSCOM
  27. 27. BBC Rushes Navigator: Hierarchical Drill-down on People Facet
  28. 28. BBC Rushes Navigator: Faceted View of All Shots
  29. 29. BBC Rushes Navigator: Searching by Subject
  30. 30. BBC Rushes Navigator: Searching by Color, Playlist composition
  31. 31. BBC Rushes Navigator: Drill-down using Subject and Color
  32. 32. Contact Information <ul><li>Valery A. Petrushin </li></ul><ul><ul><li>[email_address] </li></ul></ul><ul><li>Bradley P. Allen </li></ul><ul><ul><li>[email_address] </li></ul></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×