Semantic Analysis of Video Data to Enable Exploratory Search

  • 308 views
Uploaded on

Keynote at LinkedUp Competition Workshop 2014, at ESWC 2014, May 26, 2014, Crete, Greece. …

Keynote at LinkedUp Competition Workshop 2014, at ESWC 2014, May 26, 2014, Crete, Greece.
Abstract: Video Data has become a first class citizen of the Web. Also in the area of eLearning lecture recordings in combination with massive open online courses have become rather popular and their numbers are constantly growing. To find the appropriate video content according to the user’s information needs and to make well suited content based recommendations for videos still is an issue of ongoing research. Semantic technologies combined with state-of-the-art video analysis, as e.g. speech to text, video ocr, and visual concept detection, enable pinpoint access to video data and clear the path for content based recommender systems. Based on time-based semantically annotated video data new ways become feasible of accessing video in an exploratory way to enhance user experience. - See more at: http://linkedup-project.eu/linkedup-eswc-workshop/#sthash.QuCYDtkN.dpuf

More in: Education , Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
308
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
12
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. LinkedUp Vidi Competition Presentations and Awards 2014, ESWC 2014 Semantic Analysis of Video Data to Enable Exploratory Search Dr. Harald Sack Hasso-Plattner-Institut for IT-Systems Engineering University of Potsdam Crete, 25/06/2014 Montag, 26. Mai 14
  • 2. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 •How to Search Video Web vs. Archive •How to Open Up Video Data? Video Analysis Technologies •How to „Understand“ Video Metadata? Context-Driven Semantic Analysis •How to Use of Semantic Metadata? Exploratory Search and Intelligent Recommendations Semantic Analysis of Video Data to Enable Exploratory Search Montag, 26. Mai 14
  • 3. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam 3 Searching the Web Montag, 26. Mai 14
  • 4. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam 4 Searching the Web Montag, 26. Mai 14
  • 5. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam 5 Montag, 26. Mai 14
  • 6. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam 6 Google Knowledge Graph = “search results with semantic- search information gathered from a wide variety of sources“ Montag, 26. Mai 14
  • 7. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011 Google Multimedia Search Montag, 26. Mai 14
  • 8. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 ‣Google Multimedia Search relies on text-based metadata and link context How does Google find Multimedia? Montag, 26. Mai 14
  • 9. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 Seach by Media Content Montag, 26. Mai 14
  • 10. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 The Ordinary Archive is a Small World... Neil Armstrong Montag, 26. Mai 14
  • 11. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 But, wouldn‘t it be nice, if..... Neil Armstrong ...but maybe you are also interested in - Buzz Aldrin (1 videos) - John Glen (1 video) - Juri Gagarin (2 videos) - Richard Nixon (3 videos) - Apollo 11 (1 video) - NASA (20 videos) - Moon (14 videos) - space exploration (34 videos) - technology (1.205 videos) Sorry, no results found for ‘Neil Armstrong‘... Montag, 26. Mai 14
  • 12. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 How to Search in Video Archives? Montag, 26. Mai 14
  • 13. vfm - Seminar: Metadatenmanagement in Medienunternehmen, 05. September 2012, Bonn Jörg Waitelonis, Hasso-Plattner-Institut Potsdam Content-Based Search in Video Archives relies on text-based Metadata Current Solution: Manual Annotation Montag, 26. Mai 14
  • 14. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 image Visual Concept Detection Text Recognition Visual Analysis (Selected) Automated Multimedia Analysis Face Detection Face Detection Logo Detection audio- visual text / images Audio-Mining structural analysis Automated Speech Recognition audio event detection audio Montag, 26. Mai 14
  • 15. Annotation of Audiovisual Data Metadata Extraction Metadata (e.g. MPEG-7) ... <SpatialDecomposition> <TextAnnotation> <KeywordAnnotation> <Keyword>Astronaut</Keyword> </KeywordAnnotation> </TextAnnotation> <SpatialMask> <SubRegion> <Polygon> <Coords> 480 150 620 480 </Coords> </Polygon> </SubRegion> </SpatialMask> ... </SpatialDecomposition> ... • Video data with spatiotemporal Annotations Neil Armstrong Montag, 26. Mai 14
  • 16. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam www.yovisto.com Montag, 26. Mai 14
  • 17. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 • Authoritative Metadata • structured data • semi-structured data • natural language text • Non-authoritative Metadata • (free) user tags and comments • restricted vocabularies • (Media) Analysis Metadata • low level features • high level features • etc. How to Determine the Meaning of Metadata? Semantic Analysis reliability context pragmatics location dependency accuracy time dependency level of abstraction Montag, 26. Mai 14
  • 18. Neil Armstrong Astronaut is a Person is a Science Occupation subClassOf Employment subClassOf Entities Ontologies has an ,Neil Armstrong‘ is more than just a character string Kosmonaut same as Juri Gagarin is a is NOT a ! Montag, 26. Mai 14
  • 19. Astronaut Person Neil Armstrong Science Occupation Employment is a is a is a is a has a Web of Data Montag, 26. Mai 14
  • 20. Web of Data = Linked Open Data But what, if there is no trivial unique identification? Armstrong user tag Montag, 26. Mai 14
  • 21. Semantic Web Technologies , Dr. Harald Sack, Hasso Plattner Institute, University of Potsdam Armstrong Montag, 26. Mai 14
  • 22. Semantic Web Technologies , Dr. Harald Sack, Hasso Plattner Institute, University of Potsdam Armstrong Armstrong+ Moon Montag, 26. Mai 14
  • 23. Web of Data = Linked Open Data Understanding requires Context Armstrong Moon EagleSpace Montag, 26. Mai 14
  • 24. 42 42 42 42 24 4242 42 42 42 42 Semantic Analysis Semantics is determined by Context Context Item N.Steinmetz, H.Sack: Semantic Multimedia Information Retrieval Based on Contextual Descriptions, 2013 „Armstrong landed the Eagle on the Moon.“ Text SEMEX Multimedia Context Model Context Dimensions Temporal Context Spatial Context Provenance Context Relevance determines Ambiguity influences Accuracy influences Contextual Description Class Diversity Level of Structure Source Reliability Source Diversity Montag, 26. Mai 14
  • 25. Armstrong George Armstrong Custer Neil Armstrong The Armstrong Twins Armstrong, Florida Armstrong, Ontario Armstrong Automobile Joe Armstrong Armstrong County, Texass Armstrong Gun Craig Armstrong Armstrong (Moon Crater) Louis Armstrong Armstrong Tunnel Louis Armstrong International Airport Armstrong‘s Theorem Sir Thomas Armstrong Ian Armstrong Eagle Moon Eagle (Bird) Eagle (heraldry) USCGC Eagle The Eagle (2011 film) Eagle (song) John H. Eagle Eagle (typeface) Eagle Falls (Washington) Eagle (Moon Crater) Eagle (comic) Eagle (lunar module) Eagle TV Armstrong Tunnel The Eagle (Pub) War Eagle The Eagle (newspaper) Eagle (racehorse) Angela Eagle Linda Eagle James Philipp Eagle 95 entities448 entities Armstrong (British Columbia) Karen Armstrong Curtis Armstrong Gillian Armstrong Hilary Armstrong William L. Armstrong 156 entities Man on the Moon (film) Moon (song) Moon Son-Ri C Moon The Moon (Tarot card) Edgar Moon Moon OS Moon (Band) Moon Moon 44 Man on the Moon (soundtrack) William Moon Lottie Moon Mr. Moon (song) Man on the Moon (musical) Darvin Moon Moon 83 Francis Moon Gary Moon Robert Charles Moon Black Moon Allan Moon Ban-Ki Moon Fly me to the Moon (song) Semantic Analysis Named Entity Mapping „Armstrong landed the Eagle on the Moon.“ Consider all entities within the same context Montag, 26. Mai 14
  • 26. Armstrong George Armstrong Custer The Armstrong Twins Armstrong, Florida Armstrong, Ontario Armstrong Automobile Joe Armstrong Armstrong County, Texass Armstrong Gun Craig Armstrong Armstrong (Moon Crater) Armstrong Tunnel Louis Armstrong International Airport Armstrong‘s Theorem Sir Thomas Armstrong Ian Armstrong Eagle Moon Eagle (Bird) Eagle (heraldry) USCGC Eagle The Eagle (2011 film) Eagle (song) John H. Eagle Eagle (typeface) Eagle Falls (Washington) Eagle (Moon Crater) Eagle (comic) Eagle TV Armstrong Tunnel The Eagle (Pub) War Eagle The Eagle (newspaper) Eagle (racehorse) Angela Eagle Linda Eagle James Philipp Eagle 95 entities448 entities Armstrong (British Columbia) Karen Armstrong Curtis Armstrong Gillian Armstrong Hilary Armstrong William L. Armstrong 156 entities Man on the Moon (film) Moon (song) Moon Son-Ri C Moon The Moon (Tarot card) Edgar Moon Moon OS Moon (Band) Moon 44 Man on the Moon (soundtrack) William Moon Lottie Moon Mr. Moon (song) Man on the Moon (musical) Darvin Moon Moon 83 Francis Moon Gary Moon Robert Charles Moon Black Moon Allan Moon Ban-Ki Moon Neil Armstrong Eagle (lunar module) Moon Louis Armstrong Fly me to the Moon (song) Semantic Analysis Named Entity Mapping „Armstrong landed the Eagle on the Moon.“ N. Steinmetz, H.Sack: Semantic Multimedia Information Retrieval Based on Contextual Descriptions, 2013 Entity Selection Process (Semantic) Graph Analysis Montag, 26. Mai 14
  • 27. 42 42 42 42 24 4242 42 42 42 42 vfm - Seminar: Metadatenmanagement in Medienunternehmen, 05. September 2012, Bonn Jörg Waitelonis, Hasso-Plattner-Institut Potsdam 27 Semantically Annotated Video Data Video Analysis / Metadata Extraction time metadata metadata metadata metadata metadata e.g., person xy location yz event abc e.g., bibliographical data, geographical data, encyclopedic data, .. Entity Recognition/ Mapping N. Ludwig, H. Sack: Named Entity Recognition for User- Generated Tags. In Proc. of the 8th Int. Workshop on Text-based Information Retrieval, IEEE CS Press, 2011 Montag, 26. Mai 14
  • 28. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam 28 Entity Based Search • linguistic ambiguities of traditional keyword based search can be avoided • enables high precision and high recall retrieval http://www.yovisto.com/labs/autosuggestion/ • Query string refinement / extension • entity auto-suggestion • interpretation of natural language queries J. Osterhoff, J. Waitelonis, H. Sack, Widen the Peepholes! Entity-Based Auto-Suggestion as a rich and yet immediate Starting Point for Exploratory Search, IVDW 2012 Montag, 26. Mai 14
  • 29. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam 29 http://semex.hpi.uni-potsdam.de/mggui-dev2 search facets C. Hentschel, H. Sack, et al., Open up cultural heritage in video archives with mediaglobe, I2CS 2012 Montag, 26. Mai 14
  • 30. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam 30 Towards Exploratory Search dbpedia-owl:mission dbpedia:Neil_Armstrong dbpedia:Apollo_11 dbpedia-owl:mission category:Apollo_program dcterms:subject dbpedia:Apollo_13 dcterms:subject yago:Space_accidents_and_incidents rdf:type rdf:type dbpedia:Space_Shuttle_Challenger dbpedia-owl:mission J. Waitelonis, H. Sack: Towards exploratory video search using linked data, MTAP Volume 59, Number 2 (2012), 645-672 dbpedia:Buzz_Aldrin dbpedia:Michael_Collins Montag, 26. Mai 14
  • 31. Exploratory Search and Serendipity •Find something that you were not looking for on purpose ... dbpedia:Buzz_Aldrin dbpedia:Cookie_Monster dbpedia:Strictly_Come_Dancing dbpedia:Transformers Montag, 26. Mai 14
  • 32. Explorative Search & Intelligent Recommmendation with yovisto J. Waitelonis, H. Sack: Towards exploratory video search using linked data, Multimedia Tools and Applications, Volume 59, Number 2 (2012), pp. 645-672 Montag, 26. Mai 14
  • 33. Explorative Search & Intelligent Recommmendation with yovisto http://mediaglobe.yovisto.com:8080/ Montag, 26. Mai 14
  • 34. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 Contact: Dr. Harald Sack Hasso-Plattner-Institut für Softwaresystemtechnik Universität Potsdam Prof.-Dr.-Helmert-Str. 2-3 D-14482 Potsdam Homepage: http://www.hpi.uni-potsdam.de/meinel/team/sack.html Blog: http://yovisto.blogspot.com/ E-Mail: harald.sack@hpi.uni-potsdam.de Twitter: lysander07 / biblionomicon / yovisto Slides can be found at http://slideshare.com/lysander07/ Thank you very much for your attention! Montag, 26. Mai 14