Symposium on Future Trends in Service-Oriented Computing,HPI Potsdam, 20-21.06.2013Semantic MultimediaAnalysis and SearchD...
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011•Searching MultimediaW...
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam3Searching the WebFreitag, 21. Juni 13
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam4Searching the WebFreitag, 21. Juni 13
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam5Freitag, 21. Juni 13
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam6Google Knowledge Graph= “search resu...
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Be...
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011‣Google Multimedia Sea...
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Seach by Media Content...
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011The Ordinary Archive i...
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011But, wouldn‘t it be ni...
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011How to Search inMultim...
vfm - Seminar: Metadatenmanagement in Medienunternehmen, 05. September 2012, Bonn Jörg Waitelonis, Hasso-Plattner-Institut...
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 imageVisualConceptDet...
Structural Video Analysis• Decomposition of time-based media into meaningful mediafragments of coherent content that can b...
Video Optical Character Recognition (OCR)• Video OCR is much more difficultthan traditional print OCR• fast detection/filter...
• Face DetectionDetect candidate image regionsin a video frame that depict ahuman face• Face TrackingTrack a detected face...
Visual Concept Detection• Adaption of traditional ,Bag of Words‘approach from text retrieval• Image is expressed as vector...
Annotation of Audiovisual DataMetadata ExtractionMetadata (e.g. MPEG-7)...<SpatialDecomposition><TextAnnotation><KeywordAn...
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdamwww.yovisto.comFreitag, 21. Juni 13
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011• Authoritative Metada...
Neil ArmstrongAstronautis aPersonis aScience OccupationsubClassOfEmploymentsubClassOfEntitiesOntologieshas an,Neil Armstro...
Where does the knowledge come from...?Freitag, 21. Juni 13
Astronaut PersonNeil ArmstrongScience OccupationEmploymentis a is ais ais a has aWeb of DataFreitag, 21. Juni 13
Web of Data = Linked Open DataBut what, if there is no trivial unique identification?Armstronguser tagFreitag, 21. Juni 13
Semantic Web Technologies , Dr. Harald Sack, Hasso Plattner Institute, University of PotsdamArmstrongFreitag, 21. Juni 13
Semantic Web Technologies , Dr. Harald Sack, Hasso Plattner Institute, University of PotsdamArmstrongArmstrong+MoonFreitag...
Web of Data = Linked Open DataUnderstanding requires ContextArmstrongMoonEagleSpaceFreitag, 21. Juni 13
4242424224424242424242Semantic AnalysisSemantics is determined by ContextContext ItemN.Steinmetz, H.Sack: Semantic Multime...
ArmstrongGeorge Armstrong CusterNeil ArmstrongThe Armstrong TwinsArmstrong, FloridaArmstrong, OntarioArmstrong AutomobileJ...
Select matching entities from all possible candidate entities:• Popularity based strategies• Linguistical strategies• Stat...
ArmstrongGeorge Armstrong CusterThe Armstrong TwinsArmstrong, FloridaArmstrong, OntarioArmstrong AutomobileJoe ArmstrongAr...
4242424224424242424242vfm - Seminar: Metadatenmanagement in Medienunternehmen, 05. September 2012, Bonn Jörg Waitelonis, H...
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam34Entity Based Search• linguistic amb...
Freitag, 21. Juni 13
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam36http://mediaglobe.yovisto.com:8080/...
Freitag, 21. Juni 13
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam38Explorative Searchdbpedia-owl:missi...
Exploratory Search and Serendipity•Find something that you were not looking for on purpose ...dbpedia:Buzz_Aldrindbpedia:C...
Explorative Search & Intelligent Recommmendationwith yovistohttp://mediaglobe.yovisto.com:8080/Freitag, 21. Juni 13
Explorative Search & Intelligent Recommmendationwith yovistohttp://mediaglobe.yovisto.com:8080/Freitag, 21. Juni 13
Explorative Search & Intelligent Recommmendationwith yovistohttp://mediaglobe.yovisto.com:8080/Freitag, 21. Juni 13
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Contact:Dr. Harald Sac...
Upcoming SlideShare
Loading in …5
×

Semantic Multimedia Analysis and Search - Future SOC Symposium 2013

1,217 views
1,076 views

Published on

keynote at FutureSOC Symposium 2013, at HPI, Potsdam, 20-21.06.2013

Published in: Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,217
On SlideShare
0
From Embeds
0
Number of Embeds
16
Actions
Shares
0
Downloads
19
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Semantic Multimedia Analysis and Search - Future SOC Symposium 2013

  1. 1. Symposium on Future Trends in Service-Oriented Computing,HPI Potsdam, 20-21.06.2013Semantic MultimediaAnalysis and SearchDr. Harald SackHasso-Plattner-Institut for IT-Systems EngineeringUniversity of PotsdamPotsdam, 21/06/2013Freitag, 21. Juni 13
  2. 2. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011•Searching MultimediaWeb vs. Archive•How to Open Up Multimedia Data?Automated Multimedia Analysis•How to Determine the Meaning of(Multimedia) Metadata?Context-Driven Semantic Analysis•How to Make Use of SemanticMetadata?Exploratory Search and Intelligent RecommendationsSemantic MultimediaAnalysis and SearchFreitag, 21. Juni 13
  3. 3. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam3Searching the WebFreitag, 21. Juni 13
  4. 4. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam4Searching the WebFreitag, 21. Juni 13
  5. 5. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam5Freitag, 21. Juni 13
  6. 6. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam6Google Knowledge Graph= “search results with semantic-search information gathered froma wide variety of sources“Freitag, 21. Juni 13
  7. 7. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Google Multimedia SearchFreitag, 21. Juni 13
  8. 8. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011‣Google Multimedia Search relies on text-basedmetadata and link contextHow does Google find Multimedia?Freitag, 21. Juni 13
  9. 9. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Seach by Media ContentFreitag, 21. Juni 13
  10. 10. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011The Ordinary Archive is a Small World...Neil ArmstrongFreitag, 21. Juni 13
  11. 11. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011But, wouldn‘t it be nice, if.....Neil Armstrong...but maybe you are also interested in- Buzz Aldrin (1 videos)- John Glen (1 video)- Juri Gagarin (2 videos)- Richard Nixon (3 videos)- Apollo 11 (1 video)- NASA (20 videos)- Moon (14 videos)- space exploration (34 videos)- technology (1.205 videos)Sorry, no results found for ‘Neil Armstrong‘...Freitag, 21. Juni 13
  12. 12. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011How to Search inMultimedia Archives?Freitag, 21. Juni 13
  13. 13. vfm - Seminar: Metadatenmanagement in Medienunternehmen, 05. September 2012, Bonn Jörg Waitelonis, Hasso-Plattner-Institut PotsdamContent-Based Search in MultimediaArchives relies on text-based MetadataCurrent Solution: Manual AnnotationFreitag, 21. Juni 13
  14. 14. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 imageVisualConceptDetectionTextRecognitionVisualAnalysis(Selected) Automated Media AnalysisFaceDetectionFaceDetectionLogoDetectionaudio-visualtext / imagesAudio-MiningstructuralanalysisAutomatedSpeechRecognitionaudio eventdetectionaudioFreitag, 21. Juni 13
  15. 15. Structural Video Analysis• Decomposition of time-based media into meaningful mediafragments of coherent content that can be used as basicelement for indexing and classificationscenesshotssubshotsframesvideokeyframesFreitag, 21. Juni 13
  16. 16. Video Optical Character Recognition (OCR)• Video OCR is much more difficultthan traditional print OCR• fast detection/filtering of text candidates• verification of text candidates• script separation from background• visual quality enhancement• application of standard OCR software• spell correction w.r.t. context and temporalredundancyFreitag, 21. Juni 13
  17. 17. • Face DetectionDetect candidate image regionsin a video frame that depict ahuman face• Face TrackingTrack a detected face in videoover consecutive frames withinshot boundaries• Face ClusteringGroup faces detected and trackedin videos into visually similar setswithin a single video• Face Recognition/IdentificationReliable identification of detectedfacesVideo Face Detection, Tracking & Clusteringpersonfrontal face:90%not a personpersonprofile face:70%Freitag, 21. Juni 13
  18. 18. Visual Concept Detection• Adaption of traditional ,Bag of Words‘approach from text retrieval• Image is expressed as vector (histogram)of dictionary codeword frequencies• classification via machine learning(Support Vector Machines)Freitag, 21. Juni 13
  19. 19. Annotation of Audiovisual DataMetadata ExtractionMetadata (e.g. MPEG-7)...<SpatialDecomposition><TextAnnotation><KeywordAnnotation><Keyword>Astronaut</Keyword></KeywordAnnotation></TextAnnotation><SpatialMask><SubRegion><Polygon><Coords>480 150 620 480</Coords></Polygon></SubRegion></SpatialMask>...</SpatialDecomposition>...• Multimedia data with spatiotemporal AnnotationsNeil ArmstrongFreitag, 21. Juni 13
  20. 20. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdamwww.yovisto.comFreitag, 21. Juni 13
  21. 21. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011• Authoritative Metadata• structured data• semi-structured data• natural language text• Non-authoritative Metadata• (free) user tags and comments• restricted vocabularies• (Media) Analysis Metadata• low level features• high level features• etc.How to Determine the Meaning of Metadata?SemanticAnalysisreliabilitycontextpragmaticslocationdependencyaccuracytimedependencylevel ofabstractionFreitag, 21. Juni 13
  22. 22. Neil ArmstrongAstronautis aPersonis aScience OccupationsubClassOfEmploymentsubClassOfEntitiesOntologieshas an,Neil Armstrong‘ is more than just a character stringKosmonautsame asJuri Gagarinis ais NOT a!Freitag, 21. Juni 13
  23. 23. Where does the knowledge come from...?Freitag, 21. Juni 13
  24. 24. Astronaut PersonNeil ArmstrongScience OccupationEmploymentis a is ais ais a has aWeb of DataFreitag, 21. Juni 13
  25. 25. Web of Data = Linked Open DataBut what, if there is no trivial unique identification?Armstronguser tagFreitag, 21. Juni 13
  26. 26. Semantic Web Technologies , Dr. Harald Sack, Hasso Plattner Institute, University of PotsdamArmstrongFreitag, 21. Juni 13
  27. 27. Semantic Web Technologies , Dr. Harald Sack, Hasso Plattner Institute, University of PotsdamArmstrongArmstrong+MoonFreitag, 21. Juni 13
  28. 28. Web of Data = Linked Open DataUnderstanding requires ContextArmstrongMoonEagleSpaceFreitag, 21. Juni 13
  29. 29. 4242424224424242424242Semantic AnalysisSemantics is determined by ContextContext ItemN.Steinmetz, H.Sack: Semantic Multimedia Information Retrieval Based on Contextual Descriptions, 2013„Armstrong landed the Eagle on the Moon.“TextSEMEX Multimedia Context ModelContext DimensionsTemporalContextSpatialContextProvenanceContextRelevancedeterminesAmbiguityinfluencesAccuracyinfluencesContextual DescriptionClassDiversityLevel ofStructureSourceReliabilitySourceDiversityFreitag, 21. Juni 13
  30. 30. ArmstrongGeorge Armstrong CusterNeil ArmstrongThe Armstrong TwinsArmstrong, FloridaArmstrong, OntarioArmstrong AutomobileJoe ArmstrongArmstrong County, TexassArmstrong GunCraig ArmstrongArmstrong (Moon Crater)Louis ArmstrongArmstrong TunnelLouis Armstrong International AirportArmstrong‘s TheoremSir Thomas ArmstrongIan ArmstrongEagle MoonEagle (Bird)Eagle (heraldry)USCGC EagleThe Eagle (2011 film)Eagle (song)John H. EagleEagle (typeface)Eagle Falls (Washington)Eagle (Moon Crater)Eagle (comic)Eagle (lunar module)Eagle TVArmstrong TunnelThe Eagle (Pub)War EagleThe Eagle (newspaper)Eagle (racehorse)Angela EagleLinda EagleJames Philipp Eagle95 entities448 entitiesArmstrong (British Columbia)Karen ArmstrongCurtis ArmstrongGillian ArmstrongHilary ArmstrongWilliam L. Armstrong156 entitiesMan on the Moon (film)Moon (song)Moon Son-RiC MoonThe Moon (Tarot card)Edgar MoonMoon OSMoon (Band)MoonMoon 44Man on the Moon (soundtrack)William MoonLottie MoonMr. Moon (song)Man on the Moon (musical)Darvin MoonMoon 83Francis MoonGary MoonRobert Charles MoonBlack MoonAllan MoonBan-Ki MoonFly me to the Moon (song)Semantic AnalysisNamed Entity Mapping„Armstrong landed the Eagle on the Moon.“Consider all entities within the same contextFreitag, 21. Juni 13
  31. 31. Select matching entities from all possible candidate entities:• Popularity based strategies• Linguistical strategies• Statistical strategies• Semantic based strategiesGeneral Approach1. Make an assumption2. Do the strategies support or contradict your assumption3. Make decision according to logical and probabilistic rules/constraintsSemantic AnalysisNamed Entity RecognitionN. Ludwig, H. Sack, “Named entity recognition for user-generated tags,TIR 2011• reference text corpus(wikipedia)• link graph (wikipedia)• semantic graph(DBpedia)Entity Selection ProcessFreitag, 21. Juni 13
  32. 32. ArmstrongGeorge Armstrong CusterThe Armstrong TwinsArmstrong, FloridaArmstrong, OntarioArmstrong AutomobileJoe ArmstrongArmstrong County, TexassArmstrong GunCraig ArmstrongArmstrong (Moon Crater)Armstrong TunnelLouis Armstrong International AirportArmstrong‘s TheoremSir Thomas ArmstrongIan ArmstrongEagle MoonEagle (Bird)Eagle (heraldry)USCGC EagleThe Eagle (2011 film)Eagle (song)John H. EagleEagle (typeface)Eagle Falls (Washington)Eagle (Moon Crater)Eagle (comic)Eagle TVArmstrong TunnelThe Eagle (Pub)War EagleThe Eagle (newspaper)Eagle (racehorse)Angela EagleLinda EagleJames Philipp Eagle95 entities448 entitiesArmstrong (British Columbia)Karen ArmstrongCurtis ArmstrongGillian ArmstrongHilary ArmstrongWilliam L. Armstrong156 entitiesMan on the Moon (film)Moon (song)Moon Son-RiC MoonThe Moon (Tarot card)Edgar MoonMoon OSMoon (Band)Moon 44Man on the Moon (soundtrack)William MoonLottie MoonMr. Moon (song)Man on the Moon (musical)Darvin MoonMoon 83Francis MoonGary MoonRobert Charles MoonBlack MoonAllan MoonBan-Ki MoonNeil ArmstrongEagle (lunar module)MoonLouis ArmstrongFly me to the Moon (song)Semantic AnalysisNamed Entity Recognition„Armstrong landed the Eagle on the Moon.“N. Steinmetz, H.Sack: Semantic Multimedia Information Retrieval Based on Contextual Descriptions, 2013Entity Selection Process(Semantic) Graph AnalysisFreitag, 21. Juni 13
  33. 33. 4242424224424242424242vfm - Seminar: Metadatenmanagement in Medienunternehmen, 05. September 2012, Bonn Jörg Waitelonis, Hasso-Plattner-Institut Potsdam33Semantically Annotated MultimediaVideo Analysis /Metadata Extractiontimemetadatametadatametadatametadatametadatae.g.,person xylocation yzevent abce.g., bibliographical data,geographical data,encyclopedic data, ..Entity Recognition/MappingN. Ludwig, H. Sack:Named Entity Recognition for User-Generated Tags. In Proc. of the 8th Int.Workshop on Text-based InformationRetrieval, IEEE CS Press, 2011Freitag, 21. Juni 13
  34. 34. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam34Entity Based Search• linguistic ambiguities of traditional keyword basedsearch can be avoided• enables high precision and high recall retrievalhttp://www.yovisto.com/labs/autosuggestion/• Query string refinement / extension• entity auto-suggestion• interpretation of natural language queriesJ. Osterhoff, J. Waitelonis, H. Sack, Widen the Peepholes! Entity-Based Auto-Suggestion as a rich and yet immediate Starting Point for Exploratory Search, IVDW 2012Freitag, 21. Juni 13
  35. 35. Freitag, 21. Juni 13
  36. 36. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam36http://mediaglobe.yovisto.com:8080/mggui-dev2/search facetsC. Hentschel, H. Sack, et al.,Open up cultural heritage in video archives with mediaglobe, I2CS 2012Freitag, 21. Juni 13
  37. 37. Freitag, 21. Juni 13
  38. 38. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam38Explorative Searchdbpedia-owl:missiondbpedia:Neil_Armstrongdbpedia:Apollo_11dbpedia-owl:missioncategory:Apollo_programdcterms:subjectdbpedia:Apollo_13dcterms:subjectyago:Space_accidents_and_incidentsrdf:typerdf:typedbpedia:Space_Shuttle_Challengerdbpedia-owl:missionhttp://mediaglobe.yovisto.com:8080/J. Waitelonis, H. Sack: Towards exploratory video search using linked data, MTAP Volume 59, Number 2 (2012), 645-672dbpedia:Buzz_Aldrindbpedia:Michael_CollinsFreitag, 21. Juni 13
  39. 39. Exploratory Search and Serendipity•Find something that you were not looking for on purpose ...dbpedia:Buzz_Aldrindbpedia:Cookie_Monsterdbpedia:Strictly_Come_Dancingdbpedia:TransformersFreitag, 21. Juni 13
  40. 40. Explorative Search & Intelligent Recommmendationwith yovistohttp://mediaglobe.yovisto.com:8080/Freitag, 21. Juni 13
  41. 41. Explorative Search & Intelligent Recommmendationwith yovistohttp://mediaglobe.yovisto.com:8080/Freitag, 21. Juni 13
  42. 42. Explorative Search & Intelligent Recommmendationwith yovistohttp://mediaglobe.yovisto.com:8080/Freitag, 21. Juni 13
  43. 43. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Contact:Dr. Harald SackHasso-Plattner-Institut für SoftwaresystemtechnikUniversität PotsdamProf.-Dr.-Helmert-Str. 2-3D-14482 PotsdamHomepage: http://www.hpi.uni-potsdam.de/meinel/team/sack.htmlBlog: http://yovisto.blogspot.com/E-Mail: harald.sack@hpi.uni-potsdam.deTwitter: lysander07 / biblionomicon / yovistoSlides can be found at http://slideshare.com/lysander07/Thank you very muchfor your attention!Freitag, 21. Juni 13

×