SEMEX: Enabling Exploratory Video Search by Semantic Video Analysis

Enabling Exploratory Video Search by Semantic Video Analysis LWA 2011 Magdeburg, 30. Sep. 2011 Dr. Harald Sack Hasso-Plattner-Institut for IT-Systems Engineering University of PotsdamFreitag, 30. September 11

■ HPI was founded in October 1998 as a Public- Private-Partnership ■ HPI Research and Teaching is focussed on IT Systems Engineering ■ 10 Professors and 100 Scientiﬁc Coworkers ■ 450 Bachelor / Master Students ■ HPI is winner of CHE-Ranking 2010Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 http://hpi.uni-potsdam.de/Freitag, 30. September 11

Semantic Technologies & Multimedia Retrieval ■ Research Topics □ Semantic Web Technologies □ Ontological Engineering □ Information Retrieval □ Multimedia Analysis & Retrieval □ Social Networking □ Data/Information Visualization ■ Research ProjectsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

SEMEX - Enabling Exploratory Video Search by Semantic Video Analysis LDW 2011, Magdeburg, 30. Sep 2011 Overview (1) Searching Audiovisual Data (2) Semantic Multimedia Analysis (3) Explorative Semantic Search (4) SeMEX - Semantic Multimedia ExplorerHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

The Google Challenge...Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Google Multimedia SearchHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11

How does Google find Multimedia?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11

How does Google find Multimedia? ... <a href="/mission_pages/shuttle/shuttlemissions/sts134/ multimedia/index.html"> <IMG WIDTH="100" ALT="Close-up view of Endeavours crew cabin prior to docking with the International Space Station" TITLE="Close-up view of Endeavours crew cabin prior to docking with the International Space Station" SRC="/images/ content/549665main_2011-05-18_1600_100-75.jpg" HEIGHT="75" ALIGN="Bottom" BORDER="0" /> </a> <p><a href="/mission_pages/shuttle/shuttlemissions/sts134/ multimedia/index.html">&rsaquo; STS-134 Multimedia</a></ p> ... ‣Google Multimedia Search relies on link contextHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11

How to Search in Multimedia Archives?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

How to Search in Multimedia Archives? Step 1: Digitalization of analog data Step 2: Annotation with (textbased) metadataHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

How to Search in Multimedia Archives? • manual anotation with text-based descriptive metadataHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

How to Search in Multimedia Archives? • manual anotation with text-based descriptive metadata ...how to extract metadata in an automated way?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Automated Audiovisual AnalysisHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Automated Audiovisual Analysis Face DetectionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Automated Audiovisual Analysis Genre Analysis Classiﬁcation: Studio Indoor News Show Face DetectionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Automated Audiovisual Analysis Genre Analysis Classiﬁcation: Studio Indoor overlay News Show Face text DetectionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Automated Audiovisual Analysis Genre Analysis Classiﬁcation: Studio Indoor overlay News Show Face text Detection scene textHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Automated Audiovisual Analysis Genre Analysis Classiﬁcation: Studio Indoor overlay News Show Face Logo text Detection Detection scene textHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Automated Audiovisual Analysis Genre Analysis Classiﬁcation: Studio Indoor overlay News Show Face Logo text Detection Detection scene text Audio-Mining structural Automated speaker analysis Speech identiﬁcation RecognitionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Automated Audiovisual Analysis • Visual Analysis • Audio Analysis • Structural Analysis • Structural Analysis • Intelligent Character • Speaker Detection Recognition (ICR) • Automated Speech • Character/Logo Recognition (ASR) Detection • Character Filtering • Character Recognition • Genre Analysis & Categorization • Face / Body / Object •Detection •Tracking •ClusteringHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Structural Analysis • Decomposition of time-based media into meaningful media fragments of coherent content that can be used as basic element for indexing and classiﬁcation videoHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Structural Analysis • Decomposition of time-based media into meaningful media fragments of coherent content that can be used as basic element for indexing and classiﬁcation video scenesHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Structural Analysis • Decomposition of time-based media into meaningful media fragments of coherent content that can be used as basic element for indexing and classiﬁcation video scenes shotsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Structural Analysis • Decomposition of time-based media into meaningful media fragments of coherent content that can be used as basic element for indexing and classiﬁcation video scenes shots subshotsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Structural Analysis • Decomposition of time-based media into meaningful media fragments of coherent content that can be used as basic element for indexing and classiﬁcation video scenes shots subshots frames key framesHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Structural Analysis • Shot Boundary Detection time • Automated Identiﬁcation of • Hard Cuts • Defects, as e.g., • Drop Outs, White Outs, etc. • Soft Cuts, as e.g., • Fade-In/Out, • Dissolve, Wipe, Cross-Fade, etc. • Automated Structural Analysis based on • Analytical Shot Boundary Detection • Machine Learning Based Shot DetectionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Structural Analysis • Shot Boundary Detection • Automated Identiﬁcation of Hard Cuts based on • Luminance/Chrominance Histogram Differences & Derivatives • Edge Distribution/Density 573 574 575 576 577 578Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Structural Analysis Adaptive Threshold 1 2 20 1 3 i+W 1 X tha (i) = ↵ · 4@ Da (k, k 1)A Da (i, i 1)5 + k=i W 20 1 i+W 1 X 3 4 4@ 1)A Da(i,i-1) ...D thai = ↵ · th (L2-norm) Da (k, k Frames i (i) Histogram Difference (i) between a (i, 1) > ↵ 20 1 and i-1 of Subregion a k=i W 3 i+W 1 X tha (i) = ↵ · 4@tha(i) ... Da (k, k Threshold for Frameiai (i,1)5 + >a th↵ (i) adaptive 1)A Da (i, of Subregion Decompose Frame into a=4 Subregions Da (i + 1, i) < th↵ (i) D i 1) k=i W Hardcut: if Da (i, i 1) > th↵ (i) and Da (i + 1, i) < th↵ (i) is true for all Subregions a Da (i + 1, i) < th↵ (i) Window Size=4 (W=2) i-3 i-2 i-1 i i+1 i+2Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Structural Analysis • Shot Boundary Detection • Automated Identiﬁcation of Defects, as e.g. Drop Outs / White Outs Histogram/Chrominance Difference Analysis Drop Out Flashlight / White Out Histogram/Chrominance Difference AnalysisHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Structural Analysis • Shot Boundary Detection • Automated Identiﬁcation of Defects, as e.g., Drop Outs / White Outs ... i i+1 i+8 i+9 i+10 i+11 i+12 i+13 • Luminance/Chrominance Histogram Differences & DerivativesHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Structural Analysis • Shot Boundary Detection • Automated Identiﬁcation of Soft Cuts, as e.g. Fade Out / Fade In Fade Out Fade InHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Structural Analysis • Shot Boundary Detection • Automated Identiﬁcation of Soft Cuts, , as e.g. Fade Out / Fade In • Features applied for machine learning: • luminance histogram (Fade In / Fade Out) • luminance average Yµ and luminance variance Yσ2 follow distinct patterns 1 2 3 • image decomposition • component-based analysis to distinguish regional and global 4 5 6 changes in image content • entropy 7 8 9 • motion vectorsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Structural Analysis • Shot Boundary Detection • Automated Identiﬁcation of Soft Cuts, , as e.g. Fade Out / Fade In • Features deployed for machine learning: • luminance/chrominance histogram • entropy • motion vectorsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Structural Analysis • Shot Boundary Detection • Automated Identiﬁcation of Soft Cuts, , as e.g. Fade Out / Fade In • Features deployed for machine learning: • luminance/chrominance histogram • entropy 1 2 • motion vectors • image decomposition • compute average motion vectors for all areas • identify camera movements 3 4 (zoom, pan, etc.) and moving objectsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Automated Audiovisual Analysis • Visual Analysis • Audio Analysis • Structural Analysis • Structural Analysis • Intelligent Character • Speaker Detection Recognition (ICR) • Automated Speech • Character/Logo Recognition (ASR) Detection • Character Filtering • Character Recognition • Genre Analysis & Categorization • Face / Body / Object •Detection •Tracking •ClusteringHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Intelligent Character Recognition • Preprocessing • Character Identiﬁcation • Text Preprocessing • Text Filtering • Adaption of script geometry (Deskew) • Image quality enhancement • Optical Character Recognition (OCR) • Standard OCR software (OCRopus) • Postprocessing • Lexical analysis • Statistical / context based ﬁltering Ermittlungen nach BombenfundenHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Intelligent Character Recognition • Character Identification • Robust filter to extract text candidate frames T T T T T T T T T T • 25 fps results in 90.000 frames per 60 min • too expensive for single frame preprocessing & OCR • fast and robust text identification for preprocessing • Features used for text identification: • edge detection • DCT / Fourier Transformation • Sobel-/Canny Edge Filter • horizontal and vertical edge distribution • Local Binary Patterns (LBP) • Histogram of Oriented Gradients • stroke width analysisHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Intelligent Character Recognition • Stroke Width Transformation • based on edge ﬁltering as a preprocessing step • for each edge pixel a stroke is projected along its gradient direction until another edge pixel is hit • all pixels along the stroke will receive the same stroke width value (color)Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Intelligent Character Recognition • Stroke Width Transformation • based on edge ﬁltering as a preprocessing step • for each edge pixel a stroke is projected along its gradient direction until another edge pixel is hit • all pixels along the stroke will receive the same stroke width value (color) • connected component analysis groups pixels with similar stroke width valueHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Intelligent Character Recognition • Preprocessing • Text Preprocessing • Text Filtering Original Image Bounding BoxHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Intelligent Character Recognition • Preprocessing • Text Preprocessing • Quality Enhancement Advanced Image EnhancementHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Intelligent Character Recognition • Optical Character Recognition (OCR) • Standard OCR software (OCRopus) Standard OCR (OCRopus)Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Intelligent Character Recognition • Postprocessing • Lexical analysis • Statistical / context based ﬁltering Context-based Spell CorrectionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Automated Audiovisual Analysis • Result: Multimedia data with spatiotemporal AnnotationsMetadata Extraction time Metadata (e.g. MPEG-7) ... <Video> <TemporalDecomposition> <VideoSegment> <TextAnnotation> <KeywordAnnotation> <Keyword>Astronaut</Keyword> </KeywordAnnotation> </TextAnnotation> <MediaTime> <MediaTimePoint> T00:05:05:0F25 </MediaTimePoint> <MediaDuration> PT00H00M31S0N25F </MediaDuration> </MediaTime> ... </VideoSegment> </TemporalDecomposition> </Video> ...Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Automated Audiovisual Analysis • Result: Multimedia data with spatiotemporal AnnotationsMetadata Extraction Metadata (e.g. MPEG-7) ... <SpatialDecomposition> <TextAnnotation> <KeywordAnnotation> <Keyword>Astronaut</Keyword> </KeywordAnnotation> </TextAnnotation> <SpatialMask> <SubRegion> <Polygon> <Coords> 480 150 620 480 </Coords> </Polygon> </SubRegion> </SpatialMask> ... </SpatialDecomposition> ...Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11

... <SpatialDecomposition> <TextAnnotation> <KeywordAnnotation> <Keyword>Astronaut</Keyword> </KeywordAnnotation> </TextAnnotation> <SpatialMask> <SubRegion> <Polygon> <Coords> 480 150 620 480 </Coords> </Polygon> </SubRegion> </SpatialMask> ... </SpatialDecomposition> ... But wha t about semantic metadata .. ?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11

Multimedia Ontologies • MPEG-7 has been re-engineered to become an OWL-DL ontology (2007: Arndt et al., COMM model) • Localize a region → Draw a bounding box • Annotate the content → Interpret the content → Tag ,Astronaut‘Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11

Multimedia Ontologies Example: Tagging with an MPEG-7 Ontology Reg1 mpeg7:StillRegion rdf:type decom position Reg1 mpeg7 :spatial_ mpeg7:image mpeg7:SpatialMask mpeg7:depicts mpeg7:depicts mpeg7:polygon dbpedia:Astronaut mpeg7:Coords Man on the MoonHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11

Named Entity Recognition Neil Armstrong Entities is a is a Classes Astronaut Person Named Entity Recognition „locating and classifying atomic elements...into is a predeﬁned categories such as names, persons, organizations, locations, expressions of time, quantities, monetary values, etc.“ Science Occupation C.J.Rijsbergen, Information Retrieval (1979) is a EmploymentHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11

Named Entity Recognition Neil Armstrong is a is a Astronaut Person is a Science Occupation is a EmploymentHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11

Semantic Multimedia Analysis Video Analysis / time Metadata Extraction metadata metadata metadata metadata metadataHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Semantic Multimedia Analysis Video Analysis / time Metadata Extraction metadata metadata metadata metadata Entity Recognition/ metadata Mapping e.g., person xy location yz event abc e.g., bibliographical data, geographical data, encyclopedic data, ..Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Semantic Multimedia Analysis Named Entity Recognition • Mapping keyterms (text) to semantic entities • Context Analysis and DisambiguationHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Semantic Multimedia Analysis Named Entity Recognition • Mapping keyterms (text) to semantic entities • Context Analysis and Disambiguation Jaguar Keyterm / User TagHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Semantic Multimedia Analysis Named Entity Recognition • Mapping keyterms (text) to semantic entities • Context Analysis and Disambiguation Semantic Entities Jaguar (Car) ? Jaguar (Cat) ? Jaguar (OS) ? Jaguar Keyterm / User Tag Jaguar (Aircraft) ?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Semantic Multimedia Analysis Context Analysis and Disambiguation What deﬁnes a Context in AV-Data? • Temporal Coherence • Spatial Coherenceurring • Provenanceuationed by order static novels and Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 Freitag, 30. September 11 Figure 1. Dimensions of context deﬁnition in audio-visual

Semantic Multimedia Analysis Context Analysis and Disambiguation What deﬁnes a Context in AV-Data? • Temporal Coherence • Spatial Coherenceurring • Provenanceuation Spatialed by Dimension order static novels and Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 Freitag, 30. September 11 Figure 1. Dimensions of context deﬁnition in audio-visual

Semantic Multimedia Analysis Context Analysis and Disambiguation What deﬁnes a Context in AV-Data? • Temporal Coherence • Spatial Coherenceurring • Provenanceuation Spatialed by Dimension order static novels and Temporal Dimension Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 Freitag, 30. September 11 Figure 1. Dimensions of context deﬁnition in audio-visual

Semantic Multimedia Analysis Context Analysis and Disambiguation What deﬁnes a Context in AV-Data? • Temporal Coherence • Spatial Coherenceurring • Provenanceuation Spatialed by Dimension User-centered orderDimension static novels and Temporal Dimension Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 Freitag, 30. September 11 Figure 1. Dimensions of context deﬁnition in audio-visual

Semantic Graph Analysis Jaguar (Car) Steve McQueen Keyterm / User Tag 1956 Jaguar (OS) jaguar Jaguar (Cat) 1956 Steve jaguar rim wheel McQueen context LOD CloudHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Searching is not always just searchingHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

a simple example: I‘m looking for a book by Earnest Hemingway with the title ,For Whom the Bell Tolls‘ in the first German edition...“Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

a simple example: I‘m looking for a book by Earnest Hemingway with the title ,For Whom the Bell Tolls‘ in the first German edition...“ Wem die Ernest H Stunde schlägt. (Stockho E M I N G W A - Fischer lm usw., Berman Y. S. 8“ Verlag, n- 1941) 56 0 II 1, 25 06, 3454 8Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

...but what if... I really liked the book ,For Whom the Bell Tolls‘ but I have no idea what I should read next...Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Exploratory Search • What, if the user does not know, which query string to use? • What, if the user is looking for complex answers ? • What, if the user does not know the domain he/she is looking for? • What, if the user wants to know all(!) about a speciﬁc topic? • ...,Browsing‘ instead of ,Searching‘ • ...to find something by chance -> Serendipity • ...to get an overview • ...enable content based navigationHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011Freitag, 30. September 11

Exploratory Multimedia Search Video Analysis / time Metadata Extraction metadata metadata metadata metadata Entity Recognition/ metadata Mapping e.g., person xy location yz event abc e.g., bibliographical data, geographical data, encyclopedic data, ..Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

The Web of Data - The Semantic Web Data is a precious thing and will last longer than theHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 Berners-Lee) systems themselves. (Tim http://linkeddata.org/Freitag, 30. September 11

DBPedia - the Semantic Wikipedia dbpedia:For_Whom_the_Bell_Tolls What facts for http://dbpedia.org/page/ For_Whom_the_Bell_Tolls dbpedia:For_Whom_the_Bell_Tolls are relevant? ...use heuristicsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Exploratory Multimedia Search dbpedia-owl:author dbpedia:Ernest_Hemingway dbpedia:For_Whom_the_Bell_TollsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Exploratory Multimedia Search or uth :a wl -o dia pe db dbpedia-owl:author dbpedia:Ernest_Hemingway dbpedia:For_Whom_the_Bell_TollsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Exploratory Multimedia Search or uth :a wl -o r tho dia u l :a ow pe d ia- db pe db dbpedia-owl:author dbpedia:Ernest_Hemingway dbpedia:For_Whom_the_Bell_TollsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Exploratory Multimedia Search or uth :a wl -o r tho dia u l :a ow pe d ia- db pe db dbpedia-owl:author dbpedia-owl:author dbpedia:Ernest_Hemingway dbpedia:For_Whom_the_Bell_TollsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Exploratory Multimedia Search dbpedia-owl:author dbpedia:For_Whom_the_Bell_Tolls dbpedia:Ernest_HemingwayHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Exploratory Multimedia Search y dbpedia:Raymond_Carver - d _b edia ence bp influ d l: ow dbpedia-owl:author dbpedia:For_Whom_the_Bell_Tolls dbpedia:Ernest_HemingwayHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Exploratory Multimedia Search dbpedia:Jack_Kerouac by d_ ce en l:in a- flu ow pedi db _b y dbpedia:Raymond_Carver - d edia ence bp influ d l: ow dbpedia-owl:author dbpedia:For_Whom_the_Bell_Tolls dbpedia:Ernest_HemingwayHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Exploratory Multimedia Search dbpedia:Jack_Kerouac by d_ ce en l:in a- flu ow pedi db _b y dbpedia:Raymond_Carver - d edia ence bp influ d l: ow dbpedia-owl:author dbpedia-owl:influenced_by dbpedia:For_Whom_the_Bell_Tolls dbpedia:Ernest_Hemingway dbpedia:Jerome_D._SalingerHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Exploratory Multimedia Search dbpedia:Jerome_D._Salinger dbpedia:Jack_Kerouac dbpedia:Raymond_CarverHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Exploratory Multimedia Search dbpedia:Jerome_D._Salinger dbpedia:Jack_Kerouac dbpedia:Raymond_Carver dbpedia-owl:notableWorkHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Exploratory Multimedia Search dbpedia:Jerome_D._Salinger dbpedia:Jack_Kerouac dbpedia:Raymond_Carver dbpedia-owl:notableWork dbpedia-owl:notableWorkHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Exploratory Multimedia Search dbpedia:Jerome_D._Salinger dbpedia:Jack_Kerouac dbpedia:Raymond_Carver dbpedia-owl:notableWork dbpedia- dbpedia-owl:notableWork owl:notableWorkHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

http://bit.ly/SeMEXHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

http://mediaglobe.yovisto.com:808029Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

Contact: Dr. Harald Sack Hasso-Plattner-Institut für Softwaresystemtechnik Universität Potsdam Prof.-Dr.-Helmert-Str. 2-3 D-14482 Potsdam Homepage: ttp://www.hpi.uni-potsdam.de/meinel/team/sack.html h http://www.yovisto.com/ Blog: http://moresemantic.blogspot.com/ E-Mail: harald.sack@hpi.uni-potsdam.de joerg.waitelonis@hpi.uni-potsdam.de ch Twitter: lysander07 / biblionomicon / yovisto ve ry mu k y ou T han tio n! ur at ten f or yoHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

http://www.scoop.it	584
http://paper.li	16
http://translate.googleusercontent.com	9
http://linkeddata.uriburner.com	4
http://a0.twimg.com	2
http://tweetedtimes.com	2

SEMEX: Enabling Exploratory Video Search by Semantic Video Analysis

by Harald Sack

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 617

Statistics

SEMEX: Enabling Exploratory Video Search by Semantic Video Analysis Presentation Transcript