From Visual to Semantic Analysis

From Visual to Semantic Multimedia Analysis 6th tele-TASK Symposium 2012 HPI, Potsdam, 09.10.2012 Dr. Harald Sack Hasso-Plattner-Institut for IT-Systems Engineering University of PotsdamDienstag, 9. Oktober 12

From Visual to Semantic Multimedia Analysis • what is it all about... • Why Multimedia Retrieval is difﬁcult • Multimedia Analysis Technologies • Semantic Multimedia Analysis • what it can be used for... • Semantic Multimedia Retrieval • Data Exploration Dr. Harald Sack Hasso-Plattner-Institut for IT-Systems Engineering University of PotsdamDienstag, 9. Oktober 12

The Web is big. Really big. You just wont believe how vastly, hugely, mind-bogglingly big it is. (...according to Douglas Adams) Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, 6th tele-TASK Symposium, HPI Potsdam, 09.10.2012Dienstag, 9. Oktober 12

How do I ﬁnd something in the web? Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, 6th tele-TASK Symposium, HPI Potsdam, 09.10.2012Dienstag, 9. Oktober 12

current solution: Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, 6th tele-TASK Symposium, HPI Potsdam, 09.10.2012Dienstag, 9. Oktober 12

Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, 6th tele-TASK Symposium, HPI Potsdam, 09.10.2012Dienstag, 9. Oktober 12

what if you don‘t have text...? Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam 9Dienstag, 9. Oktober 12

today, (most) descriptive metadata is provided manually Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12

Automated Audiovisual Analysis Visual Concept Analysis Classiﬁcation: Face Studio Indoor Detection Person Identiﬁcation overlay News Show Tracking Logo Clustering Detection text scene text Audio-Mining structural Automated audio event analysis Speech detection RecognitionDienstag, 9. Oktober 12

Automated Audiovisual Analysis • Example: Video OCR H. Yang, B. Quehl, H. Sack: A Framework for Improved Video Text Detection and Recognition, Springer Multimedia Tools and Applications, 2012Dienstag, 9. Oktober 12

Video OCR • Video OCR is much more difﬁcult13 than traditional print OCR • heterogeneous/low contrast • bad lighting conditions • skewed and distorted text • compression artefacts • occlusions, partial visibility, etc. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12

Video OCR (1)Character Detection • Robust and fast ﬁlter to extract text candidate frames T T T T T T T T T T • 25 fps results in 90.000 frames per 60 min • too expensive for single frame preprocessing & OCR • fast and robust text identiﬁcation for preprocessing Analytical Character Detection • Edge Based Detection • DCT / Fourier Transformation • Sobel-/Canny Edge Filter • Histogram of Oriented Gradients • Constant Gradient Variance • Texture Based Detection •Local Binary Patterns Frame Frame with •Spatial Variance Candidate Region Based Detection Textboxes • Connected Component Analysis • Stroke Width AnalysisDienstag, 9. Oktober 12

Video OCR (2) Analytical Textbox Filtering • Horizontal & Vertical Projection Profile • Stroke Width Analysis Based Verification Frame with Candidate Frame with Verified Textboxes TextboxesDienstag, 9. Oktober 12

Video OCR Analytical Edge Based Character Detectionflow of the proposed text detection method. (b) is the vertical edge map of (a). (c) is the vertical d binary 1. Workflow ofthe result map text detection method. (b) is the vertical edge (f) shows the(c) Fig. map of (c). (e) the proposed of subsequent connected component analysis. map of (a). bprojection profile refinement. (g) is (e) the result map of subsequent connected component analysis. (b). (d) is the binary map of (c). the final detection result. Dienstag, 9. Oktober 12

Video OCR (3) Character Binarization & Normalization Original Video Frames Textbox Textbox Quality NormalizationEnhancement and BinarizationDienstag, 9. Oktober 12

Video OCR (4) Standard Optical Character Recognition • OCRopus 0.4.4 (Open Source, Apache License v2.0) • Tesseract 3.01 (Open Source, Apache License v2.0) Quality Enhanced Raw OCR Results Normalized Textboxes Ueutsche Bank WeubrandenburgDienstag, 9. Oktober 12

Video OCR (5) OCR Post Processing • OCR-adapted Spell Correction (hunspell 1.3.2, Open Source GNU lGPL) • exploits temporal redundancies for Spell Correction • exploits context for Spell Correction OCR-adapted OCR Results after Frame Raw OCR Results spell correction Spell Correction n Ueutsche Bank n+1 Deutsche Bunk n+2 eutsche Bimk Deutsche Bank n+3 Deutschi Bank n+4 Deutsche BankDienstag, 9. Oktober 12

Automated AudiovisualAnalysis Automated AV Analysis • Result: Multimedia data with spatiotemporal Annotations Metadata Extraction Metadata (e.g. MPEG-7) ... <SpatialDecomposition> <TextAnnotation> <KeywordAnnotation> <Keyword>Astronaut</Keyword> </KeywordAnnotation> </TextAnnotation> <SpatialMask> <SubRegion> <Polygon> <Coords> 480 150 620 480 </Coords> </Polygon> g rmstron </SubRegion> Neil A </SpatialMask> ... </SpatialDecomposition> ...Dienstag, 9. Oktober 12

,Neil Armstrong‘ is more than just a character string Neil Armstrong Entities is a is a Ontologies Astronaut Person is a Science Occupation is a has an Employment Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, Workshop: Interaktion und Visualisierung im Datenweb (IVDW 2012), Braunschweig, 20.09.2012Dienstag, 9. Oktober 12

Entity Mapping into the ,Web of Data‘22 Neil Armstrong is a is a Astronaut Person is a Science Occupation is a has an Employment Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12

23 A little Semantics goes a long way... Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12

Semantic Multimedia Analysis Named Entity Recognition Neil Armstrong Moon Eagle Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, Workshop: Interaktion und Visualisierung im Datenweb (IVDW 2012), Braunschweig, 20.09.2012Dienstag, 9. Oktober 12

Semantic Multimedia Analysis Semantic Analysis Named Entity Recognition - Context Dimensions for Audiovisual Media Named Entity Recognition Spatial Temporal Context Context Provenance User Context Context Context provides information for Structural Context • Disambiguation • Reliability • TrustworthinessDienstag, 9. Oktober 12

Semantic Multimedia Analysis Named Entity Recognition - Create all possible Sets of Mapping Candidates Armstrong Eagle Moon 448 entities 95 entities 156 entities Man on the Moon (film) George Armstrong Custer Eagle (Bird) Moon (song) Neil Armstrong Eagle (heraldry) Moon Son-Ri The Armstrong Twins USCGC Eagle Moon 44 C Moon Armstrong, Florida The Eagle (2011 film) Eagle (comic) The Moon (Tarot card) Craig Armstrong Armstrong, Ontario Man on the Moon (soundtrack) Eagle (song) Moon Armstrong (Moon Crater) Eagle (lunar module) Armstrong Gun The Eagle (newspaper) Man on the Moon (musical) Armstrong‘s Theorem War Eagle Mr. Moon (song) Eagle (Moon Crater) Louis Armstrong International Airport Moon (Band) The Eagle (Pub) Armstrong County, Texass Moon OS Eagle TV Eagle Falls (Washington) Moon 83 Joe Armstrong Lottie Moon Ian Armstrong Eagle (racehorse) Edgar Moon Armstrong Tunnel Armstrong Tunnel Armstrong Automobile John H. Eagle Darvin Moon Sir Thomas Armstrong Eagle (typeface) Gary Moon William Moon Louis Armstrong Angela Eagle Francis Moon Armstrong (British Columbia) Linda Eagle Robert Charles Moon Karen Armstrong Allan Moon Curtis Armstrong James Philipp Eagle Fly me to the Moon (song) Hilary Armstrong Black Moon Ban-Ki Moon Gillian Armstrong William L. ArmstrongDienstag, 9. Oktober 12

Semantic Multimedia Analysis Named Entity Recognition - Strategies for Context-based Entity Mapping • Popularity based Strategies • Linguistical Strategies • Statistical Strategies • Semantic based Strategies General Approach 1. Make an assumption 2. Do the strategies support or contradict your assumption 3. Make decision according to logical and probabilistic rulesDienstag, 9. Oktober 12

Semantic Multimedia Analysis Named Entity Recognition - Semantic Graph Analysis Armstrong Eagle Moon 448 entities 95 entities 156 entities Man on the Moon (film) George Armstrong Custer Eagle (Bird) Moon (song) Neil Armstrong Eagle (heraldry) Moon Son-Ri The Armstrong Twins USCGC Eagle Moon 44 C Moon Armstrong, Florida The Eagle (2011 film) Eagle (comic) The Moon (Tarot card) Craig Armstrong Armstrong, Ontario Moon Man on the Moon (soundtrack) Eagle (song) Armstrong (Moon Crater) Eagle (lunar module) Armstrong Gun The Eagle (newspaper) Man on the Moon (musical) Armstrong‘s Theorem War Eagle Mr. Moon (song) Eagle (Moon Crater) Louis Armstrong International Airport Moon (Band) The Eagle (Pub) Armstrong County, Texass Moon OS Eagle TV Eagle Falls (Washington) Moon 83 Joe Armstrong Lottie Moon Ian Armstrong Eagle (racehorse) Edgar Moon Armstrong Tunnel Armstrong Tunnel Armstrong Automobile John H. Eagle Darvin Moon Sir Thomas Armstrong Eagle (typeface) Gary Moon William Moon Louis Armstrong Angela Eagle Francis Moon Armstrong (British Columbia) Linda Eagle Robert Charles Moon Karen Armstrong Allan Moon Curtis Armstrong James Philipp Eagle Fly me to the Moon (song) Hilary Armstrong Black Moon Ban-Ki Moon Gillian Armstrong William L. ArmstrongDienstag, 9. Oktober 12

Semantic Multimedia Analysis Video Analysis / Metadata Extraction metadata metadata metadata metadata metadata Entity Recognition Entity Mapping e.g., bibliographical data, geographical data, encyclopedic data, ..Dienstag, 9. Oktober 12

30 ...and what can Semantic Metadata be used for? Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12

Entity Based Search31• Query string reﬁnement / extension • linguistic ambiguities of traditional keyword based• entity auto-suggestion search can be avoided• interpretation of natural language queries • enables high precision and high recall search http://www.yovisto.com/labs/autosuggestion/ Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12

Entity Based Search Faceted Search - Search result ﬁltering & navigation with semantic facets32 Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12

Explorative Search dbpedia:Michael_Collins33 dbpedia-owl:mission dbpedia:Apollo_11 dbpedia-owl:mission dcterms:subject dbpedia-owl:mission dbpedia:Neil_Armstrong dbpedia:Buzz_Collins dcterms:subject category:Apollo_program dbpedia:Apollo_13 rdf:type dbpedia:Space_Shuttle_Challenger yago:Space_accidents_and_incidents rdf:type Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12

Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12

General Problem: What is important?35 • Linked Data does not provide any information about relevancy or ranking •Development of Heuristics for Relevance Ranking of Linked Data Facts •Semantic Graph Analysis dbpedia:Neil_Armstrong •Statistics •Popularity Based Ranking J. Waitelonis, H. Sack: Towards exploratory video search using linked data, MTAP vol.59-2, 2012, pp. 645-672. A. Thalhammer, M. Knuth, H. Sack: Evaluating Entity Summarizations Using a Game-Based Ground Truth, (ISWC 2012). Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12

Explorative Search and Serendipity • Find something that you were not looking for on purpose ... dbpedia:Buzz_Collins dbpedia:Cookie_Monster dbpedia:Strictly_Come_DancingDienstag, 9. Oktober 12

http://mediaglobe.yovisto.com:8080/ Waitelonis, Sack: Augmenting Video Search with Linked Open Data,. I-Semantics 2009.Dienstag, 9. Oktober 12

So long, and Thanks for all the ﬁsh. (Douglas Adams) Contact: Harald Sack Hasso-Plattner-Institut für Softwaresystemtechnik Universität Potsdam Prof.-Dr.-Helmert-Str. 2-3 D-14482 Potsdam Homepage: ttp://www.hpi.uni-potsdam.de/meinel/team/sack.html h http://www.yovisto.com/ Blog: http://moresemantic.blogspot.com/ E-Mail: harald.sack@hpi.uni-potsdam.de Twitter: lysander07 / biblionomicon / yovisto Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, 6th tele-TASK Symposium, HPI Potsdam, 09.10.2012Dienstag, 9. Oktober 12

http://www.scoop.it	124
https://twitter.com	12
http://yovisto.blogspot.de	9
http://kred.com	6

From Visual to Semantic Analysis

by Harald Sack on Oct 09, 2012

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 151

Statistics

From Visual to Semantic Analysis Presentation Transcript