• Save
From Visual to Semantic Analysis
Upcoming SlideShare
Loading in...5
×
 

From Visual to Semantic Analysis

on

  • 2,519 views

My talk from 6th tele-TASK Symposium, 9.10.2012, at HPI, Potsdam

My talk from 6th tele-TASK Symposium, 9.10.2012, at HPI, Potsdam

Statistics

Views

Total Views
2,519
Views on SlideShare
2,350
Embed Views
169

Actions

Likes
10
Downloads
0
Comments
0

8 Embeds 169

http://www.scoop.it 127
http://yovisto.blogspot.de 16
https://twitter.com 13
http://kred.com 6
http://yovisto.blogspot.com 3
https://mail.google.com 2
http://yovisto.blogspot.in 1
http://localhost 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

From Visual to Semantic Analysis From Visual to Semantic Analysis Presentation Transcript

  • From Visual to Semantic Multimedia Analysis 6th tele-TASK Symposium 2012 HPI, Potsdam, 09.10.2012 Dr. Harald Sack Hasso-Plattner-Institut for IT-Systems Engineering University of PotsdamDienstag, 9. Oktober 12
  • From Visual to Semantic Multimedia Analysis • what is it all about... • Why Multimedia Retrieval is difficult • Multimedia Analysis Technologies • Semantic Multimedia Analysis • what it can be used for... • Semantic Multimedia Retrieval • Data Exploration Dr. Harald Sack Hasso-Plattner-Institut for IT-Systems Engineering University of PotsdamDienstag, 9. Oktober 12
  • The Web is big. Really big. You just wont believe how vastly, hugely, mind-bogglingly big it is. (...according to Douglas Adams) Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, 6th tele-TASK Symposium, HPI Potsdam, 09.10.2012Dienstag, 9. Oktober 12
  • How do I find something in the web? Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, 6th tele-TASK Symposium, HPI Potsdam, 09.10.2012Dienstag, 9. Oktober 12
  • current solution: Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, 6th tele-TASK Symposium, HPI Potsdam, 09.10.2012Dienstag, 9. Oktober 12
  • Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, 6th tele-TASK Symposium, HPI Potsdam, 09.10.2012Dienstag, 9. Oktober 12
  • Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, 6th tele-TASK Symposium, HPI Potsdam, 09.10.2012Dienstag, 9. Oktober 12
  • Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, 6th tele-TASK Symposium, HPI Potsdam, 09.10.2012Dienstag, 9. Oktober 12
  • what if you don‘t have text...? Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam 9Dienstag, 9. Oktober 12
  • today, (most) descriptive metadata is provided manually Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12
  • Automated Audiovisual Analysis Visual Concept Analysis Classification: Face Studio Indoor Detection Person Identification overlay News Show Tracking Logo Clustering Detection text scene text Audio-Mining structural Automated audio event analysis Speech detection RecognitionDienstag, 9. Oktober 12
  • Automated Audiovisual Analysis • Example: Video OCR H. Yang, B. Quehl, H. Sack: A Framework for Improved Video Text Detection and Recognition, Springer Multimedia Tools and Applications, 2012Dienstag, 9. Oktober 12
  • Video OCR • Video OCR is much more difficult13 than traditional print OCR • heterogeneous/low contrast • bad lighting conditions • skewed and distorted text • compression artefacts • occlusions, partial visibility, etc. Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12
  • Video OCR (1)Character Detection • Robust and fast filter to extract text candidate frames T T T T T T T T T T • 25 fps results in 90.000 frames per 60 min • too expensive for single frame preprocessing & OCR • fast and robust text identification for preprocessing Analytical Character Detection • Edge Based Detection • DCT / Fourier Transformation • Sobel-/Canny Edge Filter • Histogram of Oriented Gradients • Constant Gradient Variance • Texture Based Detection •Local Binary Patterns Frame Frame with •Spatial Variance Candidate Region Based Detection Textboxes • Connected Component Analysis • Stroke Width AnalysisDienstag, 9. Oktober 12
  • Video OCR (2) Analytical Textbox Filtering • Horizontal & Vertical Projection Profile • Stroke Width Analysis Based Verification Frame with Candidate Frame with Verified Textboxes TextboxesDienstag, 9. Oktober 12
  • Video OCR Analytical Edge Based Character Detectionflow of the proposed text detection method. (b) is the vertical edge map of (a). (c) is the vertical d binary 1. Workflow ofthe result map text detection method. (b) is the vertical edge (f) shows the(c) Fig. map of (c). (e) the proposed of subsequent connected component analysis. map of (a). bprojection profile refinement. (g) is (e) the result map of subsequent connected component analysis. (b). (d) is the binary map of (c). the final detection result. Dienstag, 9. Oktober 12
  • Video OCR (3) Character Binarization & Normalization Original Video Frames Textbox Textbox Quality NormalizationEnhancement and BinarizationDienstag, 9. Oktober 12
  • Video OCR (4) Standard Optical Character Recognition • OCRopus 0.4.4 (Open Source, Apache License v2.0) • Tesseract 3.01 (Open Source, Apache License v2.0) Quality Enhanced Raw OCR Results Normalized Textboxes Ueutsche Bank WeubrandenburgDienstag, 9. Oktober 12
  • Video OCR (5) OCR Post Processing • OCR-adapted Spell Correction (hunspell 1.3.2, Open Source GNU lGPL) • exploits temporal redundancies for Spell Correction • exploits context for Spell Correction OCR-adapted OCR Results after Frame Raw OCR Results spell correction Spell Correction n Ueutsche Bank n+1 Deutsche Bunk n+2 eutsche Bimk Deutsche Bank n+3 Deutschi Bank n+4 Deutsche BankDienstag, 9. Oktober 12
  • Automated AudiovisualAnalysis Automated AV Analysis • Result: Multimedia data with spatiotemporal Annotations Metadata Extraction Metadata (e.g. MPEG-7) ... <SpatialDecomposition> <TextAnnotation> <KeywordAnnotation> <Keyword>Astronaut</Keyword> </KeywordAnnotation> </TextAnnotation> <SpatialMask> <SubRegion> <Polygon> <Coords> 480 150 620 480 </Coords> </Polygon> g rmstron </SubRegion> Neil A </SpatialMask> ... </SpatialDecomposition> ...Dienstag, 9. Oktober 12
  • ,Neil Armstrong‘ is more than just a character string Neil Armstrong Entities is a is a Ontologies Astronaut Person is a Science Occupation is a has an Employment Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, Workshop: Interaktion und Visualisierung im Datenweb (IVDW 2012), Braunschweig, 20.09.2012Dienstag, 9. Oktober 12
  • Entity Mapping into the ,Web of Data‘22 Neil Armstrong is a is a Astronaut Person is a Science Occupation is a has an Employment Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12
  • 23 A little Semantics goes a long way... Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12
  • Semantic Multimedia Analysis Named Entity Recognition Neil Armstrong Moon Eagle Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, Workshop: Interaktion und Visualisierung im Datenweb (IVDW 2012), Braunschweig, 20.09.2012Dienstag, 9. Oktober 12
  • Semantic Multimedia Analysis Semantic Analysis Named Entity Recognition - Context Dimensions for Audiovisual Media Named Entity Recognition Spatial Temporal Context Context Provenance User Context Context Context provides information for Structural Context • Disambiguation • Reliability • TrustworthinessDienstag, 9. Oktober 12
  • Semantic Multimedia Analysis Named Entity Recognition - Create all possible Sets of Mapping Candidates Armstrong Eagle Moon 448 entities 95 entities 156 entities Man on the Moon (film) George Armstrong Custer Eagle (Bird) Moon (song) Neil Armstrong Eagle (heraldry) Moon Son-Ri The Armstrong Twins USCGC Eagle Moon 44 C Moon Armstrong, Florida The Eagle (2011 film) Eagle (comic) The Moon (Tarot card) Craig Armstrong Armstrong, Ontario Man on the Moon (soundtrack) Eagle (song) Moon Armstrong (Moon Crater) Eagle (lunar module) Armstrong Gun The Eagle (newspaper) Man on the Moon (musical) Armstrong‘s Theorem War Eagle Mr. Moon (song) Eagle (Moon Crater) Louis Armstrong International Airport Moon (Band) The Eagle (Pub) Armstrong County, Texass Moon OS Eagle TV Eagle Falls (Washington) Moon 83 Joe Armstrong Lottie Moon Ian Armstrong Eagle (racehorse) Edgar Moon Armstrong Tunnel Armstrong Tunnel Armstrong Automobile John H. Eagle Darvin Moon Sir Thomas Armstrong Eagle (typeface) Gary Moon William Moon Louis Armstrong Angela Eagle Francis Moon Armstrong (British Columbia) Linda Eagle Robert Charles Moon Karen Armstrong Allan Moon Curtis Armstrong James Philipp Eagle Fly me to the Moon (song) Hilary Armstrong Black Moon Ban-Ki Moon Gillian Armstrong William L. ArmstrongDienstag, 9. Oktober 12
  • Semantic Multimedia Analysis Named Entity Recognition - Strategies for Context-based Entity Mapping • Popularity based Strategies • Linguistical Strategies • Statistical Strategies • Semantic based Strategies General Approach 1. Make an assumption 2. Do the strategies support or contradict your assumption 3. Make decision according to logical and probabilistic rulesDienstag, 9. Oktober 12
  • Semantic Multimedia Analysis Named Entity Recognition - Semantic Graph Analysis Armstrong Eagle Moon 448 entities 95 entities 156 entities Man on the Moon (film) George Armstrong Custer Eagle (Bird) Moon (song) Neil Armstrong Eagle (heraldry) Moon Son-Ri The Armstrong Twins USCGC Eagle Moon 44 C Moon Armstrong, Florida The Eagle (2011 film) Eagle (comic) The Moon (Tarot card) Craig Armstrong Armstrong, Ontario Moon Man on the Moon (soundtrack) Eagle (song) Armstrong (Moon Crater) Eagle (lunar module) Armstrong Gun The Eagle (newspaper) Man on the Moon (musical) Armstrong‘s Theorem War Eagle Mr. Moon (song) Eagle (Moon Crater) Louis Armstrong International Airport Moon (Band) The Eagle (Pub) Armstrong County, Texass Moon OS Eagle TV Eagle Falls (Washington) Moon 83 Joe Armstrong Lottie Moon Ian Armstrong Eagle (racehorse) Edgar Moon Armstrong Tunnel Armstrong Tunnel Armstrong Automobile John H. Eagle Darvin Moon Sir Thomas Armstrong Eagle (typeface) Gary Moon William Moon Louis Armstrong Angela Eagle Francis Moon Armstrong (British Columbia) Linda Eagle Robert Charles Moon Karen Armstrong Allan Moon Curtis Armstrong James Philipp Eagle Fly me to the Moon (song) Hilary Armstrong Black Moon Ban-Ki Moon Gillian Armstrong William L. ArmstrongDienstag, 9. Oktober 12
  • Semantic Multimedia Analysis Video Analysis / Metadata Extraction metadata metadata metadata metadata metadata Entity Recognition Entity Mapping e.g., bibliographical data, geographical data, encyclopedic data, ..Dienstag, 9. Oktober 12
  • 30 ...and what can Semantic Metadata be used for? Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12
  • Entity Based Search31• Query string refinement / extension • linguistic ambiguities of traditional keyword based• entity auto-suggestion search can be avoided• interpretation of natural language queries • enables high precision and high recall search http://www.yovisto.com/labs/autosuggestion/ Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12
  • Entity Based Search Faceted Search - Search result filtering & navigation with semantic facets32 Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12
  • Explorative Search dbpedia:Michael_Collins33 dbpedia-owl:mission dbpedia:Apollo_11 dbpedia-owl:mission dcterms:subject dbpedia-owl:mission dbpedia:Neil_Armstrong dbpedia:Buzz_Collins dcterms:subject category:Apollo_program dbpedia:Apollo_13 rdf:type dbpedia:Space_Shuttle_Challenger yago:Space_accidents_and_incidents rdf:type Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12
  • Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12
  • General Problem: What is important?35 • Linked Data does not provide any information about relevancy or ranking •Development of Heuristics for Relevance Ranking of Linked Data Facts •Semantic Graph Analysis dbpedia:Neil_Armstrong •Statistics •Popularity Based Ranking J. Waitelonis, H. Sack: Towards exploratory video search using linked data, MTAP vol.59-2, 2012, pp. 645-672. A. Thalhammer, M. Knuth, H. Sack: Evaluating Entity Summarizations Using a Game-Based Ground Truth, (ISWC 2012). Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität PotsdamDienstag, 9. Oktober 12
  • Explorative Search and Serendipity • Find something that you were not looking for on purpose ... dbpedia:Buzz_Collins dbpedia:Cookie_Monster dbpedia:Strictly_Come_DancingDienstag, 9. Oktober 12
  • http://mediaglobe.yovisto.com:8080/ Waitelonis, Sack: Augmenting Video Search with Linked Open Data,. I-Semantics 2009.Dienstag, 9. Oktober 12
  • So long, and Thanks for all the fish. (Douglas Adams) Contact: Harald Sack Hasso-Plattner-Institut für Softwaresystemtechnik Universität Potsdam Prof.-Dr.-Helmert-Str. 2-3 D-14482 Potsdam Homepage: ttp://www.hpi.uni-potsdam.de/meinel/team/sack.html h http://www.yovisto.com/ Blog: http://moresemantic.blogspot.com/ E-Mail: harald.sack@hpi.uni-potsdam.de Twitter: lysander07 / biblionomicon / yovisto Dr. Harald Sack, Hasso-Plattner-Institut Potsdam, 6th tele-TASK Symposium, HPI Potsdam, 09.10.2012Dienstag, 9. Oktober 12