Enabling Exploratory Video                            Search by Semantic Video                                     Analysi...
■ HPI was founded in October 1998 as a Public-            Private-Partnership          ■ HPI Research and Teaching is focu...
Semantic Technologies &       Multimedia Retrieval               ■ Research Topics                     □ Semantic Web Tech...
Semantic Technologies &       Multimedia Retrieval               ■ Research Topics                     □ Semantic Web Tech...
SEMEX -                                                         Enabling Exploratory Video Search                         ...
The Google Challenge...Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 201...
Google Multimedia SearchHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web...
How does Google find Multimedia?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30....
How does Google find Multimedia?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30....
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Be...
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Be...
How does Google find Multimedia?      ...      <a href="/mission_pages/shuttle/shuttlemissions/sts134/      multimedia/ind...
How to Search in                                Multimedia Archives?Harald Sack, Hasso-Plattner-Institute for IT-Systems E...
How to Search in Multimedia Archives?                                         Step 1: Digitalization of analog data       ...
How to Search in Multimedia Archives? • manual anotation with text-based     descriptive metadataHarald Sack, Hasso-Plattn...
How to Search in Multimedia Archives? • manual anotation with text-based     descriptive metadata   ...how to extract meta...
SEMEX -                                                         Enabling Exploratory Video Search                         ...
Automated Audiovisual AnalysisHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. S...
Automated Audiovisual AnalysisHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. S...
Automated Audiovisual AnalysisHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. S...
Automated Audiovisual Analysis                                                                                            ...
Automated Audiovisual Analysis                                                                                            ...
Automated Audiovisual Analysis                                                                                            ...
Automated Audiovisual Analysis                                                                                            ...
Automated Audiovisual Analysis                                                                                            ...
Automated Audiovisual Analysis                                                                                            ...
Automated Audiovisual Analysis                                            • Visual Analysis                               ...
Automated Audiovisual Analysis                                            • Visual Analysis                               ...
Structural Analysis                      • Decomposition of time-based media into meaningful media                        ...
Structural Analysis                      • Decomposition of time-based media into meaningful media                        ...
Structural Analysis                      • Decomposition of time-based media into meaningful media                        ...
Structural Analysis                      • Decomposition of time-based media into meaningful media                        ...
Structural Analysis                      • Decomposition of time-based media into meaningful media                        ...
Structural Analysis        • Shot Boundary Detection                                                                      ...
Structural Analysis      • Shot Boundary Detection          • Automated Identification of              Hard Cuts based on  ...
Structural Analysis                                                                      Adaptive Threshold               ...
Structural Analysis      • Shot Boundary Detection          • Automated Identification of Defects, as e.g. Drop Outs / Whit...
Structural Analysis      • Shot Boundary Detection          • Automated Identification of Defects, as e.g., Drop Outs / Whi...
Structural Analysis      • Shot Boundary Detection          • Automated Identification of Soft Cuts, as e.g. Fade Out / Fad...
Structural Analysis      • Shot Boundary Detection          • Automated Identification of Soft Cuts, , as e.g. Fade Out / F...
Structural Analysis      • Shot Boundary Detection          • Automated Identification of Soft Cuts, , as e.g. Fade Out / F...
Structural Analysis      • Shot Boundary Detection          • Automated Identification of Soft Cuts, , as e.g. Fade Out / F...
Structural Analysis      • Shot Boundary Detection          • Automated Identification of Soft Cuts, , as e.g. Fade Out / F...
Automated Audiovisual Analysis                                            • Visual Analysis                               ...
Intelligent Character Recognition                       • Preprocessing                         • Character Identification ...
Intelligent Character Recognition                  • Character Identification                    • Robust filter to extract ...
Intelligent Character Recognition                    • Stroke Width Transformation                      • based on edge fil...
Intelligent Character Recognition                    • Stroke Width Transformation                      • based on edge fil...
Intelligent Character Recognition                    • Stroke Width Transformation                      • based on edge fil...
Intelligent Character Recognition                   • Preprocessing                     • Text Preprocessing              ...
Intelligent Character Recognition                   • Preprocessing                     • Text Preprocessing              ...
Intelligent Character Recognition                      • Optical Character Recognition (OCR)                        • Stan...
Intelligent Character Recognition                  • Postprocessing                    • Lexical analysis                 ...
Automated Audiovisual Analysis                    • Result: Multimedia data with spatiotemporal AnnotationsMetadata Extrac...
Automated Audiovisual Analysis                    • Result: Multimedia data with spatiotemporal AnnotationsMetadata Extrac...
...                                                           <SpatialDecomposition>                                      ...
Multimedia Ontologies      •   MPEG-7 has been re-engineered to become an OWL-DL ontology          (2007: Arndt et al., CO...
Multimedia Ontologies      Example: Tagging with an MPEG-7 Ontology                                                       ...
Named Entity Recognition                                                                                                  ...
Named Entity Recognition                                                                                                  ...
Semantic Multimedia Analysis                Video Analysis /                                                              ...
Semantic Multimedia Analysis                Video Analysis /                                                              ...
Semantic Multimedia Analysis      Named Entity Recognition                    • Mapping keyterms (text) to semantic entiti...
Semantic Multimedia Analysis      Named Entity Recognition                    • Mapping keyterms (text) to semantic entiti...
Semantic Multimedia Analysis      Named Entity Recognition                    • Mapping keyterms (text) to semantic entiti...
Semantic Multimedia Analysis            Context Analysis and Disambiguation              What defines a Context in AV-Data?...
Semantic Multimedia Analysis            Context Analysis and Disambiguation              What defines a Context in AV-Data?...
Semantic Multimedia Analysis            Context Analysis and Disambiguation              What defines a Context in AV-Data?...
Semantic Multimedia Analysis            Context Analysis and Disambiguation              What defines a Context in AV-Data?...
Semantic Graph Analysis                                                                                                   ...
SEMEX -                                                         Enabling Exploratory Video Search                         ...
Searching is not always                 just searchingHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LD...
a simple example:                I‘m looking for a book by Earnest Hemingway with the                title ,For Whom the B...
a simple example:                I‘m looking for a book by Earnest Hemingway with the                title ,For Whom the B...
...but what if...                I really liked the book ,For Whom the Bell Tolls‘                but I have no idea what ...
...but what if...                I really liked the book ,For Whom the Bell Tolls‘                but I have no idea what ...
Exploratory Search                            • What, if the user does not know, which query string to use?               ...
Exploratory Multimedia Search                Video Analysis /                                                             ...
The Web of Data - The Semantic Web                                   Data is a precious thing and will last longer than th...
DBPedia - the Semantic Wikipedia            dbpedia:For_Whom_the_Bell_Tolls                                      What fact...
Exploratory Multimedia Search                                          dbpedia-owl:author                                 ...
Exploratory Multimedia Search                                                                                             ...
Exploratory Multimedia Search                                                                                             ...
Exploratory Multimedia Search                                                                                             ...
Exploratory Multimedia Search                                     dbpedia-owl:author dbpedia:For_Whom_the_Bell_Tolls      ...
Exploratory Multimedia Search                                                                                             ...
Exploratory Multimedia Search                                                                                             ...
Exploratory Multimedia Search                                                                                             ...
Exploratory Multimedia Search                                                                                             ...
Exploratory Multimedia Search                                                                                             ...
Exploratory Multimedia Search                                                                                             ...
Exploratory Multimedia Search                                                                                             ...
SEMEX -                                                         Enabling Exploratory Video Search                         ...
http://bit.ly/SeMEXHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Fre...
http://mediaglobe.yovisto.com:808029Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg,...
SEMEX -                                                         Enabling Exploratory Video Search                         ...
Contact:                                                        Dr. Harald Sack                                           ...
Upcoming SlideShare
Loading in …5
×

SEMEX: Enabling Exploratory Video Search by Semantic Video Analysis

6,878 views

Published on

Presentation Slides from the LWA 2011 in Magdeburg at 30 Sep 2011http://lwa2011.cs.uni-magdeburg.de/

Published in: Technology, Education
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
6,878
On SlideShare
0
From Embeds
0
Number of Embeds
636
Actions
Shares
0
Downloads
78
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

SEMEX: Enabling Exploratory Video Search by Semantic Video Analysis

  1. 1. Enabling Exploratory Video Search by Semantic Video Analysis LWA 2011 Magdeburg, 30. Sep. 2011 Dr. Harald Sack Hasso-Plattner-Institut for IT-Systems Engineering University of PotsdamFreitag, 30. September 11
  2. 2. ■ HPI was founded in October 1998 as a Public- Private-Partnership ■ HPI Research and Teaching is focussed on IT Systems Engineering ■ 10 Professors and 100 Scientific Coworkers ■ 450 Bachelor / Master Students ■ HPI is winner of CHE-Ranking 2010Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 http://hpi.uni-potsdam.de/Freitag, 30. September 11
  3. 3. Semantic Technologies & Multimedia Retrieval ■ Research Topics □ Semantic Web Technologies □ Ontological Engineering □ Information Retrieval □ Multimedia Analysis & Retrieval □ Social Networking □ Data/Information Visualization ■ Research ProjectsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  4. 4. Semantic Technologies & Multimedia Retrieval ■ Research Topics □ Semantic Web Technologies □ Ontological Engineering □ Information Retrieval □ Multimedia Analysis & Retrieval □ Social Networking □ Data/Information Visualization ■ Research ProjectsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  5. 5. SEMEX - Enabling Exploratory Video Search by Semantic Video Analysis LDW 2011, Magdeburg, 30. Sep 2011 Overview (1) Searching Audiovisual Data (2) Semantic Multimedia Analysis (3) Explorative Semantic Search (4) SeMEX - Semantic Multimedia ExplorerHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  6. 6. The Google Challenge...Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  7. 7. Google Multimedia SearchHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11
  8. 8. How does Google find Multimedia?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  9. 9. How does Google find Multimedia?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  10. 10. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11
  11. 11. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11
  12. 12. How does Google find Multimedia? ... <a href="/mission_pages/shuttle/shuttlemissions/sts134/ multimedia/index.html"> <IMG WIDTH="100" ALT="Close-up view of Endeavours crew cabin prior to docking with the International Space Station" TITLE="Close-up view of Endeavours crew cabin prior to docking with the International Space Station" SRC="/images/ content/549665main_2011-05-18_1600_100-75.jpg" HEIGHT="75" ALIGN="Bottom" BORDER="0" /> </a> <p><a href="/mission_pages/shuttle/shuttlemissions/sts134/ multimedia/index.html">&rsaquo;&nbsp;STS-134 Multimedia</a></ p> ... ‣Google Multimedia Search relies on link contextHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11
  13. 13. How to Search in Multimedia Archives?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  14. 14. How to Search in Multimedia Archives? Step 1: Digitalization of analog data Step 2: Annotation with (textbased) metadataHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  15. 15. How to Search in Multimedia Archives? • manual anotation with text-based descriptive metadataHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  16. 16. How to Search in Multimedia Archives? • manual anotation with text-based descriptive metadata ...how to extract metadata in an automated way?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  17. 17. SEMEX - Enabling Exploratory Video Search by Semantic Video Analysis LDW 2011, Magdeburg, 30. Sep 2011 Overview (1) Searching Audiovisual Data (2) Semantic Multimedia Analysis (3) Explorative Semantic Search (4) SeMEX - Semantic Multimedia ExplorerHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  18. 18. Automated Audiovisual AnalysisHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  19. 19. Automated Audiovisual AnalysisHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  20. 20. Automated Audiovisual AnalysisHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  21. 21. Automated Audiovisual Analysis Face DetectionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  22. 22. Automated Audiovisual Analysis Genre Analysis Classification: Studio Indoor News Show Face DetectionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  23. 23. Automated Audiovisual Analysis Genre Analysis Classification: Studio Indoor overlay News Show Face text DetectionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  24. 24. Automated Audiovisual Analysis Genre Analysis Classification: Studio Indoor overlay News Show Face text Detection scene textHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  25. 25. Automated Audiovisual Analysis Genre Analysis Classification: Studio Indoor overlay News Show Face Logo text Detection Detection scene textHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  26. 26. Automated Audiovisual Analysis Genre Analysis Classification: Studio Indoor overlay News Show Face Logo text Detection Detection scene text Audio-Mining structural Automated speaker analysis Speech identification RecognitionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  27. 27. Automated Audiovisual Analysis • Visual Analysis • Audio Analysis • Structural Analysis • Structural Analysis • Intelligent Character • Speaker Detection Recognition (ICR) • Automated Speech • Character/Logo Recognition (ASR) Detection • Character Filtering • Character Recognition • Genre Analysis & Categorization • Face / Body / Object •Detection •Tracking •ClusteringHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  28. 28. Automated Audiovisual Analysis • Visual Analysis • Audio Analysis • Structural Analysis • Structural Analysis • Intelligent Character • Speaker Detection Recognition (ICR) • Automated Speech • Character/Logo Recognition (ASR) Detection • Character Filtering • Character Recognition • Genre Analysis & Categorization • Face / Body / Object •Detection •Tracking •ClusteringHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  29. 29. Structural Analysis • Decomposition of time-based media into meaningful media fragments of coherent content that can be used as basic element for indexing and classification videoHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  30. 30. Structural Analysis • Decomposition of time-based media into meaningful media fragments of coherent content that can be used as basic element for indexing and classification video scenesHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  31. 31. Structural Analysis • Decomposition of time-based media into meaningful media fragments of coherent content that can be used as basic element for indexing and classification video scenes shotsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  32. 32. Structural Analysis • Decomposition of time-based media into meaningful media fragments of coherent content that can be used as basic element for indexing and classification video scenes shots subshotsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  33. 33. Structural Analysis • Decomposition of time-based media into meaningful media fragments of coherent content that can be used as basic element for indexing and classification video scenes shots subshots frames key framesHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  34. 34. Structural Analysis • Shot Boundary Detection time • Automated Identification of • Hard Cuts • Defects, as e.g., • Drop Outs, White Outs, etc. • Soft Cuts, as e.g., • Fade-In/Out, • Dissolve, Wipe, Cross-Fade, etc. • Automated Structural Analysis based on • Analytical Shot Boundary Detection • Machine Learning Based Shot DetectionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  35. 35. Structural Analysis • Shot Boundary Detection • Automated Identification of Hard Cuts based on • Luminance/Chrominance Histogram Differences & Derivatives • Edge Distribution/Density 573 574 575 576 577 578Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  36. 36. Structural Analysis Adaptive Threshold 1 2 20 1 3 i+W 1 X tha (i) = ↵ · 4@ Da (k, k 1)A Da (i, i 1)5 + k=i W 20 1 i+W 1 X 3 4 4@ 1)A Da(i,i-1) ...D thai = ↵ · th (L2-norm) Da (k, k Frames i (i) Histogram Difference (i) between a (i, 1) > ↵ 20 1 and i-1 of Subregion a k=i W 3 i+W 1 X tha (i) = ↵ · 4@tha(i) ... Da (k, k Threshold for Frameiai (i,1)5 + >a th↵ (i) adaptive 1)A Da (i, of Subregion Decompose Frame into a=4 Subregions Da (i + 1, i) < th↵ (i) D i 1) k=i W Hardcut: if Da (i, i 1) > th↵ (i) and Da (i + 1, i) < th↵ (i) is true for all Subregions a Da (i + 1, i) < th↵ (i) Window Size=4 (W=2) i-3 i-2 i-1 i i+1 i+2Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  37. 37. Structural Analysis • Shot Boundary Detection • Automated Identification of Defects, as e.g. Drop Outs / White Outs Histogram/Chrominance Difference Analysis Drop Out Flashlight / White Out Histogram/Chrominance Difference AnalysisHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  38. 38. Structural Analysis • Shot Boundary Detection • Automated Identification of Defects, as e.g., Drop Outs / White Outs ... i i+1 i+8 i+9 i+10 i+11 i+12 i+13 • Luminance/Chrominance Histogram Differences & DerivativesHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  39. 39. Structural Analysis • Shot Boundary Detection • Automated Identification of Soft Cuts, as e.g. Fade Out / Fade In Fade Out Fade InHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  40. 40. Structural Analysis • Shot Boundary Detection • Automated Identification of Soft Cuts, , as e.g. Fade Out / Fade In • Features applied for machine learning: • luminance histogram (Fade In / Fade Out) • luminance average Yµ and luminance variance Yσ2 follow distinct patterns 1 2 3 • image decomposition • component-based analysis to distinguish regional and global 4 5 6 changes in image content • entropy 7 8 9 • motion vectorsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  41. 41. Structural Analysis • Shot Boundary Detection • Automated Identification of Soft Cuts, , as e.g. Fade Out / Fade In • Features deployed for machine learning: • luminance/chrominance histogram • entropy • motion vectorsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  42. 42. Structural Analysis • Shot Boundary Detection • Automated Identification of Soft Cuts, , as e.g. Fade Out / Fade In • Features deployed for machine learning: • luminance/chrominance histogram • entropy • motion vectorsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  43. 43. Structural Analysis • Shot Boundary Detection • Automated Identification of Soft Cuts, , as e.g. Fade Out / Fade In • Features deployed for machine learning: • luminance/chrominance histogram • entropy 1 2 • motion vectors • image decomposition • compute average motion vectors for all areas • identify camera movements 3 4 (zoom, pan, etc.) and moving objectsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  44. 44. Automated Audiovisual Analysis • Visual Analysis • Audio Analysis • Structural Analysis • Structural Analysis • Intelligent Character • Speaker Detection Recognition (ICR) • Automated Speech • Character/Logo Recognition (ASR) Detection • Character Filtering • Character Recognition • Genre Analysis & Categorization • Face / Body / Object •Detection •Tracking •ClusteringHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  45. 45. Intelligent Character Recognition • Preprocessing • Character Identification • Text Preprocessing • Text Filtering • Adaption of script geometry (Deskew) • Image quality enhancement • Optical Character Recognition (OCR) • Standard OCR software (OCRopus) • Postprocessing • Lexical analysis • Statistical / context based filtering Ermittlungen nach BombenfundenHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  46. 46. Intelligent Character Recognition • Character Identification • Robust filter to extract text candidate frames T T T T T T T T T T • 25 fps results in 90.000 frames per 60 min • too expensive for single frame preprocessing & OCR • fast and robust text identification for preprocessing • Features used for text identification: • edge detection • DCT / Fourier Transformation • Sobel-/Canny Edge Filter • horizontal and vertical edge distribution • Local Binary Patterns (LBP) • Histogram of Oriented Gradients • stroke width analysisHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  47. 47. Intelligent Character Recognition • Stroke Width Transformation • based on edge filtering as a preprocessing step • for each edge pixel a stroke is projected along its gradient direction until another edge pixel is hit • all pixels along the stroke will receive the same stroke width value (color)Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  48. 48. Intelligent Character Recognition • Stroke Width Transformation • based on edge filtering as a preprocessing step • for each edge pixel a stroke is projected along its gradient direction until another edge pixel is hit • all pixels along the stroke will receive the same stroke width value (color) • connected component analysis groups pixels with similar stroke width valueHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  49. 49. Intelligent Character Recognition • Stroke Width Transformation • based on edge filtering as a preprocessing step • for each edge pixel a stroke is projected along its gradient direction until another edge pixel is hit • all pixels along the stroke will receive the same stroke width value (color) • connected component analysis groups pixels with similar stroke width valueHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  50. 50. Intelligent Character Recognition • Preprocessing • Text Preprocessing • Text Filtering Original Image Bounding BoxHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  51. 51. Intelligent Character Recognition • Preprocessing • Text Preprocessing • Quality Enhancement Advanced Image EnhancementHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  52. 52. Intelligent Character Recognition • Optical Character Recognition (OCR) • Standard OCR software (OCRopus) Standard OCR (OCRopus)Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  53. 53. Intelligent Character Recognition • Postprocessing • Lexical analysis • Statistical / context based filtering Context-based Spell CorrectionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  54. 54. Automated Audiovisual Analysis • Result: Multimedia data with spatiotemporal AnnotationsMetadata Extraction time Metadata (e.g. MPEG-7) ... <Video> <TemporalDecomposition> <VideoSegment> <TextAnnotation> <KeywordAnnotation> <Keyword>Astronaut</Keyword> </KeywordAnnotation> </TextAnnotation> <MediaTime> <MediaTimePoint> T00:05:05:0F25 </MediaTimePoint> <MediaDuration> PT00H00M31S0N25F </MediaDuration> </MediaTime> ... </VideoSegment> </TemporalDecomposition> </Video> ...Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  55. 55. Automated Audiovisual Analysis • Result: Multimedia data with spatiotemporal AnnotationsMetadata Extraction Metadata (e.g. MPEG-7) ... <SpatialDecomposition> <TextAnnotation> <KeywordAnnotation> <Keyword>Astronaut</Keyword> </KeywordAnnotation> </TextAnnotation> <SpatialMask> <SubRegion> <Polygon> <Coords> 480 150 620 480 </Coords> </Polygon> </SubRegion> </SpatialMask> ... </SpatialDecomposition> ...Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11
  56. 56. ... <SpatialDecomposition> <TextAnnotation> <KeywordAnnotation> <Keyword>Astronaut</Keyword> </KeywordAnnotation> </TextAnnotation> <SpatialMask> <SubRegion> <Polygon> <Coords> 480 150 620 480 </Coords> </Polygon> </SubRegion> </SpatialMask> ... </SpatialDecomposition> ... But wha t about semantic metadata .. ?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11
  57. 57. Multimedia Ontologies • MPEG-7 has been re-engineered to become an OWL-DL ontology (2007: Arndt et al., COMM model) • Localize a region → Draw a bounding box • Annotate the content → Interpret the content → Tag ,Astronaut‘Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11
  58. 58. Multimedia Ontologies Example: Tagging with an MPEG-7 Ontology Reg1 mpeg7:StillRegion rdf:type decom position Reg1 mpeg7 :spatial_ mpeg7:image mpeg7:SpatialMask mpeg7:depicts mpeg7:depicts mpeg7:polygon dbpedia:Astronaut mpeg7:Coords Man on the MoonHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11
  59. 59. Named Entity Recognition Neil Armstrong Entities is a is a Classes Astronaut Person Named Entity Recognition „locating and classifying atomic elements...into is a predefined categories such as names, persons, organizations, locations, expressions of time, quantities, monetary values, etc.“ Science Occupation C.J.Rijsbergen, Information Retrieval (1979) is a EmploymentHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11
  60. 60. Named Entity Recognition Neil Armstrong is a is a Astronaut Person is a Science Occupation is a EmploymentHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Freitag, 30. September 11
  61. 61. Semantic Multimedia Analysis Video Analysis / time Metadata Extraction metadata metadata metadata metadata metadataHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  62. 62. Semantic Multimedia Analysis Video Analysis / time Metadata Extraction metadata metadata metadata metadata Entity Recognition/ metadata Mapping e.g., person xy location yz event abc e.g., bibliographical data, geographical data, encyclopedic data, ..Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  63. 63. Semantic Multimedia Analysis Named Entity Recognition • Mapping keyterms (text) to semantic entities • Context Analysis and DisambiguationHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  64. 64. Semantic Multimedia Analysis Named Entity Recognition • Mapping keyterms (text) to semantic entities • Context Analysis and Disambiguation Jaguar Keyterm / User TagHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  65. 65. Semantic Multimedia Analysis Named Entity Recognition • Mapping keyterms (text) to semantic entities • Context Analysis and Disambiguation Semantic Entities Jaguar (Car) ? Jaguar (Cat) ? Jaguar (OS) ? Jaguar Keyterm / User Tag Jaguar (Aircraft) ?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  66. 66. Semantic Multimedia Analysis Context Analysis and Disambiguation What defines a Context in AV-Data? • Temporal Coherence • Spatial Coherenceurring • Provenanceuationed by order static novels and Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 Freitag, 30. September 11 Figure 1. Dimensions of context definition in audio-visual
  67. 67. Semantic Multimedia Analysis Context Analysis and Disambiguation What defines a Context in AV-Data? • Temporal Coherence • Spatial Coherenceurring • Provenanceuation Spatialed by Dimension order static novels and Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 Freitag, 30. September 11 Figure 1. Dimensions of context definition in audio-visual
  68. 68. Semantic Multimedia Analysis Context Analysis and Disambiguation What defines a Context in AV-Data? • Temporal Coherence • Spatial Coherenceurring • Provenanceuation Spatialed by Dimension order static novels and Temporal Dimension Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 Freitag, 30. September 11 Figure 1. Dimensions of context definition in audio-visual
  69. 69. Semantic Multimedia Analysis Context Analysis and Disambiguation What defines a Context in AV-Data? • Temporal Coherence • Spatial Coherenceurring • Provenanceuation Spatialed by Dimension User-centered orderDimension static novels and Temporal Dimension Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 Freitag, 30. September 11 Figure 1. Dimensions of context definition in audio-visual
  70. 70. Semantic Graph Analysis Jaguar (Car) Steve McQueen Keyterm / User Tag 1956 Jaguar (OS) jaguar Jaguar (Cat) 1956 Steve jaguar rim wheel McQueen context LOD CloudHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  71. 71. SEMEX - Enabling Exploratory Video Search by Semantic Video Analysis LDW 2011, Magdeburg, 30. Sep 2011 Overview (1) Searching Audiovisual Data (2) Semantic Multimedia Analysis (3) Explorative Semantic Search (4) SeMEX - Semantic Multimedia ExplorerHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  72. 72. Searching is not always just searchingHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  73. 73. a simple example: I‘m looking for a book by Earnest Hemingway with the title ,For Whom the Bell Tolls‘ in the first German edition...“Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  74. 74. a simple example: I‘m looking for a book by Earnest Hemingway with the title ,For Whom the Bell Tolls‘ in the first German edition...“ Wem die Ernest H Stunde schlägt. (Stockho E M I N G W A - Fischer lm usw., Berman Y. S. 8“ Verlag, n- 1941) 56 0 II 1, 25 06, 3454 8Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  75. 75. ...but what if... I really liked the book ,For Whom the Bell Tolls‘ but I have no idea what I should read next...Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  76. 76. ...but what if... I really liked the book ,For Whom the Bell Tolls‘ but I have no idea what I should read next...Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  77. 77. Exploratory Search • What, if the user does not know, which query string to use? • What, if the user is looking for complex answers ? • What, if the user does not know the domain he/she is looking for? • What, if the user wants to know all(!) about a specific topic? • ...,Browsing‘ instead of ,Searching‘ • ...to find something by chance -> Serendipity • ...to get an overview • ...enable content based navigationHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011Freitag, 30. September 11
  78. 78. Exploratory Multimedia Search Video Analysis / time Metadata Extraction metadata metadata metadata metadata Entity Recognition/ metadata Mapping e.g., person xy location yz event abc e.g., bibliographical data, geographical data, encyclopedic data, ..Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  79. 79. The Web of Data - The Semantic Web Data is a precious thing and will last longer than theHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 Berners-Lee) systems themselves. (Tim http://linkeddata.org/Freitag, 30. September 11
  80. 80. DBPedia - the Semantic Wikipedia dbpedia:For_Whom_the_Bell_Tolls What facts for http://dbpedia.org/page/ For_Whom_the_Bell_Tolls dbpedia:For_Whom_the_Bell_Tolls are relevant? ...use heuristicsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  81. 81. Exploratory Multimedia Search dbpedia-owl:author dbpedia:Ernest_Hemingway dbpedia:For_Whom_the_Bell_TollsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  82. 82. Exploratory Multimedia Search or uth :a wl -o dia pe db dbpedia-owl:author dbpedia:Ernest_Hemingway dbpedia:For_Whom_the_Bell_TollsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  83. 83. Exploratory Multimedia Search or uth :a wl -o r tho dia u l :a ow pe d ia- db pe db dbpedia-owl:author dbpedia:Ernest_Hemingway dbpedia:For_Whom_the_Bell_TollsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  84. 84. Exploratory Multimedia Search or uth :a wl -o r tho dia u l :a ow pe d ia- db pe db dbpedia-owl:author dbpedia-owl:author dbpedia:Ernest_Hemingway dbpedia:For_Whom_the_Bell_TollsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  85. 85. Exploratory Multimedia Search dbpedia-owl:author dbpedia:For_Whom_the_Bell_Tolls dbpedia:Ernest_HemingwayHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  86. 86. Exploratory Multimedia Search y dbpedia:Raymond_Carver - d _b edia ence bp influ d l: ow dbpedia-owl:author dbpedia:For_Whom_the_Bell_Tolls dbpedia:Ernest_HemingwayHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  87. 87. Exploratory Multimedia Search dbpedia:Jack_Kerouac by d_ ce en l:in a- flu ow pedi db _b y dbpedia:Raymond_Carver - d edia ence bp influ d l: ow dbpedia-owl:author dbpedia:For_Whom_the_Bell_Tolls dbpedia:Ernest_HemingwayHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  88. 88. Exploratory Multimedia Search dbpedia:Jack_Kerouac by d_ ce en l:in a- flu ow pedi db _b y dbpedia:Raymond_Carver - d edia ence bp influ d l: ow dbpedia-owl:author dbpedia-owl:influenced_by dbpedia:For_Whom_the_Bell_Tolls dbpedia:Ernest_Hemingway dbpedia:Jerome_D._SalingerHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  89. 89. Exploratory Multimedia Search dbpedia:Jerome_D._Salinger dbpedia:Jack_Kerouac dbpedia:Raymond_CarverHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  90. 90. Exploratory Multimedia Search dbpedia:Jerome_D._Salinger dbpedia:Jack_Kerouac dbpedia:Raymond_Carver dbpedia-owl:notableWorkHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  91. 91. Exploratory Multimedia Search dbpedia:Jerome_D._Salinger dbpedia:Jack_Kerouac dbpedia:Raymond_Carver dbpedia-owl:notableWork dbpedia-owl:notableWorkHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  92. 92. Exploratory Multimedia Search dbpedia:Jerome_D._Salinger dbpedia:Jack_Kerouac dbpedia:Raymond_Carver dbpedia-owl:notableWork dbpedia- dbpedia-owl:notableWork owl:notableWorkHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  93. 93. SEMEX - Enabling Exploratory Video Search by Semantic Video Analysis LDW 2011, Magdeburg, 30. Sep 2011 Overview (1) Searching Audiovisual Data (2) Semantic Multimedia Analysis (3) Explorative Semantic Search (4) SeMEX - Semantic Multimedia ExplorerHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  94. 94. http://bit.ly/SeMEXHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  95. 95. http://mediaglobe.yovisto.com:808029Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  96. 96. SEMEX - Enabling Exploratory Video Search by Semantic Video Analysis LDW 2011, Magdeburg, 30. Sep 2011 Overview (1) Searching Audiovisual Data (2) Semantic Multimedia Analysis (3) Explorative Semantic Search (4) SeMEX - Semantic Multimedia ExplorerHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11
  97. 97. Contact: Dr. Harald Sack Hasso-Plattner-Institut für Softwaresystemtechnik Universität Potsdam Prof.-Dr.-Helmert-Str. 2-3 D-14482 Potsdam Homepage: ttp://www.hpi.uni-potsdam.de/meinel/team/sack.html h http://www.yovisto.com/ Blog: http://moresemantic.blogspot.com/ E-Mail: harald.sack@hpi.uni-potsdam.de joerg.waitelonis@hpi.uni-potsdam.de ch Twitter: lysander07 / biblionomicon / yovisto ve ry mu k y ou T han tio n! ur at ten f or yoHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011Freitag, 30. September 11

×