CONTENTUS—A Semantic Multimodal Search User Interface

More than the Sum of its PartsCONTENTUS—A Semantic MultimodalSearch User InterfaceJörg Waitelonis, Johannes Osterhoff, Harald SackVisual Interfaces to the Social and Semantic Web (VISSW 2011)In conjunction with the International Conference on IntelligentUser Interfaces (IUI 2011), J. Osterhoff, February 13, 2011

CONTENTUS? A Teaser2 Semantic Search Engine Cultural Heritage Domain Media types Articles, Books, Video, Audio, … People, Locations, … Part of the HPI research project Semantic Media Explorer VISSW 2011 | Johannes Osterhoff | February 13, 2011

CONTENTUS? A Teaser3 VISSW 2011 | Johannes Osterhoff | February 13, 2011

4 Issue #1 How to find the needle in the haystack? VISSW 2011 | J. Osterhoff | February 13, 2011

5Image: iStockphoto VISSW 2011 | J. Osterhoff | February 13, 2011

6 needle VISSW 2011 | J. Osterhoff | February 13, 2011

Google Results7 VISSW 2011 | J. Osterhoff | February 13, 2011

Needless to say…8 needless VISSW 2011 | J. Osterhoff | February 13, 2011

9 “needle” VISSW 2011 | J. Osterhoff | February 13, 2011

10 Jimmy Needles Image: Jay P. Lee American basketball coach 1900—1969 Person VISSW 2011 | J. Osterhoff | February 13, 2011

12 Needles, CA Location Image: Eddie Biggs VISSW 2011 | J. Osterhoff | February 13, 2011

14 “Needle” Blockbuster by John V. Soto from 2010 Film Image: IMDB VISSW 2011 | J. Osterhoff | February 13, 2011

15 Issue #1 How to find different “needles” in the haystack? VISSW 2011 | J. Osterhoff | February 13, 2011

16 Issue #2 How to present “needles” as search results? VISSW 2011 | J. Osterhoff | February 13, 2011

Google Results17 VISSW 2011 | J. Osterhoff | February 13, 2011

Flickr Results18 VISSW 2011 | J. Osterhoff | February 13, 2011

YouTube Results19 VISSW 2011 | J. Osterhoff | February 13, 2011

Wikipedia Results20 VISSW 2011 | J. Osterhoff | February 13, 2011

21 Issue #2 How to present different “needles” as distinguishable search results on one page? VISSW 2011 | J. Osterhoff | February 13, 2011

22 Why? VISSW 2011 | J. Osterhoff | February 13, 2011

23 30,000+ Libraries in Germany VISSW 2011 | J. Osterhoff | February 13, 2011

24 30,000+ Libraries in Germany VISSW 2011 | J. Osterhoff | February 13, 2011

Image: Library of Congress25 Books VISSW 2011 | J. Osterhoff | February 13, 2011

26 Articles VISSW 2011 | J. Osterhoff | February 13, 2011

Image: Library of Congress27 Video VISSW 2011 | J. Osterhoff | February 13, 2011

28 Audio VISSW 2011 | J. Osterhoff | February 13, 2011

29 Issue #3 How to make the German cultural heritage available? VISSW 2011 | J. Osterhoff | February 13, 2011

30 Authors i.e. Bertolt Brecht VISSW 2011 | J. Osterhoff | February 13, 2011

31 Years i.e. 1848 VISSW 2011 | J. Osterhoff | February 13, 2011

Peoplei.e. from Ulm

Image: Wikipedia (Softeis)33 Locations among many others… VISSW 2011 | J. Osterhoff | February 13, 2011

Image: Wikipedia34 Music Style VISSW 2011 | J. Osterhoff | February 13, 2011

Process Chain35 HPI Research VISSW 2011 | J. Osterhoff | February 13, 2011

On the Interface36 Media Assets Books, Articles, Video, Audio Cultural Categorizations Authors, Year of Creation, People, Locations, Music Styles, Organizations etc. VISSW 2011 | J. Osterhoff | February 13, 2011

37 Issue #3 How to make the German cultural heritage available in meaningful information retrieval? VISSW 2011 | J. Osterhoff | February 13, 2011

38 VISSW 2011 | J. Osterhoff | February 13, 2011

39 1. What is CONTENTUS? 2. Interface Design 3. Demonstration of the System 4. Summary VISSW 2011 | J. Osterhoff | February 13, 2011

What is CONTENTUS? Introduction40 ■ a semantic search engine and a multimedia retrieval system VISSW 2011 | Johannes Osterhoff | February 13, 2011

What is CONTENTUS? Introduction41 ■ a semantic search engine and a multimedia retrieval system ■ web-based graphical user interface VISSW 2011 | Johannes Osterhoff | February 13, 2011

What is CONTENTUS? Introduction42 ■ a semantic search engine and a multimedia retrieval system ■ web-based graphical user interface ■ content-based suggestions and a multi-modal presentation of search results to support semantic search VISSW 2011 | Johannes Osterhoff | February 13, 2011

What is CONTENTUS? Introduction43 ■ a semantic search engine and a multimedia retrieval system ■ web-based graphical user interface ■ content-based suggestions and a multi-modal presentation of search results to support semantic search ■ developed in the context of the German research project THESEUS VISSW 2011 | Johannes Osterhoff | February 13, 2011

What is CONTENTUS? Introduction44 ■ Combination of a facetted browsing and breadcrumb-based navigation ■ A time-line enables time-based filtering of search results ■ Personalized search (in the next demonstrator) VISSW 2011 | Johannes Osterhoff | February 13, 2011

What is CONTENTUS? Introduction45 “More than the sum of its parts” VISSW 2011 | J. Osterhoff | February 13, 2011

What is CONTENTUS? Introduction46 More efficient and more pleasant user experience. VISSW 2011 | J. Osterhoff | February 13, 2011

What is CONTENTUS? Context47 ■ new technologies for managing, organizing, and searching multimedia content gets more efficient ■ tremendous diversity of multimedia data affects the retrieval and search process ■ image and video search technology is evolving rapidly VISSW 2011 | Johannes Osterhoff | February 13, 2011

What is CONTENTUS? Context48 ■ semantic search technology promises more accurate and complete search results ■ new user interfaces are required to enable query formulation and intelligent visualization of search results VISSW 2011 | J. Osterhoff | February 13, 2011

What is CONTENTUS? Technologies49 ■ all types of media assets ■ technologies  media optimization  content analysis  semantic metadata generation  speech transcripts  OCR  extraction of semantic entities (i.e. people, places, events) ■ highly heterogeneous collection of semantically annotated data VISSW 2011 | Johannes Osterhoff | February 13, 2011

User Interface Paradigms51 No research on a single aspect of current approaches in semantic multimedia retrieval… VISSW 2011 | Johannes Osterhoff | February 13, 2011

User Interface Paradigms52 But: exploiting synergies by combining state-of-the-art technologies instead. VISSW 2011 | Johannes Osterhoff | February 13, 2011

User Interface Paradigms53 semantic multimedia technologies VISSW 2011 | Johannes Osterhoff | February 13, 2011

User Interface Paradigms54 semantic + multimedia technologies VISSW 2011 | Johannes Osterhoff | February 13, 2011

User Interface Paradigms55 semantic + multimedia technologies + interface design VISSW 2011 | Johannes Osterhoff | February 13, 2011

User Interface Paradigms56 combination of multimedia and semantic search = new challenges in visualizing search results and facilitating navigation VISSW 2011 | J. Osterhoff | February 13, 2011

User Interface Paradigms57  compare different media at one glance  how to summarize content most meaningfully in one single search result item  time-based information (audio, video) VISSW 2011 | J. Osterhoff | February 13, 2011

User Interface Paradigms58 multimodal VISSW 2011 | Johannes Osterhoff | February 13, 2011

User Interface Paradigms59 multimodal = articles, books, video, audio, people, locations on one result-page VISSW 2011 | Johannes Osterhoff | February 13, 2011

User Interface Paradigms60 The route with the user‟s experience gets more important than the original goal of fact finding. VISSW 2011 | Johannes Osterhoff | February 13, 2011

User Interface Paradigms61 main goal satisfying search experience for the user VISSW 2011 | J. Osterhoff | February 13, 2011

63 Search Area

64 Search Field breadcrumbs

67 Result Area

68 media types history collection organize and filter

69 time-line faceted suggestions refine and filter

search results70

search results71 article article video audio location person video

76 less important search results

User Interface Detail Page for Books78 Detail View VISSW 2011 | Johannes Osterhoff | February 13, 2011

User Interface Detail Page for Books79 full page other pages articles on this page text snippet (OCR) detected entities Detail View VISSW 2011 | Johannes Osterhoff | February 13, 2011

81 Demo http://mediaglobe.yovisto.com:8081/contentus/ VISSW 2011 | Johannes Osterhoff | February 13, 2011

Summary83 Issue #1 Issue #1 How to find the needle How to find different in the haystack? “needles” in the haystack? Issue #2 Issue #2 How to present How to present “needles” different “needles” as as search results? distinguishable search results? Issue #3 Issue #3 How to make the German How to make the German cultural heritage available? Cultural Heritage available in meaningful Information Retrieval? VISSW 2011 | J. Osterhoff | February 13, 2011

Future Work84 ■ Feedback that lets the user know why exactly the listed items have been presented ■ Possibility to connect with friends and to share results with them ■ Suggestions based on friends„ likes ■ State-of-the-art multi-platform web design ■ Next demonstrator to be presented at IBC 2011 Amsterdam VISSW 2011 | Johannes Osterhoff | February 13, 2011

85 VISSW 2011 | Johannes Osterhoff | February 13, 2011

Person Book86 Location Audio Video VISSW 2011 | Johannes Osterhoff | February 13, 2011

87 Thank you! Johannes P Osterhoff, M.A. johannes.osterhoff@hpi.uni-potsdam.de Hasso-Plattner-Institute, Prof.-Dr.-Helmert-Str. 2-3, 14482 Potsdam, Germany VISSW 2011 | Johannes Osterhoff | February 13, 2011

CONTENTUS—A Semantic Multimodal Search User Interface

by Johannes P Osterhoff on Mar 08, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 14

Statistics

CONTENTUS—A Semantic Multimodal Search User Interface — Presentation Transcript