The document describes CONTENTUS, a semantic multimodal search interface. CONTENTUS combines faceted browsing and breadcrumb navigation with a timeline to filter search results from a heterogeneous collection of semantically annotated cultural heritage media. It aims to provide a more efficient and pleasant user experience by presenting different media types together as search results, such as articles, books, videos and audios alongside people, locations and other semantic entities. The interface design brings together semantic and multimedia technologies to address challenges in visualizing diverse search outputs in a distinguishable yet integrated manner.

1. More than the Sum of its Parts
CONTENTUS—
A Semantic Multimodal
Search User Interface
Jörg Waitelonis, Johannes Osterhoff, Harald Sack
Visual Interfaces to the Social and Semantic Web (VISSW 2011)
In conjunction with the International Conference on Intelligent
User Interfaces (IUI 2011), J. Osterhoff, February 13, 2011

2. CONTENTUS?
A Teaser
2
Semantic Search Engine
Cultural Heritage Domain
Media types
Articles, Books, Video, Audio, …
People, Locations, …
Part of the HPI research project
Semantic Media Explorer
VISSW 2011 | Johannes Osterhoff | February 13, 2011

3. CONTENTUS?
A Teaser
3
VISSW 2011 | Johannes Osterhoff | February 13, 2011

4. 4
Issue #1
How to find the needle
in the haystack?
VISSW 2011 | J. Osterhoff | February 13, 2011

5. 5
Image: iStockphoto
VISSW 2011 | J. Osterhoff | February 13, 2011

6. 6
needle
VISSW 2011 | J. Osterhoff | February 13, 2011

7. Google Results
7
VISSW 2011 | J. Osterhoff | February 13, 2011

8. Needless to say…
8
needless
VISSW 2011 | J. Osterhoff | February 13, 2011

9. 9
“needle”
VISSW 2011 | J. Osterhoff | February 13, 2011

10. 10
Jimmy Needles
Image: Jay P. Lee
American basketball coach
1900—1969
Person
VISSW 2011 | J. Osterhoff | February 13, 2011

11. 11
“needle”
VISSW 2011 | J. Osterhoff | February 13, 2011

12. 12
Needles, CA
Location
Image: Eddie Biggs
VISSW 2011 | J. Osterhoff | February 13, 2011

13. 13
“needle”
VISSW 2011 | J. Osterhoff | February 13, 2011

14. 14
“Needle”
Blockbuster
by John V. Soto from 2010
Film
Image: IMDB
VISSW 2011 | J. Osterhoff | February 13, 2011

15. 15
Issue #1
How to find different “needles”
in the haystack?
VISSW 2011 | J. Osterhoff | February 13, 2011

16. 16
Issue #2
How to present
“needles”
as search results?
VISSW 2011 | J. Osterhoff | February 13, 2011

17. Google Results
17
VISSW 2011 | J. Osterhoff | February 13, 2011

18. Flickr Results
18
VISSW 2011 | J. Osterhoff | February 13, 2011

19. YouTube Results
19
VISSW 2011 | J. Osterhoff | February 13, 2011

20. Wikipedia Results
20
VISSW 2011 | J. Osterhoff | February 13, 2011

21. 21
Issue #2
How to present
different “needles”
as distinguishable
search results
on one page?
VISSW 2011 | J. Osterhoff | February 13, 2011

22. 22
Why?
VISSW 2011 | J. Osterhoff | February 13, 2011

23. 23
30,000+
Libraries in Germany
VISSW 2011 | J. Osterhoff | February 13, 2011

24. 24
30,000+
Libraries in Germany
VISSW 2011 | J. Osterhoff | February 13, 2011

25. Image: Library of Congress
25
Books
VISSW 2011 | J. Osterhoff | February 13, 2011

26. 26
Articles
VISSW 2011 | J. Osterhoff | February 13, 2011

27. Image: Library of Congress
27
Video
VISSW 2011 | J. Osterhoff | February 13, 2011

28. 28
Audio
VISSW 2011 | J. Osterhoff | February 13, 2011

29. 29
Issue #3
How to make the German
cultural heritage available?
VISSW 2011 | J. Osterhoff | February 13, 2011

30. 30
Authors
i.e. Bertolt Brecht
VISSW 2011 | J. Osterhoff | February 13, 2011

31. 31
Years
i.e. 1848
VISSW 2011 | J. Osterhoff | February 13, 2011

32. People
i.e. from Ulm

33. Image: Wikipedia (Softeis)
33
Locations
among many others…
VISSW 2011 | J. Osterhoff | February 13, 2011

34. Image: Wikipedia
34
Music Style
VISSW 2011 | J. Osterhoff | February 13, 2011

35. Process Chain
35
HPI Research
VISSW 2011 | J. Osterhoff | February 13, 2011

36. On the Interface
36
Media Assets
Books, Articles, Video, Audio
Cultural Categorizations
Authors, Year of Creation,
People, Locations, Music Styles,
Organizations etc.
VISSW 2011 | J. Osterhoff | February 13, 2011

37. 37
Issue #3
How to make the German
cultural heritage available
in meaningful
information retrieval?
VISSW 2011 | J. Osterhoff | February 13, 2011

38. 38
VISSW 2011 | J. Osterhoff | February 13, 2011

39. 39
1. What is CONTENTUS?
2. Interface Design
3. Demonstration of the System
4. Summary
VISSW 2011 | J. Osterhoff | February 13, 2011

40. What is CONTENTUS?
Introduction
40
■ a semantic search engine and a multimedia
retrieval system
VISSW 2011 | Johannes Osterhoff | February 13, 2011

41. What is CONTENTUS?
Introduction
41
■ a semantic search engine and a multimedia
retrieval system
■ web-based graphical user interface
VISSW 2011 | Johannes Osterhoff | February 13, 2011

42. What is CONTENTUS?
Introduction
42
■ a semantic search engine and a multimedia
retrieval system
■ web-based graphical user interface
■ content-based suggestions and
a multi-modal presentation of
search results to support semantic search
VISSW 2011 | Johannes Osterhoff | February 13, 2011

43. What is CONTENTUS?
Introduction
43
■ a semantic search engine and a multimedia
retrieval system
■ web-based graphical user interface
■ content-based suggestions and
a multi-modal presentation of
search results to support semantic search
■ developed in the context of the German
research project THESEUS
VISSW 2011 | Johannes Osterhoff | February 13, 2011

44. What is CONTENTUS?
Introduction
44
■ Combination of a facetted browsing and
breadcrumb-based navigation
■ A time-line enables time-based filtering
of search results
■ Personalized search (in the next demonstrator)
VISSW 2011 | Johannes Osterhoff | February 13, 2011

45. What is CONTENTUS?
Introduction
45
“More than
the sum of its parts”
VISSW 2011 | J. Osterhoff | February 13, 2011

46. What is CONTENTUS?
Introduction
46
More efficient
and more pleasant
user experience.
VISSW 2011 | J. Osterhoff | February 13, 2011

47. What is CONTENTUS?
Context
47
■ new technologies for managing,
organizing, and searching multimedia
content gets more efficient
■ tremendous diversity of multimedia
data affects the retrieval and search
process
■ image and video search technology is
evolving rapidly
VISSW 2011 | Johannes Osterhoff | February 13, 2011

48. What is CONTENTUS?
Context
48
■ semantic search technology promises
more accurate and complete search
results
■ new user interfaces
are required to enable query formulation
and intelligent visualization of search
results
VISSW 2011 | J. Osterhoff | February 13, 2011

49. What is CONTENTUS?
Technologies
49
■ all types of media assets
■ technologies
 media optimization
 content analysis
 semantic metadata generation
 speech transcripts
 OCR
 extraction of semantic entities (i.e. people, places, events)
■ highly heterogeneous collection of semantically
annotated data
VISSW 2011 | Johannes Osterhoff | February 13, 2011

50. 50
1. What is CONTENTUS?
2. Interface Design
3. Demonstration of the System
4. Summary
VISSW 2011 | J. Osterhoff | February 13, 2011

51. User Interface
Paradigms
51
No research on
a single aspect of
current approaches
in semantic multimedia
retrieval…
VISSW 2011 | Johannes Osterhoff | February 13, 2011

52. User Interface
Paradigms
52
But:
exploiting synergies
by combining
state-of-the-art
technologies instead.
VISSW 2011 | Johannes Osterhoff | February 13, 2011

53. User Interface
Paradigms
53
semantic
multimedia
technologies
VISSW 2011 | Johannes Osterhoff | February 13, 2011

54. User Interface
Paradigms
54
semantic +
multimedia
technologies
VISSW 2011 | Johannes Osterhoff | February 13, 2011

55. User Interface
Paradigms
55
semantic +
multimedia
technologies +
interface design
VISSW 2011 | Johannes Osterhoff | February 13, 2011

56. User Interface
Paradigms
56
combination of multimedia
and semantic search
= new challenges in visualizing
search results and facilitating
navigation
VISSW 2011 | J. Osterhoff | February 13, 2011

57. User Interface
Paradigms
57
 compare different media at one glance
 how to summarize content most
meaningfully in one single search
result item
 time-based information (audio, video)
VISSW 2011 | J. Osterhoff | February 13, 2011

58. User Interface
Paradigms
58
multimodal
VISSW 2011 | Johannes Osterhoff | February 13, 2011

59. User Interface
Paradigms
59
multimodal
= articles, books,
video, audio, people,
locations on one
result-page
VISSW 2011 | Johannes Osterhoff | February 13, 2011

60. User Interface
Paradigms
60
The route with the
user‟s experience
gets more important
than the original goal
of fact finding.
VISSW 2011 | Johannes Osterhoff | February 13, 2011

61. User Interface
Paradigms
61
main goal
satisfying search experience
for the user
VISSW 2011 | J. Osterhoff | February 13, 2011

62. 62

63. 63 Search Area

64. 64 Search Field breadcrumbs

65. 65

66. 66

67. 67
Result Area

68. 68
media types
history
collection
organize and filter

69. 69
time-line
faceted
suggestions
refine and filter

70. search results
70

71. search results
71
article
article video
audio location
person video

72. 72

73. 73

74. 74

75. 75

76. 76
less important
search results

77. 77

78. User Interface
Detail Page for Books
78
Detail View
VISSW 2011 | Johannes Osterhoff | February 13, 2011

79. User Interface
Detail Page for Books
79
full page
other pages
articles on this page
text snippet (OCR)
detected entities
Detail View
VISSW 2011 | Johannes Osterhoff | February 13, 2011

80. 80
1. What is CONTENTUS?
2. Interface Design
3. Demonstration of the System
4. Summary
VISSW 2011 | J. Osterhoff | February 13, 2011

81. 81
Demo
http://mediaglobe.yovisto.com:8081/contentus/
VISSW 2011 | Johannes Osterhoff | February 13, 2011

82. 82
1. What is CONTENTUS?
2. Interface Design
3. Demonstration of the System
4. Summary
VISSW 2011 | J. Osterhoff | February 13, 2011

83. Summary
83
Issue #1 Issue #1
How to find the needle How to find different
in the haystack? “needles” in the haystack?
Issue #2 Issue #2
How to present How to present
“needles” different “needles” as
as search results? distinguishable search
results?
Issue #3 Issue #3
How to make the German How to make the German
cultural heritage available? Cultural Heritage available
in meaningful Information
Retrieval?
VISSW 2011 | J. Osterhoff | February 13, 2011

84. Future Work
84
■ Feedback that lets the user know why exactly the listed
items have been presented
■ Possibility to connect with friends and to share results
with them
■ Suggestions based on friends„ likes
■ State-of-the-art multi-platform web design
■ Next demonstrator to be presented at
IBC 2011 Amsterdam
VISSW 2011 | Johannes Osterhoff | February 13, 2011

85. 85
VISSW 2011 | Johannes Osterhoff | February 13, 2011

86. Person
Book
86
Location
Audio
Video
VISSW 2011 | Johannes Osterhoff | February 13, 2011

87. 87
Thank you!
Johannes P Osterhoff, M.A.
johannes.osterhoff@hpi.uni-potsdam.de
Hasso-Plattner-Institute, Prof.-Dr.-Helmert-Str. 2-3, 14482 Potsdam, Germany
VISSW 2011 | Johannes Osterhoff | February 13, 2011