Multimedia Information Seeking Through
Search and Hyperlinking
Gareth J. F. Jones
Centre for Next Generation Localisation
...
Overview
Background and Motivation
MediaEval Multimedia Evaluation Benchmark
Search and Hyperlinking
Search and Hyperlinki...
Background and Motivation
Introduction
Search and hyperlink navigation are well established
approaches to information seek...
Background and Motivation
Introduction
Search and hyperlink navigation are well established
approaches to information seek...
Background and Motivation
Introduction
Search and hyperlink navigation are well established
approaches to information seek...
Background and Motivation
Introduction
Search and hyperlink navigation are well established
approaches to information seek...
Background and Motivation
Introduction
Existing research on multimedia information retrieval
mainly focused on:
Background and Motivation
Introduction
Existing research on multimedia information retrieval
mainly focused on:
search for...
Background and Motivation
Introduction
Existing research on multimedia information retrieval
mainly focused on:
search for...
Background and Motivation
Introduction
Existing research on multimedia information retrieval
mainly focused on:
search for...
Background and Motivation
Introduction
Existing research on multimedia information retrieval
mainly focused on:
search for...
Background and Motivation
Introduction
Existing research on multimedia information retrieval
mainly focused on:
search for...
Background and Motivation
Current Video Recommendation
Search for “Things to do in London”
Background and Motivation
Vision of Hyperlinked Video
Background and Motivation
Vision of Hyperlinked Video
Background and Motivation
Search and Hyperlinking
These studies take place within the context of the
MediaEval Multimedia ...
MediaEval Multimedia Evaluation Benchmark
MediaEval Multimedia Evaluation Benchmark
An evaluation benchmarking initiative ...
MediaEval Multimedia Evaluation Benchmark
MediaEval Multimedia Evaluation Benchmark
TREC style annual evaluation cycle:
ca...
MediaEval Multimedia Evaluation Benchmark
MediaEval Multimedia Evaluation Benchmark
TREC style annual evaluation cycle:
ca...
MediaEval Multimedia Evaluation Benchmark
MediaEval Workshops
MediaEval 2012 workshop in Pisa, Italy - co-located with
ECC...
MediaEval Multimedia Evaluation Benchmark
MediaEval Workshops
MediaEval 2012 workshop in Pisa, Italy - co-located with
ECC...
Search and Hyperlinking
Search and Hyperlinking
Audio-visual archive exploration scenario.
Video-to-video linking.
Start w...
Search and Hyperlinking
Search and Hyperlinking
Video, e.g. 2 hours
Interesting segment,
e.g. 10 minutes
Anchor: segment f...
Search and Hyperlinking
Search and Hyperlinking
Search and Hyperlinking
Search and Hyperlinking
MediaEval 2012 Search and Hyperlinking task (dataset
blip.tv):
Search: ret...
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2012
Multimodal search where relevance can be related to both...
Search and Hyperlinking
Blip10000 dataset
The Blip10000 dataset was created by the PetaMedia EU
Network of Excellence:
Con...
Search and Hyperlinking
Test Query Set
30 textual queries consisting of natural language type
statements and search engine...
Search and Hyperlinking
Evaluation
Average Precision
AP =
1
n
.
N
r=1
P[r]
Search and Hyperlinking
Evaluation
Average Precision
AP =
1
n
.
N
r=1
P[r]
Generalized Average Precision:
GAP =
1
n
.
N
r=...
Search and Hyperlinking
Evaluation
Segment Precision (SP[r]) at rank r:
Search and Hyperlinking
Evaluation
Segment Precision (SP[r]) at rank r:
Average Segment Precision:
ASP =
1
n
.
N
r=1
SP[r]...
Search and Hyperlinking
Hyperlinking
Linking effectiveness evaluated using MAP.
Relevance of each of the top 10 proposed l...
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Working with 270 hours of BBC archival data:
one complet...
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Search queries created by 30 public users with computer
...
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Users given a general introduction on video search using...
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Users given a general introduction on video search using...
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Users given a general introduction on video search using...
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Users given a general introduction on video search using...
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Users given a general introduction on video search using...
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Users given a general introduction on video search using...
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Search and Hyperlinking
Specification of Anchors
Search and Hyperlinking
Classification of Selected Clips
Search and Hyperlinking
Classification of Anchors
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Set of 50 known-item experimental queries selected
Corre...
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Set of 50 known-item experimental queries selected
Corre...
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Set of 50 known-item experimental queries selected
Corre...
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Set of 50 known-item experimental queries selected
Corre...
Conclusions and Research Prospects
Conclusions and Research Prospects
Search is a more established task, but little resear...
Conclusions and Research Prospects
References
M.Larson, M.Soleymani, M.Eskevich, P.Serdyukov, R.Ordelman and G.J.F.Jones,
...
Conclusions and Research Prospects
Acknowledgements
Coordinators of the MediaEval Search and Hyperlinking task:
Robin Aly,...
Upcoming SlideShare
Loading in...5
×

Gareth Jones 1209

418
-1

Published on

Научно-технический семинар Гарета Джонса 12 сентября

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
418
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
5
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Gareth Jones 1209

  1. 1. Multimedia Information Seeking Through Search and Hyperlinking Gareth J. F. Jones Centre for Next Generation Localisation School of Computing, Dublin City University, Dublin, Ireland
  2. 2. Overview Background and Motivation MediaEval Multimedia Evaluation Benchmark Search and Hyperlinking Search and Hyperlinking at MediaEval 2012 Search and Hyperlinking at MediaEval 2013 Conclusions and Research Prospects
  3. 3. Background and Motivation Introduction Search and hyperlink navigation are well established approaches to information seeking on the textual web.
  4. 4. Background and Motivation Introduction Search and hyperlink navigation are well established approaches to information seeking on the textual web. Increasing amounts of online digital multimedia content is creating new challenges and opportunities for information access.
  5. 5. Background and Motivation Introduction Search and hyperlink navigation are well established approaches to information seeking on the textual web. Increasing amounts of online digital multimedia content is creating new challenges and opportunities for information access. Realizing the potential of this content requires users to be provided with engaging ways to interact with it, helping them to discover, browse, navigate and search.
  6. 6. Background and Motivation Introduction Search and hyperlink navigation are well established approaches to information seeking on the textual web. Increasing amounts of online digital multimedia content is creating new challenges and opportunities for information access. Realizing the potential of this content requires users to be provided with engaging ways to interact with it, helping them to discover, browse, navigate and search. Different classes of users, such as media professionals, students, researchers and home users will potentially have different needs and preferred means of access.
  7. 7. Background and Motivation Introduction Existing research on multimedia information retrieval mainly focused on:
  8. 8. Background and Motivation Introduction Existing research on multimedia information retrieval mainly focused on: search for individual relevant items - generally focusing on visual information needs and features; or
  9. 9. Background and Motivation Introduction Existing research on multimedia information retrieval mainly focused on: search for individual relevant items - generally focusing on visual information needs and features; or linking of content based on sharing visual features without attention to user information needs, e.g. linking videos containing the same individuals.
  10. 10. Background and Motivation Introduction Existing research on multimedia information retrieval mainly focused on: search for individual relevant items - generally focusing on visual information needs and features; or linking of content based on sharing visual features without attention to user information needs, e.g. linking videos containing the same individuals. Linking of multimedia content to text resources, e.g. wikipedia.
  11. 11. Background and Motivation Introduction Existing research on multimedia information retrieval mainly focused on: search for individual relevant items - generally focusing on visual information needs and features; or linking of content based on sharing visual features without attention to user information needs, e.g. linking videos containing the same individuals. Linking of multimedia content to text resources, e.g. wikipedia. Social remmendation based on user behaviour.
  12. 12. Background and Motivation Introduction Existing research on multimedia information retrieval mainly focused on: search for individual relevant items - generally focusing on visual information needs and features; or linking of content based on sharing visual features without attention to user information needs, e.g. linking videos containing the same individuals. Linking of multimedia content to text resources, e.g. wikipedia. Social remmendation based on user behaviour. Our work in search and hyperlinking aims to explore multimodal access over multimedia content, and linking of video content to support potential user interests or needs.
  13. 13. Background and Motivation Current Video Recommendation Search for “Things to do in London”
  14. 14. Background and Motivation Vision of Hyperlinked Video
  15. 15. Background and Motivation Vision of Hyperlinked Video
  16. 16. Background and Motivation Search and Hyperlinking These studies take place within the context of the MediaEval Multimedia Evaluation Benchmark campaign (www.multimediaeval.org).
  17. 17. MediaEval Multimedia Evaluation Benchmark MediaEval Multimedia Evaluation Benchmark An evaluation benchmarking initiative focusing on information access and indexing tasks for multimedia content. Established in 2010, following on from earlier VideoCLEF tasks at CLEF 2008 and CLEF 2009. Evaluates new algorithms primarily for novel and emerging tasks linked to a use case. Emphasis on the “multi” in multimedia: audio, speech, visual content, tags, users, context.
  18. 18. MediaEval Multimedia Evaluation Benchmark MediaEval Multimedia Evaluation Benchmark TREC style annual evaluation cycle: call for task proposals, call for participation, data release, participants submit results, organisers release evaluation of results, participants gather at a workshop to discuss the results of the task.
  19. 19. MediaEval Multimedia Evaluation Benchmark MediaEval Multimedia Evaluation Benchmark TREC style annual evaluation cycle: call for task proposals, call for participation, data release, participants submit results, organisers release evaluation of results, participants gather at a workshop to discuss the results of the task. Search and Hyperlinking task at MediaEval 2012 and MediaEval 2013.
  20. 20. MediaEval Multimedia Evaluation Benchmark MediaEval Workshops MediaEval 2012 workshop in Pisa, Italy - co-located with ECCV 2012.
  21. 21. MediaEval Multimedia Evaluation Benchmark MediaEval Workshops MediaEval 2012 workshop in Pisa, Italy - co-located with ECCV 2012. MediaEval 2013 workshop in Barcelona, Spain - co-located with ACM Multimedia 2013.
  22. 22. Search and Hyperlinking Search and Hyperlinking Audio-visual archive exploration scenario. Video-to-video linking. Start with archive search assuming an “information need”. Assume search becomes serendipitous or exploratory by following hyperlinks.
  23. 23. Search and Hyperlinking Search and Hyperlinking Video, e.g. 2 hours Interesting segment, e.g. 10 minutes Anchor: segment for which a user would like to link from (e.g. 1 minute), e.g. “I want to know more about ...” Hyperlink Target: relevant segment for a given anchor, e.g. 5 minutes
  24. 24. Search and Hyperlinking Search and Hyperlinking
  25. 25. Search and Hyperlinking Search and Hyperlinking MediaEval 2012 Search and Hyperlinking task (dataset blip.tv): Search: retrieve known-item video segment given a natural language description Hyperlinking: link known-item video segment to similar segments in the collection MediaEval 2013 Search and Hyperlinking task (dataset BBC archive): Search: retrieve known-item video segment given a natural language description Hyperlinking: link user-defined anchor within the known-item to relevant target video segments in the collection
  26. 26. Search and Hyperlinking Search and Hyperlinking at MediaEval 2012 Multimodal search where relevance can be related to both visual and/or audio information streams. Models scenario where user seeks single relevant known-item fragment of multimedia content. Once the relevant fragment has been located the user can engage in browsing or exploratory search by following automatically created hyperlinks to related multimedia fragments within the same collection.
  27. 27. Search and Hyperlinking Blip10000 dataset The Blip10000 dataset was created by the PetaMedia EU Network of Excellence: Contains 14,838 Creative Commons videos crawled from blip.tv, and corresponding user provided metadata. Total of ca. 3,260 hours of data: Divided into 5,288 development videos, and 9,550 test videos. Automatically created visual features: shot boundaries - ave. shot length 30 secs single keyframe for each shot visual concepts from a set of 589 Two automatic speech recognition transcripts created: LIMSI/Vocapia and LIUM. Comparative experiments using only English language dataset: 4,890 test videos.
  28. 28. Search and Hyperlinking Test Query Set 30 textual queries consisting of natural language type statements and search engine style queries. Textual information collected via crowdsourcing using the Amazon Mechanical Turk (AMT) platform; multimodal features added manually afterwards. Query set for video selected at random from the top 10 genre categories. Turkers required to: locate an interesting region of the video mark the beginning and end points of the interesting region transcribe the words spoken exactly create natural language and search engine type queries which they would use to refind this region of the video
  29. 29. Search and Hyperlinking Evaluation Average Precision AP = 1 n . N r=1 P[r]
  30. 30. Search and Hyperlinking Evaluation Average Precision AP = 1 n . N r=1 P[r] Generalized Average Precision: GAP = 1 n . N r=1 P[r] · 1 − Distance Granularity · 0.1
  31. 31. Search and Hyperlinking Evaluation Segment Precision (SP[r]) at rank r:
  32. 32. Search and Hyperlinking Evaluation Segment Precision (SP[r]) at rank r: Average Segment Precision: ASP = 1 n . N r=1 SP[r] · rel(sr ) SP[r] = Segment Precision [r], rel(sr ) = 1, if relevant content is present, otherwise rel(sr ) = 0
  33. 33. Search and Hyperlinking Hyperlinking Linking effectiveness evaluated using MAP. Relevance of each of the top 10 proposed links for each submitted run evaluated using AMT. Separate Qrel file created for each run; pooled unified Qrel created by merging run qrels. Turkers given the following options: Video segments totally unrealted Video segments related, same topic or focus, but different information Video segments related, different perspective or view on the same information Video segments are basically the same.
  34. 34. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Working with 270 hours of BBC archival data: one complete week material not owned by the BBC excluded Task designed to explore the information needs and task activities of home users. Expected to favour exploratory search options.
  35. 35. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Search queries created by 30 public users with computer and online experience. Recruited using external agency. Profile: Nationality: UK Age Group: 16-30 Computer familiarity: high Online activity: high
  36. 36. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Users given a general introduction on video search using the AXES Pro system.
  37. 37. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Users given a general introduction on video search using the AXES Pro system. Scope and type of the collection explained, and users allowed to explore the collection.
  38. 38. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Users given a general introduction on video search using the AXES Pro system. Scope and type of the collection explained, and users allowed to explore the collection. User creates text query expressing information need which should be satisfied by the available dataset.
  39. 39. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Users given a general introduction on video search using the AXES Pro system. Scope and type of the collection explained, and users allowed to explore the collection. User creates text query expressing information need which should be satisfied by the available dataset. User searches for a relevant video, reformulating their query if necessary.
  40. 40. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Users given a general introduction on video search using the AXES Pro system. Scope and type of the collection explained, and users allowed to explore the collection. User creates text query expressing information need which should be satisfied by the available dataset. User searches for a relevant video, reformulating their query if necessary. Ask the user to identify anchors within their chosen video: specify region, spoken words or whole segmennts.
  41. 41. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Users given a general introduction on video search using the AXES Pro system. Scope and type of the collection explained, and users allowed to explore the collection. User creates text query expressing information need which should be satisfied by the available dataset. User searches for a relevant video, reformulating their query if necessary. Ask the user to identify anchors within their chosen video: specify region, spoken words or whole segmennts. Explain reasons why these anchors were selected.
  42. 42. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013
  43. 43. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013
  44. 44. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013
  45. 45. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013
  46. 46. Search and Hyperlinking Specification of Anchors
  47. 47. Search and Hyperlinking Classification of Selected Clips
  48. 48. Search and Hyperlinking Classification of Anchors
  49. 49. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Set of 50 known-item experimental queries selected Corresponding set of 100 anchors selected
  50. 50. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Set of 50 known-item experimental queries selected Corresponding set of 100 anchors selected Run submissions 10th September!
  51. 51. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Set of 50 known-item experimental queries selected Corresponding set of 100 anchors selected Run submissions 10th September! Relevance assessment of links ongoing using returning test volunteers at BBC and using AMT (to increase amount of relevance assessment in time available)
  52. 52. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Set of 50 known-item experimental queries selected Corresponding set of 100 anchors selected Run submissions 10th September! Relevance assessment of links ongoing using returning test volunteers at BBC and using AMT (to increase amount of relevance assessment in time available) MediaEval 2013 workshop 18th-19th October
  53. 53. Conclusions and Research Prospects Conclusions and Research Prospects Search is a more established task, but little research so far on identifying reliable jump-in points. Promising results from Search and Hyperlinking in 2012 encouraged us to move to user studies with BBC data. Work so far with volunteer subjects suggests that users will make use of video hyperlinks. Results and other findings from Search and Hyperlinking in 2013 will be used to help develop focused research agenda for 2014 and beyond.
  54. 54. Conclusions and Research Prospects References M.Larson, M.Soleymani, M.Eskevich, P.Serdyukov, R.Ordelman and G.J.F.Jones, The Community and the Crowd: Developing large-scale data collections for multimedia benchmarking. IEEE Multimedia. Vol.19, No.3, pp15-23, 2012. M.Eskevich, W.Magdy and G.J.F.Jones. New Metrics for Meaningful Evaluation of Informally Structured Speech Retrieval, In Proceedings of the 34th European Conference on Information Retrieval (ECIR 2012), Barcelona, April 2012. M.Eskevich, G.J.F.Jones, S.Chen, R.Aly, R.Ordelman and M.Larson, Search and Hyperlinking Task at MediaEval 2012, Proceedings of the MediaEval 2012 Workshop, Pisa, Italy, October 2012. M.Eskevich, G.J.F.Jones., R.Aly, R.J.F. Ordelman, S.Chen, D.Nadeem, C.Guinaudeau, G.Gravier, P.Sebillot, T.de Nies, P.Debevere, R.van de Walle, P.Galuscakova, P.Pecina and M.Larson, Information Seeking through Search and Hyperlinking, In Proceedings of ACM International Conference on Multimedia Retrieval (ICMR 2013), Dallas, Texas, USA, April 2013. R.Aly, R.Ordelman, M.Eskevitch, G.J.F.Jones and S.Chen, Linking inside a video collection – what and how to measure?, Proceedings of the First Worldwide Web Workshop on Linked Media (LiME-2013) at WWW 2013, Rio de Janeiro, Brazil, May 2013.
  55. 55. Conclusions and Research Prospects Acknowledgements Coordinators of the MediaEval Search and Hyperlinking task: Robin Aly, Maria Eskevich, Gareth Jones, Roeland Ordelman Search and Hyperlinking activities supported by:
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×