Gareth Jones 1209
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Gareth Jones 1209

on

  • 573 views

Научно-технический семинар Гарета Джонса 12 сентября

Научно-технический семинар Гарета Джонса 12 сентября

Statistics

Views

Total Views
573
Views on SlideShare
380
Embed Views
193

Actions

Likes
1
Downloads
1
Comments
0

3 Embeds 193

http://tech.yandex.ru 173
http://events.yandex.ru 18
https://tech.yandex.ru 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Gareth Jones 1209 Presentation Transcript

  • 1. Multimedia Information Seeking Through Search and Hyperlinking Gareth J. F. Jones Centre for Next Generation Localisation School of Computing, Dublin City University, Dublin, Ireland
  • 2. Overview Background and Motivation MediaEval Multimedia Evaluation Benchmark Search and Hyperlinking Search and Hyperlinking at MediaEval 2012 Search and Hyperlinking at MediaEval 2013 Conclusions and Research Prospects
  • 3. Background and Motivation Introduction Search and hyperlink navigation are well established approaches to information seeking on the textual web.
  • 4. Background and Motivation Introduction Search and hyperlink navigation are well established approaches to information seeking on the textual web. Increasing amounts of online digital multimedia content is creating new challenges and opportunities for information access.
  • 5. Background and Motivation Introduction Search and hyperlink navigation are well established approaches to information seeking on the textual web. Increasing amounts of online digital multimedia content is creating new challenges and opportunities for information access. Realizing the potential of this content requires users to be provided with engaging ways to interact with it, helping them to discover, browse, navigate and search.
  • 6. Background and Motivation Introduction Search and hyperlink navigation are well established approaches to information seeking on the textual web. Increasing amounts of online digital multimedia content is creating new challenges and opportunities for information access. Realizing the potential of this content requires users to be provided with engaging ways to interact with it, helping them to discover, browse, navigate and search. Different classes of users, such as media professionals, students, researchers and home users will potentially have different needs and preferred means of access.
  • 7. Background and Motivation Introduction Existing research on multimedia information retrieval mainly focused on:
  • 8. Background and Motivation Introduction Existing research on multimedia information retrieval mainly focused on: search for individual relevant items - generally focusing on visual information needs and features; or
  • 9. Background and Motivation Introduction Existing research on multimedia information retrieval mainly focused on: search for individual relevant items - generally focusing on visual information needs and features; or linking of content based on sharing visual features without attention to user information needs, e.g. linking videos containing the same individuals.
  • 10. Background and Motivation Introduction Existing research on multimedia information retrieval mainly focused on: search for individual relevant items - generally focusing on visual information needs and features; or linking of content based on sharing visual features without attention to user information needs, e.g. linking videos containing the same individuals. Linking of multimedia content to text resources, e.g. wikipedia.
  • 11. Background and Motivation Introduction Existing research on multimedia information retrieval mainly focused on: search for individual relevant items - generally focusing on visual information needs and features; or linking of content based on sharing visual features without attention to user information needs, e.g. linking videos containing the same individuals. Linking of multimedia content to text resources, e.g. wikipedia. Social remmendation based on user behaviour.
  • 12. Background and Motivation Introduction Existing research on multimedia information retrieval mainly focused on: search for individual relevant items - generally focusing on visual information needs and features; or linking of content based on sharing visual features without attention to user information needs, e.g. linking videos containing the same individuals. Linking of multimedia content to text resources, e.g. wikipedia. Social remmendation based on user behaviour. Our work in search and hyperlinking aims to explore multimodal access over multimedia content, and linking of video content to support potential user interests or needs.
  • 13. Background and Motivation Current Video Recommendation Search for “Things to do in London”
  • 14. Background and Motivation Vision of Hyperlinked Video
  • 15. Background and Motivation Vision of Hyperlinked Video
  • 16. Background and Motivation Search and Hyperlinking These studies take place within the context of the MediaEval Multimedia Evaluation Benchmark campaign (www.multimediaeval.org).
  • 17. MediaEval Multimedia Evaluation Benchmark MediaEval Multimedia Evaluation Benchmark An evaluation benchmarking initiative focusing on information access and indexing tasks for multimedia content. Established in 2010, following on from earlier VideoCLEF tasks at CLEF 2008 and CLEF 2009. Evaluates new algorithms primarily for novel and emerging tasks linked to a use case. Emphasis on the “multi” in multimedia: audio, speech, visual content, tags, users, context.
  • 18. MediaEval Multimedia Evaluation Benchmark MediaEval Multimedia Evaluation Benchmark TREC style annual evaluation cycle: call for task proposals, call for participation, data release, participants submit results, organisers release evaluation of results, participants gather at a workshop to discuss the results of the task.
  • 19. MediaEval Multimedia Evaluation Benchmark MediaEval Multimedia Evaluation Benchmark TREC style annual evaluation cycle: call for task proposals, call for participation, data release, participants submit results, organisers release evaluation of results, participants gather at a workshop to discuss the results of the task. Search and Hyperlinking task at MediaEval 2012 and MediaEval 2013.
  • 20. MediaEval Multimedia Evaluation Benchmark MediaEval Workshops MediaEval 2012 workshop in Pisa, Italy - co-located with ECCV 2012.
  • 21. MediaEval Multimedia Evaluation Benchmark MediaEval Workshops MediaEval 2012 workshop in Pisa, Italy - co-located with ECCV 2012. MediaEval 2013 workshop in Barcelona, Spain - co-located with ACM Multimedia 2013.
  • 22. Search and Hyperlinking Search and Hyperlinking Audio-visual archive exploration scenario. Video-to-video linking. Start with archive search assuming an “information need”. Assume search becomes serendipitous or exploratory by following hyperlinks.
  • 23. Search and Hyperlinking Search and Hyperlinking Video, e.g. 2 hours Interesting segment, e.g. 10 minutes Anchor: segment for which a user would like to link from (e.g. 1 minute), e.g. “I want to know more about ...” Hyperlink Target: relevant segment for a given anchor, e.g. 5 minutes
  • 24. Search and Hyperlinking Search and Hyperlinking
  • 25. Search and Hyperlinking Search and Hyperlinking MediaEval 2012 Search and Hyperlinking task (dataset blip.tv): Search: retrieve known-item video segment given a natural language description Hyperlinking: link known-item video segment to similar segments in the collection MediaEval 2013 Search and Hyperlinking task (dataset BBC archive): Search: retrieve known-item video segment given a natural language description Hyperlinking: link user-defined anchor within the known-item to relevant target video segments in the collection
  • 26. Search and Hyperlinking Search and Hyperlinking at MediaEval 2012 Multimodal search where relevance can be related to both visual and/or audio information streams. Models scenario where user seeks single relevant known-item fragment of multimedia content. Once the relevant fragment has been located the user can engage in browsing or exploratory search by following automatically created hyperlinks to related multimedia fragments within the same collection.
  • 27. Search and Hyperlinking Blip10000 dataset The Blip10000 dataset was created by the PetaMedia EU Network of Excellence: Contains 14,838 Creative Commons videos crawled from blip.tv, and corresponding user provided metadata. Total of ca. 3,260 hours of data: Divided into 5,288 development videos, and 9,550 test videos. Automatically created visual features: shot boundaries - ave. shot length 30 secs single keyframe for each shot visual concepts from a set of 589 Two automatic speech recognition transcripts created: LIMSI/Vocapia and LIUM. Comparative experiments using only English language dataset: 4,890 test videos.
  • 28. Search and Hyperlinking Test Query Set 30 textual queries consisting of natural language type statements and search engine style queries. Textual information collected via crowdsourcing using the Amazon Mechanical Turk (AMT) platform; multimodal features added manually afterwards. Query set for video selected at random from the top 10 genre categories. Turkers required to: locate an interesting region of the video mark the beginning and end points of the interesting region transcribe the words spoken exactly create natural language and search engine type queries which they would use to refind this region of the video
  • 29. Search and Hyperlinking Evaluation Average Precision AP = 1 n . N r=1 P[r]
  • 30. Search and Hyperlinking Evaluation Average Precision AP = 1 n . N r=1 P[r] Generalized Average Precision: GAP = 1 n . N r=1 P[r] · 1 − Distance Granularity · 0.1
  • 31. Search and Hyperlinking Evaluation Segment Precision (SP[r]) at rank r:
  • 32. Search and Hyperlinking Evaluation Segment Precision (SP[r]) at rank r: Average Segment Precision: ASP = 1 n . N r=1 SP[r] · rel(sr ) SP[r] = Segment Precision [r], rel(sr ) = 1, if relevant content is present, otherwise rel(sr ) = 0
  • 33. Search and Hyperlinking Hyperlinking Linking effectiveness evaluated using MAP. Relevance of each of the top 10 proposed links for each submitted run evaluated using AMT. Separate Qrel file created for each run; pooled unified Qrel created by merging run qrels. Turkers given the following options: Video segments totally unrealted Video segments related, same topic or focus, but different information Video segments related, different perspective or view on the same information Video segments are basically the same.
  • 34. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Working with 270 hours of BBC archival data: one complete week material not owned by the BBC excluded Task designed to explore the information needs and task activities of home users. Expected to favour exploratory search options.
  • 35. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Search queries created by 30 public users with computer and online experience. Recruited using external agency. Profile: Nationality: UK Age Group: 16-30 Computer familiarity: high Online activity: high
  • 36. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Users given a general introduction on video search using the AXES Pro system.
  • 37. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Users given a general introduction on video search using the AXES Pro system. Scope and type of the collection explained, and users allowed to explore the collection.
  • 38. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Users given a general introduction on video search using the AXES Pro system. Scope and type of the collection explained, and users allowed to explore the collection. User creates text query expressing information need which should be satisfied by the available dataset.
  • 39. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Users given a general introduction on video search using the AXES Pro system. Scope and type of the collection explained, and users allowed to explore the collection. User creates text query expressing information need which should be satisfied by the available dataset. User searches for a relevant video, reformulating their query if necessary.
  • 40. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Users given a general introduction on video search using the AXES Pro system. Scope and type of the collection explained, and users allowed to explore the collection. User creates text query expressing information need which should be satisfied by the available dataset. User searches for a relevant video, reformulating their query if necessary. Ask the user to identify anchors within their chosen video: specify region, spoken words or whole segmennts.
  • 41. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Users given a general introduction on video search using the AXES Pro system. Scope and type of the collection explained, and users allowed to explore the collection. User creates text query expressing information need which should be satisfied by the available dataset. User searches for a relevant video, reformulating their query if necessary. Ask the user to identify anchors within their chosen video: specify region, spoken words or whole segmennts. Explain reasons why these anchors were selected.
  • 42. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013
  • 43. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013
  • 44. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013
  • 45. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013
  • 46. Search and Hyperlinking Specification of Anchors
  • 47. Search and Hyperlinking Classification of Selected Clips
  • 48. Search and Hyperlinking Classification of Anchors
  • 49. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Set of 50 known-item experimental queries selected Corresponding set of 100 anchors selected
  • 50. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Set of 50 known-item experimental queries selected Corresponding set of 100 anchors selected Run submissions 10th September!
  • 51. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Set of 50 known-item experimental queries selected Corresponding set of 100 anchors selected Run submissions 10th September! Relevance assessment of links ongoing using returning test volunteers at BBC and using AMT (to increase amount of relevance assessment in time available)
  • 52. Search and Hyperlinking Search and Hyperlinking at MediaEval 2013 Set of 50 known-item experimental queries selected Corresponding set of 100 anchors selected Run submissions 10th September! Relevance assessment of links ongoing using returning test volunteers at BBC and using AMT (to increase amount of relevance assessment in time available) MediaEval 2013 workshop 18th-19th October
  • 53. Conclusions and Research Prospects Conclusions and Research Prospects Search is a more established task, but little research so far on identifying reliable jump-in points. Promising results from Search and Hyperlinking in 2012 encouraged us to move to user studies with BBC data. Work so far with volunteer subjects suggests that users will make use of video hyperlinks. Results and other findings from Search and Hyperlinking in 2013 will be used to help develop focused research agenda for 2014 and beyond.
  • 54. Conclusions and Research Prospects References M.Larson, M.Soleymani, M.Eskevich, P.Serdyukov, R.Ordelman and G.J.F.Jones, The Community and the Crowd: Developing large-scale data collections for multimedia benchmarking. IEEE Multimedia. Vol.19, No.3, pp15-23, 2012. M.Eskevich, W.Magdy and G.J.F.Jones. New Metrics for Meaningful Evaluation of Informally Structured Speech Retrieval, In Proceedings of the 34th European Conference on Information Retrieval (ECIR 2012), Barcelona, April 2012. M.Eskevich, G.J.F.Jones, S.Chen, R.Aly, R.Ordelman and M.Larson, Search and Hyperlinking Task at MediaEval 2012, Proceedings of the MediaEval 2012 Workshop, Pisa, Italy, October 2012. M.Eskevich, G.J.F.Jones., R.Aly, R.J.F. Ordelman, S.Chen, D.Nadeem, C.Guinaudeau, G.Gravier, P.Sebillot, T.de Nies, P.Debevere, R.van de Walle, P.Galuscakova, P.Pecina and M.Larson, Information Seeking through Search and Hyperlinking, In Proceedings of ACM International Conference on Multimedia Retrieval (ICMR 2013), Dallas, Texas, USA, April 2013. R.Aly, R.Ordelman, M.Eskevitch, G.J.F.Jones and S.Chen, Linking inside a video collection – what and how to measure?, Proceedings of the First Worldwide Web Workshop on Linked Media (LiME-2013) at WWW 2013, Rio de Janeiro, Brazil, May 2013.
  • 55. Conclusions and Research Prospects Acknowledgements Coordinators of the MediaEval Search and Hyperlinking task: Robin Aly, Maria Eskevich, Gareth Jones, Roeland Ordelman Search and Hyperlinking activities supported by: