Your SlideShare is downloading. ×
Search and Hyperlinking Task at MediaEval 2012
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Search and Hyperlinking Task at MediaEval 2012

102
views

Published on

The Search and Hyperlinking Task was one of the Brave New Tasks at MediaEval 2012. The Task consisted of two sub-tasks which focused on search and linking in retrieval from a collection of …

The Search and Hyperlinking Task was one of the Brave New Tasks at MediaEval 2012. The Task consisted of two sub-tasks which focused on search and linking in retrieval from a collection of semi-professional video content. These tasks followed up on research carried out within the MediaEval 2011 Rich Speech Retrieval (RSR) Task and the VideoCLEF 2009 Linking Task.

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
102
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Search and Hyperlinking Task at MediaEval 2012 Search and Hyperlinking Task at MediaEval 2012Maria Eskevich1 , Gareth J.F. Jones1 , Shu Chen1 Robin Aly2 , Roeland Ordelman2 Martha Larson3 1 Dublin City University, Dublin, Ireland 2 University of Twente, The Netherlands 3 Delft University of Technology, Delft, The Netherlands
  • 2. Search and Hyperlinking Task at MediaEval 2012Background
  • 3. Search and Hyperlinking Task at MediaEval 2012Brave New Task
  • 4. Search and Hyperlinking Task at MediaEval 2012Brave New Task and Crowdsourcing
  • 5. Search and Hyperlinking Task at MediaEval 2012Brave New Task: Search and Hyperlinking Previous work: MediaEval 2011: Rich Speech Retrieval Task VideoCLEF 2009: Linking Task
  • 6. Search and Hyperlinking Task at MediaEval 2012Brave New Task: Search and Hyperlinking Previous work: MediaEval 2011: Rich Speech Retrieval Task VideoCLEF 2009: Linking Task What is Brave New about the Search and Hyperlinking? Unified scenario for 2 tasks: Search output results as input for Linking Use of crowdsourcing for results assessment: workers define the relevance of the video segments, the HIT can be run for each new submission and enrich overall results Search: ME10WWW dataset − > blip10000
  • 7. Search and Hyperlinking Task at MediaEval 2012Sub-tasks Search sub-task: Known-item search: Textual queries only Multimodal queries (text + video clues) Required runs: 1 submission for 1-best output of each ASR transcript Linking sub-task Ad-hoc search for videos that can potentially be linked to the anchor videos (ground truth of the Search subtask) Required runs: use of ASR 1-best transcripts as representation *Additional runs: use team output of Search subtask as video anchors for Linking sub-task.
  • 8. Search and Hyperlinking Task at MediaEval 2012Data: blip10000 ASR Transcript: LIMSI/Vocapia: Confusion networks LIUM: 1-best Lattices Confision networks Video clues: Shot boundaries (TU Berlin) Concept-based descriptors based on a list of 589 concepts (University of Oxford) Face detection results (INRIA)
  • 9. Search and Hyperlinking Task at MediaEval 2012Evaluation: Search sub-task
  • 10. Search and Hyperlinking Task at MediaEval 2012Evaluation: Search sub-task
  • 11. Search and Hyperlinking Task at MediaEval 2012Evaluation: Search sub-task
  • 12. Search and Hyperlinking Task at MediaEval 2012Evaluation: Search sub-task
  • 13. Search and Hyperlinking Task at MediaEval 2012Evaluation: Search sub-task
  • 14. Search and Hyperlinking Task at MediaEval 2012Evaluation: Search sub-task Mean Reciprocal Rank (MRR): 1 RR = RANK Mean Generalized Average Precision (mGAP): 1 GAP = . PENALTY RANK
  • 15. Search and Hyperlinking Task at MediaEval 2012Evaluation: Search sub-task Mean Average Segment Precision (MASP): Ranking + Length of (ir)relevant content Segment Precision (SP[r ]) at rank r :
  • 16. Search and Hyperlinking Task at MediaEval 2012Evaluation: Search sub-task Mean Average Segment Precision (MASP): Ranking + Length of (ir)relevant content Segment Precision (SP[r ]) at rank r :
  • 17. Search and Hyperlinking Task at MediaEval 2012Evaluation: Search sub-task Mean Average Segment Precision (MASP): Ranking + Length of (ir)relevant content Segment Precision (SP[r ]) at rank r : Average Segment Precision:
  • 18. Search and Hyperlinking Task at MediaEval 2012Evaluation: Search sub-task Mean Average Segment Precision (MASP): Ranking + Length of (ir)relevant content Segment Precision (SP[r ]) at rank r : Average Segment Precision: N 1 ASP = . SP[r ] · rel(sr ) n r =1 rel(sr ) = 1, if relevant content is present, otherwise rel(sr ) = 0
  • 19. Search and Hyperlinking Task at MediaEval 2012Relevance Evaluation via Crowdsourcing:Amazon MTurk HIT
  • 20. Search and Hyperlinking Task at MediaEval 2012Evaluation: Linking sub-task
  • 21. Search and Hyperlinking Task at MediaEval 2012Evaluation: Linking sub-task
  • 22. Search and Hyperlinking Task at MediaEval 2012Evaluation: Linking sub-task
  • 23. Search and Hyperlinking Task at MediaEval 2012Evaluation: Linking sub-task
  • 24. Search and Hyperlinking Task at MediaEval 2012Relevance Evaluation via CrowdsourcingDetails Assessment level: Videos Provided segments: Length varies across runs and participants
  • 25. Search and Hyperlinking Task at MediaEval 2012Relevance Evaluation via CrowdsourcingDetails Assessment level: Videos Provided segments: Length varies across runs and participants Workers explanation of relevance judgement: Relevant: ”Same” videos Videos are on the same topic Same program different topics Irrelevant: Same program different topics Different topics Different video styles: interview vs presentation
  • 26. Search and Hyperlinking Task at MediaEval 2012Participants Group Search Linking Charles University in Prague (CUNI) 5 – Dublin City University (DCU) 4 6 Ghent University (MMLab) 4 4+4 INRIA/IRISA – 5 University of Twente (UTwente) 3 1
  • 27. Search and Hyperlinking Task at MediaEval 2012Participants Group Search Linking Charles University in Prague (CUNI) 5 – Dublin City University (DCU) 4 6 Ghent University (MMLab) 4 4+4 INRIA/IRISA – 5 University of Twente (UTwente) 3 1
  • 28. Search and Hyperlinking Task at MediaEval 2012Search sub-task results
  • 29. Search and Hyperlinking Task at MediaEval 2012Linking sub-task results
  • 30. Search and Hyperlinking Task at MediaEval 2012Thank you for your attention! Participants presentations: Search sub-task: Maria Eskevich, Dublin City University Petra Galuscakova, Charles University in Prague Search and Linking sub-task: Danish Nadeem, University of Twente Linking sub-task: Camille Guinaudeau, INRIA/IRISA Shu Chen, Dublin City University Search and Linking sub-task: Tom de Nies, University of Chent
  • 31. Search and Hyperlinking Task at MediaEval 2012Thank you for your attention!Buon compleanno Robin! Participants presentations: Search sub-task: Maria Eskevich, Dublin City University Petra Galuscakova, Charles University in Prague Search and Linking sub-task: Danish Nadeem, University of Twente Linking sub-task: Camille Guinaudeau, INRIA/IRISA Shu Chen, Dublin City University Search and Linking sub-task: Tom de Nies, University of Chent