Overview of the Search and Hyperlinking Task at the MediaEval evaluation workshop 2013

1. Search and Hyperlinking
Overview
Robin Aly, Maria Eskevich,
Roeland Ordelman, Gareth J.F. Jones

2. Outline
•
•
•
•
•
The search and hyperlinking task
Dataset (Videos + user input)
Ground truth creation
Evaluation procedure
Results
5/13/13
LIME workshop - WWW2013

3. ME 2013 Search & Hyperlinking
• ME 2012 S&HL “brave new” task:
– Search: retrieve known-item video segment given a
natural language description
– Linking: link known-item video segment to similar
segments in the collection (blib.tv)
• ME 2013 S&HL “regular” task
– Search: retrieve known video segment (known-item)
given a textual query and visual cues
– Linking: link user-defined anchor within the knownitem to relevant target video segments
7/2/13
DGA workshop - July 2013, Paris

4. Terminology
• Video (e.g, 2 hours)
• Interesting segment (e.g. 10 min)
• Anchor: segment for which a user
requests a link (e.g., 1 min)
“I want to know more about this”
• Hyperlink
• Target: relevant segment for given
anchor (e.g., 5 min)
7/2/13
DGA workshop - July 2013, Paris

5. ME 2013 Search & Hyperlinking
Jump-in point
7/2/13
Anchor
DGA workshop - July 2013, Paris

6. Dataset: Video collection
• copyright cleared broadcasts from the period
of 01.04.2008 – 11.05.2008
• 1667 hours, 2323 videos
• ~200 videos were rebroadcast
5/13/13
LIME workshop - WWW2013

7. Dataset: Query
• 28 Users
- Policemen, Hair dresser, Bouncer, Sales manger,
Student, Self-employed
• Two hour session:
– Browse the archive
– Define known-item
– Define anchor
5/13/13
LIME workshop - WWW2013

8. AXES PRO
system
Formulate
Information
need
Text search
7/2/13
Visual search
DGA workshop - July 2013, Paris

10. Select anchor
using sliders
Describe
anchor
Describe
target
7/2/13
DGA workshop - July 2013, Paris

12. Dataset: outcome (1/2)
• 50 known-items
<top>
<itemId>item_9</itemId>
<queryText>animal park, kenya marathon , wildlife reserve</queryText>
<visualQueues>animals, kenya wildlife reserve,marathon</visualQueues>
</top>
5/13/13
LIME workshop - WWW2013

13. Dataset: outcome (2/2)
• 98 anchors
<anchor>
<anchorId>anchor_3</anchorId>
<startTime>5.29</startTime>
<endTime>6.03</endTime>
<item>
<fileName>v20080506_093000_...</fileName>
</item
</anchor>
5/13/13
LIME workshop - WWW2013

14. Ground truth creation
• Search sub-task: done (known-items)
• Judgments linking sub-task:
– top-10 of one run per participant (7) by users
locally at BBC
– top-10 of most runs using Amazon's mechanical
turk
– ~70% Agreement
– Similar measurements
5/13/13
LIME workshop - WWW2013

17. Unifying Qrels
Case1: Merge relevant
Judgment 1
Judgment 2
Result
5/13/13
LIME workshop - WWW2013
Case2: Prioritize relevant
over non-relevant

18. Ground truth statistics

21. Evaluation: search sub-task
• Depending on jump-in point
• Measures: MRR, MGAP, MSAP
• Rebroadcasts not considered
5/13/13

22. Evaluation: linking sub-task
• Measures: P@5,10,20 and map
• Resolving conflicts
Judged
segment
Submitted
segment
Result:

23. RESULTS

25. Per Task

26. Results: search sub-task

27. Results: hyperlinking sub-task

28. Conclusions
•
•
•
•
•
Task defined by real users
Innovative elements perceived as interesting
Crowdsourcing valid evaluation strategy
Experimental setup works
Provides a promising starting point for further
research

29. The Search and Hyperlinking task was funded by
We are grateful to
Jana Eggink and
Andy O'Dwyer
from the BBC for preparing the collection and hosting the user trials.
... and of course Martha for advise & crowdsourcing access.