Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Set-­‐up	
  of	
  the	
  	
  
Search	
  &	
  Hyperlinking	
  Task	
  
	
  in	
  the	
  	
  
	
  Series	
  
	
  
	
  Evalua...
Video	
  hyperlinking	
  
•  Topic	
  with	
  increasing	
  aEen@on	
  
•  Relate	
  video	
  (fragments)	
  to	
  other	
...
MediaEval
•  a	
  mul$media	
  benchmarking	
  ini@a@ve	
  running	
  several	
  
tasks	
  
•  evaluates	
  new	
  algorit...
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
Video	
  linking	
  scenario	
  
•  Audiovisual	
  archive	
  explora$on	
  scenario	
  
•  Video-­‐to-­‐video	
  linking	...
Terminology	
  
•  Video	
  (e.g,	
  2	
  hours)	
  
•  Interes@ng	
  segment	
  (e.g.	
  10	
  min)	
  
•  Anchor:	
  seg...
ME	
  2013	
  Search	
  &	
  Hyperlinking	
  
•  ME	
  2012	
  S&HL	
  “brave	
  new”	
  task:	
  	
  
–  Search:	
  retri...
ME	
  2013	
  Search	
  &	
  Hyperlinking	
  
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
Anchor	
  
J...
Task	
  set-­‐up	
  
1.  Organizers:	
  provide	
  anchors	
  to	
  par@cipants	
  
2.  Par@cipants:	
  find	
  relevant	
 ...
1.	
  Provide	
  anchors	
  	
  
•  Requirements:	
  
– Large	
  video	
  data	
  set	
  
– Real	
  ‘users’	
  selec@ng	
 ...
Researchers	
  &	
  
Educators	
  
	
  
AXES	
  Research	
  
system	
  
Journalists	
   Research	
  
Academic	
  
research...
Users	
  
•  Recruitment	
  Agency	
  
•  In	
  an@cipa@on	
  on	
  a	
  rather	
  complex	
  task:	
  
public	
  users	
 ...
ANCHOR	
  SELECTION	
  
BBC	
  User	
  Trials	
  
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
Search	
  for	
  an	
  interes@ng	
  video	
  
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
potentially...
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
Formulate	
  
Informa@on	
  
need	
  
Text	
  search	
   V...
Select	
  an	
  interes@ng	
  clip	
  
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
BBC ARCHIVE
potenti...
Select	
  an	
  anchor	
  
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
BBC ARCHIVE
potentially interes...
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
Describe	
  	
  
target	
  
Describe	
  
anchor	
  
Select...
Trial	
  evalua@on	
  
•  Most	
  par@cipants	
  understood	
  the	
  task	
  
•  Not	
  difficult	
  
•  Technology	
  woul...
Classifica@on	
  of	
  selected	
  clips	
  
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
0	
  
5	
  
10...
Classifica@on	
  of	
  anchors	
  
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
0	
  
10	
  
20	
  
30	
...
2.	
  Automa@c	
  target	
  selec@on	
  
•  ~	
  15	
  Par@cipants	
  
•  Data	
  
–  50	
  NL	
  queries,	
  50	
  anchor...
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
BBC ARCHIVE
potentially interesting video
interesting clip...
3.	
  Evaluate	
  target	
  relevance	
  	
  
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
BBC ARCHIVE
...
Relevance	
  assessment	
  
•  Default	
  procedure:	
  
–  Par@cipants	
  of	
  1st	
  user	
  trials	
  	
  
–  Evaluate...
Assessment	
  interface	
  
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
Original	
  anchor	
  
&	
  ta...
Evalua@on	
  metrics	
  	
  
•  MAP	
  evalua@on	
  for	
  both	
  searching	
  and	
  linking	
  
•  Issues:	
  
–  MAP	
...
4.	
  Discuss	
  at	
  workshop	
  
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
h3p://www.mul:mediaeva...
7/2/13	
   DGA	
  workshop	
  -­‐	
  July	
  2013,	
  Paris	
  
Work	
  funded	
  by:	
  
Task	
  Organizers:	
  
	
  
Rob...
Presentation on MediaEval Search & Linking task 2013
Upcoming SlideShare
Loading in …5
×

Presentation on MediaEval Search & Linking task 2013

453 views

Published on

Presentation on the set-up of the MediaEval Search & Linking task 2013

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Presentation on MediaEval Search & Linking task 2013

  1. 1. Set-­‐up  of  the     Search  &  Hyperlinking  Task    in  the      Series      Evalua'on  of  Video  Hyperlinking   Roeland  Ordelman  -­‐  Audiovisual  Content  Access     University  of  Twente,  The  Netherlands   Netherlands  Ins@tute  for  Sound  and  Vision   Cross-­‐Media  Interac@on  
  2. 2. Video  hyperlinking   •  Topic  with  increasing  aEen@on   •  Relate  video  (fragments)  to  other  mul@media  sources   •  Various  perspec@ves  and  approaches,  e.g.,:   –  Recommenda@on/Detail-­‐on-­‐demand     –  Seman@c/Event  Web   –  Storytelling/narra@ves   –  Summariza@on/Clustering   •  Link  up  communi@es  by  means  of  special  sessions,   workshops  and  benchmarks   –  Mul@media  Modeling  (MMM  2014)  –  Dublin.  Special  session  on   Mul@media  Hyperlinking  and  Retrieval  –  CfP  due  01  August  2013   hEp://mmm2014.org/special-­‐sessions/   7/2/13   DGA  workshop  -­‐  July  2013,  Paris  
  3. 3. MediaEval •  a  mul$media  benchmarking  ini@a@ve  running  several   tasks   •  evaluates  new  algorithms  for  mul@media  access  and   retrieval  linked  to  a  use  case   •  emphasis  on  the  "mul$"  in  mul@media:  audio,   speech,  visual  content,  tags,  users,  context.     •  new  evalua@on  paradigms  involving  users     h3p://www.mul:mediaeval.org  
  4. 4. 7/2/13   DGA  workshop  -­‐  July  2013,  Paris  
  5. 5. Video  linking  scenario   •  Audiovisual  archive  explora$on  scenario   •  Video-­‐to-­‐video  linking   •  Start  with  archive  search:          assuming  an  “informa$on  need”   •  Poten@ally  search  may  go  into:        serendipitous  or  exploratory  type  of  behavior   •  Create  meaningful  cross-­‐sec:ons  through  archive   •  Audiovisual  archive  interlinked  on  the  fragment  level   •  Automated  “Storytelling”  type  of  approaches   7/2/13   DGA  workshop  -­‐  July  2013,  Paris  
  6. 6. Terminology   •  Video  (e.g,  2  hours)   •  Interes@ng  segment  (e.g.  10  min)   •  Anchor:  segment  for  which  a  user   requests  a  link  (e.g.,  1  min)    “I  want  to  know  more  about  this”   •  Hyperlink   •  Target:  relevant  segment  for  given   anchor  (e.g.,  5  min)   7/2/13   DGA  workshop  -­‐  July  2013,  Paris  
  7. 7. ME  2013  Search  &  Hyperlinking   •  ME  2012  S&HL  “brave  new”  task:     –  Search:  retrieve  known-­‐item  video  segment  given  a   natural  language  descrip@on     –  Linking:  link  known-­‐item  video  segment  to  similar   segments  in  the  collec@on  (blib.tv)   •  ME  2013  S&HL  “regular”  task   –  Search:  retrieve  known-­‐item  video  segment  given  a   natural  language  descrip@on     –  Linking:  link  user-­‐defined  anchor  within  the  known-­‐ item  to  relevant  target  video  segments  in  the   collec@on  (bbc  archive)   7/2/13   DGA  workshop  -­‐  July  2013,  Paris  
  8. 8. ME  2013  Search  &  Hyperlinking   7/2/13   DGA  workshop  -­‐  July  2013,  Paris   Anchor   Jump-­‐in  point  
  9. 9. Task  set-­‐up   1.  Organizers:  provide  anchors  to  par@cipants   2.  Par@cipants:  find  relevant  targets     3.  Organizers:  evaluate  relevance  targets   4.  Organizers+Par@cipants:  discuss  at  workshop     7/2/13   DGA  workshop  -­‐  July  2013,  Paris  
  10. 10. 1.  Provide  anchors     •  Requirements:   – Large  video  data  set   – Real  ‘users’  selec@ng  anchors   •  Data:   – 270h  of  BBC  archival  data   •  One  complete  week   •  Excluded:  material  not  owned  by  BBC  (e.g.,  Dr.  Who)     •  Users:  …   7/2/13   DGA  workshop  -­‐  July  2013,  Paris  
  11. 11. Researchers  &   Educators     AXES  Research   system   Journalists   Research   Academic   researchers  &   students   Inves:gate   Academic   educators   Educate   Home  users     AXES  Home  system   Home  users   Entertainment,   Edutainment   Main  group   User   Target   Media   Professionals       AXES  Pro  system   Broadcast   Professionals   Reuse   Media  Archivists   Annotate   User  requirements  study   7/2/13   DGA  workshop  -­‐  July  2013,  Paris   !   !!  
  12. 12. Users   •  Recruitment  Agency   •  In  an@cipa@on  on  a  rather  complex  task:   public  users  with  computer  &  online   experience   •  30  Public  Users   – Na@onality:  UK   – Age  group:  16-­‐30   – Computer  familiarity:  high   – Online  ac@vity:  high   7/2/13   DGA  workshop  -­‐  July  2013,  Paris  
  13. 13. ANCHOR  SELECTION   BBC  User  Trials   7/2/13   DGA  workshop  -­‐  July  2013,  Paris  
  14. 14. Search  for  an  interes@ng  video   7/2/13   DGA  workshop  -­‐  July  2013,  Paris   potentially interesting videoSEARCH BBC ARCHIVE
  15. 15. 7/2/13   DGA  workshop  -­‐  July  2013,  Paris   Formulate   Informa@on   need   Text  search   Visual  search   AXES  PRO   system  
  16. 16. Select  an  interes@ng  clip   7/2/13   DGA  workshop  -­‐  July  2013,  Paris   BBC ARCHIVE potentially interesting video interesting clip SEARCH
  17. 17. Select  an  anchor   7/2/13   DGA  workshop  -­‐  July  2013,  Paris   BBC ARCHIVE potentially interesting video interesting clip interesting segment for linking SEARCH
  18. 18. 7/2/13   DGA  workshop  -­‐  July  2013,  Paris   Describe     target   Describe   anchor   Select  anchor   using  sliders  
  19. 19. Trial  evalua@on   •  Most  par@cipants  understood  the  task   •  Not  difficult   •  Technology  would  be  a  very  nice-­‐to-­‐have   men@oning  the  “serendipitous”  type  of   browsing  behavior   •  Par@cipants  want  music  as  link  anchors  (which   we  did  not  an@cipate  on!)   7/2/13   DGA  workshop  -­‐  July  2013,  Paris  
  20. 20. Classifica@on  of  selected  clips   7/2/13   DGA  workshop  -­‐  July  2013,  Paris   0   5   10   15   20   25  
  21. 21. Classifica@on  of  anchors   7/2/13   DGA  workshop  -­‐  July  2013,  Paris   0   10   20   30   40   50   60   70   80   90   Whole  Scene          Speech          Other          Object,   moving          Object,  sta:c   First  FA  cup     Final  at  Wembley   Mayor  Bloomberg  of   New  York  talking   about  Ken  Livingstone  
  22. 22. 2.  Automa@c  target  selec@on   •  ~  15  Par@cipants   •  Data   –  50  NL  queries,  50  anchors     –  270h  of  BBC  archival  data  (used  for  anchor  selec@on)   –  +  One  month  before  ~1000h   •  Metadata   –  Archival  metadata   –  Sub@tles   –  Speech  transcripts  (provided  by  LIMSI  &  LIUM)   –  Shot  segmenta@on   –  (Self-­‐derived  features)   7/2/13   DGA  workshop  -­‐  July  2013,  Paris  
  23. 23. 7/2/13   DGA  workshop  -­‐  July  2013,  Paris   BBC ARCHIVE potentially interesting video interesting clip interesting segment for linking related video clip video hyperlink SEARCH LINK
  24. 24. 3.  Evaluate  target  relevance     7/2/13   DGA  workshop  -­‐  July  2013,  Paris   BBC ARCHIVE potentially interesting video interesting clip interesting segment for linking related video clip video hyperlink SEARCH LINK Relevance Assessment
  25. 25. Relevance  assessment   •  Default  procedure:   –  Par@cipants  of  1st  user  trials     –  Evaluate  for  their  own  anchors  the  automa@cally  generated  targets  from   ME  par@cipants   •  Risks:   –  No  shows   –  Par@cipants  can  only  evaluate  top-­‐N  (N-­‐max=5?)  targets  for  a  given   anchor   •  Fall-­‐back  op@on:   –  Use  Amazon  Mechanical  Turk  approach   –  (same  as  we  did  in  ME-­‐2012  task)   –  Drawback   •  relevance  assessment  not  based  on  personal  trail     “informa@on  need”  >>  “clip  selec@on”  >>  “anchor  selec@on”  >>    “target  evalua@on”   7/2/13   DGA  workshop  -­‐  July  2013,  Paris  
  26. 26. Assessment  interface   7/2/13   DGA  workshop  -­‐  July  2013,  Paris   Original  anchor   &  target   descrip@on   ANCHOR   SUGGESTED  TARGETS   Relevance   assessment  &   comments  
  27. 27. Evalua@on  metrics     •  MAP  evalua@on  for  both  searching  and  linking   •  Issues:   –  MAP  does  not  provide  means  to  evaluate   segmenta@on  (e.g.,  correct  video  but  jump-­‐in  point  is   incorrect)   –  limit  relevant  results  to  a  window  of  N  (N=60  seconds)   before  and  awer  the  actual  beginning  of  the  relevant   segment   •  Addi@onal  metric:   –  Mean  Generalized  Average  Precision  (mGAP)   •  awards  runs  that  are  closer  to  the  jump-­‐in  point  of  the   relevant  content   7/2/13   DGA  workshop  -­‐  July  2013,  Paris  
  28. 28. 4.  Discuss  at  workshop   7/2/13   DGA  workshop  -­‐  July  2013,  Paris   h3p://www.mul:mediaeval.org   18-­‐19  October  2013   Barcelona,  Spain  
  29. 29. 7/2/13   DGA  workshop  -­‐  July  2013,  Paris   Work  funded  by:   Task  Organizers:     Robin  Aly           Maria  Eskevitch   Gareth  Jones         Roeland  Ordelman           h3p://www.mul:mediaeval.org  

×