SlideShare a Scribd company logo
Search and Hyperlinking Task at MediaEval 2012




    Search and Hyperlinking Task
         at MediaEval 2012

Maria Eskevich1 , Gareth J.F. Jones1 , Shu Chen1
       Robin Aly2 , Roeland Ordelman2
                Martha Larson3
                 1   Dublin City University, Dublin, Ireland
             2   University of Twente, The Netherlands
    3   Delft University of Technology, Delft, The Netherlands
Search and Hyperlinking Task at MediaEval 2012


Background
Search and Hyperlinking Task at MediaEval 2012


Brave New Task
Search and Hyperlinking Task at MediaEval 2012


Brave New Task and Crowdsourcing
Search and Hyperlinking Task at MediaEval 2012


Brave New Task: Search and Hyperlinking

     Previous work:
       MediaEval 2011: Rich Speech Retrieval Task
       VideoCLEF 2009: Linking Task
Search and Hyperlinking Task at MediaEval 2012


Brave New Task: Search and Hyperlinking

     Previous work:
        MediaEval 2011: Rich Speech Retrieval Task
        VideoCLEF 2009: Linking Task

      What is Brave New about the Search and Hyperlinking?
        Unified scenario for 2 tasks:
        Search output results as input for Linking
        Use of crowdsourcing for results assessment:
        workers define the relevance of the video segments,
        the HIT can be run for each new submission and
        enrich overall results
        Search: ME10WWW dataset − > blip10000
Search and Hyperlinking Task at MediaEval 2012


Sub-tasks

       Search sub-task:
        Known-item search:
             Textual queries only
             Multimodal queries (text + video clues)
        Required runs:
        1 submission for 1-best output of each ASR transcript
       Linking sub-task
        Ad-hoc search for videos that can potentially be linked to
        the anchor videos (ground truth of the Search subtask)
        Required runs:
        use of ASR 1-best transcripts as representation
        *Additional runs:
        use team output of Search subtask as video anchors for
        Linking sub-task.
Search and Hyperlinking Task at MediaEval 2012


Data: blip10000


       ASR Transcript:
         LIMSI/Vocapia:
             Confusion networks
         LIUM:
             1-best
             Lattices
             Confision networks
       Video clues:
         Shot boundaries (TU Berlin)

         Concept-based descriptors based on a list of 589 concepts
         (University of Oxford)
         Face detection results (INRIA)
Search and Hyperlinking Task at MediaEval 2012


Evaluation: Search sub-task
Search and Hyperlinking Task at MediaEval 2012


Evaluation: Search sub-task
Search and Hyperlinking Task at MediaEval 2012


Evaluation: Search sub-task
Search and Hyperlinking Task at MediaEval 2012


Evaluation: Search sub-task
Search and Hyperlinking Task at MediaEval 2012


Evaluation: Search sub-task
Search and Hyperlinking Task at MediaEval 2012


Evaluation: Search sub-task




        Mean Reciprocal Rank (MRR):

                                  1
                                RR =
                                RANK
        Mean Generalized Average Precision (mGAP):

                                    1
                    GAP =              . PENALTY
                                  RANK
Search and Hyperlinking Task at MediaEval 2012


Evaluation: Search sub-task

        Mean Average Segment Precision (MASP):
        Ranking + Length of (ir)relevant content

        Segment Precision (SP[r ]) at rank r :
Search and Hyperlinking Task at MediaEval 2012


Evaluation: Search sub-task

        Mean Average Segment Precision (MASP):
        Ranking + Length of (ir)relevant content

        Segment Precision (SP[r ]) at rank r :
Search and Hyperlinking Task at MediaEval 2012


Evaluation: Search sub-task

        Mean Average Segment Precision (MASP):
        Ranking + Length of (ir)relevant content

        Segment Precision (SP[r ]) at rank r :




      Average Segment Precision:
Search and Hyperlinking Task at MediaEval 2012


Evaluation: Search sub-task

        Mean Average Segment Precision (MASP):
        Ranking + Length of (ir)relevant content

        Segment Precision (SP[r ]) at rank r :




      Average Segment Precision:
                                             N
                                      1
                        ASP =           .          SP[r ] · rel(sr )
                                      n
                                            r =1

      rel(sr ) = 1, if relevant content is present,
      otherwise rel(sr ) = 0
Search and Hyperlinking Task at MediaEval 2012


Relevance Evaluation via Crowdsourcing:
Amazon MTurk HIT
Search and Hyperlinking Task at MediaEval 2012


Evaluation: Linking sub-task
Search and Hyperlinking Task at MediaEval 2012


Evaluation: Linking sub-task
Search and Hyperlinking Task at MediaEval 2012


Evaluation: Linking sub-task
Search and Hyperlinking Task at MediaEval 2012


Evaluation: Linking sub-task
Search and Hyperlinking Task at MediaEval 2012


Relevance Evaluation via Crowdsourcing
Details

       Assessment level:
         Videos
         Provided segments:
         Length varies across runs and participants
Search and Hyperlinking Task at MediaEval 2012


Relevance Evaluation via Crowdsourcing
Details

       Assessment level:
         Videos
         Provided segments:
         Length varies across runs and participants

       Workers explanation of relevance judgement:
         Relevant:
              ”Same” videos
              Videos are on the same topic
              Same program different topics
         Irrelevant:
              Same program different topics
              Different topics
              Different video styles: interview vs presentation
Search and Hyperlinking Task at MediaEval 2012


Participants




                      Group                Search Linking
       Charles University in Prague (CUNI)    5      –
          Dublin City University (DCU)        4      6
           Ghent University (MMLab)           4     4+4
                  INRIA/IRISA                 –      5
         University of Twente (UTwente)       3      1
Search and Hyperlinking Task at MediaEval 2012


Participants




                      Group                Search Linking
       Charles University in Prague (CUNI)    5      –
          Dublin City University (DCU)        4      6
           Ghent University (MMLab)           4     4+4
                  INRIA/IRISA                 –      5
         University of Twente (UTwente)       3      1
Search and Hyperlinking Task at MediaEval 2012


Search sub-task results
Search and Hyperlinking Task at MediaEval 2012


Linking sub-task results
Search and Hyperlinking Task at MediaEval 2012


Thank you for your attention!



      Participants presentations:
        Search sub-task:
          Maria Eskevich, Dublin City University
          Petra Galuscakova, Charles University in Prague
        Search and Linking sub-task:
          Danish Nadeem, University of Twente
        Linking sub-task:
          Camille Guinaudeau, INRIA/IRISA
          Shu Chen, Dublin City University
        Search and Linking sub-task:
          Tom de Nies, University of Chent
Search and Hyperlinking Task at MediaEval 2012


Thank you for your attention!
Buon compleanno Robin!

      Participants presentations:
        Search sub-task:
          Maria Eskevich, Dublin City University
          Petra Galuscakova, Charles University in Prague
        Search and Linking sub-task:
          Danish Nadeem, University of Twente
        Linking sub-task:
          Camille Guinaudeau, INRIA/IRISA
          Shu Chen, Dublin City University
        Search and Linking sub-task:
          Tom de Nies, University of Chent

More Related Content

What's hot

Seq2seq Model to Tokenize the Chinese Language
Seq2seq Model to Tokenize the Chinese LanguageSeq2seq Model to Tokenize the Chinese Language
Seq2seq Model to Tokenize the Chinese Language
Jinho Choi
 
Overview of the 2014 ALTA Shared Task: Identifying Expressions of Locations i...
Overview of the 2014 ALTA Shared Task: Identifying Expressions of Locations i...Overview of the 2014 ALTA Shared Task: Identifying Expressions of Locations i...
Overview of the 2014 ALTA Shared Task: Identifying Expressions of Locations i...
Diego Molla-Aliod
 
Finding Help with Programming Errors: An Exploratory Study of Novice Software...
Finding Help with Programming Errors: An Exploratory Study of Novice Software...Finding Help with Programming Errors: An Exploratory Study of Novice Software...
Finding Help with Programming Errors: An Exploratory Study of Novice Software...
Preetha Chatterjee
 
From TREC to Watson: is open domain question answering a solved problem?
From TREC to Watson: is open domain question answering a solved problem?From TREC to Watson: is open domain question answering a solved problem?
From TREC to Watson: is open domain question answering a solved problem?
Constantin Orasan
 
Question answering
Question answeringQuestion answering
Question answering
Nafiseh Navabpour
 
Deep Natural Language Processing for Search and Recommender Systems
Deep Natural Language Processing for Search and Recommender SystemsDeep Natural Language Processing for Search and Recommender Systems
Deep Natural Language Processing for Search and Recommender Systems
Huiji Gao
 
Mining Code Examples with Descriptive Text from Software Artifacts
Mining Code Examples with Descriptive Text from Software ArtifactsMining Code Examples with Descriptive Text from Software Artifacts
Mining Code Examples with Descriptive Text from Software Artifacts
Preetha Chatterjee
 
Using Interactive GA for Requirements Prioritization
Using Interactive GA for Requirements PrioritizationUsing Interactive GA for Requirements Prioritization
Using Interactive GA for Requirements Prioritization
Francis Palma
 
Search and Hyperlinking Overview @MediaEval2014
Search and Hyperlinking Overview @MediaEval2014Search and Hyperlinking Overview @MediaEval2014
Search and Hyperlinking Overview @MediaEval2014Maria Eskevich
 
Extracting Archival-Quality Information from Software-Related Chats
Extracting Archival-Quality Information from Software-Related ChatsExtracting Archival-Quality Information from Software-Related Chats
Extracting Archival-Quality Information from Software-Related Chats
Preetha Chatterjee
 

What's hot (10)

Seq2seq Model to Tokenize the Chinese Language
Seq2seq Model to Tokenize the Chinese LanguageSeq2seq Model to Tokenize the Chinese Language
Seq2seq Model to Tokenize the Chinese Language
 
Overview of the 2014 ALTA Shared Task: Identifying Expressions of Locations i...
Overview of the 2014 ALTA Shared Task: Identifying Expressions of Locations i...Overview of the 2014 ALTA Shared Task: Identifying Expressions of Locations i...
Overview of the 2014 ALTA Shared Task: Identifying Expressions of Locations i...
 
Finding Help with Programming Errors: An Exploratory Study of Novice Software...
Finding Help with Programming Errors: An Exploratory Study of Novice Software...Finding Help with Programming Errors: An Exploratory Study of Novice Software...
Finding Help with Programming Errors: An Exploratory Study of Novice Software...
 
From TREC to Watson: is open domain question answering a solved problem?
From TREC to Watson: is open domain question answering a solved problem?From TREC to Watson: is open domain question answering a solved problem?
From TREC to Watson: is open domain question answering a solved problem?
 
Question answering
Question answeringQuestion answering
Question answering
 
Deep Natural Language Processing for Search and Recommender Systems
Deep Natural Language Processing for Search and Recommender SystemsDeep Natural Language Processing for Search and Recommender Systems
Deep Natural Language Processing for Search and Recommender Systems
 
Mining Code Examples with Descriptive Text from Software Artifacts
Mining Code Examples with Descriptive Text from Software ArtifactsMining Code Examples with Descriptive Text from Software Artifacts
Mining Code Examples with Descriptive Text from Software Artifacts
 
Using Interactive GA for Requirements Prioritization
Using Interactive GA for Requirements PrioritizationUsing Interactive GA for Requirements Prioritization
Using Interactive GA for Requirements Prioritization
 
Search and Hyperlinking Overview @MediaEval2014
Search and Hyperlinking Overview @MediaEval2014Search and Hyperlinking Overview @MediaEval2014
Search and Hyperlinking Overview @MediaEval2014
 
Extracting Archival-Quality Information from Software-Related Chats
Extracting Archival-Quality Information from Software-Related ChatsExtracting Archival-Quality Information from Software-Related Chats
Extracting Archival-Quality Information from Software-Related Chats
 

Viewers also liked

My favorite apps 2010
My favorite apps 2010My favorite apps 2010
My favorite apps 2010
Milpitas High School
 
авторская песня
авторская песняавторская песня
авторская песня
Татьяна Л
 
empresa inteligente
empresa inteligenteempresa inteligente
empresa inteligente
guestb26ebc
 
Assembly Man
Assembly ManAssembly Man
Assembly Man
David Holzel
 
Nativos
NativosNativos

Viewers also liked (8)

My favorite apps 2010
My favorite apps 2010My favorite apps 2010
My favorite apps 2010
 
André carvalho
André carvalhoAndré carvalho
André carvalho
 
Estadística i
Estadística iEstadística i
Estadística i
 
Tiempos modernos
Tiempos modernosTiempos modernos
Tiempos modernos
 
авторская песня
авторская песняавторская песня
авторская песня
 
empresa inteligente
empresa inteligenteempresa inteligente
empresa inteligente
 
Assembly Man
Assembly ManAssembly Man
Assembly Man
 
Nativos
NativosNativos
Nativos
 

Similar to Search and Hyperlinking Task at MediaEval 2012

Gareth Jones 1209
Gareth Jones 1209Gareth Jones 1209
Gareth Jones 1209
Yandex
 
Profile-based Dataset Recommendation for RDF Data Linking
Profile-based Dataset Recommendation for RDF Data Linking  Profile-based Dataset Recommendation for RDF Data Linking
Profile-based Dataset Recommendation for RDF Data Linking
Mohamed BEN ELLEFI
 
Developing an evaluation strategy to gain insights into the ROER4D multi-nati...
Developing an evaluation strategy to gain insights into the ROER4D multi-nati...Developing an evaluation strategy to gain insights into the ROER4D multi-nati...
Developing an evaluation strategy to gain insights into the ROER4D multi-nati...
SarahG_SS
 
VII Jornadas eMadrid "Education in exponential times". "Analysing and Alterin...
VII Jornadas eMadrid "Education in exponential times". "Analysing and Alterin...VII Jornadas eMadrid "Education in exponential times". "Analysing and Alterin...
VII Jornadas eMadrid "Education in exponential times". "Analysing and Alterin...
eMadrid network
 
SE@M 2010: Automatic Keywords Extraction - a Basis for Content Recommendation
SE@M 2010: Automatic Keywords Extraction - a Basis for Content RecommendationSE@M 2010: Automatic Keywords Extraction - a Basis for Content Recommendation
SE@M 2010: Automatic Keywords Extraction - a Basis for Content Recommendation
Ivana Bosnic
 
DCU Search Runs at MediaEval 2012: Search and Hyperlinking Task
DCU Search Runs at MediaEval 2012: Search and Hyperlinking TaskDCU Search Runs at MediaEval 2012: Search and Hyperlinking Task
DCU Search Runs at MediaEval 2012: Search and Hyperlinking TaskMediaEval2012
 
iLastic: Linked Data Generation Workflow and User Interface for iMinds Schola...
iLastic: Linked Data Generation Workflow and User Interface for iMinds Schola...iLastic: Linked Data Generation Workflow and User Interface for iMinds Schola...
iLastic: Linked Data Generation Workflow and User Interface for iMinds Schola...
andimou
 
A Context-Aware Retrieval System for Mobile Applications
A Context-Aware Retrieval System for Mobile ApplicationsA Context-Aware Retrieval System for Mobile Applications
A Context-Aware Retrieval System for Mobile Applications
marcopavan83
 
Arnold_Kumar_NLC2014
Arnold_Kumar_NLC2014Arnold_Kumar_NLC2014
Arnold_Kumar_NLC2014
Patricia Arnold
 
Developing an evaluation strategy to gain insights into the ROER4D multi-nati...
Developing an evaluation strategy to gain insights into the ROER4D multi-nati...Developing an evaluation strategy to gain insights into the ROER4D multi-nati...
Developing an evaluation strategy to gain insights into the ROER4D multi-nati...
Open Education Global (OEGlobal)
 
A Semantic-enhanced Inference Framework for Heterogeneous Resources Management
A Semantic-enhanced Inference Framework for Heterogeneous Resources ManagementA Semantic-enhanced Inference Framework for Heterogeneous Resources Management
A Semantic-enhanced Inference Framework for Heterogeneous Resources Management
Silvia Giannini
 
Online Index Extraction from Linked Open Data Sources
Online Index Extraction from Linked Open Data SourcesOnline Index Extraction from Linked Open Data Sources
Online Index Extraction from Linked Open Data Sources
Fabio Benedetti
 
Contextual Recommendation of Social Updates, a tag-based framework
Contextual Recommendation of Social Updates, a tag-based frameworkContextual Recommendation of Social Updates, a tag-based framework
Contextual Recommendation of Social Updates, a tag-based framework
Adrien Joly
 
Knowledge discoverylaurahollink
Knowledge discoverylaurahollinkKnowledge discoverylaurahollink
Knowledge discoverylaurahollink
SSSW
 
LinkedUp - Linked Data & Education
LinkedUp - Linked Data & EducationLinkedUp - Linked Data & Education
LinkedUp - Linked Data & Education
Stefan Dietze
 
Resource Discovery Landscape
Resource Discovery LandscapeResource Discovery Landscape
Resource Discovery Landscape
Andy Powell
 
The Personal Networks of Novice Librarian Researchers
The Personal Networks of Novice Librarian ResearchersThe Personal Networks of Novice Librarian Researchers
The Personal Networks of Novice Librarian Researchers
IRDL
 
Project Credit: Melissa Haendel - On the Nature of Credit
Project Credit: Melissa Haendel - On the Nature of CreditProject Credit: Melissa Haendel - On the Nature of Credit
Project Credit: Melissa Haendel - On the Nature of Credit
CASRAI
 
On the nature of Credit
On the nature of CreditOn the nature of Credit
On the nature of Credit
mhaendel
 
The current oer search dilemma
The current oer search dilemmaThe current oer search dilemma
The current oer search dilemma
Ishan Abeywardena, Ph.D.
 

Similar to Search and Hyperlinking Task at MediaEval 2012 (20)

Gareth Jones 1209
Gareth Jones 1209Gareth Jones 1209
Gareth Jones 1209
 
Profile-based Dataset Recommendation for RDF Data Linking
Profile-based Dataset Recommendation for RDF Data Linking  Profile-based Dataset Recommendation for RDF Data Linking
Profile-based Dataset Recommendation for RDF Data Linking
 
Developing an evaluation strategy to gain insights into the ROER4D multi-nati...
Developing an evaluation strategy to gain insights into the ROER4D multi-nati...Developing an evaluation strategy to gain insights into the ROER4D multi-nati...
Developing an evaluation strategy to gain insights into the ROER4D multi-nati...
 
VII Jornadas eMadrid "Education in exponential times". "Analysing and Alterin...
VII Jornadas eMadrid "Education in exponential times". "Analysing and Alterin...VII Jornadas eMadrid "Education in exponential times". "Analysing and Alterin...
VII Jornadas eMadrid "Education in exponential times". "Analysing and Alterin...
 
SE@M 2010: Automatic Keywords Extraction - a Basis for Content Recommendation
SE@M 2010: Automatic Keywords Extraction - a Basis for Content RecommendationSE@M 2010: Automatic Keywords Extraction - a Basis for Content Recommendation
SE@M 2010: Automatic Keywords Extraction - a Basis for Content Recommendation
 
DCU Search Runs at MediaEval 2012: Search and Hyperlinking Task
DCU Search Runs at MediaEval 2012: Search and Hyperlinking TaskDCU Search Runs at MediaEval 2012: Search and Hyperlinking Task
DCU Search Runs at MediaEval 2012: Search and Hyperlinking Task
 
iLastic: Linked Data Generation Workflow and User Interface for iMinds Schola...
iLastic: Linked Data Generation Workflow and User Interface for iMinds Schola...iLastic: Linked Data Generation Workflow and User Interface for iMinds Schola...
iLastic: Linked Data Generation Workflow and User Interface for iMinds Schola...
 
A Context-Aware Retrieval System for Mobile Applications
A Context-Aware Retrieval System for Mobile ApplicationsA Context-Aware Retrieval System for Mobile Applications
A Context-Aware Retrieval System for Mobile Applications
 
Arnold_Kumar_NLC2014
Arnold_Kumar_NLC2014Arnold_Kumar_NLC2014
Arnold_Kumar_NLC2014
 
Developing an evaluation strategy to gain insights into the ROER4D multi-nati...
Developing an evaluation strategy to gain insights into the ROER4D multi-nati...Developing an evaluation strategy to gain insights into the ROER4D multi-nati...
Developing an evaluation strategy to gain insights into the ROER4D multi-nati...
 
A Semantic-enhanced Inference Framework for Heterogeneous Resources Management
A Semantic-enhanced Inference Framework for Heterogeneous Resources ManagementA Semantic-enhanced Inference Framework for Heterogeneous Resources Management
A Semantic-enhanced Inference Framework for Heterogeneous Resources Management
 
Online Index Extraction from Linked Open Data Sources
Online Index Extraction from Linked Open Data SourcesOnline Index Extraction from Linked Open Data Sources
Online Index Extraction from Linked Open Data Sources
 
Contextual Recommendation of Social Updates, a tag-based framework
Contextual Recommendation of Social Updates, a tag-based frameworkContextual Recommendation of Social Updates, a tag-based framework
Contextual Recommendation of Social Updates, a tag-based framework
 
Knowledge discoverylaurahollink
Knowledge discoverylaurahollinkKnowledge discoverylaurahollink
Knowledge discoverylaurahollink
 
LinkedUp - Linked Data & Education
LinkedUp - Linked Data & EducationLinkedUp - Linked Data & Education
LinkedUp - Linked Data & Education
 
Resource Discovery Landscape
Resource Discovery LandscapeResource Discovery Landscape
Resource Discovery Landscape
 
The Personal Networks of Novice Librarian Researchers
The Personal Networks of Novice Librarian ResearchersThe Personal Networks of Novice Librarian Researchers
The Personal Networks of Novice Librarian Researchers
 
Project Credit: Melissa Haendel - On the Nature of Credit
Project Credit: Melissa Haendel - On the Nature of CreditProject Credit: Melissa Haendel - On the Nature of Credit
Project Credit: Melissa Haendel - On the Nature of Credit
 
On the nature of Credit
On the nature of CreditOn the nature of Credit
On the nature of Credit
 
The current oer search dilemma
The current oer search dilemmaThe current oer search dilemma
The current oer search dilemma
 

More from Maria Eskevich

Video Hyperlinking (LNK) Task at TRECVid 2016
Video Hyperlinking (LNK) Task at TRECVid 2016Video Hyperlinking (LNK) Task at TRECVid 2016
Video Hyperlinking (LNK) Task at TRECVid 2016
Maria Eskevich
 
Focus on spoken content in multimedia retrieval
Focus on spoken content in multimedia retrievalFocus on spoken content in multimedia retrieval
Focus on spoken content in multimedia retrieval
Maria Eskevich
 
Audio/Video Search: Why? What? How?
Audio/Video Search: Why? What? How?Audio/Video Search: Why? What? How?
Audio/Video Search: Why? What? How?
Maria Eskevich
 
Creating a Data Collection for Evaluating Rich Speech Retrieval (LREC 2012)
Creating a Data Collection for Evaluating Rich Speech Retrieval (LREC 2012)Creating a Data Collection for Evaluating Rich Speech Retrieval (LREC 2012)
Creating a Data Collection for Evaluating Rich Speech Retrieval (LREC 2012)
Maria Eskevich
 
DCU at the NTCIR-9 SpokenDoc Passage Retrieval Task
DCU at the NTCIR-9 SpokenDoc Passage Retrieval TaskDCU at the NTCIR-9 SpokenDoc Passage Retrieval Task
DCU at the NTCIR-9 SpokenDoc Passage Retrieval Task
Maria Eskevich
 
Towards Methods for Efficient Access to Spoken Content in the AMI Corpus (SSC...
Towards Methods for Efficient Access to Spoken Content in the AMI Corpus (SSC...Towards Methods for Efficient Access to Spoken Content in the AMI Corpus (SSC...
Towards Methods for Efficient Access to Spoken Content in the AMI Corpus (SSC...Maria Eskevich
 
Comparing Retrieval Effectiveness of Alternative Content Segmentation Methods...
Comparing Retrieval Effectiveness of Alternative Content Segmentation Methods...Comparing Retrieval Effectiveness of Alternative Content Segmentation Methods...
Comparing Retrieval Effectiveness of Alternative Content Segmentation Methods...
Maria Eskevich
 
New Metrics for Meaningful Evaluation of Informally Structured Speech Retriev...
New Metrics for Meaningful Evaluation of Informally Structured Speech Retriev...New Metrics for Meaningful Evaluation of Informally Structured Speech Retriev...
New Metrics for Meaningful Evaluation of Informally Structured Speech Retriev...
Maria Eskevich
 

More from Maria Eskevich (8)

Video Hyperlinking (LNK) Task at TRECVid 2016
Video Hyperlinking (LNK) Task at TRECVid 2016Video Hyperlinking (LNK) Task at TRECVid 2016
Video Hyperlinking (LNK) Task at TRECVid 2016
 
Focus on spoken content in multimedia retrieval
Focus on spoken content in multimedia retrievalFocus on spoken content in multimedia retrieval
Focus on spoken content in multimedia retrieval
 
Audio/Video Search: Why? What? How?
Audio/Video Search: Why? What? How?Audio/Video Search: Why? What? How?
Audio/Video Search: Why? What? How?
 
Creating a Data Collection for Evaluating Rich Speech Retrieval (LREC 2012)
Creating a Data Collection for Evaluating Rich Speech Retrieval (LREC 2012)Creating a Data Collection for Evaluating Rich Speech Retrieval (LREC 2012)
Creating a Data Collection for Evaluating Rich Speech Retrieval (LREC 2012)
 
DCU at the NTCIR-9 SpokenDoc Passage Retrieval Task
DCU at the NTCIR-9 SpokenDoc Passage Retrieval TaskDCU at the NTCIR-9 SpokenDoc Passage Retrieval Task
DCU at the NTCIR-9 SpokenDoc Passage Retrieval Task
 
Towards Methods for Efficient Access to Spoken Content in the AMI Corpus (SSC...
Towards Methods for Efficient Access to Spoken Content in the AMI Corpus (SSC...Towards Methods for Efficient Access to Spoken Content in the AMI Corpus (SSC...
Towards Methods for Efficient Access to Spoken Content in the AMI Corpus (SSC...
 
Comparing Retrieval Effectiveness of Alternative Content Segmentation Methods...
Comparing Retrieval Effectiveness of Alternative Content Segmentation Methods...Comparing Retrieval Effectiveness of Alternative Content Segmentation Methods...
Comparing Retrieval Effectiveness of Alternative Content Segmentation Methods...
 
New Metrics for Meaningful Evaluation of Informally Structured Speech Retriev...
New Metrics for Meaningful Evaluation of Informally Structured Speech Retriev...New Metrics for Meaningful Evaluation of Informally Structured Speech Retriev...
New Metrics for Meaningful Evaluation of Informally Structured Speech Retriev...
 

Recently uploaded

Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
Rohit Gautam
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
DianaGray10
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Vladimir Iglovikov, Ph.D.
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
Kari Kakkonen
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
Alpen-Adria-Universität
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
James Anderson
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neo4j
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
Pierluigi Pugliese
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
SOFTTECHHUB
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Neo4j
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Neo4j
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
DianaGray10
 
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
Pixlogix Infotech
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 

Recently uploaded (20)

Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
 
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 

Search and Hyperlinking Task at MediaEval 2012

  • 1. Search and Hyperlinking Task at MediaEval 2012 Search and Hyperlinking Task at MediaEval 2012 Maria Eskevich1 , Gareth J.F. Jones1 , Shu Chen1 Robin Aly2 , Roeland Ordelman2 Martha Larson3 1 Dublin City University, Dublin, Ireland 2 University of Twente, The Netherlands 3 Delft University of Technology, Delft, The Netherlands
  • 2. Search and Hyperlinking Task at MediaEval 2012 Background
  • 3. Search and Hyperlinking Task at MediaEval 2012 Brave New Task
  • 4. Search and Hyperlinking Task at MediaEval 2012 Brave New Task and Crowdsourcing
  • 5. Search and Hyperlinking Task at MediaEval 2012 Brave New Task: Search and Hyperlinking Previous work: MediaEval 2011: Rich Speech Retrieval Task VideoCLEF 2009: Linking Task
  • 6. Search and Hyperlinking Task at MediaEval 2012 Brave New Task: Search and Hyperlinking Previous work: MediaEval 2011: Rich Speech Retrieval Task VideoCLEF 2009: Linking Task What is Brave New about the Search and Hyperlinking? Unified scenario for 2 tasks: Search output results as input for Linking Use of crowdsourcing for results assessment: workers define the relevance of the video segments, the HIT can be run for each new submission and enrich overall results Search: ME10WWW dataset − > blip10000
  • 7. Search and Hyperlinking Task at MediaEval 2012 Sub-tasks Search sub-task: Known-item search: Textual queries only Multimodal queries (text + video clues) Required runs: 1 submission for 1-best output of each ASR transcript Linking sub-task Ad-hoc search for videos that can potentially be linked to the anchor videos (ground truth of the Search subtask) Required runs: use of ASR 1-best transcripts as representation *Additional runs: use team output of Search subtask as video anchors for Linking sub-task.
  • 8. Search and Hyperlinking Task at MediaEval 2012 Data: blip10000 ASR Transcript: LIMSI/Vocapia: Confusion networks LIUM: 1-best Lattices Confision networks Video clues: Shot boundaries (TU Berlin) Concept-based descriptors based on a list of 589 concepts (University of Oxford) Face detection results (INRIA)
  • 9. Search and Hyperlinking Task at MediaEval 2012 Evaluation: Search sub-task
  • 10. Search and Hyperlinking Task at MediaEval 2012 Evaluation: Search sub-task
  • 11. Search and Hyperlinking Task at MediaEval 2012 Evaluation: Search sub-task
  • 12. Search and Hyperlinking Task at MediaEval 2012 Evaluation: Search sub-task
  • 13. Search and Hyperlinking Task at MediaEval 2012 Evaluation: Search sub-task
  • 14. Search and Hyperlinking Task at MediaEval 2012 Evaluation: Search sub-task Mean Reciprocal Rank (MRR): 1 RR = RANK Mean Generalized Average Precision (mGAP): 1 GAP = . PENALTY RANK
  • 15. Search and Hyperlinking Task at MediaEval 2012 Evaluation: Search sub-task Mean Average Segment Precision (MASP): Ranking + Length of (ir)relevant content Segment Precision (SP[r ]) at rank r :
  • 16. Search and Hyperlinking Task at MediaEval 2012 Evaluation: Search sub-task Mean Average Segment Precision (MASP): Ranking + Length of (ir)relevant content Segment Precision (SP[r ]) at rank r :
  • 17. Search and Hyperlinking Task at MediaEval 2012 Evaluation: Search sub-task Mean Average Segment Precision (MASP): Ranking + Length of (ir)relevant content Segment Precision (SP[r ]) at rank r : Average Segment Precision:
  • 18. Search and Hyperlinking Task at MediaEval 2012 Evaluation: Search sub-task Mean Average Segment Precision (MASP): Ranking + Length of (ir)relevant content Segment Precision (SP[r ]) at rank r : Average Segment Precision: N 1 ASP = . SP[r ] · rel(sr ) n r =1 rel(sr ) = 1, if relevant content is present, otherwise rel(sr ) = 0
  • 19. Search and Hyperlinking Task at MediaEval 2012 Relevance Evaluation via Crowdsourcing: Amazon MTurk HIT
  • 20. Search and Hyperlinking Task at MediaEval 2012 Evaluation: Linking sub-task
  • 21. Search and Hyperlinking Task at MediaEval 2012 Evaluation: Linking sub-task
  • 22. Search and Hyperlinking Task at MediaEval 2012 Evaluation: Linking sub-task
  • 23. Search and Hyperlinking Task at MediaEval 2012 Evaluation: Linking sub-task
  • 24. Search and Hyperlinking Task at MediaEval 2012 Relevance Evaluation via Crowdsourcing Details Assessment level: Videos Provided segments: Length varies across runs and participants
  • 25. Search and Hyperlinking Task at MediaEval 2012 Relevance Evaluation via Crowdsourcing Details Assessment level: Videos Provided segments: Length varies across runs and participants Workers explanation of relevance judgement: Relevant: ”Same” videos Videos are on the same topic Same program different topics Irrelevant: Same program different topics Different topics Different video styles: interview vs presentation
  • 26. Search and Hyperlinking Task at MediaEval 2012 Participants Group Search Linking Charles University in Prague (CUNI) 5 – Dublin City University (DCU) 4 6 Ghent University (MMLab) 4 4+4 INRIA/IRISA – 5 University of Twente (UTwente) 3 1
  • 27. Search and Hyperlinking Task at MediaEval 2012 Participants Group Search Linking Charles University in Prague (CUNI) 5 – Dublin City University (DCU) 4 6 Ghent University (MMLab) 4 4+4 INRIA/IRISA – 5 University of Twente (UTwente) 3 1
  • 28. Search and Hyperlinking Task at MediaEval 2012 Search sub-task results
  • 29. Search and Hyperlinking Task at MediaEval 2012 Linking sub-task results
  • 30. Search and Hyperlinking Task at MediaEval 2012 Thank you for your attention! Participants presentations: Search sub-task: Maria Eskevich, Dublin City University Petra Galuscakova, Charles University in Prague Search and Linking sub-task: Danish Nadeem, University of Twente Linking sub-task: Camille Guinaudeau, INRIA/IRISA Shu Chen, Dublin City University Search and Linking sub-task: Tom de Nies, University of Chent
  • 31. Search and Hyperlinking Task at MediaEval 2012 Thank you for your attention! Buon compleanno Robin! Participants presentations: Search sub-task: Maria Eskevich, Dublin City University Petra Galuscakova, Charles University in Prague Search and Linking sub-task: Danish Nadeem, University of Twente Linking sub-task: Camille Guinaudeau, INRIA/IRISA Shu Chen, Dublin City University Search and Linking sub-task: Tom de Nies, University of Chent