0
A Large Scale Concept Ontology
       for Multimedia Understanding
   Milind Naphade, John R. Smith, Alexander Hauptmann, ...
Central Idea
•   Collaborative activity of three
    critical communities – Users,
    Library Scientists and Knowledge
  ...
Central Idea
•   Collaborative activity of three                      Users (Analysts,
    critical communities – Users,  ...
Central Idea
•   Collaborative activity of three                         Users (Analysts,
    critical communities – Users...
Central Idea
•   Collaborative activity of three                         Users (Analysts,
    critical communities – Users...
Central Idea
•   Collaborative activity of three                          Users (Analysts,
    critical communities – User...
Problem
  • Users and analysts require richly annotated video content for
       accomplishing required access and analysi...
Workshop Goals




Page
                 PNN   MITRE
Workshop Goals
  •    Organize series of workshops that bring together three critical
       communities – Users, Library ...
Workshop Goals
  •    Organize series of workshops that bring together three critical
       communities – Users, Library ...
Workshop Goals
  •    Organize series of workshops that bring together three critical
       communities – Users, Library ...
Workshop Goals
  •    Organize series of workshops that bring together three critical
       communities – Users, Library ...
Workshop Goals
  •    Organize series of workshops that bring together three critical
       communities – Users, Library ...
Workshop Goals
  •    Organize series of workshops that bring together three critical
       communities – Users, Library ...
Workshop Goals
  •    Organize series of workshops that bring together three critical
       communities – Users, Library ...
Workshop Goals
  •    Organize series of workshops that bring together three critical
       communities – Users, Library ...
Workshop Goals
  •    Organize series of workshops that bring together three critical
       communities – Users, Library ...
Workshop Goals
  •    Organize series of workshops that bring together three critical
       communities – Users, Library ...
Workshop Goals
  •    Organize series of workshops that bring together three critical
       communities – Users, Library ...
Workshop Goals
  •    Organize series of workshops that bring together three critical
       communities – Users, Library ...
Workshop Goals
  •    Organize series of workshops that bring together three critical
       communities – Users, Library ...
Workshop Format and Duration




Page
                               PNN   MITRE
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
   annotation, experimentation, and...
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
    annotation, experimentation, an...
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
    annotation, experimentation, an...
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
    annotation, experimentation, an...
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
    annotation, experimentation, an...
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
    annotation, experimentation, an...
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
    annotation, experimentation, an...
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
    annotation, experimentation, an...
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
    annotation, experimentation, an...
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
    annotation, experimentation, an...
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
    annotation, experimentation, an...
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
    annotation, experimentation, an...
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
    annotation, experimentation, an...
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
    annotation, experimentation, an...
Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
    annotation, experimentation, an...
Broadcast News Video Content Description Ontology
                                         •   Why the Focus on Broadcast ...
Broadcast News Video Content Description Ontology
                                                                    •   ...
Broadcast News Video Content Description Ontology
                                                                    •   ...
Approach (Pre-workshop and 1st workshop)




Page
                                  PNN   MITRE
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input




Page
                                   PNN ...
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input
     - Solicit input on user needs and existing ...
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input
     - Solicit input on user needs and existing ...
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input
     - Solicit input on user needs and existing ...
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input
     - Solicit input on user needs and existing ...
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input
     - Solicit input on user needs and existing ...
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input
     - Solicit input on user needs and existing ...
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input
     - Solicit input on user needs and existing ...
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input
     - Solicit input on user needs and existing ...
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input
     - Solicit input on user needs and existing ...
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input
     - Solicit input on user needs and existing ...
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input
     - Solicit input on user needs and existing ...
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input
     - Solicit input on user needs and existing ...
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input
     - Solicit input on user needs and existing ...
Approach (Pre-workshop and 1st workshop)
•   Pre-workshop: Call for Input
     - Solicit input on user needs and existing ...
Approach (Ad-hoc Tasks and 2nd workshop)




Page
                                      PNN   MITRE
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group




Page
                                      PNN   MITRE
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation




Page
                         ...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Approach (Ad-hoc Tasks and 2nd workshop)
•      Ad hoc Group
        Task 1: Annotation
        -              Annotate be...
Input


              Task
              s

               Output
             Documents
Page
       PNN    MITRE
Users:                                                                                     Ex. Ontology & catalog systems:...
Users:                                                                                                                    ...
Input


              Task
              s

               Output
             Documents

Page
       PNN    MITRE
Systems:
                                                          Technical capabilities      Video logging
            ...
Systems:
                                                                                Technical capabilities           ...
Workshop 2: Evaluation




                                   Input


                                   Task
            ...
Ontology
                                                          Query Evaluation
                                      ...
Ontology
                                                                Query Evaluation
                                ...
Domain and Data Sets
 • Candidate data set:
       - TRECVID Corpus (>200 hours of video broadcast news from CNN
         ...
Evaluation Methods
   •   Require benchmarks and metrics for evaluating:
         - Utility of ontology – coverage of quer...
Confirmed Participants – Knowledge Experts and Users




Page
                                              PNN   MITRE
Confirmed Participants – Knowledge Experts and Users
Library Sciences and Knowledge
  representation (definition of
  lexi...
Confirmed Participants – Knowledge Experts and Users
Library Sciences and Knowledge
  representation (definition of
  lexi...
Confirmed Participants – Knowledge Experts and Users
Library Sciences and Knowledge           Standardization and Benchmar...
Confirmed Participants – Knowledge Experts and Users
Library Sciences and Knowledge           Standardization and Benchmar...
Confirmed Participants – Technical Team

Theoretical Analysis:            Experimentation: (Help       Prototyping: (Help ...
Impact and Outcome
  •    First of a Kind Ontology of 1000 or more semantic concepts that have been
       evaluated for t...
Summary of Key Questions
   •   How easy was it to create annotations
         - (man-hours/hr of video?)
   •   How well ...
Video Event Ontology (VEO) & VEML

•      A Video Event Ontology was developed in the ARDA workshop on
       video event ...
Upcoming SlideShare
Loading in...5
×

Lscom

814

Published on

Published in: Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
814
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
20
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Lscom"

  1. 1. A Large Scale Concept Ontology for Multimedia Understanding Milind Naphade, John R. Smith, Alexander Hauptmann, Shih-Fu Chang & Edward Chang IBM Research, Carnegie Mellon University, Columbia University & University of California at Santa Barbara naphade@us.ibm.com jsmith@us.ibm.com alex@cs.cmu.edu sfchang@ee.columbia.edu echang@xanadu.ece.ucsb.edu April 2005 NRRC NWRRC MITRE
  2. 2. Central Idea • Collaborative activity of three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers, Algorithm, System and Solution Designers – to create a user-driven concept ontology for analysis of video broadcast news Page PNN MITRE
  3. 3. Central Idea • Collaborative activity of three Users (Analysts, critical communities – Users, Broadcasters) Intelligence Library Scientists and Knowledge Community, Experts, and Technical Broadcasting Researchers, Algorithm, System Corporations and Solution Designers – to create a user-driven concept ontology for analysis of video broadcast news Page PNN MITRE
  4. 4. Central Idea • Collaborative activity of three Users (Analysts, critical communities – Users, Broadcasters) Intelligence Library Scientists and Knowledge Community, Experts, and Technical Broadcasting Researchers, Algorithm, System Corporations and Solution Designers – to create a user-driven concept ontology for analysis of video broadcast news Vision, Machine Learning, Detection Analytics Technical Researchers, Algorithm Designers & System Developers Page PNN MITRE
  5. 5. Central Idea • Collaborative activity of three Users (Analysts, critical communities – Users, Broadcasters) Intelligence Library Scientists and Knowledge Community, Experts, and Technical Broadcasting Researchers, Algorithm, System Corporations and Solution Designers – to create a user-driven concept ontology for analysis of video broadcast news Knowledge Vision, Machine Representation, Learning, Detection Library Scientists Analytics Standardization Technical Researchers, Algorithm Ontology Experts Designers & System Developers Page PNN MITRE
  6. 6. Central Idea • Collaborative activity of three Users (Analysts, critical communities – Users, Broadcasters) Intelligence Library Scientists and Knowledge Community, Experts, and Technical Broadcasting Researchers, Algorithm, System Corporations and Solution Designers – to create a user-driven concept ontology for analysis of video broadcast news Lexicon and Ontology 1000 or more Knowledge concepts Vision, Machine Representation, Learning, Detection Library Scientists Analytics Standardization Technical Researchers, Algorithm Ontology Experts Designers & System Developers Page PNN MITRE
  7. 7. Problem • Users and analysts require richly annotated video content for accomplishing required access and analysis functions over massive amount of video content. • Big Barriers: - Research community needs to advance technology for bridging gap from low-level features to semantics - Lack of large scale useful well-defined semantic lexicon - Lack of user-centric ontology - Lack of corpora annotated with rich lexicon - Lack of feasibility studies for any ontology if defined • Examples: - The TRECVID lexicon defined from a frequentist perspective. Its not user-centric. • No effort to date to design lexicon by joint partnership between different communities (users, knowledge experts, technical) Page PNN MITRE
  8. 8. Workshop Goals Page PNN MITRE
  9. 9. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news Page PNN MITRE
  10. 10. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size Page PNN MITRE
  11. 11. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: Page PNN MITRE
  12. 12. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices Page PNN MITRE
  13. 13. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements Page PNN MITRE
  14. 14. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain Page PNN MITRE
  15. 15. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain - Solicit input on technical capabilities Page PNN MITRE
  16. 16. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain - Solicit input on technical capabilities - Analyze technical capabilities for concept modeling and detection Page PNN MITRE
  17. 17. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain - Solicit input on technical capabilities - Analyze technical capabilities for concept modeling and detection - Form benchmark and define annotation tasks Page PNN MITRE
  18. 18. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain - Solicit input on technical capabilities - Analyze technical capabilities for concept modeling and detection - Form benchmark and define annotation tasks - Annotate benchmark dataset Page PNN MITRE
  19. 19. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain - Solicit input on technical capabilities - Analyze technical capabilities for concept modeling and detection - Form benchmark and define annotation tasks - Annotate benchmark dataset - Perform benchmark concept modeling, detection and evaluation Page PNN MITRE
  20. 20. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain - Solicit input on technical capabilities - Analyze technical capabilities for concept modeling and detection - Form benchmark and define annotation tasks - Annotate benchmark dataset - Perform benchmark concept modeling, detection and evaluation - Analyze concept detection performance and revise concept ontology Page PNN MITRE
  21. 21. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain - Solicit input on technical capabilities - Analyze technical capabilities for concept modeling and detection - Form benchmark and define annotation tasks - Annotate benchmark dataset - Perform benchmark concept modeling, detection and evaluation - Analyze concept detection performance and revise concept ontology - Conduct gap analysis and identify outstanding research challenges Page PNN MITRE
  22. 22. Workshop Format and Duration Page PNN MITRE
  23. 23. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks Page PNN MITRE
  24. 24. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain Page PNN MITRE
  25. 25. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: Page PNN MITRE
  26. 26. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices Page PNN MITRE
  27. 27. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): Page PNN MITRE
  28. 28. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs Page PNN MITRE
  29. 29. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis Page PNN MITRE
  30. 30. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks Page PNN MITRE
  31. 31. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks • Task 1: Annotation Page PNN MITRE
  32. 32. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks • Task 1: Annotation • Task 2: Experimentation Page PNN MITRE
  33. 33. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks • Task 1: Annotation • Task 2: Experimentation • Task 3: Evaluation Page PNN MITRE
  34. 34. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks • Task 1: Annotation • Task 2: Experimentation • Task 3: Evaluation - Ontology Evaluation Workshop (two-weeks): Page PNN MITRE
  35. 35. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks • Task 1: Annotation • Task 2: Experimentation • Task 3: Evaluation - Ontology Evaluation Workshop (two-weeks): • Part 1: Validation and Refinement Page PNN MITRE
  36. 36. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks • Task 1: Annotation • Task 2: Experimentation • Task 3: Evaluation - Ontology Evaluation Workshop (two-weeks): • Part 1: Validation and Refinement • Part 2: Outstanding Challenges and Recommendations Page PNN MITRE
  37. 37. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks • Task 1: Annotation • Task 2: Experimentation • Task 3: Evaluation - Ontology Evaluation Workshop (two-weeks): • Part 1: Validation and Refinement • Part 2: Outstanding Challenges and Recommendations • Substantial off-line tasks for annotation and experimentation require Page PNN MITRE
  38. 38. Broadcast News Video Content Description Ontology • Why the Focus on Broadcast News Domain? Broadcast News Ontology - Critical mass of users, content providers, applications - Good content availability (TRECVID, LDC, FBIS) - Shares large set of core concepts with other domains • News Ontology Formalism: Production Broadcast Grammars - Entity-Relationship (E-R) Graphs News Content - RDF, DAML / DAML+OIL, W3C OWL News - MPEG-7, MediaNet, VEML Domain • Seed Representations: - TRECVID-2003 News Lexicon (Annotation Forum) Core - Library of Congress TGM-I Video - CNN, BBC Classification Systems MPEG-7 Video Annotation Tool Page PNN MITRE
  39. 39. Broadcast News Video Content Description Ontology • Why the Focus on Broadcast News Domain? Broadcast News Ontology - Critical mass of users, content providers, applications - Good content availability (TRECVID, LDC, FBIS) - Shares large set of core concepts with other domains • News Ontology Formalism: Production Broadcast Grammars - Entity-Relationship (E-R) Graphs News Content - RDF, DAML / DAML+OIL, W3C OWL News - MPEG-7, MediaNet, VEML Domain • Seed Representations: - TRECVID-2003 News Lexicon (Annotation Forum) Core - Library of Congress TGM-I Video - CNN, BBC Classification Systems Concepts Objects Sites Actions Person Outdoors Indoors People Face News Monolog News News Anchor Studio Subject Dialog Crowd MPEG-7 Video Annotation Tool Page PNN MITRE
  40. 40. Broadcast News Video Content Description Ontology • Why the Focus on Broadcast News Domain? Broadcast News Ontology - Critical mass of users, content providers, applications - Good content availability (TRECVID, LDC, FBIS) - Shares large set of core concepts with other domains • News Ontology Formalism: Production Broadcast Grammars - Entity-Relationship (E-R) Graphs News Content - RDF, DAML / DAML+OIL, W3C OWL News - MPEG-7, MediaNet, VEML Domain • Seed Representations: - TRECVID-2003 News Lexicon (Annotation Forum) Core - Library of Congress TGM-I Video - CNN, BBC Classification Systems Concepts Objects Sites Actions Person Outdoors Indoors People Face News Monolog News News Anchor Studio Subject Dialog Crowd MPEG-7 Video Annotation Tool Page PNN MITRE
  41. 41. Approach (Pre-workshop and 1st workshop) Page PNN MITRE
  42. 42. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input Page PNN MITRE
  43. 43. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices Page PNN MITRE
  44. 44. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Page PNN MITRE
  45. 45. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs Page PNN MITRE
  46. 46. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work Page PNN MITRE
  47. 47. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Page PNN MITRE
  48. 48. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 Page PNN MITRE
  49. 49. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices Page PNN MITRE
  50. 50. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices • Domain Concepts and Ontology System Page PNN MITRE
  51. 51. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices • Domain Concepts and Ontology System • Video Concept Ontology Page PNN MITRE
  52. 52. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices • Domain Concepts and Ontology System • Video Concept Ontology Part 2: Technical Analysis Page PNN MITRE
  53. 53. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices • Domain Concepts and Ontology System • Video Concept Ontology Part 2: Technical Analysis • Analyze technical capabilities for concept modeling and detection Page PNN MITRE
  54. 54. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices • Domain Concepts and Ontology System • Video Concept Ontology Part 2: Technical Analysis • Analyze technical capabilities for concept modeling and detection • Form benchmark and define annotation tasks Page PNN MITRE
  55. 55. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices • Domain Concepts and Ontology System • Video Concept Ontology Part 2: Technical Analysis • Analyze technical capabilities for concept modeling and detection • Form benchmark and define annotation tasks Output: Version 1 Page PNN MITRE
  56. 56. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices • Domain Concepts and Ontology System • Video Concept Ontology Part 2: Technical Analysis • Analyze technical capabilities for concept modeling and detection • Form benchmark and define annotation tasks Output: Version 1 • Benchmark (Use cases, Annotation) Page PNN MITRE
  57. 57. Approach (Ad-hoc Tasks and 2nd workshop) Page PNN MITRE
  58. 58. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Page PNN MITRE
  59. 59. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation Page PNN MITRE
  60. 60. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Page PNN MITRE
  61. 61. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation Page PNN MITRE
  62. 62. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Page PNN MITRE
  63. 63. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation Page PNN MITRE
  64. 64. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Page PNN MITRE
  65. 65. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: Page PNN MITRE
  66. 66. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 Page PNN MITRE
  67. 67. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 Page PNN MITRE
  68. 68. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Page PNN MITRE
  69. 69. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - Page PNN MITRE
  70. 70. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Page PNN MITRE
  71. 71. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation Page PNN MITRE
  72. 72. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Page PNN MITRE
  73. 73. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Output Page PNN MITRE
  74. 74. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Output - Domain Concepts v.2 and Ontology System v.2 Page PNN MITRE
  75. 75. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Output - Domain Concepts v.2 and Ontology System v.2 - Video Concept Ontology v.2 Page PNN MITRE
  76. 76. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Output - Domain Concepts v.2 and Ontology System v.2 - Video Concept Ontology v.2 Part 2: Outstanding Challenges Page PNN MITRE
  77. 77. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Output - Domain Concepts v.2 and Ontology System v.2 - Video Concept Ontology v.2 Part 2: Outstanding Challenges - Conduct gap analysis and identify outstanding research challenges Page PNN MITRE
  78. 78. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Output - Domain Concepts v.2 and Ontology System v.2 - Video Concept Ontology v.2 Part 2: Outstanding Challenges - Conduct gap analysis and identify outstanding research challenges Output: Page PNN MITRE
  79. 79. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Output - Domain Concepts v.2 and Ontology System v.2 - Video Concept Ontology v.2 Part 2: Outstanding Challenges - Conduct gap analysis and identify outstanding research challenges Output: - Research Challenges v.1 Page PNN MITRE
  80. 80. Input Task s Output Documents Page PNN MITRE
  81. 81. Users: Ex. Ontology & catalog systems: Catalog  IC analysts  RDF Use cases Existing practices terms  Broadcasters  Topic Maps User needs Ex. Usage:  DAML / DAML+OIL  Searching  W3C OWL Ontology  Browsing  Dublin Core Queries Queries systems  Navigation  OCLC  Threading  Open Directory Project  Summarization  MARC  MODS  MediaNet Pre-Workshop:  VEML Call for Input Ex. Catalog terms Prior work Appl.  LOC TGM-I Analysis Analysis  MPEG-7 classif. Schemes Ex. Queries  TRECVID  BBC Input Task s Output Documents Page PNN MITRE
  82. 82. Users: Ex. Ontology & catalog systems: Catalog  IC analysts  RDF Use cases Existing practices terms  Broadcasters  Topic Maps User needs Ex. Usage:  DAML / DAML+OIL  Searching  W3C OWL Ontology  Browsing  Dublin Core Queries Queries systems  Navigation  OCLC  Threading  Open Directory Project  Summarization  MARC  MODS  MediaNet Pre-Workshop:  VEML Call for Input Ex. Catalog terms Prior work Appl.  LOC TGM-I Analysis Analysis  MPEG-7 classif. Schemes Ex. Queries  TRECVID  BBC • Identifies • Documents indexing Existing Requirements requirements for existing Practices Study v.1 broadcast news practices for Study v.1 video including indexing example queries broadcast news Workshop 1: User Needs Modeling Concept Analysis Analysis • Specifies • Identifies and ontology system Ontology Domain defines domain for broadcast System Concepts concepts, terms, news video Study v.1 Study v.1 classification domain systems for broadcast news Input video (ex. Ontology objects, actions, Design sites, events) Task s • Specifies draft lexicon and Output Video Concept ontology for Documents Ontology broadcast news Page v.1 video domain PNN MITRE
  83. 83. Input Task s Output Documents Page PNN MITRE
  84. 84. Systems: Technical capabilities  Video logging Video Concept Systems &  Video retrieval Ontology Prototypes Ex. State-of-art techniques: Workshop 1: Draft v.1  Content-based search Benchmark Technical Analysis  Segmentation State-of-art Results  Tracking Techniques  High-level feature detection  Story segmentation Ex. Benchmarks:  TRECVID Benchmark • Defines Formation benchmark for Technical concept Analysis detection for video broadcast news (dataset, detection and search tasks, Benchmark metrics) Draft v.1 Input Task s Output Documents Page PNN MITRE
  85. 85. Systems: Technical capabilities  Video logging Video Concept Systems &  Video retrieval Ontology Prototypes Ex. State-of-art techniques: Workshop 1: Draft v.1  Content-based search Benchmark Technical Analysis  Segmentation State-of-art Results  Tracking Techniques  High-level feature detection  Story segmentation Ex. Benchmarks:  TRECVID Benchmark • Defines Formation benchmark for Technical concept Analysis detection for video broadcast news (dataset, detection and search tasks, Benchmark metrics) Draft v.1 • Identifies and maps Ad Hoc Task 1: Feature Concept techniques for Extraction Modeling modeling and Annotation detecting each • Identifies v. 1 v.1 Annotation of draft features Task 1 concepts required for modeling each of draft • Refines concepts Ad Hoc Task 2,3: benchmark to Input include Benchmark Experimentation Experimentation annotated Draft v.2 ground-truth for Task experimentation s and evaluation Query Evaluation Concept with Automatic Output Detection Detection v.1 Documents Evaluation v.1 Ontology Page Evaluation v.1 PNN MITRE
  86. 86. Workshop 2: Evaluation Input Task s Output Documents Page PNN MITRE
  87. 87. Ontology Query Evaluation Evaluation v.1 Concept with Automatic Detection Workshop 2: Evaluation Detection v.1 Evaluation v.1 Analysis • Revises • Revises lexicon ontology system Ontology Domain based on based on System Concepts performance performance Study v.2 Study v.2 analysis analysis Ontology Re-design Requirements Study v.1 • Refines lexicon Video Concept and ontology for Ontology broadcast news v.2 video domain Input Task s Output Documents Page PNN MITRE
  88. 88. Ontology Query Evaluation Evaluation v.1 Concept with Automatic Detection Workshop 2: Evaluation Detection v.1 Evaluation v.1 Analysis • Revises • Revises lexicon ontology system Ontology Domain based on based on System Concepts performance performance Study v.2 Study v.2 analysis analysis Ontology Re-design Requirements Workshop 2: Study v.1 Outstanding Challenges • Refines lexicon Video Concept and ontology for Ontology broadcast news v.2 video domain Gap Analysis Input Task • Identifies and s • Recommendations defines Recommendations Research technology gaps for ontology v.1 Challenge v.1 and challenges exploitation and Output for future solution design Documents research Page PNN MITRE
  89. 89. Domain and Data Sets • Candidate data set: - TRECVID Corpus (>200 hours of video broadcast news from CNN and ABC). Has the following advantages • availability • generalization capability better with than other domains • # of research groups up to speed on this domain for tools/detectors • TREC established some benchmark and evaluation metrics already. - Will avoid letting domain specifics influence the design of ontology to an extent where the ontology starts catering to artifacts of the BN domain. - Will seek other sources such as FBIS, WNC etc. • Annotation issues: - Plan to leverage prior video annotation efforts where possible (e.g., TRECVID annotation forum) - Hands-on annotation effort will induce discussions and requires refinements of concepts meanings Page PNN MITRE
  90. 90. Evaluation Methods • Require benchmarks and metrics for evaluating: - Utility of ontology – coverage of queries in terms of quality and quantity - Feasibility of ontology: • Accuracy of concept detection and degree of automation (amount of training) • Effectiveness of query systems using automatically extracted concepts • Metrics of Retrieval Effectiveness - Precision & Recall Curves, Average Precision, Precision at Fixed Depth • Metrics of Lexicon Effectiveness - Number of Use Cases that can be answered by lexicon successfully - Mean average precision across the set of use cases • Evaluate at multiple levels of granularity: - Individual concept, classes, hierarchies Page PNN MITRE
  91. 91. Confirmed Participants – Knowledge Experts and Users Page PNN MITRE
  92. 92. Confirmed Participants – Knowledge Experts and Users Library Sciences and Knowledge representation (definition of lexicon):  Corrine Jorgensen, School of Information Studies, Florida State University  Barbara Tillett, Chief of Cataloging Policy and Support, Library of Congress  Jerry Hobbs, USC / ISI  Michael Witbrock, Cycorp  Ronald Murray, Preservation Reformatting Division, Library of Congress Page PNN MITRE
  93. 93. Confirmed Participants – Knowledge Experts and Users Library Sciences and Knowledge representation (definition of lexicon):  Corrine Jorgensen, School of Information Studies, Florida State University  Barbara Tillett, Chief of Cataloging Policy and Support, Library of Congress  Jerry Hobbs, USC / ISI  Michael Witbrock, Cycorp  Ronald Murray, Preservation Reformatting Division, Library of Congress R&D Agencies  John Prange, ARDA  Sankar Basu, Div. of Computing and Comm. Foundations, NSF  Maria Zemankova, Div. of Inform. and Intell. Systems., NSF Page PNN MITRE
  94. 94. Confirmed Participants – Knowledge Experts and Users Library Sciences and Knowledge Standardization and Benchmarking representation (definition of (theoretical and empirical lexicon): evaluation):  Corrine Jorgensen, School of  Paul Over, NIST Information Studies, Florida State  John Garofolo, NIST University  Donna Harman, NIST  Barbara Tillett, Chief of Cataloging  David Day, MITRE Policy and Support, Library of  John R. Smith, IBM Research Congress  Jerry Hobbs, USC / ISI  Michael Witbrock, Cycorp  Ronald Murray, Preservation Reformatting Division, Library of Congress R&D Agencies  John Prange, ARDA  Sankar Basu, Div. of Computing and Comm. Foundations, NSF  Maria Zemankova, Div. of Inform. and Intell. Systems., NSF Page PNN MITRE
  95. 95. Confirmed Participants – Knowledge Experts and Users Library Sciences and Knowledge Standardization and Benchmarking representation (definition of (theoretical and empirical lexicon): evaluation):  Corrine Jorgensen, School of  Paul Over, NIST Information Studies, Florida State  John Garofolo, NIST University  Donna Harman, NIST  Barbara Tillett, Chief of Cataloging  David Day, MITRE Policy and Support, Library of  John R. Smith, IBM Research Congress  Jerry Hobbs, USC / ISI  Michael Witbrock, Cycorp  Ronald Murray, Preservation Reformatting Division, Library of User Communities (interpretation of Congress use cases for lexicon definition, broadcasters help getting query logs for finding useful lexical entries) R&D Agencies  Joanne Evans, British Broadcasting  John Prange, ARDA Corporation  Sankar Basu, Div. of Computing and  Chris Porter, Getty Images Comm. Foundations, NSF  ARDA and analysts  Maria Zemankova, Div. of Inform. and Intell. Systems., NSF Page PNN MITRE
  96. 96. Confirmed Participants – Technical Team Theoretical Analysis: Experimentation: (Help Prototyping: (Help with address evaluation issues prototyping tools for (Help conduct analysis for lexicon, ontology and annotation, evaluation, during initial lexicon and concept evaluation) querying, summarization ontology design)  Alexander and statistics gathering)  Shih-Fu Chang, Hauptmann, CMU  Milind R. Naphade, IBM Columbia University Research  Alan Smeaton, Dublin  Ramesh Jain, Georgia  Edward Chang, City University Institute of Technology UCSB  HongJiang Zhang,  Thomas Huang, UIUC  Nevenka Dimitrova,  Microsoft Research Edward Delp, Purdue Phillips Research University  Ajay Divakaran, MERL  Rainer Lienhart, Intel  Wessel Kraaij,  Apostol Natsev, IBM Information Systems Research Division, TNO TPD  Tat-Seng Chua, NUS  Ching-Yung Lin, IBM  Ram Nevatia, USC Research  John Kender,  Mubarak Shah, Columbia University University of Central Florida Page PNN MITRE
  97. 97. Impact and Outcome • First of a Kind Ontology of 1000 or more semantic concepts that have been evaluated for their usability and feasibility by different communities including UC, OC, MC. • Annotated corpus (200 hours) and ontology can be further exploited for future TRECVID, VACE, MPEG-7 activities. Core semantic primitives, that can be included in various video description standards/languages such as MPEG-7. • Empirical and theoretical study of automatic concept detection performance for elements of this large ontology. Use of current state of the art detection wherever possible. Use of simulation where the detection is not available. • Use cases (queries) testing and expansion into ontology • Reports documenting use cases, existing practices, research challenges and recommendations • Prototype systems and tools for annotation, query formulation and evaluation • Guidelines on manual and automatic multimedia query formulation techniques going from use-cases to concepts. • Categorization of classes of concepts based on feasibility, detection performance and difficulty in automation BOTTOMLINE: All this is driven by the user Page PNN MITRE
  98. 98. Summary of Key Questions • How easy was it to create annotations - (man-hours/hr of video?) • How well does the lexicon 'partition' the collection • Given perfect annotations/classification: - How well does the lexicon aid with queries/tasks • How good is automatic annotation of the sample collection - What fraction of perfect annotations accuracy is obtained for the queries/tasks • How much is automatic classification performance of a given lexical item a function of training data - Estimate how much training data would get this lexical item to 60%, 80%, 90%, 95%? • What lexicon changes are necessary or desirable? Page PNN MITRE
  99. 99. Video Event Ontology (VEO) & VEML • A Video Event Ontology was developed in the ARDA workshop on video event ontologies for surveillance and meetings allows natural, hierarchical representation of complex spatio-temporal events common in the physical world by a composition of simpler (primitive) events • VEML – XML-derived Video Event Markup Language used to annotate data by instantiating a class defined in that ontology. Example: We will attempt to use or adapt their notation to the extent possible • (http://www.veml.org:8668//space/2003-10-08/StealingByBlocking.veml) • Broadcast video news ontology is likely to have little overlap with the complex surveillance events described in the VEO, except for some basic concepts. We expect our ontology to be broader, but much shallower • Our broadcast news ontology is largely applicable to any edited broadcast video (e.g. documentaries, talk shows, movies) and somewhat applicable to video in general (including surveillance, UAV and home videos). Page PNN MITRE
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×