Uploaded on

 

More in: Education , Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
744
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
19
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. A Large Scale Concept Ontology for Multimedia Understanding Milind Naphade, John R. Smith, Alexander Hauptmann, Shih-Fu Chang & Edward Chang IBM Research, Carnegie Mellon University, Columbia University & University of California at Santa Barbara naphade@us.ibm.com jsmith@us.ibm.com alex@cs.cmu.edu sfchang@ee.columbia.edu echang@xanadu.ece.ucsb.edu April 2005 NRRC NWRRC MITRE
  • 2. Central Idea • Collaborative activity of three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers, Algorithm, System and Solution Designers – to create a user-driven concept ontology for analysis of video broadcast news Page PNN MITRE
  • 3. Central Idea • Collaborative activity of three Users (Analysts, critical communities – Users, Broadcasters) Intelligence Library Scientists and Knowledge Community, Experts, and Technical Broadcasting Researchers, Algorithm, System Corporations and Solution Designers – to create a user-driven concept ontology for analysis of video broadcast news Page PNN MITRE
  • 4. Central Idea • Collaborative activity of three Users (Analysts, critical communities – Users, Broadcasters) Intelligence Library Scientists and Knowledge Community, Experts, and Technical Broadcasting Researchers, Algorithm, System Corporations and Solution Designers – to create a user-driven concept ontology for analysis of video broadcast news Vision, Machine Learning, Detection Analytics Technical Researchers, Algorithm Designers & System Developers Page PNN MITRE
  • 5. Central Idea • Collaborative activity of three Users (Analysts, critical communities – Users, Broadcasters) Intelligence Library Scientists and Knowledge Community, Experts, and Technical Broadcasting Researchers, Algorithm, System Corporations and Solution Designers – to create a user-driven concept ontology for analysis of video broadcast news Knowledge Vision, Machine Representation, Learning, Detection Library Scientists Analytics Standardization Technical Researchers, Algorithm Ontology Experts Designers & System Developers Page PNN MITRE
  • 6. Central Idea • Collaborative activity of three Users (Analysts, critical communities – Users, Broadcasters) Intelligence Library Scientists and Knowledge Community, Experts, and Technical Broadcasting Researchers, Algorithm, System Corporations and Solution Designers – to create a user-driven concept ontology for analysis of video broadcast news Lexicon and Ontology 1000 or more Knowledge concepts Vision, Machine Representation, Learning, Detection Library Scientists Analytics Standardization Technical Researchers, Algorithm Ontology Experts Designers & System Developers Page PNN MITRE
  • 7. Problem • Users and analysts require richly annotated video content for accomplishing required access and analysis functions over massive amount of video content. • Big Barriers: - Research community needs to advance technology for bridging gap from low-level features to semantics - Lack of large scale useful well-defined semantic lexicon - Lack of user-centric ontology - Lack of corpora annotated with rich lexicon - Lack of feasibility studies for any ontology if defined • Examples: - The TRECVID lexicon defined from a frequentist perspective. Its not user-centric. • No effort to date to design lexicon by joint partnership between different communities (users, knowledge experts, technical) Page PNN MITRE
  • 8. Workshop Goals Page PNN MITRE
  • 9. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news Page PNN MITRE
  • 10. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size Page PNN MITRE
  • 11. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: Page PNN MITRE
  • 12. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices Page PNN MITRE
  • 13. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements Page PNN MITRE
  • 14. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain Page PNN MITRE
  • 15. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain - Solicit input on technical capabilities Page PNN MITRE
  • 16. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain - Solicit input on technical capabilities - Analyze technical capabilities for concept modeling and detection Page PNN MITRE
  • 17. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain - Solicit input on technical capabilities - Analyze technical capabilities for concept modeling and detection - Form benchmark and define annotation tasks Page PNN MITRE
  • 18. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain - Solicit input on technical capabilities - Analyze technical capabilities for concept modeling and detection - Form benchmark and define annotation tasks - Annotate benchmark dataset Page PNN MITRE
  • 19. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain - Solicit input on technical capabilities - Analyze technical capabilities for concept modeling and detection - Form benchmark and define annotation tasks - Annotate benchmark dataset - Perform benchmark concept modeling, detection and evaluation Page PNN MITRE
  • 20. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain - Solicit input on technical capabilities - Analyze technical capabilities for concept modeling and detection - Form benchmark and define annotation tasks - Annotate benchmark dataset - Perform benchmark concept modeling, detection and evaluation - Analyze concept detection performance and revise concept ontology Page PNN MITRE
  • 21. Workshop Goals • Organize series of workshops that bring together three critical communities – Users, Library Scientists and Knowledge Experts, and Technical Researchers – to create a ontology on order of 1000 concepts for analysis of video broadcast news • Ensure impact through focused collaboration of these different communities to achieve balance of usefulness, feasibility and size • Specific Tasks: - Solicit input on user needs and existing practices - Analyze applications, prior work, concept modeling requirements - Develop draft concept ontology for video broadcast news domain - Solicit input on technical capabilities - Analyze technical capabilities for concept modeling and detection - Form benchmark and define annotation tasks - Annotate benchmark dataset - Perform benchmark concept modeling, detection and evaluation - Analyze concept detection performance and revise concept ontology - Conduct gap analysis and identify outstanding research challenges Page PNN MITRE
  • 22. Workshop Format and Duration Page PNN MITRE
  • 23. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks Page PNN MITRE
  • 24. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain Page PNN MITRE
  • 25. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: Page PNN MITRE
  • 26. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices Page PNN MITRE
  • 27. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): Page PNN MITRE
  • 28. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs Page PNN MITRE
  • 29. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis Page PNN MITRE
  • 30. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks Page PNN MITRE
  • 31. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks • Task 1: Annotation Page PNN MITRE
  • 32. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks • Task 1: Annotation • Task 2: Experimentation Page PNN MITRE
  • 33. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks • Task 1: Annotation • Task 2: Experimentation • Task 3: Evaluation Page PNN MITRE
  • 34. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks • Task 1: Annotation • Task 2: Experimentation • Task 3: Evaluation - Ontology Evaluation Workshop (two-weeks): Page PNN MITRE
  • 35. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks • Task 1: Annotation • Task 2: Experimentation • Task 3: Evaluation - Ontology Evaluation Workshop (two-weeks): • Part 1: Validation and Refinement Page PNN MITRE
  • 36. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks • Task 1: Annotation • Task 2: Experimentation • Task 3: Evaluation - Ontology Evaluation Workshop (two-weeks): • Part 1: Validation and Refinement • Part 2: Outstanding Challenges and Recommendations Page PNN MITRE
  • 37. Workshop Format and Duration • Propose to hold two multi-week workshops accompanied by annotation, experimentation, and prototyping tasks • Focus on video broadcast news domain • Workshop Organization: - Pre-workshop 1: Call for Input on User Needs and Existing Practices - Ontology Definition Workshop (two-weeks): • Part 1: User Needs • Part 2: Technical Analysis - Ad hoc Tasks • Task 1: Annotation • Task 2: Experimentation • Task 3: Evaluation - Ontology Evaluation Workshop (two-weeks): • Part 1: Validation and Refinement • Part 2: Outstanding Challenges and Recommendations • Substantial off-line tasks for annotation and experimentation require Page PNN MITRE
  • 38. Broadcast News Video Content Description Ontology • Why the Focus on Broadcast News Domain? Broadcast News Ontology - Critical mass of users, content providers, applications - Good content availability (TRECVID, LDC, FBIS) - Shares large set of core concepts with other domains • News Ontology Formalism: Production Broadcast Grammars - Entity-Relationship (E-R) Graphs News Content - RDF, DAML / DAML+OIL, W3C OWL News - MPEG-7, MediaNet, VEML Domain • Seed Representations: - TRECVID-2003 News Lexicon (Annotation Forum) Core - Library of Congress TGM-I Video - CNN, BBC Classification Systems MPEG-7 Video Annotation Tool Page PNN MITRE
  • 39. Broadcast News Video Content Description Ontology • Why the Focus on Broadcast News Domain? Broadcast News Ontology - Critical mass of users, content providers, applications - Good content availability (TRECVID, LDC, FBIS) - Shares large set of core concepts with other domains • News Ontology Formalism: Production Broadcast Grammars - Entity-Relationship (E-R) Graphs News Content - RDF, DAML / DAML+OIL, W3C OWL News - MPEG-7, MediaNet, VEML Domain • Seed Representations: - TRECVID-2003 News Lexicon (Annotation Forum) Core - Library of Congress TGM-I Video - CNN, BBC Classification Systems Concepts Objects Sites Actions Person Outdoors Indoors People Face News Monolog News News Anchor Studio Subject Dialog Crowd MPEG-7 Video Annotation Tool Page PNN MITRE
  • 40. Broadcast News Video Content Description Ontology • Why the Focus on Broadcast News Domain? Broadcast News Ontology - Critical mass of users, content providers, applications - Good content availability (TRECVID, LDC, FBIS) - Shares large set of core concepts with other domains • News Ontology Formalism: Production Broadcast Grammars - Entity-Relationship (E-R) Graphs News Content - RDF, DAML / DAML+OIL, W3C OWL News - MPEG-7, MediaNet, VEML Domain • Seed Representations: - TRECVID-2003 News Lexicon (Annotation Forum) Core - Library of Congress TGM-I Video - CNN, BBC Classification Systems Concepts Objects Sites Actions Person Outdoors Indoors People Face News Monolog News News Anchor Studio Subject Dialog Crowd MPEG-7 Video Annotation Tool Page PNN MITRE
  • 41. Approach (Pre-workshop and 1st workshop) Page PNN MITRE
  • 42. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input Page PNN MITRE
  • 43. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices Page PNN MITRE
  • 44. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Page PNN MITRE
  • 45. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs Page PNN MITRE
  • 46. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work Page PNN MITRE
  • 47. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Page PNN MITRE
  • 48. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 Page PNN MITRE
  • 49. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices Page PNN MITRE
  • 50. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices • Domain Concepts and Ontology System Page PNN MITRE
  • 51. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices • Domain Concepts and Ontology System • Video Concept Ontology Page PNN MITRE
  • 52. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices • Domain Concepts and Ontology System • Video Concept Ontology Part 2: Technical Analysis Page PNN MITRE
  • 53. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices • Domain Concepts and Ontology System • Video Concept Ontology Part 2: Technical Analysis • Analyze technical capabilities for concept modeling and detection Page PNN MITRE
  • 54. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices • Domain Concepts and Ontology System • Video Concept Ontology Part 2: Technical Analysis • Analyze technical capabilities for concept modeling and detection • Form benchmark and define annotation tasks Page PNN MITRE
  • 55. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices • Domain Concepts and Ontology System • Video Concept Ontology Part 2: Technical Analysis • Analyze technical capabilities for concept modeling and detection • Form benchmark and define annotation tasks Output: Version 1 Page PNN MITRE
  • 56. Approach (Pre-workshop and 1st workshop) • Pre-workshop: Call for Input - Solicit input on user needs and existing practices • Ontology Definition Workshop Part 1: User Needs • Analyze use cases, concept modeling requirements, prior lexicon and ontology work • Develop draft concept ontology for video broadcast news domain Output: Version 1 • Requirements and Existing Practices • Domain Concepts and Ontology System • Video Concept Ontology Part 2: Technical Analysis • Analyze technical capabilities for concept modeling and detection • Form benchmark and define annotation tasks Output: Version 1 • Benchmark (Use cases, Annotation) Page PNN MITRE
  • 57. Approach (Ad-hoc Tasks and 2nd workshop) Page PNN MITRE
  • 58. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Page PNN MITRE
  • 59. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation Page PNN MITRE
  • 60. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Page PNN MITRE
  • 61. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation Page PNN MITRE
  • 62. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Page PNN MITRE
  • 63. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation Page PNN MITRE
  • 64. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Page PNN MITRE
  • 65. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: Page PNN MITRE
  • 66. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 Page PNN MITRE
  • 67. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 Page PNN MITRE
  • 68. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Page PNN MITRE
  • 69. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - Page PNN MITRE
  • 70. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Page PNN MITRE
  • 71. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation Page PNN MITRE
  • 72. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Page PNN MITRE
  • 73. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Output Page PNN MITRE
  • 74. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Output - Domain Concepts v.2 and Ontology System v.2 Page PNN MITRE
  • 75. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Output - Domain Concepts v.2 and Ontology System v.2 - Video Concept Ontology v.2 Page PNN MITRE
  • 76. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Output - Domain Concepts v.2 and Ontology System v.2 - Video Concept Ontology v.2 Part 2: Outstanding Challenges Page PNN MITRE
  • 77. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Output - Domain Concepts v.2 and Ontology System v.2 - Video Concept Ontology v.2 Part 2: Outstanding Challenges - Conduct gap analysis and identify outstanding research challenges Page PNN MITRE
  • 78. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Output - Domain Concepts v.2 and Ontology System v.2 - Video Concept Ontology v.2 Part 2: Outstanding Challenges - Conduct gap analysis and identify outstanding research challenges Output: Page PNN MITRE
  • 79. Approach (Ad-hoc Tasks and 2nd workshop) • Ad hoc Group Task 1: Annotation - Annotate benchmark dataset Task 2: Experimentation - Perform benchmark concept modeling and detection Task 3: Evaluation - Evaluation of concept detection, ontology and use of automatic detection for use cases and evaluation Output: - Benchmark v.2 - Concept Detection Evaluation v.1 - Ontology Evaluation v.1 Query Answering Effectiveness with Automated Detection Evaluation v.1 - • Ontology Evaluation Workshop Part 1: Validation - Analyze evaluation of ontology, concept detection and its application to use case answering. Output - Domain Concepts v.2 and Ontology System v.2 - Video Concept Ontology v.2 Part 2: Outstanding Challenges - Conduct gap analysis and identify outstanding research challenges Output: - Research Challenges v.1 Page PNN MITRE
  • 80. Input Task s Output Documents Page PNN MITRE
  • 81. Users: Ex. Ontology & catalog systems: Catalog  IC analysts  RDF Use cases Existing practices terms  Broadcasters  Topic Maps User needs Ex. Usage:  DAML / DAML+OIL  Searching  W3C OWL Ontology  Browsing  Dublin Core Queries Queries systems  Navigation  OCLC  Threading  Open Directory Project  Summarization  MARC  MODS  MediaNet Pre-Workshop:  VEML Call for Input Ex. Catalog terms Prior work Appl.  LOC TGM-I Analysis Analysis  MPEG-7 classif. Schemes Ex. Queries  TRECVID  BBC Input Task s Output Documents Page PNN MITRE
  • 82. Users: Ex. Ontology & catalog systems: Catalog  IC analysts  RDF Use cases Existing practices terms  Broadcasters  Topic Maps User needs Ex. Usage:  DAML / DAML+OIL  Searching  W3C OWL Ontology  Browsing  Dublin Core Queries Queries systems  Navigation  OCLC  Threading  Open Directory Project  Summarization  MARC  MODS  MediaNet Pre-Workshop:  VEML Call for Input Ex. Catalog terms Prior work Appl.  LOC TGM-I Analysis Analysis  MPEG-7 classif. Schemes Ex. Queries  TRECVID  BBC • Identifies • Documents indexing Existing Requirements requirements for existing Practices Study v.1 broadcast news practices for Study v.1 video including indexing example queries broadcast news Workshop 1: User Needs Modeling Concept Analysis Analysis • Specifies • Identifies and ontology system Ontology Domain defines domain for broadcast System Concepts concepts, terms, news video Study v.1 Study v.1 classification domain systems for broadcast news Input video (ex. Ontology objects, actions, Design sites, events) Task s • Specifies draft lexicon and Output Video Concept ontology for Documents Ontology broadcast news Page v.1 video domain PNN MITRE
  • 83. Input Task s Output Documents Page PNN MITRE
  • 84. Systems: Technical capabilities  Video logging Video Concept Systems &  Video retrieval Ontology Prototypes Ex. State-of-art techniques: Workshop 1: Draft v.1  Content-based search Benchmark Technical Analysis  Segmentation State-of-art Results  Tracking Techniques  High-level feature detection  Story segmentation Ex. Benchmarks:  TRECVID Benchmark • Defines Formation benchmark for Technical concept Analysis detection for video broadcast news (dataset, detection and search tasks, Benchmark metrics) Draft v.1 Input Task s Output Documents Page PNN MITRE
  • 85. Systems: Technical capabilities  Video logging Video Concept Systems &  Video retrieval Ontology Prototypes Ex. State-of-art techniques: Workshop 1: Draft v.1  Content-based search Benchmark Technical Analysis  Segmentation State-of-art Results  Tracking Techniques  High-level feature detection  Story segmentation Ex. Benchmarks:  TRECVID Benchmark • Defines Formation benchmark for Technical concept Analysis detection for video broadcast news (dataset, detection and search tasks, Benchmark metrics) Draft v.1 • Identifies and maps Ad Hoc Task 1: Feature Concept techniques for Extraction Modeling modeling and Annotation detecting each • Identifies v. 1 v.1 Annotation of draft features Task 1 concepts required for modeling each of draft • Refines concepts Ad Hoc Task 2,3: benchmark to Input include Benchmark Experimentation Experimentation annotated Draft v.2 ground-truth for Task experimentation s and evaluation Query Evaluation Concept with Automatic Output Detection Detection v.1 Documents Evaluation v.1 Ontology Page Evaluation v.1 PNN MITRE
  • 86. Workshop 2: Evaluation Input Task s Output Documents Page PNN MITRE
  • 87. Ontology Query Evaluation Evaluation v.1 Concept with Automatic Detection Workshop 2: Evaluation Detection v.1 Evaluation v.1 Analysis • Revises • Revises lexicon ontology system Ontology Domain based on based on System Concepts performance performance Study v.2 Study v.2 analysis analysis Ontology Re-design Requirements Study v.1 • Refines lexicon Video Concept and ontology for Ontology broadcast news v.2 video domain Input Task s Output Documents Page PNN MITRE
  • 88. Ontology Query Evaluation Evaluation v.1 Concept with Automatic Detection Workshop 2: Evaluation Detection v.1 Evaluation v.1 Analysis • Revises • Revises lexicon ontology system Ontology Domain based on based on System Concepts performance performance Study v.2 Study v.2 analysis analysis Ontology Re-design Requirements Workshop 2: Study v.1 Outstanding Challenges • Refines lexicon Video Concept and ontology for Ontology broadcast news v.2 video domain Gap Analysis Input Task • Identifies and s • Recommendations defines Recommendations Research technology gaps for ontology v.1 Challenge v.1 and challenges exploitation and Output for future solution design Documents research Page PNN MITRE
  • 89. Domain and Data Sets • Candidate data set: - TRECVID Corpus (>200 hours of video broadcast news from CNN and ABC). Has the following advantages • availability • generalization capability better with than other domains • # of research groups up to speed on this domain for tools/detectors • TREC established some benchmark and evaluation metrics already. - Will avoid letting domain specifics influence the design of ontology to an extent where the ontology starts catering to artifacts of the BN domain. - Will seek other sources such as FBIS, WNC etc. • Annotation issues: - Plan to leverage prior video annotation efforts where possible (e.g., TRECVID annotation forum) - Hands-on annotation effort will induce discussions and requires refinements of concepts meanings Page PNN MITRE
  • 90. Evaluation Methods • Require benchmarks and metrics for evaluating: - Utility of ontology – coverage of queries in terms of quality and quantity - Feasibility of ontology: • Accuracy of concept detection and degree of automation (amount of training) • Effectiveness of query systems using automatically extracted concepts • Metrics of Retrieval Effectiveness - Precision & Recall Curves, Average Precision, Precision at Fixed Depth • Metrics of Lexicon Effectiveness - Number of Use Cases that can be answered by lexicon successfully - Mean average precision across the set of use cases • Evaluate at multiple levels of granularity: - Individual concept, classes, hierarchies Page PNN MITRE
  • 91. Confirmed Participants – Knowledge Experts and Users Page PNN MITRE
  • 92. Confirmed Participants – Knowledge Experts and Users Library Sciences and Knowledge representation (definition of lexicon):  Corrine Jorgensen, School of Information Studies, Florida State University  Barbara Tillett, Chief of Cataloging Policy and Support, Library of Congress  Jerry Hobbs, USC / ISI  Michael Witbrock, Cycorp  Ronald Murray, Preservation Reformatting Division, Library of Congress Page PNN MITRE
  • 93. Confirmed Participants – Knowledge Experts and Users Library Sciences and Knowledge representation (definition of lexicon):  Corrine Jorgensen, School of Information Studies, Florida State University  Barbara Tillett, Chief of Cataloging Policy and Support, Library of Congress  Jerry Hobbs, USC / ISI  Michael Witbrock, Cycorp  Ronald Murray, Preservation Reformatting Division, Library of Congress R&D Agencies  John Prange, ARDA  Sankar Basu, Div. of Computing and Comm. Foundations, NSF  Maria Zemankova, Div. of Inform. and Intell. Systems., NSF Page PNN MITRE
  • 94. Confirmed Participants – Knowledge Experts and Users Library Sciences and Knowledge Standardization and Benchmarking representation (definition of (theoretical and empirical lexicon): evaluation):  Corrine Jorgensen, School of  Paul Over, NIST Information Studies, Florida State  John Garofolo, NIST University  Donna Harman, NIST  Barbara Tillett, Chief of Cataloging  David Day, MITRE Policy and Support, Library of  John R. Smith, IBM Research Congress  Jerry Hobbs, USC / ISI  Michael Witbrock, Cycorp  Ronald Murray, Preservation Reformatting Division, Library of Congress R&D Agencies  John Prange, ARDA  Sankar Basu, Div. of Computing and Comm. Foundations, NSF  Maria Zemankova, Div. of Inform. and Intell. Systems., NSF Page PNN MITRE
  • 95. Confirmed Participants – Knowledge Experts and Users Library Sciences and Knowledge Standardization and Benchmarking representation (definition of (theoretical and empirical lexicon): evaluation):  Corrine Jorgensen, School of  Paul Over, NIST Information Studies, Florida State  John Garofolo, NIST University  Donna Harman, NIST  Barbara Tillett, Chief of Cataloging  David Day, MITRE Policy and Support, Library of  John R. Smith, IBM Research Congress  Jerry Hobbs, USC / ISI  Michael Witbrock, Cycorp  Ronald Murray, Preservation Reformatting Division, Library of User Communities (interpretation of Congress use cases for lexicon definition, broadcasters help getting query logs for finding useful lexical entries) R&D Agencies  Joanne Evans, British Broadcasting  John Prange, ARDA Corporation  Sankar Basu, Div. of Computing and  Chris Porter, Getty Images Comm. Foundations, NSF  ARDA and analysts  Maria Zemankova, Div. of Inform. and Intell. Systems., NSF Page PNN MITRE
  • 96. Confirmed Participants – Technical Team Theoretical Analysis: Experimentation: (Help Prototyping: (Help with address evaluation issues prototyping tools for (Help conduct analysis for lexicon, ontology and annotation, evaluation, during initial lexicon and concept evaluation) querying, summarization ontology design)  Alexander and statistics gathering)  Shih-Fu Chang, Hauptmann, CMU  Milind R. Naphade, IBM Columbia University Research  Alan Smeaton, Dublin  Ramesh Jain, Georgia  Edward Chang, City University Institute of Technology UCSB  HongJiang Zhang,  Thomas Huang, UIUC  Nevenka Dimitrova,  Microsoft Research Edward Delp, Purdue Phillips Research University  Ajay Divakaran, MERL  Rainer Lienhart, Intel  Wessel Kraaij,  Apostol Natsev, IBM Information Systems Research Division, TNO TPD  Tat-Seng Chua, NUS  Ching-Yung Lin, IBM  Ram Nevatia, USC Research  John Kender,  Mubarak Shah, Columbia University University of Central Florida Page PNN MITRE
  • 97. Impact and Outcome • First of a Kind Ontology of 1000 or more semantic concepts that have been evaluated for their usability and feasibility by different communities including UC, OC, MC. • Annotated corpus (200 hours) and ontology can be further exploited for future TRECVID, VACE, MPEG-7 activities. Core semantic primitives, that can be included in various video description standards/languages such as MPEG-7. • Empirical and theoretical study of automatic concept detection performance for elements of this large ontology. Use of current state of the art detection wherever possible. Use of simulation where the detection is not available. • Use cases (queries) testing and expansion into ontology • Reports documenting use cases, existing practices, research challenges and recommendations • Prototype systems and tools for annotation, query formulation and evaluation • Guidelines on manual and automatic multimedia query formulation techniques going from use-cases to concepts. • Categorization of classes of concepts based on feasibility, detection performance and difficulty in automation BOTTOMLINE: All this is driven by the user Page PNN MITRE
  • 98. Summary of Key Questions • How easy was it to create annotations - (man-hours/hr of video?) • How well does the lexicon 'partition' the collection • Given perfect annotations/classification: - How well does the lexicon aid with queries/tasks • How good is automatic annotation of the sample collection - What fraction of perfect annotations accuracy is obtained for the queries/tasks • How much is automatic classification performance of a given lexical item a function of training data - Estimate how much training data would get this lexical item to 60%, 80%, 90%, 95%? • What lexicon changes are necessary or desirable? Page PNN MITRE
  • 99. Video Event Ontology (VEO) & VEML • A Video Event Ontology was developed in the ARDA workshop on video event ontologies for surveillance and meetings allows natural, hierarchical representation of complex spatio-temporal events common in the physical world by a composition of simpler (primitive) events • VEML – XML-derived Video Event Markup Language used to annotate data by instantiating a class defined in that ontology. Example: We will attempt to use or adapt their notation to the extent possible • (http://www.veml.org:8668//space/2003-10-08/StealingByBlocking.veml) • Broadcast video news ontology is likely to have little overlap with the complex surveillance events described in the VEO, except for some basic concepts. We expect our ontology to be broader, but much shallower • Our broadcast news ontology is largely applicable to any edited broadcast video (e.g. documentaries, talk shows, movies) and somewhat applicable to video in general (including surveillance, UAV and home videos). Page PNN MITRE