• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Untangling the semantic structure in a broadcast video archive
 

Untangling the semantic structure in a broadcast video archive

on

  • 681 views

Invited talk

Invited talk
Workshop on Interactive Information Access: Untangling Tasks and Technologies
At Centrum voor Wiskunde en Informatica (CWI), Amsterdam, The Netherlands
On Dec. 6, 2010

Statistics

Views

Total Views
681
Views on SlideShare
652
Embed Views
29

Actions

Likes
0
Downloads
1
Comments
0

1 Embed 29

http://homepages.cwi.nl 29

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Untangling the semantic structure in a broadcast video archive Untangling the semantic structure in a broadcast video archive Presentation Transcript

    • Workshop on Interactive Information AccessUntangling Tasks and TechnologiesUntangling the semantic structurein a broadcast video archive Ichiro IDE Nagoya University, Japan University of Amsterdam, The NetherlandsDecember 7, 2010
    • 2 2Introduction• Online digital video archive is becoming a reality • Efficient retrieval and browsing • Effective reuse• Our aims: • Extract the semantic structure between video data • Rearrange video segments and generate new contents • Provide a browsing / editing interface based on the extracted semantic structure • (Semi-) automatic rearrangement of retrieved results for answering queries Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 3 3NII news video archive RAID disk Servers PCs for capturing Client MPEG-1/2 Closed-caption Client PC decoder decoder PC Video archiving server DBMS server Video archive Metadata MPEG-1 MPEG-2 Closed-caption Story Video Video text boundary [970 GB] [5.9 TB] [79 MB] [46 k stories] Data processing server• Program: NHK News7• Period: March 16, 2001– (1,700– hours) Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 4 4Overview of the talk Exploring news stories along the topic thread structure § Cross-language detection of related news stories by text and near-duplicate video segments § Structuring a broadcast video archive based on near-duplicate video segments Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 5 5 Exploring news stories along the topic thread structure I. Ide, H. Mo, N. Katayama, S. Satoh:“Exploiting topic thread structures in a news video archive for the semi-automatic generation of video summaries”, 2006 IEEE Int. Conf. on Multimedia and Expo (ICME2006), July 2006 I. Ide, T. Kinoshita, T. Takahashi, S. Satoh, H. Murase: “mediaWalker: A video archive explorer based on time-series semantic structure”, 15th ACM Int. Multimedia Conf. Demo Session, Sept. 2007 I.Ide , T. Kinoshita, T. Takahashi, H. Mo, N. Katayama, S. Satoh, H. Murase: “Exploiting the chronological semantic structure in a large-scale broadcast news video archive for its efficient exploration”, APSIPA Annual Summit and Conf. (ASC) 2010, to appear in Dec. 2010 Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 6 6Semantic structures in news videoIntra- & Inter-video structure• Story tracking / Topic threading Intra-video structured videos Video-1 Story-3 Video-2 Story-1 Story-3  Inter-video Video-3 Story-1 Story-2 structure Video-4 Story-2 Story-3 Video-5 Story-1 Story-2 Story-5 Thread-2 Thread-1 …  Reveals the semantic structure throughout the archive Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 7 7Example of a topic thread structure Period: 100 days Origin May 1, 2003 Story #1 [Cluster-view] Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 8 8Contents of a topic thread structure SARS outbreak Chinese gov. worries Chinese gov. watches In Beijing the spread in rural areas the spread in rural areas Spreads in Calms down mainland China in Taiwan WHO sends a WHO declaresmission to Beijing the ceaseSlows downs in mainland Calms down inChina, spreads in Taiwan mainland China, reports fromToronto Taiwanese doctor foundinfected after traveling Japan Search for Anti-SARS conference Infection in Japan held in Beijing Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 9 9Browsing news video by the threadstructure: mediaWalker Demo Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 10 10Towards Video Story-Telling From here To here I want to know how it developed• Generate a summarized video that explains how the story developed between two news stories • Select a path (semi-)automatically • Summarize the video streams along the path Currently under work with Frank Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 14 14 Cross-language detection of related news storiesby text and near-duplicate video segments A. Ogawa, T. Takahashi, I. Ide, H. Murase: “Cross-lingual retrieval of identical news events by near-duplicate video segment detection”, 14th Intl. Multimedia Modeling Conference (MMM2008), Jan. 2008 Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 15 15Cross-language news story detection• Definition ― Detect news stories in different channels (especially in different languages) discussing the same event• Problem Near ― Text-based approach duplicate • Low MT * ASR quality (Though, recently improving…) (Though, recently improving…) • Different view-point, culture• Proposed method • Detect near-duplicate video segments to complement text information on Interactive Information Access Workshop on Interactive Information Access Workshop
    • 16 16Comparison of news video streams• Identical event should be broadcast in a close timing • Compare news programs broadcast within +/- 24 hours Compare only the center part to avoid super-imposed captions Cope with color differences by histogram averaging Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 17 17Example of news stories on a same event <<Keywords>> operation [25], US army [20], Fallujah [18], militaryNov 9, 2004 Story # 1 force [12], troops [7], military strategy [7], attack [5],19:01 (GMT+9) -- Iraqi army [5], general citizens [5], Iraq [4], … <<Keywords>> city [9], Jean [6], Aaron [6], Iraqi [4], phone, call [3],Nov 8, 2004 Story # 1 army forces [3], casualties [3], …22:03 (GMT-5) -- Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 18 18Cross-language news browsing interface:topicTraveller Demo Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 20 20Result• Dataset – 18 pairs of (JP: 1  US: 2) – Ground truth: manually given Sum of Text only Image only text and Image Recall 83% (38/46) 96% (20/46) 43% (20/46)Precision 72% (38/53) 90% (44/49) 77% (20/26) Advantage of using image information Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 21 21 Structuring a broadcast video archivebased on near-duplicate video segments I. Ide, Y. Shamoto, D. Deguchi, T. Takahashi, H. Murase: “Classification of near-duplicate video segments based on their appearance patterns”, 20th Int. Conf. on Pattern Recognition (ICPR2010), Aug. 2010. Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 22 22Structuring a broadcast video archive• Structure? – For browsing / retrieval – Differs among programs / genres• Applications – Advertisement database – Related contents detection • Related news, … – Periodic contents detection • Sub-program structure  Handle in a unified framework Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 23 23Example of appearance patternsAdvertisementRelated news Sub-program• Different distributions for different types Demo Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 24 24Classes of near-duplicate segment types 1) Advertisement 2) Related news 3) Sub-program 4) Rebroadcast 5) Similar framing 6) Extracted segment Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 25 25Near-duplicate detection experiment• Data set – 1 week of broadcast from 6 channels in Tokyo area  Total: 1,008 hours• Computer environment – Cluster computer • 40 CPU (Intel Xeon 3.4Ghz, Main Memory: 1.0 GB)• Computation cost – CPU time: 133 days – Actual time: 4 days• Result – 3,597,943 pairs (40,928 unique segments) Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 26 26Automatic classification of classes• Classification rules Unique ND segment set – Features of near-duplicate Rebroadcast video segments within a unique segment set Advertisement • Appearance period • Appeared channels Sub- program • Appearance interval • Length of the segment Similar framing • Periodic or not Extracted Related • Extracted segment or not segment news Extracted segment Original segment Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 27 27Evaluation• 100 unique segment sets per class (61 sets for rebroadcast) Manual classification 1) 2) 3) 4) 5) 6) Misc. Automatic classification 1) Advertisement 92% 1% 2% 0% 0% 0% 5% 2) Related news 1% 51% 7% 0% 5% 17% 19% 3) Sub-program 0% 0% 65% 0% 2% 0% 33% 4) Rebroadcast 0% 0% 0% 36% 0% 0% 64% 5) Similar framing 0% 0% 0% 0% 63% 6% 31% 6) Extracted segment 1% 49% 2% 0% 0% 35% 13%  Accuracy: 57% Cover rate: 77% Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 28 28Future directions• Now we have structured the archives in various ways  Consider how to exploit the structure• Reorganize the video data based on an external “scenario” – News video archive  Wikipedia description  (Semi-)automatic Documentary generation – Cooking video archive  Plain recipe text  Multimedia supplementation to a text recipe … Workshop on Interactive Information Access Workshop on Interactive Information Access
    • 29 29Summary• Introduced works on analyzing the semantic structures in large-scale news video archives and interfaces for efficient understanding of its contents.Thanks to:• Nagoya Univ: Profs. Hiroshi Murase, Daisuke Deguchi Akira Ogawa, Yuji Shamoto, Tomoki Okuoka• NII: Profs. Shin’ichi Satoh, Norio Katayama, Hiroshi Mo• Gifu Shotoku Gakuen Univ.: Prof. Tomokazu Takahashi• NetCompass Ltd.: Tomoyoshi Kinoshita, Takeharu HaraigawaFunded by:• JSPS, MEXT, MRI Inc., Kayamori Information Science Fund, Hoso Bunka Foundation Workshop on Interactive Information Access Workshop on Interactive Information Access