0
Workshop on Interactive Information AccessUntangling Tasks and TechnologiesUntangling the semantic structurein a broadcast...
2                                                                    2Introduction• Online digital video archive is becomi...
3                                                                                 3NII news video archive                 ...
4                                                               4Overview of the talk              Exploring news stories ...
5                                                                                                                     5   ...
6                                                                      6Semantic structures in news videoIntra- & Inter-vi...
7                                                             7Example of a topic thread structure  Period: 100 days    Or...
8                                                                                                 8Contents of a topic thr...
9                                                          9Browsing news video by the threadstructure: mediaWalker       ...
10                                                                10Towards Video Story-Telling  From here                ...
14                                                                                                    14         Cross-lan...
15                                                                  15Cross-language news story detection• Definition  ― D...
16                                                                16Comparison of news video streams• Identical event shou...
17                                                                                 17Example of news stories on a same eve...
18                                                         18Cross-language news browsing interface:topicTraveller        ...
20                                                                 20Result• Dataset  – 18 pairs of (JP: 1  US: 2)  – Gr...
21                                                                                            21 Structuring a broadcast v...
22                                                                 22Structuring a broadcast video archive• Structure?  – ...
23                                                                 23Example of appearance patternsAdvertisementRelated ne...
24                                                                24Classes of near-duplicate segment types 1) Advertiseme...
25                                                                 25Near-duplicate detection experiment• Data set  – 1 we...
26                                                                               26Automatic classification of classes• Cl...
27                                                                                                     27Evaluation• 100 u...
28                                                               28Future directions• Now we have structured the archives ...
29                                                                            29Summary• Introduced works on analyzing the...
Upcoming SlideShare
Loading in...5
×

Untangling the semantic structure in a broadcast video archive

614

Published on

Invited talk
Workshop on Interactive Information Access: Untangling Tasks and Technologies
At Centrum voor Wiskunde en Informatica (CWI), Amsterdam, The Netherlands
On Dec. 6, 2010

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
614
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Untangling the semantic structure in a broadcast video archive"

  1. 1. Workshop on Interactive Information AccessUntangling Tasks and TechnologiesUntangling the semantic structurein a broadcast video archive Ichiro IDE Nagoya University, Japan University of Amsterdam, The NetherlandsDecember 7, 2010
  2. 2. 2 2Introduction• Online digital video archive is becoming a reality • Efficient retrieval and browsing • Effective reuse• Our aims: • Extract the semantic structure between video data • Rearrange video segments and generate new contents • Provide a browsing / editing interface based on the extracted semantic structure • (Semi-) automatic rearrangement of retrieved results for answering queries Workshop on Interactive Information Access Workshop on Interactive Information Access
  3. 3. 3 3NII news video archive RAID disk Servers PCs for capturing Client MPEG-1/2 Closed-caption Client PC decoder decoder PC Video archiving server DBMS server Video archive Metadata MPEG-1 MPEG-2 Closed-caption Story Video Video text boundary [970 GB] [5.9 TB] [79 MB] [46 k stories] Data processing server• Program: NHK News7• Period: March 16, 2001– (1,700– hours) Workshop on Interactive Information Access Workshop on Interactive Information Access
  4. 4. 4 4Overview of the talk Exploring news stories along the topic thread structure § Cross-language detection of related news stories by text and near-duplicate video segments § Structuring a broadcast video archive based on near-duplicate video segments Workshop on Interactive Information Access Workshop on Interactive Information Access
  5. 5. 5 5 Exploring news stories along the topic thread structure I. Ide, H. Mo, N. Katayama, S. Satoh:“Exploiting topic thread structures in a news video archive for the semi-automatic generation of video summaries”, 2006 IEEE Int. Conf. on Multimedia and Expo (ICME2006), July 2006 I. Ide, T. Kinoshita, T. Takahashi, S. Satoh, H. Murase: “mediaWalker: A video archive explorer based on time-series semantic structure”, 15th ACM Int. Multimedia Conf. Demo Session, Sept. 2007 I.Ide , T. Kinoshita, T. Takahashi, H. Mo, N. Katayama, S. Satoh, H. Murase: “Exploiting the chronological semantic structure in a large-scale broadcast news video archive for its efficient exploration”, APSIPA Annual Summit and Conf. (ASC) 2010, to appear in Dec. 2010 Workshop on Interactive Information Access Workshop on Interactive Information Access
  6. 6. 6 6Semantic structures in news videoIntra- & Inter-video structure• Story tracking / Topic threading Intra-video structured videos Video-1 Story-3 Video-2 Story-1 Story-3  Inter-video Video-3 Story-1 Story-2 structure Video-4 Story-2 Story-3 Video-5 Story-1 Story-2 Story-5 Thread-2 Thread-1 …  Reveals the semantic structure throughout the archive Workshop on Interactive Information Access Workshop on Interactive Information Access
  7. 7. 7 7Example of a topic thread structure Period: 100 days Origin May 1, 2003 Story #1 [Cluster-view] Workshop on Interactive Information Access Workshop on Interactive Information Access
  8. 8. 8 8Contents of a topic thread structure SARS outbreak Chinese gov. worries Chinese gov. watches In Beijing the spread in rural areas the spread in rural areas Spreads in Calms down mainland China in Taiwan WHO sends a WHO declaresmission to Beijing the ceaseSlows downs in mainland Calms down inChina, spreads in Taiwan mainland China, reports fromToronto Taiwanese doctor foundinfected after traveling Japan Search for Anti-SARS conference Infection in Japan held in Beijing Workshop on Interactive Information Access Workshop on Interactive Information Access
  9. 9. 9 9Browsing news video by the threadstructure: mediaWalker Demo Workshop on Interactive Information Access Workshop on Interactive Information Access
  10. 10. 10 10Towards Video Story-Telling From here To here I want to know how it developed• Generate a summarized video that explains how the story developed between two news stories • Select a path (semi-)automatically • Summarize the video streams along the path Currently under work with Frank Workshop on Interactive Information Access Workshop on Interactive Information Access
  11. 11. 14 14 Cross-language detection of related news storiesby text and near-duplicate video segments A. Ogawa, T. Takahashi, I. Ide, H. Murase: “Cross-lingual retrieval of identical news events by near-duplicate video segment detection”, 14th Intl. Multimedia Modeling Conference (MMM2008), Jan. 2008 Workshop on Interactive Information Access Workshop on Interactive Information Access
  12. 12. 15 15Cross-language news story detection• Definition ― Detect news stories in different channels (especially in different languages) discussing the same event• Problem Near ― Text-based approach duplicate • Low MT * ASR quality (Though, recently improving…) (Though, recently improving…) • Different view-point, culture• Proposed method • Detect near-duplicate video segments to complement text information on Interactive Information Access Workshop on Interactive Information Access Workshop
  13. 13. 16 16Comparison of news video streams• Identical event should be broadcast in a close timing • Compare news programs broadcast within +/- 24 hours Compare only the center part to avoid super-imposed captions Cope with color differences by histogram averaging Workshop on Interactive Information Access Workshop on Interactive Information Access
  14. 14. 17 17Example of news stories on a same event <<Keywords>> operation [25], US army [20], Fallujah [18], militaryNov 9, 2004 Story # 1 force [12], troops [7], military strategy [7], attack [5],19:01 (GMT+9) -- Iraqi army [5], general citizens [5], Iraq [4], … <<Keywords>> city [9], Jean [6], Aaron [6], Iraqi [4], phone, call [3],Nov 8, 2004 Story # 1 army forces [3], casualties [3], …22:03 (GMT-5) -- Workshop on Interactive Information Access Workshop on Interactive Information Access
  15. 15. 18 18Cross-language news browsing interface:topicTraveller Demo Workshop on Interactive Information Access Workshop on Interactive Information Access
  16. 16. 20 20Result• Dataset – 18 pairs of (JP: 1  US: 2) – Ground truth: manually given Sum of Text only Image only text and Image Recall 83% (38/46) 96% (20/46) 43% (20/46)Precision 72% (38/53) 90% (44/49) 77% (20/26) Advantage of using image information Workshop on Interactive Information Access Workshop on Interactive Information Access
  17. 17. 21 21 Structuring a broadcast video archivebased on near-duplicate video segments I. Ide, Y. Shamoto, D. Deguchi, T. Takahashi, H. Murase: “Classification of near-duplicate video segments based on their appearance patterns”, 20th Int. Conf. on Pattern Recognition (ICPR2010), Aug. 2010. Workshop on Interactive Information Access Workshop on Interactive Information Access
  18. 18. 22 22Structuring a broadcast video archive• Structure? – For browsing / retrieval – Differs among programs / genres• Applications – Advertisement database – Related contents detection • Related news, … – Periodic contents detection • Sub-program structure  Handle in a unified framework Workshop on Interactive Information Access Workshop on Interactive Information Access
  19. 19. 23 23Example of appearance patternsAdvertisementRelated news Sub-program• Different distributions for different types Demo Workshop on Interactive Information Access Workshop on Interactive Information Access
  20. 20. 24 24Classes of near-duplicate segment types 1) Advertisement 2) Related news 3) Sub-program 4) Rebroadcast 5) Similar framing 6) Extracted segment Workshop on Interactive Information Access Workshop on Interactive Information Access
  21. 21. 25 25Near-duplicate detection experiment• Data set – 1 week of broadcast from 6 channels in Tokyo area  Total: 1,008 hours• Computer environment – Cluster computer • 40 CPU (Intel Xeon 3.4Ghz, Main Memory: 1.0 GB)• Computation cost – CPU time: 133 days – Actual time: 4 days• Result – 3,597,943 pairs (40,928 unique segments) Workshop on Interactive Information Access Workshop on Interactive Information Access
  22. 22. 26 26Automatic classification of classes• Classification rules Unique ND segment set – Features of near-duplicate Rebroadcast video segments within a unique segment set Advertisement • Appearance period • Appeared channels Sub- program • Appearance interval • Length of the segment Similar framing • Periodic or not Extracted Related • Extracted segment or not segment news Extracted segment Original segment Workshop on Interactive Information Access Workshop on Interactive Information Access
  23. 23. 27 27Evaluation• 100 unique segment sets per class (61 sets for rebroadcast) Manual classification 1) 2) 3) 4) 5) 6) Misc. Automatic classification 1) Advertisement 92% 1% 2% 0% 0% 0% 5% 2) Related news 1% 51% 7% 0% 5% 17% 19% 3) Sub-program 0% 0% 65% 0% 2% 0% 33% 4) Rebroadcast 0% 0% 0% 36% 0% 0% 64% 5) Similar framing 0% 0% 0% 0% 63% 6% 31% 6) Extracted segment 1% 49% 2% 0% 0% 35% 13%  Accuracy: 57% Cover rate: 77% Workshop on Interactive Information Access Workshop on Interactive Information Access
  24. 24. 28 28Future directions• Now we have structured the archives in various ways  Consider how to exploit the structure• Reorganize the video data based on an external “scenario” – News video archive  Wikipedia description  (Semi-)automatic Documentary generation – Cooking video archive  Plain recipe text  Multimedia supplementation to a text recipe … Workshop on Interactive Information Access Workshop on Interactive Information Access
  25. 25. 29 29Summary• Introduced works on analyzing the semantic structures in large-scale news video archives and interfaces for efficient understanding of its contents.Thanks to:• Nagoya Univ: Profs. Hiroshi Murase, Daisuke Deguchi Akira Ogawa, Yuji Shamoto, Tomoki Okuoka• NII: Profs. Shin’ichi Satoh, Norio Katayama, Hiroshi Mo• Gifu Shotoku Gakuen Univ.: Prof. Tomokazu Takahashi• NetCompass Ltd.: Tomoyoshi Kinoshita, Takeharu HaraigawaFunded by:• JSPS, MEXT, MRI Inc., Kayamori Information Science Fund, Hoso Bunka Foundation Workshop on Interactive Information Access Workshop on Interactive Information Access
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×