FIAT-IFTA 2013 - Television linked to the web: the case for audiovisual archives

3,212 views
3,128 views

Published on

FIAT-IFTA 2013 - Television linked to the web: the case for audiovisual archives

  1. 1. Television linked to the web: the case for audiovisual archives Johan Oomen and Lotte Belice Baltussen @johanoomen and @lottebelice Netherlands Institute for Sound and Vision FIAT/IFTA | Dubai, United Arab Emirates – 27 October 2013 www.linkedtv.eu
  2. 2. www.linkedtv.eu multimedia content analysis new distribution platforms lots of archive content 2
  3. 3. www.linkedtv.eu multimedia content analysis new distribution platforms where the magic happens! lots of archive content 2
  4. 4. Where the magic happens www.linkedtv.eu multimedia content analysis new distribution platforms lots of archive content 3
  5. 5. Audiovisual broadcasts
  6. 6. Audiovisual broadcasts
  7. 7. Audiovisual broadcasts Professional annotation
  8. 8. Audiovisual broadcasts Professional annotation Search engine
  9. 9. Audiovisual broadcasts Professional annotation Television makers Search engine
  10. 10. Audiovisual broadcasts Professional annotation Academics Television makers Search engine
  11. 11. Audiovisual broadcasts General public Professional annotation Academics Television makers Search engine
  12. 12. Audiovisual broadcasts General public Professional annotation Machine analysis Academics Data gathering Television makers Search engine
  13. 13. multimedia content analysis Data gathering Crowdsourcing Twitter News Importer Subtitles Wikis
  14. 14. Data Gathering: Pros and Cons
  15. 15. Data Gathering: Pros and Cons • Large-scale data for some programs • Noisy data (sometimes) • Access to opinions about television • Not created for the archive • Puts programs in larger national context • • Unclear how the data can be used (sometimes) Affordable
  16. 16. multimedia content analysis Machine Analysis Multimedia content analysis Low-level features
  17. 17. multimedia content analysis Machine Analysis Concept detectors Audio transcripts today sniper fire disrupted the funeral of an eleven year old ethnic albanian boy he was killed yesterday while while chopping wood his family blames serb police before his death louisiana state police now say six workers were killed after a natural gas well exploded and caught fire about forty five miles east of shreveport four others were injured in yesterday's blast a police spokesman says the derek started to melt in the intense heat the Speaker recognition Multimedia content analysis Low-level features Face recognition
  18. 18. Machine Analysis: Pros and Cons
  19. 19. • Machine Analysis: Pros and Cons Extremely detailed, millisecond-level descriptions • Directly describe video content • • Affordable Large scale! • • Noisy • No summarisation of what is important • Confusing for users Limited to what the machine “knows”
  20. 20. Media fragments www.linkedtv.eu 10 linkedtv.eu
  21. 21. Where the magic happens www.linkedtv.eu multimedia content analysis new distribution platforms lots of archive content 11
  22. 22. Context guestimates www.linkedtv.eu thousands of hrs of new television content produced by Dutch public broadcasters every year Hours of archival content of Sound and Vision used in this new content 12 linkedtv.eu
  23. 23. How do broadcast professionals use archive content? www.linkedtv.eu Number of orders § Query log analysis.* (50k user sessions, 264k queries) 4000 3500 3000 2500 2000 1500 1000 500 0 broadcast order story order fragment order >30 29-30 28-29 27-28 26-27 25-26 24-25 23-24 22-23 21-22 20-21 19-20 18-19 17-18 16-17 15-16 14-15 13-14 12-13 11-12 10-11 9-10 8-9 7-8 6-7 5-6 4-5 3-4 2-3 1-2 0-1 Order duration in minutes § § § 33% programs 17% stories, and fragments 50% audiovisual fragments with a start and end time specified by the users themselves. 13 * Paper: Search Behavior of Media Professionals at an Audiovisual Archive: ATransaction Log Analysis. By Bouke Huurnink
  24. 24. Preference for new material www.linkedtv.eu § 46 % of the orders are for footage broadcast more than 1 year before the order date § 12% of orders are for material that is over 10 years old. § => a lot of high quality content is not used. 14
  25. 25. Context guestimates www.linkedtv.eu thousands of hrs of new television content produced by Dutch public broadcasters every year Hours of archival content of Sound and Vision used in this new content 15 linkedtv.eu
  26. 26. Context – the potential www.linkedtv.eu Total av-collection Sound and Vision 750k of hours (& growing daily) 16 linkedtv.eu
  27. 27. Context – the potential: on-demand personalised services Total av-collection Sound and Vision 750k hours (& growing daily) 17 linkedtv.eu www.linkedtv.eu
  28. 28. Where the magic happens www.linkedtv.eu multimedia content analysis new distribution platforms lots of archive content 18
  29. 29. Web and TV exist in parallel www.linkedtv.eu § 86% of TV viewers surf the Web at the same time (Yahoo!, USA) § 88% of USA consumers use mobile as second screen (Business Insider, May 2013) § Manual efforts to link TV program to additional content (Shazam, IntoNow) 19 linkedtv.eu
  30. 30. Samsung advert www.linkedtv.eu 20 http://www.youtube.com/watch?v=UwmoBmlkaGQ
  31. 31. Examples – BBC Interactive Video Player www.linkedtv.eu 21 linkedtv.eu
  32. 32. Examples – Hulu Face Match www.linkedtv.eu 22 linkedtv.eu
  33. 33. The LinkedTV vision www.linkedtv.eu LinkedTV: interweaving Web and TV into a single experience § Mixing lean back and lean forward § how to maintain immersion while removing fourth wall illusion? § will this work across genres and audiences? § Does second screen help or hinder? 23 linkedtv.eu
  34. 34. www.linkedtv.eu Research Organisations SME Technology Providers User Interface and Scenario Partners Technical coordinator Studies 24 Market linkedtv.eu
  35. 35. LinkedTV – Use cases www.linkedtv.eu Hyperlinked Documentary: Antiques Interactive § Program watched predominantly by 50+ers § AVRO: Tussen Kunst en Kitsch (TKK) § Objects, locations and experts provide anchors to related materials (targets). 25 Interactive News § Professional news content produced by RBB for general audience § local news show rbb Aktuell § People, locations and events can be linked to previous events.
  36. 36. Use case – Antiques Interactive Bert and Anne www.linkedtv.eu “I like to relax and sit back when I watch Antiques Road Show. It’s always exciting when something’s worth a lot. In some cases, I like to look up extra info about an object or art style after the show on the PC, but if something doesn’t interest me, I’ll just read a magazine.” “As an antiques shop owner, by work is also my hobby. I’m always looking up information about art objects on my PC or tablet. I like watching my colleagues talking about are on Antiques Road Show and I use my tablet to bookmark specific objects or other information that I like. I also do this for Anne, and send her overviews.” 26
  37. 37. How LinkedTV helps Bert and Anne www.linkedtv.eu Video Object and word detection Connection to concepts Selecton of related concepts Selection of related content Presentation engine 27 linkedtv.eu
  38. 38. How LinkedTV helps Bert and Anne www.linkedtv.eu Video Object and word detection Connection to concepts Selecton of related concepts Selection of related content “...painting of Jan Sluijters....” 27 Presentation engine linkedtv.eu
  39. 39. How LinkedTV helps Bert and Anne www.linkedtv.eu Video Object and word detection Connection to concepts Selecton of related concepts Selection of related content “...painting of Jan Sluijters....” dbpedia.org/resource/Jan_Sluyters 27 linkedtv.eu Presentation engine
  40. 40. How LinkedTV helps Bert and Anne www.linkedtv.eu Video Object and word detection Paintings by Jan Sluijters Connection to concepts Selecton of related concepts Selection of related content “...painting of Jan Sluijters....” dbpedia.org/resource/Jan_Sluyters 27 linkedtv.eu Presentation engine
  41. 41. How LinkedTV helps Bert and Anne www.linkedtv.eu Video Object and word detection Paintings by Jan Sluijters Connection to concepts Selecton of related concepts Selection of related content “...painting of Jan Sluijters....” dbpedia.org/resource/Jan_Sluyters 27 linkedtv.eu Presentation engine
  42. 42. How LinkedTV helps Bert and Anne www.linkedtv.eu Video Object and word detection Paintings by Jan Sluijters Connection to concepts Selecton of related concepts Selection of related content “...painting of Jan Sluijters....” dbpedia.org/resource/Jan_Sluyters 27 linkedtv.eu Presentation engine
  43. 43. From one screen... www.linkedtv.eu 28 http://www3.springfieldwebtv.nl/view.html?path=/view.html&&presentation=/ domain/springfieldwebtv/user/linkedtv/collection/1/presentation/2
  44. 44. From one screen... www.linkedtv.eu 28 http://www3.springfieldwebtv.nl/view.html?path=/view.html&&presentation=/ domain/springfieldwebtv/user/linkedtv/collection/1/presentation/2
  45. 45. From one screen... www.linkedtv.eu 28 http://www3.springfieldwebtv.nl/view.html?path=/view.html&&presentation=/ domain/springfieldwebtv/user/linkedtv/collection/1/presentation/2
  46. 46. § www.linkedtv.eu 29
  47. 47. ...to paper prototype... www.linkedtv.eu 30 linkedtv.eu
  48. 48. www.linkedtv.eu 31
  49. 49. ...to second screen demo www.linkedtv.eu 32 linkedtv.eu
  50. 50. How do we give broadcasters some control? www.linkedtv.eu Hypervideo analysis Paintings by Jan Sluijters Generation of media fragments Named Entity Recognition Enriching NEs with web content Editor tool “...painting of Jan Sluijters....” dbpedia.org/resource/Jan_Sluyters 33 linkedtv.eu Presentation engine
  51. 51. [Useful tool example] editor tool www.linkedtv.eu § § § Broadcasters want to have some control over noisy data. Editors have the expertise to determine relevance + The user interface can only show a limited amount of links, and there are many § Research: sample 805 videos, average 55 NEs per video. 55 NEs = many, many possible links depending on the size of the white list [Li et al., 2013] t1 w Link NE - person: Jan Sluijters , Roeland Ordelman1 hite l is Link white li Link white li st 2 st 3 Link white list 4 NE – art syle: Fauvism Link white list 5 whi Link NE - person: Emiel Aardewerk te list Link white li Link white li 34 6 st 7 st etc etc
  52. 52. Editor tool overview – selecting entities and enrichments 35 www.linkedtv.eu
  53. 53. Editor tool overview – selecting entities and enrichments 36 www.linkedtv.eu
  54. 54. Editor tool overview – editor‘s choices www.linkedtv.eu 37
  55. 55. Future work in LinkedTV www.linkedtv.eu The project ends in May 2015. The coming year, we’ll focus on: •Improving noisy data from automatic enrichments • eg. through aggregating multiple entities and visual cues •Multiple user studies and iterative development of front-end functionalities and interface based on feedback •Making enrichments more relevant for users by further refining personalisation technologies •Further Market studies and business model development 38
  56. 56. Online resources! www.linkedtv.eu 39
  57. 57. Do stuff now! Media Fragment Enricher , Roeland Ordelman1 40 Eurecom’s tool: http://mfe.synote.org/mfe/ www.linkedtv.eu
  58. 58. Lets explore! www.linkedtv.eu multimedia content analysis new distribution platforms lots of archive content 41
  59. 59. Lets explore! www.linkedtv.eu multimedia content analysis new distribution platforms where the magic happens! lots of archive content 41
  60. 60. Thanks! www.linkedtv.eu Find me here: @johanoomen| joomen@beeldengeluid.nl @lottebelice lbbaltussen@beeldengeluid.nl Credits: §All LinkedTV partners §Slide hat-tip to: Lynda Hardman (CWI), Lyndon Nixon (MODUL), Raphaël Troncy , Roeland Ordelman1 (Eurecom), Bouke Huurnink (Sound and Vision) § § § LinkedTV overview: From SmartTV to LinkedTV. slideshare.net/linkedtv/linked-tv-introduction The importance of Linked Media to the Future Web. slideshare.net/linkedtv/www-linked-mediakeynote Describing media assets: media fragment specification and description. slideshare.net/troncy/ describing-media-assets-media-fragment-specification-and-description §Heaps of thanks to AVRO for so generously letting LinkedTV use their content. Links: §www.linkedtv.eu. See demo’s, tools, research, scenario’s and more! §twitter.com/BenGlabs §www.avro.nl 42

×